Raul Ochoa
1f038ac1f4
Moves from host queues to user queues
...
- Existing jobs are moved before start processing them.
- Uses a new queue prefix to avoid collisions.
- Pub/Sub also changes communication channel.
- Job subscriber emits user+host on new jobs.
- Batch processor is faulty. See TODO in batch.js.
2016-10-12 21:32:29 +02:00
Raul Ochoa
aa68dff24b
Use constants for tests
2016-10-12 18:44:46 +02:00
Raul Ochoa
f7d1f9426c
Use constants for queues
2016-10-12 17:53:03 +02:00
Raul Ochoa
189aff2aa9
Only log message on empty queue
2016-10-12 17:42:46 +02:00
Raul Ochoa
6bb2abde0d
Only start lock renewal on lock acquisition
2016-10-12 17:01:24 +02:00
Raul Ochoa
b86f82d3ca
Batch.stop removes all listeners
2016-10-12 16:43:18 +02:00
Raul Ochoa
782caaf125
Run acceptance tests at the end
2016-10-12 16:41:33 +02:00
Raul Ochoa
c21f373291
Tests for locker
2016-10-12 13:15:55 +02:00
Raul Ochoa
75fc21241f
Locker TTL is configured
2016-10-12 13:11:20 +02:00
Raul Ochoa
88f6d46d00
Reuse existing redlock
...
Return not connected clients to pool
2016-10-12 13:10:18 +02:00
Raul Ochoa
3f1b67993c
Locker keep refreshing lock by itself
2016-10-12 12:30:13 +02:00
Raul Ochoa
67566c1d0e
Callback in subscriber unsubscribe errors
2016-10-12 12:29:54 +02:00
Raul Ochoa
c74f9bcce0
More aggressive on seek interval
2016-10-12 12:29:18 +02:00
Raul Ochoa
98185e55cf
Remove Job Queue Pool and use internal structure
...
- We don't need to create a different job queue per host.
- Batch locks on message instead of dequeue.
2016-10-12 12:26:50 +02:00
Raul Ochoa
167ecc6965
Set NODE_ENV environment var to test
2016-10-12 01:41:01 +02:00
Raul Ochoa
e1d0ffc7dd
Logger set to fatal on test environment
2016-10-12 01:40:35 +02:00
Raul Ochoa
e401c01d78
Only log on non-test environments
2016-10-12 01:40:14 +02:00
Raul Ochoa
22d8e48f53
Only lock on dequeue
2016-10-12 00:10:40 +02:00
Raul Ochoa
81393190f7
Add callback to jobseeker result from initial load
2016-10-11 19:59:11 +02:00
Raul Ochoa
8bc52b09cf
Remove console call
2016-10-11 19:46:27 +02:00
Raul Ochoa
dc1a23e886
Add error handler for channel subscriber
2016-10-11 19:45:43 +02:00
Raul Ochoa
2822b68198
onJobHandler receives host with job
...
Queue seeker only returns hosts, not mixing responsibilities
2016-10-11 19:45:26 +02:00
Raul Ochoa
01cf6f244f
Share redis pool for pubsub
2016-10-11 19:41:58 +02:00
Raul Ochoa
ecc6bf0400
Use real on message handler
2016-10-11 19:04:12 +02:00
Raul Ochoa
611508c654
Hide queue seeker behind job subscriber
2016-10-11 19:01:39 +02:00
Raul Ochoa
e7c4ee32df
Share redis channel config
2016-10-11 18:41:59 +02:00
Raul Ochoa
d15c7ab0de
Always return client to pool
2016-10-11 18:30:35 +02:00
Raul Ochoa
e4b1711e8e
pub/sub package
2016-10-11 18:28:46 +02:00
Raul Ochoa
6f30ff95d8
Stubs next version
2016-10-11 08:59:00 +02:00
Raul Ochoa
e7068b8468
Release 1.38.0
2016-10-11 08:57:36 +02:00
Raul Ochoa
d50fd89df1
Bump version
2016-10-11 08:54:33 +02:00
Raul Ochoa
b6c2d60b71
Merge pull request #373 from CartoDB/batch-queue-leader
...
Batch queue leader
2016-10-10 20:01:12 +02:00
Raul Ochoa
2c064041a1
Add dist lock to run all jobs by host in order
...
It uses http://redis.io/topics/distlock
Which is not perfect: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
2016-10-10 19:54:59 +02:00
Raul Ochoa
0de5d94617
Use debug with same params, no considering job status
2016-10-10 19:53:59 +02:00
Raul Ochoa
90c489119b
Add distributed lock implementation with redis distlock
2016-10-10 19:51:11 +02:00
Raul Ochoa
56a632347b
Inject publisher
2016-10-10 19:47:50 +02:00
Raul Ochoa
f036e29b8f
Make test stable
2016-10-10 19:47:18 +02:00
Raul Ochoa
41d7823227
Drop table if exists
2016-10-10 19:46:53 +02:00
Raul Ochoa
a0ace8d034
Use name in test
2016-10-10 19:46:39 +02:00
Raul Ochoa
66820a67bb
Make possible to specify a name for batch
2016-10-10 19:46:07 +02:00
Raul Ochoa
f4950f73c0
Add test to validate multiple consumers query order scenario
2016-10-10 15:56:04 +02:00
Raul Ochoa
ece95afc95
Add test client to run plain queries
2016-10-10 15:37:46 +02:00
Raul Ochoa
46579093dd
Fix test scenario description
2016-10-10 15:37:19 +02:00
Raul Ochoa
8ac656492f
Move to support directory
2016-10-10 15:22:50 +02:00
Raul Ochoa
faa02a79b9
Test for 578c43b1a8
2016-10-10 15:17:05 +02:00
Raul Ochoa
fd6eb49fe2
Merge pull request #372 from CartoDB/batch-queries-limits
...
Batch queries limits
2016-10-10 12:25:26 +02:00
Raul Ochoa
d8c3181054
Update news
2016-10-10 12:11:21 +02:00
Raul Ochoa
deb1ccf876
DRY job final statuses
2016-10-10 12:09:13 +02:00
Raul Ochoa
8a4f54bb87
Allow users to set max statement_timeout for their queries
2016-10-10 12:01:36 +02:00
Raul Ochoa
5401a7edff
Timeout is passed into query runner
2016-10-10 12:00:54 +02:00
