Raul Ochoa
4203696e1e
Bump version and update news
2016-10-17 15:04:48 +02:00
Raul Ochoa
cdde1be29e
Re-use redis pool as much as possible
2016-10-17 15:02:34 +02:00
Raul Ochoa
4da7363b72
Merge pull request #374 from CartoDB/batch-user-queues
...
Moves from host queues to user queues
2016-10-17 13:22:37 +02:00
Raul Ochoa
431f72873a
250 queued jobs as default limit
2016-10-17 13:00:23 +02:00
Raul Ochoa
180ba19df5
Fix host queue seeking
2016-10-17 12:51:01 +02:00
Raul Ochoa
39bb7e6249
Lock resources by host+user
...
This allows to run multiple jobs in parallel but guarantees order by user
2016-10-17 12:34:52 +02:00
Raul Ochoa
4dad54d004
Add test to validate query order by user
2016-10-17 12:33:49 +02:00
Raul Ochoa
b8a57460dc
Handle stop gracefully
2016-10-17 12:28:01 +02:00
Raul Ochoa
8b9a30eb75
Queue seeker was not _finding_ queues when only one present
2016-10-17 12:27:06 +02:00
Raul Ochoa
f6dc991ab5
Allow to override configuration per API call
2016-10-17 10:52:35 +02:00
Raul Ochoa
cb23b7f46c
Make test finish quicker
2016-10-17 10:52:07 +02:00
Raul Ochoa
c62fe29160
Load config on object creation
2016-10-17 10:51:50 +02:00
Raul Ochoa
6179327486
Rename
2016-10-14 13:10:27 +02:00
Raul Ochoa
b8c63f5ffc
Rename
2016-10-14 12:56:41 +02:00
Raul Ochoa
5bb7d8fa1c
Merge branch 'master' into batch-user-queues
2016-10-14 12:33:37 +02:00
Raul Ochoa
b510a3684d
Add option to skip script downloading
2016-10-14 12:28:43 +02:00
Raul Ochoa
9d952a8050
Stubs next version
2016-10-13 15:20:38 +02:00
Raul Ochoa
39b4baed5f
Release 1.38.2
2016-10-13 15:19:11 +02:00
Raul Ochoa
a6cc4444d2
Update news
2016-10-13 15:18:48 +02:00
Raul Ochoa
a8802d1163
redis-distlock acquires and releases redis clients by operation
2016-10-13 13:48:06 +02:00
Raul Ochoa
05eda290be
Create one client for queue-seeker and share per seek cycle
2016-10-13 13:09:56 +02:00
Raul Ochoa
df32c1a2bb
Stubs next version
2016-10-13 10:17:35 +02:00
Raul Ochoa
24eb1ad10e
Release 1.38.1
2016-10-13 10:16:08 +02:00
Raul Ochoa
734a147dc3
Update news
2016-10-13 10:08:44 +02:00
Raul Ochoa
32b4d66d1f
Merge pull request #375 from CartoDB/batch-subscriber-improvements
...
Batch subscriber improvements
2016-10-13 09:59:43 +02:00
Raul Ochoa
1e442b37ab
Allow to set a max number of queued jobs per user
2016-10-12 22:40:35 +02:00
Raul Ochoa
1f038ac1f4
Moves from host queues to user queues
...
- Existing jobs are moved before start processing them.
- Uses a new queue prefix to avoid collisions.
- Pub/Sub also changes communication channel.
- Job subscriber emits user+host on new jobs.
- Batch processor is faulty. See TODO in batch.js.
2016-10-12 21:32:29 +02:00
Raul Ochoa
aa68dff24b
Use constants for tests
2016-10-12 18:44:46 +02:00
Raul Ochoa
f7d1f9426c
Use constants for queues
2016-10-12 17:53:03 +02:00
Raul Ochoa
189aff2aa9
Only log message on empty queue
2016-10-12 17:42:46 +02:00
Raul Ochoa
6bb2abde0d
Only start lock renewal on lock acquisition
2016-10-12 17:01:24 +02:00
Raul Ochoa
b86f82d3ca
Batch.stop removes all listeners
2016-10-12 16:43:18 +02:00
Raul Ochoa
782caaf125
Run acceptance tests at the end
2016-10-12 16:41:33 +02:00
Raul Ochoa
c21f373291
Tests for locker
2016-10-12 13:15:55 +02:00
Raul Ochoa
75fc21241f
Locker TTL is configured
2016-10-12 13:11:20 +02:00
Raul Ochoa
88f6d46d00
Reuse existing redlock
...
Return not connected clients to pool
2016-10-12 13:10:18 +02:00
Raul Ochoa
3f1b67993c
Locker keep refreshing lock by itself
2016-10-12 12:30:13 +02:00
Raul Ochoa
67566c1d0e
Callback in subscriber unsubscribe errors
2016-10-12 12:29:54 +02:00
Raul Ochoa
c74f9bcce0
More aggressive on seek interval
2016-10-12 12:29:18 +02:00
Raul Ochoa
98185e55cf
Remove Job Queue Pool and use internal structure
...
- We don't need to create a different job queue per host.
- Batch locks on message instead of dequeue.
2016-10-12 12:26:50 +02:00
Raul Ochoa
167ecc6965
Set NODE_ENV environment var to test
2016-10-12 01:41:01 +02:00
Raul Ochoa
e1d0ffc7dd
Logger set to fatal on test environment
2016-10-12 01:40:35 +02:00
Raul Ochoa
e401c01d78
Only log on non-test environments
2016-10-12 01:40:14 +02:00
Raul Ochoa
22d8e48f53
Only lock on dequeue
2016-10-12 00:10:40 +02:00
Raul Ochoa
81393190f7
Add callback to jobseeker result from initial load
2016-10-11 19:59:11 +02:00
Raul Ochoa
8bc52b09cf
Remove console call
2016-10-11 19:46:27 +02:00
Raul Ochoa
dc1a23e886
Add error handler for channel subscriber
2016-10-11 19:45:43 +02:00
Raul Ochoa
2822b68198
onJobHandler receives host with job
...
Queue seeker only returns hosts, not mixing responsibilities
2016-10-11 19:45:26 +02:00
Raul Ochoa
01cf6f244f
Share redis pool for pubsub
2016-10-11 19:41:58 +02:00
Raul Ochoa
ecc6bf0400
Use real on message handler
2016-10-11 19:04:12 +02:00
