cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,870 Commits 1 Branch 118 Tags 9.7 MiB
2e2e233b68
Commit Graph

59 Commits

Author SHA1 Message Date
Daniel García Aubert
2e2e233b68 Use 'bunyan' logger instead of 'debug' to log debug messages 2019-04-04 14:31:41 +02:00
Daniel García Aubert
87eefb9cb7 Typo 2019-04-03 18:53:58 +02:00
Daniel García Aubert
544d85310f Typo 2019-04-03 18:50:05 +02:00
Daniel García Aubert
7f6f91fc15 Clear job from WIP while draining 2019-04-03 18:49:11 +02:00
Daniel García Aubert
c1dff1e395 Improve naming 2017-04-04 10:34:07 +02:00
Daniel García Aubert
36d5dc1719 Queue seeker refactor: 
- Extract from job subcriber everything related to queue dicovering
 - Job Queue is responsible of queue dicovering and exposes functions to scan user queues.
 - Batch creates an interval to consume queues
 - Now Batch.start() does:
  * Finds existing queues from metadata, adding to queue index the ones that are not indexed yet
  * Binds callback to process jobs for every queue found.
  * Inits queue consumer interval
  * Subscribes to PUB/SUB channel
 2017-04-03 19:26:21 +02:00
Daniel García Aubert
24ff1cf808 Discover and add to index queues of users with jobs on batch service startup 2017-04-03 12:34:30 +02:00
Daniel García Aubert
4a2b7258be Do not return error if job could not added to work-in-progress list 2016-11-07 12:04:47 +01:00
Daniel García Aubert
cfeabf94c7 Fix jshint typo 2016-11-02 11:33:26 +01:00
Daniel García Aubert
92b01f1163 Hide add/clear job from work in progress list 2016-10-28 17:45:13 +02:00
Daniel García Aubert
e007b1c22a Fix jshint typo 2016-10-28 17:09:31 +02:00
Daniel García Aubert
aa1527b371 Use clear work-in-progress after finishing jobs 2016-10-28 15:58:41 +02:00
Daniel García Aubert
5b8108d4a8 Use job service to add jobs to work-in-progress list 2016-10-27 18:00:56 +02:00
Daniel García Aubert
f6dffb81cb Use default name 2016-10-27 16:55:41 +02:00
Raul Ochoa
594aba6179 Stop migrating old queues by default 2016-10-21 16:07:27 +02:00
Raul Ochoa
95b3a8adf1 Be explicit about queue status 2016-10-19 16:58:31 +02:00
Raul Ochoa
b164ec8c86 Better debugging 2016-10-19 16:58:00 +02:00
Raul Ochoa
ac65c1c39a Rename 2016-10-19 10:36:13 +02:00
Raul Ochoa
3a57331a54 Delegate job scheduling 
There is a host scheduler managing the host locking.

When it can acquire a lock over the host it will delegate
all the tasks related to that host to the same scheduler.

This scheduler will take care of how many jobs it will submit,
and in which order. It's also responsible for guaranteeing the
execution order per user.

Capacity planner dictates how many jobs can be run at the
same time in a given host. There are two simple strategies:

1. Infinity: it will attempt to run as many jobs as different users.
2. One: it will run just one job at the same time.

Missing things:
 - Handle lock renewal failures.
 - Fair scheduling for pending/waiting users.
 - Capacity based on real resources.
 2016-10-18 20:43:15 +02:00
Raul Ochoa
d1e3be2e22 Do not emit job:status from batch 2016-10-18 20:19:44 +02:00
Raul Ochoa
ef6cd24bf3 Correct debug 2016-10-18 11:18:11 +02:00
Raul Ochoa
ac7bad43a5 Lock by host instead of host + user 
- Host lock only released if there are no pending jobs.
- Will allow to schedule jobs by host.
 2016-10-17 19:03:55 +02:00
Raul Ochoa
761fbe5205 Separate job draining from processing 2016-10-17 18:44:47 +02:00
Raul Ochoa
cdde1be29e Re-use redis pool as much as possible 2016-10-17 15:02:34 +02:00
Raul Ochoa
39bb7e6249 Lock resources by host+user 
This allows to run multiple jobs in parallel but guarantees order by user
 2016-10-17 12:34:52 +02:00
Raul Ochoa
1f038ac1f4 Moves from host queues to user queues 
- Existing jobs are moved before start processing them.
 - Uses a new queue prefix to avoid collisions.
 - Pub/Sub also changes communication channel.
 - Job subscriber emits user+host on new jobs.
 - Batch processor is faulty. See TODO in batch.js.
 2016-10-12 21:32:29 +02:00
Raul Ochoa
189aff2aa9 Only log message on empty queue 2016-10-12 17:42:46 +02:00
Raul Ochoa
b86f82d3ca Batch.stop removes all listeners 2016-10-12 16:43:18 +02:00
Raul Ochoa
75fc21241f Locker TTL is configured 2016-10-12 13:11:20 +02:00
Raul Ochoa
98185e55cf Remove Job Queue Pool and use internal structure 
- We don't need to create a different job queue per host.
- Batch locks on message instead of dequeue.
 2016-10-12 12:26:50 +02:00
Raul Ochoa
22d8e48f53 Only lock on dequeue 2016-10-12 00:10:40 +02:00
Raul Ochoa
2822b68198 onJobHandler receives host with job 
Queue seeker only returns hosts, not mixing responsibilities
 2016-10-11 19:45:26 +02:00
Raul Ochoa
2c064041a1 Add dist lock to run all jobs by host in order 
It uses http://redis.io/topics/distlock
Which is not perfect: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
 2016-10-10 19:54:59 +02:00
Raul Ochoa
0de5d94617 Use debug with same params, no considering job status 2016-10-10 19:53:59 +02:00
Raul Ochoa
56a632347b Inject publisher 2016-10-10 19:47:50 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Daniel García Aubert
aa0ce62a85 Implement batch logger to log query times when queries are defined with id 2016-09-29 15:09:36 +02:00
Daniel García Aubert
0586f45413 Added callback to job subscriber to allow to batch service emit ready event 2016-07-22 13:47:14 +02:00
Daniel García Aubert
eb37fd305f Moved forever module 2016-05-18 11:55:58 +02:00
Daniel García Aubert
8b7d481b9a Avoided console usage, favor debug 2016-05-18 11:06:49 +02:00
Daniel García Aubert
cc7dd7a0d2 Job model refactor 2016-05-13 18:50:55 +02:00
Daniel García Aubert
4a83de09af Added job status enumerable to avoid hardcodes strings 2016-04-05 18:50:04 +02:00
Daniel García Aubert
da16f32789 Added uknown status to jobs when drain fails 2016-01-25 20:07:41 +01:00
Daniel García Aubert
e9bd93cf3f Now batch service is linked to express app in order to make it accesible from main app module to expose drain mechanism 2016-01-25 16:28:19 +01:00
Daniel García Aubert
1e49fbc33d Stopped batch service when draining 2016-01-25 15:32:00 +01:00
Daniel García Aubert
20fd536659 Now batch stop is synchronous, changed signature in test. 
Fixed minor issue when draining the last job, queue is got before cancel the job.
 2016-01-25 14:51:37 +01:00
Daniel García Aubert
c308551d05 Implemented drain to stop batch service gracefully. 
Split job acceptance test
 2016-01-13 16:25:25 +01:00
Daniel García Aubert
8d29e6b45c Improved log messages 2016-01-08 16:24:53 +01:00
Daniel García Aubert
20f00d58d9 Refactored batch service to avoid event noise, doing in callback way 2016-01-08 15:47:59 +01:00
Daniel García Aubert
f9f52d2bd1 Improved object creation in batch service 2016-01-08 11:32:01 +01:00