Simon Martín
fdc542f7b5
BunyanLogger
2018-05-23 17:24:48 +02:00
Daniel García Aubert
36d5dc1719
Queue seeker refactor:
...
- Extract from job subcriber everything related to queue dicovering
- Job Queue is responsible of queue dicovering and exposes functions to scan user queues.
- Batch creates an interval to consume queues
- Now Batch.start() does:
* Finds existing queues from metadata, adding to queue index the ones that are not indexed yet
* Binds callback to process jobs for every queue found.
* Inits queue consumer interval
* Subscribes to PUB/SUB channel
2017-04-03 19:26:21 +02:00
Raul Ochoa
16e9e709b8
Cancel with user statement_timeout limit from redis
2016-10-21 14:35:24 +02:00
Raul Ochoa
cdde1be29e
Re-use redis pool as much as possible
2016-10-17 15:02:34 +02:00
Raul Ochoa
1f038ac1f4
Moves from host queues to user queues
...
- Existing jobs are moved before start processing them.
- Uses a new queue prefix to avoid collisions.
- Pub/Sub also changes communication channel.
- Job subscriber emits user+host on new jobs.
- Batch processor is faulty. See TODO in batch.js.
2016-10-12 21:32:29 +02:00
Raul Ochoa
98185e55cf
Remove Job Queue Pool and use internal structure
...
- We don't need to create a different job queue per host.
- Batch locks on message instead of dequeue.
2016-10-12 12:26:50 +02:00
Raul Ochoa
01cf6f244f
Share redis pool for pubsub
2016-10-11 19:41:58 +02:00
Raul Ochoa
611508c654
Hide queue seeker behind job subscriber
2016-10-11 19:01:39 +02:00
Raul Ochoa
e4b1711e8e
pub/sub package
2016-10-11 18:28:46 +02:00
Raul Ochoa
56a632347b
Inject publisher
2016-10-10 19:47:50 +02:00
Raul Ochoa
66820a67bb
Make possible to specify a name for batch
2016-10-10 19:46:07 +02:00
Daniel García Aubert
aa0ce62a85
Implement batch logger to log query times when queries are defined with id
2016-09-29 15:09:36 +02:00
Raul Ochoa
d33fe5ac21
Stop indexing jobs per user
...
Removes .list() from job backend
2016-08-30 19:01:23 +02:00
Daniel García Aubert
5eaad4d5d9
Uses redis-mpool for pubsub in Batch API
2016-07-07 14:14:46 +02:00
Daniel García Aubert
a1f31df92e
Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job
2016-06-29 18:29:53 +02:00
Daniel García Aubert
0873b6fcaa
Merge branch 'master' into batch-add-profile
2016-06-03 12:03:17 +02:00
Daniel García Aubert
976bf5b039
Implemented profiling for job-runner and job-controller
2016-05-26 19:44:59 +02:00
Daniel García Aubert
64ad284c9c
WIP: adding metrics to Batch API
2016-05-24 11:19:00 +02:00
Daniel García Aubert
636686386e
Moved userDatabaseMetadataService from job_runner to query_runner
2016-05-23 17:51:56 +02:00
Daniel García Aubert
d2d3ba8159
Passed tests
2016-05-16 01:22:47 +02:00
Daniel García Aubert
cc7dd7a0d2
Job model refactor
2016-05-13 18:50:55 +02:00
Daniel
423e729e11
Merge pull request #283 from CartoDB/batch-improve-startup
...
Improved initialization in Batch API
2016-04-05 15:46:21 +02:00
Daniel García Aubert
b62edcb64f
Refactored batch-api startup. Isolated code related to queue discovering, api much clear and concise.
2016-04-04 16:05:33 +02:00
dgaubert
e8a44136c5
Now Batch API looks for queues with jobs to consume at service initialization time
2016-03-31 17:37:35 +02:00
dgaubert
fd9bfe277e
Implemented multiquery jobs for Batch API
2016-03-31 12:39:03 +02:00
Daniel García Aubert
e6f714a51b
Implemented some new unit test
2016-01-22 19:31:25 +01:00
Daniel García Aubert
c308551d05
Implemented drain to stop batch service gracefully.
...
Split job acceptance test
2016-01-13 16:25:25 +01:00
Daniel García Aubert
20f00d58d9
Refactored batch service to avoid event noise, doing in callback way
2016-01-08 15:47:59 +01:00
Daniel García Aubert
f9f52d2bd1
Improved object creation in batch service
2016-01-08 11:32:01 +01:00
Daniel García Aubert
3c23bf12e7
Fixed bug whether a job consumer is empty. And refactored batch module, now is a common object intead of module function
2015-12-22 20:12:10 +01:00
Daniel García Aubert
3762ad7a39
Now jobs are stored in redis instead of user's database
2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964
Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout
2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a
Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it.
2015-12-16 15:57:58 +01:00
Daniel García Aubert
d0787d03f7
Reused matadata backend instance for batch service
2015-12-14 10:36:16 +01:00
Daniel García Aubert
6cc48bf9dd
Implemented batch service
2015-12-09 20:17:45 +01:00
Daniel García Aubert
0c6edc573a
Used cartodb-redis instead redis-mpool
2015-12-09 12:18:33 +01:00
Daniel García Aubert
00721bcd02
Implementing batch service
2015-12-09 00:02:08 +01:00
