Daniel García Aubert
|
2d86fc2ce3
|
In fallback jobs, used fallback_status property instead of an array of statuses
|
2016-05-23 15:05:14 +02:00 |
|
Daniel García Aubert
|
f7392c48cc
|
Improved naming in job's constructors
|
2016-05-23 11:37:09 +02:00 |
|
Daniel García Aubert
|
fef194e969
|
Fix bad use of jshint's hacks
|
2016-05-23 11:20:42 +02:00 |
|
Ubuntu
|
75fc80ecca
|
Merge branch 'batch-fallback-job' into 293-no-longer-running-job
|
2016-05-23 09:13:31 +00:00 |
|
Daniel García Aubert
|
99d1efc739
|
Made host not mandatory for job model
|
2016-05-23 11:13:03 +02:00 |
|
Daniel García Aubert
|
60ab3eb6ba
|
Fixes #293, set cancel status to a job that actually is no running but has running status.
|
2016-05-23 10:26:09 +02:00 |
|
Daniel García Aubert
|
d3e7f4b9f0
|
Fixed missing failed_reason in multiple jobs
|
2016-05-19 17:04:36 +02:00 |
|
Daniel García Aubert
|
e04787a070
|
Fixed missing failed_reason in failing jobs
|
2016-05-19 17:02:48 +02:00 |
|
Daniel García Aubert
|
62c8a9d315
|
Refactor and fixed issue, when cancelling a fallback job, it is not cancelled properly
|
2016-05-19 14:10:18 +02:00 |
|
Daniel García Aubert
|
5c4ced34e6
|
Added acceptance test for job cancelling
|
2016-05-18 13:25:08 +02:00 |
|
Daniel García Aubert
|
eb37fd305f
|
Moved forever module
|
2016-05-18 11:55:58 +02:00 |
|
Daniel García Aubert
|
b94a9c7c0a
|
Fixed conflicts
|
2016-05-18 11:48:39 +02:00 |
|
Daniel García Aubert
|
8b7d481b9a
|
Avoided console usage, favor debug
|
2016-05-18 11:06:49 +02:00 |
|
Daniel García Aubert
|
9441017ed3
|
Moved job classes to a models folder
|
2016-05-17 20:00:45 +02:00 |
|
Daniel García Aubert
|
26cd778583
|
Improved param name
|
2016-05-17 19:55:33 +02:00 |
|
Daniel García Aubert
|
b06eaae5e6
|
Use REDIS_DB as constant
|
2016-05-17 19:48:55 +02:00 |
|
Daniel García Aubert
|
672b8ef537
|
Made pure functions
|
2016-05-17 19:44:45 +02:00 |
|
Daniel García Aubert
|
40d82112a8
|
Used REDIS_PREFIX as constant
|
2016-05-17 19:41:31 +02:00 |
|
Daniel García Aubert
|
3bac26cf2f
|
Used JOB_PREFIX as constant
|
2016-05-17 19:27:05 +02:00 |
|
Daniel García Aubert
|
ef9dfae189
|
Moved job models to its own namespace
|
2016-05-17 18:59:39 +02:00 |
|
Daniel García Aubert
|
6acc02e998
|
Improved fallback setStatus method and added acceptance test
|
2016-05-17 18:51:02 +02:00 |
|
Daniel García Aubert
|
b2eac6b880
|
Implemented test for fallback jobs
|
2016-05-17 01:00:27 +02:00 |
|
Daniel García Aubert
|
be3d9d2316
|
Fallback jobs WIP
|
2016-05-16 13:49:56 +02:00 |
|
Daniel García Aubert
|
050f077e8c
|
Avoid null values
|
2016-05-16 11:56:44 +02:00 |
|
Daniel García Aubert
|
d2d3ba8159
|
Passed tests
|
2016-05-16 01:22:47 +02:00 |
|
Daniel García Aubert
|
cc7dd7a0d2
|
Job model refactor
|
2016-05-13 18:50:55 +02:00 |
|
Daniel García Aubert
|
61bbdd6db4
|
Removed unused code
|
2016-04-05 18:52:12 +02:00 |
|
Daniel García Aubert
|
4a83de09af
|
Added job status enumerable to avoid hardcodes strings
|
2016-04-05 18:50:04 +02:00 |
|
Daniel
|
423e729e11
|
Merge pull request #283 from CartoDB/batch-improve-startup
Improved initialization in Batch API
|
2016-04-05 15:46:21 +02:00 |
|
Daniel García Aubert
|
1414e0a9dd
|
Fix #276, reload subcription to PUBSUB channel and looks for queues every 10 minutes
|
2016-04-04 20:00:18 +02:00 |
|
Daniel García Aubert
|
b62edcb64f
|
Refactored batch-api startup. Isolated code related to queue discovering, api much clear and concise.
|
2016-04-04 16:05:33 +02:00 |
|
dgaubert
|
ed070e3040
|
Used SCAN instead of KEYS for looking for queues when Batch API starts
|
2016-04-01 15:03:32 +02:00 |
|
dgaubert
|
e2cc09f1dd
|
Allowed modification to multiquery-jobs if its status is pending
|
2016-04-01 11:33:27 +02:00 |
|
dgaubert
|
c886e05b87
|
Improved condition in job backend
|
2016-03-31 19:14:38 +02:00 |
|
dgaubert
|
5c5c151237
|
Added failed_reason to query in multiquery jobs
|
2016-03-31 19:03:38 +02:00 |
|
dgaubert
|
e8a44136c5
|
Now Batch API looks for queues with jobs to consume at service initialization time
|
2016-03-31 17:37:35 +02:00 |
|
dgaubert
|
270cdf8969
|
Set pending to multiquery job when draining
|
2016-03-31 15:11:35 +02:00 |
|
dgaubert
|
f95e8ef7f2
|
Fixed job status when failing
|
2016-03-31 13:36:58 +02:00 |
|
dgaubert
|
fd9bfe277e
|
Implemented multiquery jobs for Batch API
|
2016-03-31 12:39:03 +02:00 |
|
dgaubert
|
ef65350771
|
Returns job from redis after change any job status
|
2016-03-30 16:44:55 +02:00 |
|
Daniel García Aubert
|
c220b157b4
|
Removed extra empty lines
|
2016-03-21 19:47:07 +01:00 |
|
Daniel García Aubert
|
efca278bec
|
Fixed error updating multiquery job
|
2016-03-21 19:43:28 +01:00 |
|
Daniel García Aubert
|
aefd32ee06
|
Improved message in failed_error property for multi-query jobs
|
2016-03-21 15:10:05 +01:00 |
|
Daniel García Aubert
|
ee0821a518
|
Changed error message in batch api
|
2016-03-21 12:29:43 +01:00 |
|
Daniel García Aubert
|
689f97c1c7
|
Improved function naming and query error message.
|
2016-03-21 12:09:54 +01:00 |
|
Daniel García Aubert
|
705e649b80
|
Fixed style typos
Please enter the commit message for your changes. Lines starting
|
2016-03-18 18:19:00 +01:00 |
|
Daniel García Aubert
|
5e0b86b181
|
Improved error message in multi-jobs
|
2016-03-18 16:23:40 +01:00 |
|
Daniel García Aubert
|
dcab147e90
|
Improved error handling for multi-jobs and cancelation.
|
2016-03-18 16:05:36 +01:00 |
|
Daniel García Aubert
|
056f22b156
|
Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series
|
2016-03-18 14:57:18 +01:00 |
|
Daniel García Aubert
|
fbaebfe8f2
|
Fixed weird message in Batch SQL API
|
2016-03-02 10:31:14 +01:00 |
|
Daniel García Aubert
|
da16f32789
|
Added uknown status to jobs when drain fails
|
2016-01-25 20:07:41 +01:00 |
|
Daniel García Aubert
|
e9bd93cf3f
|
Now batch service is linked to express app in order to make it accesible from main app module to expose drain mechanism
|
2016-01-25 16:28:19 +01:00 |
|
Daniel García Aubert
|
1e49fbc33d
|
Stopped batch service when draining
|
2016-01-25 15:32:00 +01:00 |
|
Daniel García Aubert
|
20fd536659
|
Now batch stop is synchronous, changed signature in test.
Fixed minor issue when draining the last job, queue is got before cancel the job.
|
2016-01-25 14:51:37 +01:00 |
|
Daniel García Aubert
|
e6f714a51b
|
Implemented some new unit test
|
2016-01-22 19:31:25 +01:00 |
|
Daniel García Aubert
|
cb638bb0fe
|
Fixed issue with Postgres error codes, implemented some test for job controller
|
2016-01-22 18:22:21 +01:00 |
|
Daniel García Aubert
|
5c4d239467
|
Fixed unit test for job queue and moved isJobFound function as private method in job backend
|
2016-01-22 12:43:41 +01:00 |
|
Daniel García Aubert
|
31ac48ef1a
|
Changed job's ttl to be configurable
|
2016-01-21 15:33:42 +01:00 |
|
Daniel García Aubert
|
d07af1740a
|
Fixed bug after cancel a job by user the job runner's callback is never called
|
2016-01-17 19:28:15 +01:00 |
|
Daniel García Aubert
|
a89e3776ec
|
Removed unused callback in before hook in test and added proper configuration from user database config
|
2016-01-14 14:08:34 +01:00 |
|
Daniel García Aubert
|
c308551d05
|
Implemented drain to stop batch service gracefully.
Split job acceptance test
|
2016-01-13 16:25:25 +01:00 |
|
Daniel García Aubert
|
8d29e6b45c
|
Improved log messages
|
2016-01-08 16:24:53 +01:00 |
|
Daniel García Aubert
|
20f00d58d9
|
Refactored batch service to avoid event noise, doing in callback way
|
2016-01-08 15:47:59 +01:00 |
|
Daniel García Aubert
|
f9f52d2bd1
|
Improved object creation in batch service
|
2016-01-08 11:32:01 +01:00 |
|
Daniel García Aubert
|
44d4fd1bb8
|
Minor improvement in job runner creation
|
2016-01-07 19:16:39 +01:00 |
|
Daniel García Aubert
|
f89c44586c
|
Fixed bug related to growing file descriptors when batch api processes jobs
|
2016-01-07 18:36:32 +01:00 |
|
Daniel García Aubert
|
e6a4e0f7eb
|
Returned 201 HTTP code in job creation
Fixed bug listing jobs, now user job index is cleaned before respond to client
|
2016-01-07 12:07:30 +01:00 |
|
Daniel García Aubert
|
681772bf37
|
Implemented test for Job Controller and implemented batch stop service
|
2016-01-05 17:42:28 +01:00 |
|
Daniel García Aubert
|
246655de94
|
Changed redis data structure for users jobs
|
2016-01-04 19:08:13 +01:00 |
|
Daniel García Aubert
|
13ffaab0d5
|
Resolved minor bug when user tries to cancel a job and included a ttl to jobs after its resolution
|
2016-01-04 15:20:06 +01:00 |
|
Daniel García Aubert
|
534da04270
|
Improved job cancelation and fixed minor bugs
|
2015-12-31 17:01:06 +01:00 |
|
Daniel García Aubert
|
35650985db
|
Implemented job modification whether job is pending
|
2015-12-31 15:42:31 +01:00 |
|
Daniel García Aubert
|
ef354bdb4d
|
Improved queue consuption avoiding possible stack overflow
|
2015-12-31 12:33:11 +01:00 |
|
Daniel García Aubert
|
ada39d84b8
|
Implemented job cancelation in Batch API
|
2015-12-30 20:16:18 +01:00 |
|
Daniel García Aubert
|
41a01bc0e3
|
Fixed typo in batch service
|
2015-12-30 11:51:32 +01:00 |
|
Daniel García Aubert
|
8c42019641
|
Improved namespace in redis for batch's stuff
|
2015-12-29 15:46:04 +01:00 |
|
Daniel García Aubert
|
851c393f75
|
Refactored batch manager avoiding concurrence issues
|
2015-12-29 10:19:10 +01:00 |
|
Daniel García Aubert
|
c4f96b4a59
|
Implemented support for listing jobs for one user
|
2015-12-24 17:42:49 +01:00 |
|
Daniel García Aubert
|
15d76f7cc7
|
Added postgres default port
|
2015-12-23 18:59:43 +01:00 |
|
Daniel García Aubert
|
ecb76ca1de
|
Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs.
|
2015-12-23 17:29:11 +01:00 |
|
Daniel García Aubert
|
b16c0983c6
|
Implemented tests for batch API
|
2015-12-22 23:13:33 +01:00 |
|
Daniel García Aubert
|
3c23bf12e7
|
Fixed bug whether a job consumer is empty. And refactored batch module, now is a common object intead of module function
|
2015-12-22 20:12:10 +01:00 |
|
Daniel García Aubert
|
f41c43aa38
|
Disabled statement timeout bypassing pgbouncer connection
|
2015-12-22 15:43:00 +01:00 |
|
Daniel García Aubert
|
3762ad7a39
|
Now jobs are stored in redis instead of user's database
|
2015-12-21 19:57:10 +01:00 |
|
Daniel García Aubert
|
fb8feeb964
|
Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout
|
2015-12-16 18:13:48 +01:00 |
|
Daniel García Aubert
|
43f759e96a
|
Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it.
|
2015-12-16 15:57:58 +01:00 |
|
Daniel García Aubert
|
d0787d03f7
|
Reused matadata backend instance for batch service
|
2015-12-14 10:36:16 +01:00 |
|
Daniel García Aubert
|
fe9280b0f7
|
Fixed bug in job manager when reenqueue a job when there is no slot to run it
|
2015-12-11 18:08:19 +01:00 |
|
Daniel García Aubert
|
1002375439
|
Fixed issue when retrieving user metadata was not parsed properly
|
2015-12-10 18:09:43 +01:00 |
|
Daniel García Aubert
|
f5ca879ce3
|
Added entry point for main app to batch api
|
2015-12-10 15:08:31 +01:00 |
|
Daniel García Aubert
|
a941d99b2f
|
Now long running queries with statement SELECT are piped to a new table
|
2015-12-09 22:05:04 +01:00 |
|
Daniel García Aubert
|
6cc48bf9dd
|
Implemented batch service
|
2015-12-09 20:17:45 +01:00 |
|
Daniel García Aubert
|
0c6edc573a
|
Used cartodb-redis instead redis-mpool
|
2015-12-09 12:18:33 +01:00 |
|
Daniel García Aubert
|
00721bcd02
|
Implementing batch service
|
2015-12-09 00:02:08 +01:00 |
|