cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,578 Commits 1 Branch 118 Tags 9.7 MiB
f7d1f9426c
Commit Graph

187 Commits

Author SHA1 Message Date
Raul Ochoa
05ada98124 Remove .list() from job service 2016-08-30 18:51:51 +02:00
Raul Ochoa
4c8d734bbf Remove update method from job service 2016-08-30 17:46:32 +02:00
Daniel García Aubert
9f50475ad1 Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API 2016-08-30 13:50:03 +02:00
Daniel García Aubert
2932227e8b Improved naming for jobs TTL constant 2016-08-30 13:49:16 +02:00
Daniel García Aubert
02a252940a Improved naming for jobs TTL constant 2016-08-30 10:11:49 +02:00
Daniel García Aubert
0586f45413 Added callback to job subscriber to allow to batch service emit ready event 2016-07-22 13:47:14 +02:00
Daniel García Aubert
89c3681be0 Fix bug when checking if a job is found 2016-07-19 12:34:06 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
74d83a457e Now batch publisher sends a ping to server before publishing and create a new connection if error. 
Batch publisher and subscriber logs (if debug enabled) both outcoming and incoming messages to give more visibility.
 2016-07-07 10:44:17 +02:00
Raul Ochoa
be0f059f01 Add <%= job_id %> template support for onerror and onsuccess fallback queries 2016-06-30 17:41:02 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Raul Ochoa
85fa9d3c2b Merge branch 'master' into batch-onerror-template 
Conflicts:
	NEWS.md
 2016-06-29 16:25:05 +02:00
Raul Ochoa
a3117a2f01 Add <%= error_message %> template support for onerror fallback queries 2016-06-29 14:22:23 +02:00
Raul Ochoa
1d8f5539a7 Adds start and end time for batch queries with fallback 2016-06-29 13:56:45 +02:00
Daniel García Aubert
e387458801 Now profiler is passed as argument instead to be a member property of job runner 2016-06-22 16:10:42 +02:00
Daniel
39ceba6707 Merge pull request #312 from CartoDB/batch-add-profile 
Adds profiling to batch and job controller
 2016-06-08 19:31:36 +02:00
Daniel
ddd40f83f8 Merge pull request #316 from CartoDB/fix-error-handling-job-runner 
Fixed error handling in job query runner
 2016-06-08 11:13:59 +02:00
Daniel
58055080c9 Merge pull request #311 from CartoDB/309-skipped-status 
Fixes #309, added skipped status to fallback-jobs
 2016-06-08 11:12:59 +02:00
Daniel García Aubert
0873b6fcaa Merge branch 'master' into batch-add-profile 2016-06-03 12:03:17 +02:00
Daniel García Aubert
cc1a5641ea Fixed conflicts in merge 2016-06-03 11:43:21 +02:00
Daniel García Aubert
b6967f98f2 Fixed error handling in job query runner 2016-06-03 10:44:16 +02:00
Daniel García Aubert
7bd4f46935 Used right method to check if there is more queries from query in fallback job 2016-06-02 19:55:04 +02:00
Daniel García Aubert
62cb63f132 Refactored job fallback method 2016-06-02 19:39:48 +02:00
Daniel García Aubert
2a2127f4e1 Improved falback job to get the status of last finished query 2016-06-02 18:20:53 +02:00
Daniel García Aubert
86d02ab0b7 Refactored job fallback initialization 2016-06-02 14:28:35 +02:00
Daniel García Aubert
cd439757b7 Fix typo 2016-06-02 13:47:45 +02:00
Daniel García Aubert
25b9103af9 Removed duplicated code, method isValidTransition is encapsulated into a new object and job and query inherits from it 2016-06-02 12:26:20 +02:00
Daniel García Aubert
481a82500b Refactored fallback jobs 2016-06-01 10:39:01 +02:00
Daniel García Aubert
7de1e323b0 Avoid to use job_id to profile job timing 2016-05-30 12:43:42 +02:00
Daniel García Aubert
cf1e072c17 Improved steps in profiling 2016-05-27 12:41:24 +02:00
Daniel García Aubert
d3aa4a4140 Merge branch 'master' into batch-move-user-metadata-service 2016-05-27 11:47:28 +02:00
Daniel García Aubert
976bf5b039 Implemented profiling for job-runner and job-controller 2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577 Merge branch 'master' into batch-add-profile 2016-05-26 17:46:52 +02:00
Daniel García Aubert
1a0e2b681b Fixes #309, added skipped status to fallback-jobs 2016-05-26 17:37:37 +02:00
Daniel García Aubert
e079491a7e Fixed issue with status transition fallback jobs 2016-05-25 17:00:27 +02:00
Daniel García Aubert
40ed7d4dc2 Merge branch 'master' into 293-no-longer-running-job 2016-05-24 11:47:26 +02:00
Daniel García Aubert
232c8147de Adjust max-complexity value to linter 2016-05-24 11:26:23 +02:00
Daniel García Aubert
64ad284c9c WIP: adding metrics to Batch API 2016-05-24 11:19:00 +02:00
Daniel García Aubert
3caa1373bf Removed useless condition 2016-05-23 18:47:45 +02:00
Daniel García Aubert
636686386e Moved userDatabaseMetadataService from job_runner to query_runner 2016-05-23 17:51:56 +02:00
Daniel García Aubert
a60c0e1927 Moved job-factory to a proper place 2016-05-23 15:20:14 +02:00
Daniel García Aubert
7fce6c5dfc Avoid hacky 2016-05-23 15:13:31 +02:00
Daniel García Aubert
2d86fc2ce3 In fallback jobs, used fallback_status property instead of an array of statuses 2016-05-23 15:05:14 +02:00
Daniel García Aubert
f7392c48cc Improved naming in job's constructors 2016-05-23 11:37:09 +02:00
Daniel García Aubert
fef194e969 Fix bad use of jshint's hacks 2016-05-23 11:20:42 +02:00
Ubuntu
75fc80ecca Merge branch 'batch-fallback-job' into 293-no-longer-running-job 2016-05-23 09:13:31 +00:00
Daniel García Aubert
99d1efc739 Made host not mandatory for job model 2016-05-23 11:13:03 +02:00
Daniel García Aubert
60ab3eb6ba Fixes #293, set cancel status to a job that actually is no running but has running status. 2016-05-23 10:26:09 +02:00
Daniel García Aubert
d3e7f4b9f0 Fixed missing failed_reason in multiple jobs 2016-05-19 17:04:36 +02:00
Daniel García Aubert
e04787a070 Fixed missing failed_reason in failing jobs 2016-05-19 17:02:48 +02:00
Daniel García Aubert
62c8a9d315 Refactor and fixed issue, when cancelling a fallback job, it is not cancelled properly 2016-05-19 14:10:18 +02:00
Daniel García Aubert
5c4ced34e6 Added acceptance test for job cancelling 2016-05-18 13:25:08 +02:00
Daniel García Aubert
eb37fd305f Moved forever module 2016-05-18 11:55:58 +02:00
Daniel García Aubert
b94a9c7c0a Fixed conflicts 2016-05-18 11:48:39 +02:00
Daniel García Aubert
8b7d481b9a Avoided console usage, favor debug 2016-05-18 11:06:49 +02:00
Daniel García Aubert
9441017ed3 Moved job classes to a models folder 2016-05-17 20:00:45 +02:00
Daniel García Aubert
26cd778583 Improved param name 2016-05-17 19:55:33 +02:00
Daniel García Aubert
b06eaae5e6 Use REDIS_DB as constant 2016-05-17 19:48:55 +02:00
Daniel García Aubert
672b8ef537 Made pure functions 2016-05-17 19:44:45 +02:00
Daniel García Aubert
40d82112a8 Used REDIS_PREFIX as constant 2016-05-17 19:41:31 +02:00
Daniel García Aubert
3bac26cf2f Used JOB_PREFIX as constant 2016-05-17 19:27:05 +02:00
Daniel García Aubert
ef9dfae189 Moved job models to its own namespace 2016-05-17 18:59:39 +02:00
Daniel García Aubert
6acc02e998 Improved fallback setStatus method and added acceptance test 2016-05-17 18:51:02 +02:00
Daniel García Aubert
b2eac6b880 Implemented test for fallback jobs 2016-05-17 01:00:27 +02:00
Daniel García Aubert
be3d9d2316 Fallback jobs WIP 2016-05-16 13:49:56 +02:00
Daniel García Aubert
050f077e8c Avoid null values 2016-05-16 11:56:44 +02:00
Daniel García Aubert
d2d3ba8159 Passed tests 2016-05-16 01:22:47 +02:00
Daniel García Aubert
cc7dd7a0d2 Job model refactor 2016-05-13 18:50:55 +02:00
Daniel García Aubert
61bbdd6db4 Removed unused code 2016-04-05 18:52:12 +02:00
Daniel García Aubert
4a83de09af Added job status enumerable to avoid hardcodes strings 2016-04-05 18:50:04 +02:00
Daniel
423e729e11 Merge pull request #283 from CartoDB/batch-improve-startup 
Improved initialization in Batch API
 2016-04-05 15:46:21 +02:00
Daniel García Aubert
1414e0a9dd Fix #276, reload subcription to PUBSUB channel and looks for queues every 10 minutes 2016-04-04 20:00:18 +02:00
Daniel García Aubert
b62edcb64f Refactored batch-api startup. Isolated code related to queue discovering, api much clear and concise. 2016-04-04 16:05:33 +02:00
dgaubert
ed070e3040 Used SCAN instead of KEYS for looking for queues when Batch API starts 2016-04-01 15:03:32 +02:00
dgaubert
e2cc09f1dd Allowed modification to multiquery-jobs if its status is pending 2016-04-01 11:33:27 +02:00
dgaubert
c886e05b87 Improved condition in job backend 2016-03-31 19:14:38 +02:00
dgaubert
5c5c151237 Added failed_reason to query in multiquery jobs 2016-03-31 19:03:38 +02:00
dgaubert
e8a44136c5 Now Batch API looks for queues with jobs to consume at service initialization time 2016-03-31 17:37:35 +02:00
dgaubert
270cdf8969 Set pending to multiquery job when draining 2016-03-31 15:11:35 +02:00
dgaubert
f95e8ef7f2 Fixed job status when failing 2016-03-31 13:36:58 +02:00
dgaubert
fd9bfe277e Implemented multiquery jobs for Batch API 2016-03-31 12:39:03 +02:00
dgaubert
ef65350771 Returns job from redis after change any job status 2016-03-30 16:44:55 +02:00
Daniel García Aubert
c220b157b4 Removed extra empty lines 2016-03-21 19:47:07 +01:00
Daniel García Aubert
efca278bec Fixed error updating multiquery job 2016-03-21 19:43:28 +01:00
Daniel García Aubert
aefd32ee06 Improved message in failed_error property for multi-query jobs 2016-03-21 15:10:05 +01:00
Daniel García Aubert
ee0821a518 Changed error message in batch api 2016-03-21 12:29:43 +01:00
Daniel García Aubert
689f97c1c7 Improved function naming and query error message. 2016-03-21 12:09:54 +01:00
Daniel García Aubert
705e649b80 Fixed style typos 
Please enter the commit message for your changes. Lines starting
 2016-03-18 18:19:00 +01:00
Daniel García Aubert
5e0b86b181 Improved error message in multi-jobs 2016-03-18 16:23:40 +01:00
Daniel García Aubert
dcab147e90 Improved error handling for multi-jobs and cancelation. 2016-03-18 16:05:36 +01:00
Daniel García Aubert
056f22b156 Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series 2016-03-18 14:57:18 +01:00
Daniel García Aubert
fbaebfe8f2 Fixed weird message in Batch SQL API 2016-03-02 10:31:14 +01:00
Daniel García Aubert
da16f32789 Added uknown status to jobs when drain fails 2016-01-25 20:07:41 +01:00
Daniel García Aubert
e9bd93cf3f Now batch service is linked to express app in order to make it accesible from main app module to expose drain mechanism 2016-01-25 16:28:19 +01:00
Daniel García Aubert
1e49fbc33d Stopped batch service when draining 2016-01-25 15:32:00 +01:00
Daniel García Aubert
20fd536659 Now batch stop is synchronous, changed signature in test. 
Fixed minor issue when draining the last job, queue is got before cancel the job.
 2016-01-25 14:51:37 +01:00
Daniel García Aubert
e6f714a51b Implemented some new unit test 2016-01-22 19:31:25 +01:00
Daniel García Aubert
cb638bb0fe Fixed issue with Postgres error codes, implemented some test for job controller 2016-01-22 18:22:21 +01:00
Daniel García Aubert
5c4d239467 Fixed unit test for job queue and moved isJobFound function as private method in job backend 2016-01-22 12:43:41 +01:00