cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,013 Commits 1 Branch 118 Tags 9.7 MiB
df523960a2
Commit Graph

306 Commits

Author SHA1 Message Date
Raul Ochoa
1d20f11f0c Remove unused var 2016-10-06 18:44:21 +02:00
Raul Ochoa
578c43b1a8 Multiple queries jobs pushed as first job between queries 2016-10-06 18:27:38 +02:00
Raul Ochoa
e108d0df57 Debug query to run 2016-10-06 18:24:28 +02:00
Raul Ochoa
7c7320061f Merge branch 'master' into limit-batch-queries 2016-10-06 12:46:34 +02:00
Raul Ochoa
26fe6a1626 Add readme for batch queries feature 2016-10-06 12:27:38 +02:00
Raul Ochoa
eb2768c197 Add dbhost attribute to batch queries logs 2016-10-05 19:09:10 +02:00
Raul Ochoa
6c2db4385c Batch Queries: use path instead of stream to be able to reopen FD 2016-10-03 13:28:13 +02:00
Raul Ochoa
857ba747d0 Rename 2016-09-30 18:45:33 +02:00
Raul Ochoa
b269418db4 Tag logs 2016-09-30 18:45:15 +02:00
Raul Ochoa
72fb851db7 Fix typo 2016-09-30 16:55:33 +02:00
Raul Ochoa
20573a7f67 Always log queries from fallback jobs 
- Add query id if exists
- Only log analyses for expected format
- Log with query start and end times
 2016-09-30 16:54:10 +02:00
Raul Ochoa
ba34412ce3 Return on boolean on log 2016-09-30 16:51:34 +02:00
Raul Ochoa
9555a2cbde Bring back log logic for FallbackJob 2016-09-30 16:47:37 +02:00
Raul Ochoa
60546de147 Move log logic to each job 2016-09-30 16:44:39 +02:00
Daniel García Aubert
ed27b67cec Simplified batch logger construction 2016-09-29 15:17:25 +02:00
Daniel García Aubert
aa0ce62a85 Implement batch logger to log query times when queries are defined with id 2016-09-29 15:09:36 +02:00
Daniel García Aubert
4f3d361226 Fixes #356, set limit to batch queries to 12h 2016-09-13 12:32:41 +02:00
Raul Ochoa
461728d3e2 Remove user indexer 2016-08-30 19:08:06 +02:00
Raul Ochoa
d33fe5ac21 Stop indexing jobs per user 
Removes .list() from job backend
 2016-08-30 19:01:23 +02:00
Raul Ochoa
05ada98124 Remove .list() from job service 2016-08-30 18:51:51 +02:00
Raul Ochoa
4c8d734bbf Remove update method from job service 2016-08-30 17:46:32 +02:00
Daniel García Aubert
9f50475ad1 Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API 2016-08-30 13:50:03 +02:00
Daniel García Aubert
2932227e8b Improved naming for jobs TTL constant 2016-08-30 13:49:16 +02:00
Daniel García Aubert
02a252940a Improved naming for jobs TTL constant 2016-08-30 10:11:49 +02:00
Daniel García Aubert
0586f45413 Added callback to job subscriber to allow to batch service emit ready event 2016-07-22 13:47:14 +02:00
Daniel García Aubert
89c3681be0 Fix bug when checking if a job is found 2016-07-19 12:34:06 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
74d83a457e Now batch publisher sends a ping to server before publishing and create a new connection if error. 
Batch publisher and subscriber logs (if debug enabled) both outcoming and incoming messages to give more visibility.
 2016-07-07 10:44:17 +02:00
Raul Ochoa
be0f059f01 Add <%= job_id %> template support for onerror and onsuccess fallback queries 2016-06-30 17:41:02 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Raul Ochoa
85fa9d3c2b Merge branch 'master' into batch-onerror-template 
Conflicts:
	NEWS.md
 2016-06-29 16:25:05 +02:00
Raul Ochoa
a3117a2f01 Add <%= error_message %> template support for onerror fallback queries 2016-06-29 14:22:23 +02:00
Raul Ochoa
1d8f5539a7 Adds start and end time for batch queries with fallback 2016-06-29 13:56:45 +02:00
Daniel García Aubert
e387458801 Now profiler is passed as argument instead to be a member property of job runner 2016-06-22 16:10:42 +02:00
Daniel
39ceba6707 Merge pull request #312 from CartoDB/batch-add-profile 
Adds profiling to batch and job controller
 2016-06-08 19:31:36 +02:00
Daniel
ddd40f83f8 Merge pull request #316 from CartoDB/fix-error-handling-job-runner 
Fixed error handling in job query runner
 2016-06-08 11:13:59 +02:00
Daniel
58055080c9 Merge pull request #311 from CartoDB/309-skipped-status 
Fixes #309, added skipped status to fallback-jobs
 2016-06-08 11:12:59 +02:00
Daniel García Aubert
0873b6fcaa Merge branch 'master' into batch-add-profile 2016-06-03 12:03:17 +02:00
Daniel García Aubert
cc1a5641ea Fixed conflicts in merge 2016-06-03 11:43:21 +02:00
Daniel García Aubert
b6967f98f2 Fixed error handling in job query runner 2016-06-03 10:44:16 +02:00
Daniel García Aubert
7bd4f46935 Used right method to check if there is more queries from query in fallback job 2016-06-02 19:55:04 +02:00
Daniel García Aubert
62cb63f132 Refactored job fallback method 2016-06-02 19:39:48 +02:00
Daniel García Aubert
2a2127f4e1 Improved falback job to get the status of last finished query 2016-06-02 18:20:53 +02:00
Daniel García Aubert
86d02ab0b7 Refactored job fallback initialization 2016-06-02 14:28:35 +02:00
Daniel García Aubert
cd439757b7 Fix typo 2016-06-02 13:47:45 +02:00
Daniel García Aubert
25b9103af9 Removed duplicated code, method isValidTransition is encapsulated into a new object and job and query inherits from it 2016-06-02 12:26:20 +02:00
Daniel García Aubert
481a82500b Refactored fallback jobs 2016-06-01 10:39:01 +02:00
Daniel García Aubert
7de1e323b0 Avoid to use job_id to profile job timing 2016-05-30 12:43:42 +02:00
Daniel García Aubert
cf1e072c17 Improved steps in profiling 2016-05-27 12:41:24 +02:00
Daniel García Aubert
d3aa4a4140 Merge branch 'master' into batch-move-user-metadata-service 2016-05-27 11:47:28 +02:00
Daniel García Aubert
976bf5b039 Implemented profiling for job-runner and job-controller 2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577 Merge branch 'master' into batch-add-profile 2016-05-26 17:46:52 +02:00
Daniel García Aubert
1a0e2b681b Fixes #309, added skipped status to fallback-jobs 2016-05-26 17:37:37 +02:00
Daniel García Aubert
e079491a7e Fixed issue with status transition fallback jobs 2016-05-25 17:00:27 +02:00
Daniel García Aubert
40ed7d4dc2 Merge branch 'master' into 293-no-longer-running-job 2016-05-24 11:47:26 +02:00
Daniel García Aubert
232c8147de Adjust max-complexity value to linter 2016-05-24 11:26:23 +02:00
Daniel García Aubert
64ad284c9c WIP: adding metrics to Batch API 2016-05-24 11:19:00 +02:00
Daniel García Aubert
3caa1373bf Removed useless condition 2016-05-23 18:47:45 +02:00
Daniel García Aubert
636686386e Moved userDatabaseMetadataService from job_runner to query_runner 2016-05-23 17:51:56 +02:00
Daniel García Aubert
a60c0e1927 Moved job-factory to a proper place 2016-05-23 15:20:14 +02:00
Daniel García Aubert
7fce6c5dfc Avoid hacky 2016-05-23 15:13:31 +02:00
Daniel García Aubert
2d86fc2ce3 In fallback jobs, used fallback_status property instead of an array of statuses 2016-05-23 15:05:14 +02:00
Daniel García Aubert
f7392c48cc Improved naming in job's constructors 2016-05-23 11:37:09 +02:00
Daniel García Aubert
fef194e969 Fix bad use of jshint's hacks 2016-05-23 11:20:42 +02:00
Ubuntu
75fc80ecca Merge branch 'batch-fallback-job' into 293-no-longer-running-job 2016-05-23 09:13:31 +00:00
Daniel García Aubert
99d1efc739 Made host not mandatory for job model 2016-05-23 11:13:03 +02:00
Daniel García Aubert
60ab3eb6ba Fixes #293, set cancel status to a job that actually is no running but has running status. 2016-05-23 10:26:09 +02:00
Daniel García Aubert
d3e7f4b9f0 Fixed missing failed_reason in multiple jobs 2016-05-19 17:04:36 +02:00
Daniel García Aubert
e04787a070 Fixed missing failed_reason in failing jobs 2016-05-19 17:02:48 +02:00
Daniel García Aubert
62c8a9d315 Refactor and fixed issue, when cancelling a fallback job, it is not cancelled properly 2016-05-19 14:10:18 +02:00
Daniel García Aubert
5c4ced34e6 Added acceptance test for job cancelling 2016-05-18 13:25:08 +02:00
Daniel García Aubert
eb37fd305f Moved forever module 2016-05-18 11:55:58 +02:00
Daniel García Aubert
b94a9c7c0a Fixed conflicts 2016-05-18 11:48:39 +02:00
Daniel García Aubert
8b7d481b9a Avoided console usage, favor debug 2016-05-18 11:06:49 +02:00
Daniel García Aubert
9441017ed3 Moved job classes to a models folder 2016-05-17 20:00:45 +02:00
Daniel García Aubert
26cd778583 Improved param name 2016-05-17 19:55:33 +02:00
Daniel García Aubert
b06eaae5e6 Use REDIS_DB as constant 2016-05-17 19:48:55 +02:00
Daniel García Aubert
672b8ef537 Made pure functions 2016-05-17 19:44:45 +02:00
Daniel García Aubert
40d82112a8 Used REDIS_PREFIX as constant 2016-05-17 19:41:31 +02:00
Daniel García Aubert
3bac26cf2f Used JOB_PREFIX as constant 2016-05-17 19:27:05 +02:00
Daniel García Aubert
ef9dfae189 Moved job models to its own namespace 2016-05-17 18:59:39 +02:00
Daniel García Aubert
6acc02e998 Improved fallback setStatus method and added acceptance test 2016-05-17 18:51:02 +02:00
Daniel García Aubert
b2eac6b880 Implemented test for fallback jobs 2016-05-17 01:00:27 +02:00
Daniel García Aubert
be3d9d2316 Fallback jobs WIP 2016-05-16 13:49:56 +02:00
Daniel García Aubert
050f077e8c Avoid null values 2016-05-16 11:56:44 +02:00
Daniel García Aubert
d2d3ba8159 Passed tests 2016-05-16 01:22:47 +02:00
Daniel García Aubert
cc7dd7a0d2 Job model refactor 2016-05-13 18:50:55 +02:00
Daniel García Aubert
61bbdd6db4 Removed unused code 2016-04-05 18:52:12 +02:00
Daniel García Aubert
4a83de09af Added job status enumerable to avoid hardcodes strings 2016-04-05 18:50:04 +02:00
Daniel
423e729e11 Merge pull request #283 from CartoDB/batch-improve-startup 
Improved initialization in Batch API
 2016-04-05 15:46:21 +02:00
Daniel García Aubert
1414e0a9dd Fix #276, reload subcription to PUBSUB channel and looks for queues every 10 minutes 2016-04-04 20:00:18 +02:00
Daniel García Aubert
b62edcb64f Refactored batch-api startup. Isolated code related to queue discovering, api much clear and concise. 2016-04-04 16:05:33 +02:00
dgaubert
ed070e3040 Used SCAN instead of KEYS for looking for queues when Batch API starts 2016-04-01 15:03:32 +02:00
dgaubert
e2cc09f1dd Allowed modification to multiquery-jobs if its status is pending 2016-04-01 11:33:27 +02:00
dgaubert
c886e05b87 Improved condition in job backend 2016-03-31 19:14:38 +02:00
dgaubert
5c5c151237 Added failed_reason to query in multiquery jobs 2016-03-31 19:03:38 +02:00
dgaubert
e8a44136c5 Now Batch API looks for queues with jobs to consume at service initialization time 2016-03-31 17:37:35 +02:00
dgaubert
270cdf8969 Set pending to multiquery job when draining 2016-03-31 15:11:35 +02:00
dgaubert
f95e8ef7f2 Fixed job status when failing 2016-03-31 13:36:58 +02:00
dgaubert
fd9bfe277e Implemented multiquery jobs for Batch API 2016-03-31 12:39:03 +02:00
dgaubert
ef65350771 Returns job from redis after change any job status 2016-03-30 16:44:55 +02:00
Daniel García Aubert
c220b157b4 Removed extra empty lines 2016-03-21 19:47:07 +01:00
Daniel García Aubert
efca278bec Fixed error updating multiquery job 2016-03-21 19:43:28 +01:00
Daniel García Aubert
aefd32ee06 Improved message in failed_error property for multi-query jobs 2016-03-21 15:10:05 +01:00
Daniel García Aubert
ee0821a518 Changed error message in batch api 2016-03-21 12:29:43 +01:00
Daniel García Aubert
689f97c1c7 Improved function naming and query error message. 2016-03-21 12:09:54 +01:00
Daniel García Aubert
705e649b80 Fixed style typos 
Please enter the commit message for your changes. Lines starting
 2016-03-18 18:19:00 +01:00
Daniel García Aubert
5e0b86b181 Improved error message in multi-jobs 2016-03-18 16:23:40 +01:00
Daniel García Aubert
dcab147e90 Improved error handling for multi-jobs and cancelation. 2016-03-18 16:05:36 +01:00
Daniel García Aubert
056f22b156 Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series 2016-03-18 14:57:18 +01:00
Daniel García Aubert
fbaebfe8f2 Fixed weird message in Batch SQL API 2016-03-02 10:31:14 +01:00
Daniel García Aubert
da16f32789 Added uknown status to jobs when drain fails 2016-01-25 20:07:41 +01:00
Daniel García Aubert
e9bd93cf3f Now batch service is linked to express app in order to make it accesible from main app module to expose drain mechanism 2016-01-25 16:28:19 +01:00
Daniel García Aubert
1e49fbc33d Stopped batch service when draining 2016-01-25 15:32:00 +01:00
Daniel García Aubert
20fd536659 Now batch stop is synchronous, changed signature in test. 
Fixed minor issue when draining the last job, queue is got before cancel the job.
 2016-01-25 14:51:37 +01:00
Daniel García Aubert
e6f714a51b Implemented some new unit test 2016-01-22 19:31:25 +01:00
Daniel García Aubert
cb638bb0fe Fixed issue with Postgres error codes, implemented some test for job controller 2016-01-22 18:22:21 +01:00
Daniel García Aubert
5c4d239467 Fixed unit test for job queue and moved isJobFound function as private method in job backend 2016-01-22 12:43:41 +01:00
Daniel García Aubert
31ac48ef1a Changed job's ttl to be configurable 2016-01-21 15:33:42 +01:00
Daniel García Aubert
d07af1740a Fixed bug after cancel a job by user the job runner's callback is never called 2016-01-17 19:28:15 +01:00
Daniel García Aubert
a89e3776ec Removed unused callback in before hook in test and added proper configuration from user database config 2016-01-14 14:08:34 +01:00
Daniel García Aubert
c308551d05 Implemented drain to stop batch service gracefully. 
Split job acceptance test
 2016-01-13 16:25:25 +01:00
Daniel García Aubert
8d29e6b45c Improved log messages 2016-01-08 16:24:53 +01:00
Daniel García Aubert
20f00d58d9 Refactored batch service to avoid event noise, doing in callback way 2016-01-08 15:47:59 +01:00
Daniel García Aubert
f9f52d2bd1 Improved object creation in batch service 2016-01-08 11:32:01 +01:00
Daniel García Aubert
44d4fd1bb8 Minor improvement in job runner creation 2016-01-07 19:16:39 +01:00
Daniel García Aubert
f89c44586c Fixed bug related to growing file descriptors when batch api processes jobs 2016-01-07 18:36:32 +01:00
Daniel García Aubert
e6a4e0f7eb Returned 201 HTTP code in job creation 
Fixed bug listing jobs, now user job index is cleaned before respond to client
 2016-01-07 12:07:30 +01:00
Daniel García Aubert
681772bf37 Implemented test for Job Controller and implemented batch stop service 2016-01-05 17:42:28 +01:00
Daniel García Aubert
246655de94 Changed redis data structure for users jobs 2016-01-04 19:08:13 +01:00
Daniel García Aubert
13ffaab0d5 Resolved minor bug when user tries to cancel a job and included a ttl to jobs after its resolution 2016-01-04 15:20:06 +01:00
Daniel García Aubert
534da04270 Improved job cancelation and fixed minor bugs 2015-12-31 17:01:06 +01:00
Daniel García Aubert
35650985db Implemented job modification whether job is pending 2015-12-31 15:42:31 +01:00
Daniel García Aubert
ef354bdb4d Improved queue consuption avoiding possible stack overflow 2015-12-31 12:33:11 +01:00
Daniel García Aubert
ada39d84b8 Implemented job cancelation in Batch API 2015-12-30 20:16:18 +01:00
Daniel García Aubert
41a01bc0e3 Fixed typo in batch service 2015-12-30 11:51:32 +01:00
Daniel García Aubert
8c42019641 Improved namespace in redis for batch's stuff 2015-12-29 15:46:04 +01:00
Daniel García Aubert
851c393f75 Refactored batch manager avoiding concurrence issues 2015-12-29 10:19:10 +01:00
Daniel García Aubert
c4f96b4a59 Implemented support for listing jobs for one user 2015-12-24 17:42:49 +01:00
Daniel García Aubert
15d76f7cc7 Added postgres default port 2015-12-23 18:59:43 +01:00
Daniel García Aubert
ecb76ca1de Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs. 2015-12-23 17:29:11 +01:00
Daniel García Aubert
b16c0983c6 Implemented tests for batch API 2015-12-22 23:13:33 +01:00
Daniel García Aubert
3c23bf12e7 Fixed bug whether a job consumer is empty. And refactored batch module, now is a common object intead of module function 2015-12-22 20:12:10 +01:00
Daniel García Aubert
f41c43aa38 Disabled statement timeout bypassing pgbouncer connection 2015-12-22 15:43:00 +01:00
Daniel García Aubert
3762ad7a39 Now jobs are stored in redis instead of user's database 2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964 Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout 2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it. 2015-12-16 15:57:58 +01:00
Daniel García Aubert
d0787d03f7 Reused matadata backend instance for batch service 2015-12-14 10:36:16 +01:00
Daniel García Aubert
fe9280b0f7 Fixed bug in job manager when reenqueue a job when there is no slot to run it 2015-12-11 18:08:19 +01:00
Daniel García Aubert
1002375439 Fixed issue when retrieving user metadata was not parsed properly 2015-12-10 18:09:43 +01:00
Daniel García Aubert
f5ca879ce3 Added entry point for main app to batch api 2015-12-10 15:08:31 +01:00
Daniel García Aubert
a941d99b2f Now long running queries with statement SELECT are piped to a new table 2015-12-09 22:05:04 +01:00
Daniel García Aubert
6cc48bf9dd Implemented batch service 2015-12-09 20:17:45 +01:00
Daniel García Aubert
0c6edc573a Used cartodb-redis instead redis-mpool 2015-12-09 12:18:33 +01:00
Daniel García Aubert
00721bcd02 Implementing batch service 2015-12-09 00:02:08 +01:00