cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,634 Commits 1 Branch 118 Tags 9.7 MiB
1ee0878631
Commit Graph

219 Commits

Author SHA1 Message Date
Raul Ochoa
611508c654 Hide queue seeker behind job subscriber 2016-10-11 19:01:39 +02:00
Raul Ochoa
e7c4ee32df Share redis channel config 2016-10-11 18:41:59 +02:00
Raul Ochoa
d15c7ab0de Always return client to pool 2016-10-11 18:30:35 +02:00
Raul Ochoa
e4b1711e8e pub/sub package 2016-10-11 18:28:46 +02:00
Raul Ochoa
2c064041a1 Add dist lock to run all jobs by host in order 
It uses http://redis.io/topics/distlock
Which is not perfect: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
 2016-10-10 19:54:59 +02:00
Raul Ochoa
0de5d94617 Use debug with same params, no considering job status 2016-10-10 19:53:59 +02:00
Raul Ochoa
90c489119b Add distributed lock implementation with redis distlock 2016-10-10 19:51:11 +02:00
Raul Ochoa
56a632347b Inject publisher 2016-10-10 19:47:50 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Raul Ochoa
deb1ccf876 DRY job final statuses 2016-10-10 12:09:13 +02:00
Raul Ochoa
8a4f54bb87 Allow users to set max statement_timeout for their queries 2016-10-10 12:01:36 +02:00
Raul Ochoa
5401a7edff Timeout is passed into query runner 2016-10-10 12:00:54 +02:00
Raul Ochoa
51d4ff0698 Differentiate between statement timeout and user cancelled query 2016-10-10 11:58:44 +02:00
Raul Ochoa
1d20f11f0c Remove unused var 2016-10-06 18:44:21 +02:00
Raul Ochoa
578c43b1a8 Multiple queries jobs pushed as first job between queries 2016-10-06 18:27:38 +02:00
Raul Ochoa
e108d0df57 Debug query to run 2016-10-06 18:24:28 +02:00
Raul Ochoa
7c7320061f Merge branch 'master' into limit-batch-queries 2016-10-06 12:46:34 +02:00
Raul Ochoa
26fe6a1626 Add readme for batch queries feature 2016-10-06 12:27:38 +02:00
Raul Ochoa
eb2768c197 Add dbhost attribute to batch queries logs 2016-10-05 19:09:10 +02:00
Raul Ochoa
6c2db4385c Batch Queries: use path instead of stream to be able to reopen FD 2016-10-03 13:28:13 +02:00
Raul Ochoa
857ba747d0 Rename 2016-09-30 18:45:33 +02:00
Raul Ochoa
b269418db4 Tag logs 2016-09-30 18:45:15 +02:00
Raul Ochoa
72fb851db7 Fix typo 2016-09-30 16:55:33 +02:00
Raul Ochoa
20573a7f67 Always log queries from fallback jobs 
- Add query id if exists
- Only log analyses for expected format
- Log with query start and end times
 2016-09-30 16:54:10 +02:00
Raul Ochoa
ba34412ce3 Return on boolean on log 2016-09-30 16:51:34 +02:00
Raul Ochoa
9555a2cbde Bring back log logic for FallbackJob 2016-09-30 16:47:37 +02:00
Raul Ochoa
60546de147 Move log logic to each job 2016-09-30 16:44:39 +02:00
Daniel García Aubert
ed27b67cec Simplified batch logger construction 2016-09-29 15:17:25 +02:00
Daniel García Aubert
aa0ce62a85 Implement batch logger to log query times when queries are defined with id 2016-09-29 15:09:36 +02:00
Daniel García Aubert
4f3d361226 Fixes #356, set limit to batch queries to 12h 2016-09-13 12:32:41 +02:00
Raul Ochoa
461728d3e2 Remove user indexer 2016-08-30 19:08:06 +02:00
Raul Ochoa
d33fe5ac21 Stop indexing jobs per user 
Removes .list() from job backend
 2016-08-30 19:01:23 +02:00
Raul Ochoa
05ada98124 Remove .list() from job service 2016-08-30 18:51:51 +02:00
Raul Ochoa
4c8d734bbf Remove update method from job service 2016-08-30 17:46:32 +02:00
Daniel García Aubert
9f50475ad1 Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API 2016-08-30 13:50:03 +02:00
Daniel García Aubert
2932227e8b Improved naming for jobs TTL constant 2016-08-30 13:49:16 +02:00
Daniel García Aubert
02a252940a Improved naming for jobs TTL constant 2016-08-30 10:11:49 +02:00
Daniel García Aubert
0586f45413 Added callback to job subscriber to allow to batch service emit ready event 2016-07-22 13:47:14 +02:00
Daniel García Aubert
89c3681be0 Fix bug when checking if a job is found 2016-07-19 12:34:06 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
74d83a457e Now batch publisher sends a ping to server before publishing and create a new connection if error. 
Batch publisher and subscriber logs (if debug enabled) both outcoming and incoming messages to give more visibility.
 2016-07-07 10:44:17 +02:00
Raul Ochoa
be0f059f01 Add <%= job_id %> template support for onerror and onsuccess fallback queries 2016-06-30 17:41:02 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Raul Ochoa
85fa9d3c2b Merge branch 'master' into batch-onerror-template 
Conflicts:
	NEWS.md
 2016-06-29 16:25:05 +02:00
Raul Ochoa
a3117a2f01 Add <%= error_message %> template support for onerror fallback queries 2016-06-29 14:22:23 +02:00
Raul Ochoa
1d8f5539a7 Adds start and end time for batch queries with fallback 2016-06-29 13:56:45 +02:00
Daniel García Aubert
e387458801 Now profiler is passed as argument instead to be a member property of job runner 2016-06-22 16:10:42 +02:00
Daniel
39ceba6707 Merge pull request #312 from CartoDB/batch-add-profile 
Adds profiling to batch and job controller
 2016-06-08 19:31:36 +02:00
Daniel
ddd40f83f8 Merge pull request #316 from CartoDB/fix-error-handling-job-runner 
Fixed error handling in job query runner
 2016-06-08 11:13:59 +02:00
Daniel
58055080c9 Merge pull request #311 from CartoDB/309-skipped-status 
Fixes #309, added skipped status to fallback-jobs
 2016-06-08 11:12:59 +02:00
Daniel García Aubert
0873b6fcaa Merge branch 'master' into batch-add-profile 2016-06-03 12:03:17 +02:00
Daniel García Aubert
cc1a5641ea Fixed conflicts in merge 2016-06-03 11:43:21 +02:00
Daniel García Aubert
b6967f98f2 Fixed error handling in job query runner 2016-06-03 10:44:16 +02:00
Daniel García Aubert
7bd4f46935 Used right method to check if there is more queries from query in fallback job 2016-06-02 19:55:04 +02:00
Daniel García Aubert
62cb63f132 Refactored job fallback method 2016-06-02 19:39:48 +02:00
Daniel García Aubert
2a2127f4e1 Improved falback job to get the status of last finished query 2016-06-02 18:20:53 +02:00
Daniel García Aubert
86d02ab0b7 Refactored job fallback initialization 2016-06-02 14:28:35 +02:00
Daniel García Aubert
cd439757b7 Fix typo 2016-06-02 13:47:45 +02:00
Daniel García Aubert
25b9103af9 Removed duplicated code, method isValidTransition is encapsulated into a new object and job and query inherits from it 2016-06-02 12:26:20 +02:00
Daniel García Aubert
481a82500b Refactored fallback jobs 2016-06-01 10:39:01 +02:00
Daniel García Aubert
7de1e323b0 Avoid to use job_id to profile job timing 2016-05-30 12:43:42 +02:00
Daniel García Aubert
cf1e072c17 Improved steps in profiling 2016-05-27 12:41:24 +02:00
Daniel García Aubert
d3aa4a4140 Merge branch 'master' into batch-move-user-metadata-service 2016-05-27 11:47:28 +02:00
Daniel García Aubert
976bf5b039 Implemented profiling for job-runner and job-controller 2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577 Merge branch 'master' into batch-add-profile 2016-05-26 17:46:52 +02:00
Daniel García Aubert
1a0e2b681b Fixes #309, added skipped status to fallback-jobs 2016-05-26 17:37:37 +02:00
Daniel García Aubert
e079491a7e Fixed issue with status transition fallback jobs 2016-05-25 17:00:27 +02:00
Daniel García Aubert
40ed7d4dc2 Merge branch 'master' into 293-no-longer-running-job 2016-05-24 11:47:26 +02:00
Daniel García Aubert
232c8147de Adjust max-complexity value to linter 2016-05-24 11:26:23 +02:00
Daniel García Aubert
64ad284c9c WIP: adding metrics to Batch API 2016-05-24 11:19:00 +02:00
Daniel García Aubert
3caa1373bf Removed useless condition 2016-05-23 18:47:45 +02:00
Daniel García Aubert
636686386e Moved userDatabaseMetadataService from job_runner to query_runner 2016-05-23 17:51:56 +02:00
Daniel García Aubert
a60c0e1927 Moved job-factory to a proper place 2016-05-23 15:20:14 +02:00
Daniel García Aubert
7fce6c5dfc Avoid hacky 2016-05-23 15:13:31 +02:00
Daniel García Aubert
2d86fc2ce3 In fallback jobs, used fallback_status property instead of an array of statuses 2016-05-23 15:05:14 +02:00
Daniel García Aubert
f7392c48cc Improved naming in job's constructors 2016-05-23 11:37:09 +02:00
Daniel García Aubert
fef194e969 Fix bad use of jshint's hacks 2016-05-23 11:20:42 +02:00
Ubuntu
75fc80ecca Merge branch 'batch-fallback-job' into 293-no-longer-running-job 2016-05-23 09:13:31 +00:00
Daniel García Aubert
99d1efc739 Made host not mandatory for job model 2016-05-23 11:13:03 +02:00
Daniel García Aubert
60ab3eb6ba Fixes #293, set cancel status to a job that actually is no running but has running status. 2016-05-23 10:26:09 +02:00
Daniel García Aubert
d3e7f4b9f0 Fixed missing failed_reason in multiple jobs 2016-05-19 17:04:36 +02:00
Daniel García Aubert
e04787a070 Fixed missing failed_reason in failing jobs 2016-05-19 17:02:48 +02:00
Daniel García Aubert
62c8a9d315 Refactor and fixed issue, when cancelling a fallback job, it is not cancelled properly 2016-05-19 14:10:18 +02:00
Daniel García Aubert
5c4ced34e6 Added acceptance test for job cancelling 2016-05-18 13:25:08 +02:00
Daniel García Aubert
eb37fd305f Moved forever module 2016-05-18 11:55:58 +02:00
Daniel García Aubert
b94a9c7c0a Fixed conflicts 2016-05-18 11:48:39 +02:00
Daniel García Aubert
8b7d481b9a Avoided console usage, favor debug 2016-05-18 11:06:49 +02:00
Daniel García Aubert
9441017ed3 Moved job classes to a models folder 2016-05-17 20:00:45 +02:00
Daniel García Aubert
26cd778583 Improved param name 2016-05-17 19:55:33 +02:00
Daniel García Aubert
b06eaae5e6 Use REDIS_DB as constant 2016-05-17 19:48:55 +02:00
Daniel García Aubert
672b8ef537 Made pure functions 2016-05-17 19:44:45 +02:00
Daniel García Aubert
40d82112a8 Used REDIS_PREFIX as constant 2016-05-17 19:41:31 +02:00
Daniel García Aubert
3bac26cf2f Used JOB_PREFIX as constant 2016-05-17 19:27:05 +02:00
Daniel García Aubert
ef9dfae189 Moved job models to its own namespace 2016-05-17 18:59:39 +02:00
Daniel García Aubert
6acc02e998 Improved fallback setStatus method and added acceptance test 2016-05-17 18:51:02 +02:00
Daniel García Aubert
b2eac6b880 Implemented test for fallback jobs 2016-05-17 01:00:27 +02:00
Daniel García Aubert
be3d9d2316 Fallback jobs WIP 2016-05-16 13:49:56 +02:00
Daniel García Aubert
050f077e8c Avoid null values 2016-05-16 11:56:44 +02:00
Daniel García Aubert
d2d3ba8159 Passed tests 2016-05-16 01:22:47 +02:00