cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,551 Commits 1 Branch 118 Tags 9.7 MiB
e7068b8468
Commit Graph

165 Commits

Author SHA1 Message Date
Raul Ochoa
2c064041a1 Add dist lock to run all jobs by host in order 
It uses http://redis.io/topics/distlock
Which is not perfect: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
 2016-10-10 19:54:59 +02:00
Raul Ochoa
0de5d94617 Use debug with same params, no considering job status 2016-10-10 19:53:59 +02:00
Raul Ochoa
90c489119b Add distributed lock implementation with redis distlock 2016-10-10 19:51:11 +02:00
Raul Ochoa
56a632347b Inject publisher 2016-10-10 19:47:50 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Raul Ochoa
deb1ccf876 DRY job final statuses 2016-10-10 12:09:13 +02:00
Raul Ochoa
8a4f54bb87 Allow users to set max statement_timeout for their queries 2016-10-10 12:01:36 +02:00
Raul Ochoa
5401a7edff Timeout is passed into query runner 2016-10-10 12:00:54 +02:00
Raul Ochoa
51d4ff0698 Differentiate between statement timeout and user cancelled query 2016-10-10 11:58:44 +02:00
Raul Ochoa
1d20f11f0c Remove unused var 2016-10-06 18:44:21 +02:00
Raul Ochoa
578c43b1a8 Multiple queries jobs pushed as first job between queries 2016-10-06 18:27:38 +02:00
Raul Ochoa
e108d0df57 Debug query to run 2016-10-06 18:24:28 +02:00
Raul Ochoa
7c7320061f Merge branch 'master' into limit-batch-queries 2016-10-06 12:46:34 +02:00
Raul Ochoa
26fe6a1626 Add readme for batch queries feature 2016-10-06 12:27:38 +02:00
Raul Ochoa
eb2768c197 Add dbhost attribute to batch queries logs 2016-10-05 19:09:10 +02:00
Raul Ochoa
6c2db4385c Batch Queries: use path instead of stream to be able to reopen FD 2016-10-03 13:28:13 +02:00
Raul Ochoa
857ba747d0 Rename 2016-09-30 18:45:33 +02:00
Raul Ochoa
b269418db4 Tag logs 2016-09-30 18:45:15 +02:00
Raul Ochoa
72fb851db7 Fix typo 2016-09-30 16:55:33 +02:00
Raul Ochoa
20573a7f67 Always log queries from fallback jobs 
- Add query id if exists
- Only log analyses for expected format
- Log with query start and end times
 2016-09-30 16:54:10 +02:00
Raul Ochoa
ba34412ce3 Return on boolean on log 2016-09-30 16:51:34 +02:00
Raul Ochoa
9555a2cbde Bring back log logic for FallbackJob 2016-09-30 16:47:37 +02:00
Raul Ochoa
60546de147 Move log logic to each job 2016-09-30 16:44:39 +02:00
Daniel García Aubert
ed27b67cec Simplified batch logger construction 2016-09-29 15:17:25 +02:00
Daniel García Aubert
aa0ce62a85 Implement batch logger to log query times when queries are defined with id 2016-09-29 15:09:36 +02:00
Daniel García Aubert
4f3d361226 Fixes #356, set limit to batch queries to 12h 2016-09-13 12:32:41 +02:00
Raul Ochoa
461728d3e2 Remove user indexer 2016-08-30 19:08:06 +02:00
Raul Ochoa
d33fe5ac21 Stop indexing jobs per user 
Removes .list() from job backend
 2016-08-30 19:01:23 +02:00
Raul Ochoa
05ada98124 Remove .list() from job service 2016-08-30 18:51:51 +02:00
Raul Ochoa
4c8d734bbf Remove update method from job service 2016-08-30 17:46:32 +02:00
Daniel García Aubert
9f50475ad1 Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API 2016-08-30 13:50:03 +02:00
Daniel García Aubert
2932227e8b Improved naming for jobs TTL constant 2016-08-30 13:49:16 +02:00
Daniel García Aubert
02a252940a Improved naming for jobs TTL constant 2016-08-30 10:11:49 +02:00
Daniel García Aubert
0586f45413 Added callback to job subscriber to allow to batch service emit ready event 2016-07-22 13:47:14 +02:00
Daniel García Aubert
89c3681be0 Fix bug when checking if a job is found 2016-07-19 12:34:06 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
74d83a457e Now batch publisher sends a ping to server before publishing and create a new connection if error. 
Batch publisher and subscriber logs (if debug enabled) both outcoming and incoming messages to give more visibility.
 2016-07-07 10:44:17 +02:00
Raul Ochoa
be0f059f01 Add <%= job_id %> template support for onerror and onsuccess fallback queries 2016-06-30 17:41:02 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Raul Ochoa
85fa9d3c2b Merge branch 'master' into batch-onerror-template 
Conflicts:
	NEWS.md
 2016-06-29 16:25:05 +02:00
Raul Ochoa
a3117a2f01 Add <%= error_message %> template support for onerror fallback queries 2016-06-29 14:22:23 +02:00
Raul Ochoa
1d8f5539a7 Adds start and end time for batch queries with fallback 2016-06-29 13:56:45 +02:00
Daniel García Aubert
e387458801 Now profiler is passed as argument instead to be a member property of job runner 2016-06-22 16:10:42 +02:00
Daniel
39ceba6707 Merge pull request #312 from CartoDB/batch-add-profile 
Adds profiling to batch and job controller
 2016-06-08 19:31:36 +02:00
Daniel
ddd40f83f8 Merge pull request #316 from CartoDB/fix-error-handling-job-runner 
Fixed error handling in job query runner
 2016-06-08 11:13:59 +02:00
Daniel
58055080c9 Merge pull request #311 from CartoDB/309-skipped-status 
Fixes #309, added skipped status to fallback-jobs
 2016-06-08 11:12:59 +02:00
Daniel García Aubert
0873b6fcaa Merge branch 'master' into batch-add-profile 2016-06-03 12:03:17 +02:00
Daniel García Aubert
cc1a5641ea Fixed conflicts in merge 2016-06-03 11:43:21 +02:00
Daniel García Aubert
b6967f98f2 Fixed error handling in job query runner 2016-06-03 10:44:16 +02:00