cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,909 Commits 1 Branch 118 Tags 9.7 MiB
14846b254e
Commit Graph

274 Commits

Author SHA1 Message Date
Raul Ochoa
66cc137d04 Split http capacity between simple and load 
- Simple will use 'available_cores' from response.
- Load will use 'cores' and 'relative_load'.
 2016-10-20 11:12:08 +02:00
Raul Ochoa
0af5cf703a Allow to configure capacity strategy 
- HTTP strategy: mechanism to compute load from db host.
- Fixed strategy: hardcoded number of queries to run at the same time, via configuration.
 2016-10-19 18:42:53 +02:00
Raul Ochoa
9596ac4730 Scheduler handles new tasks when there is free slots 2016-10-19 16:59:27 +02:00
Raul Ochoa
95b3a8adf1 Be explicit about queue status 2016-10-19 16:58:31 +02:00
Raul Ochoa
b164ec8c86 Better debugging 2016-10-19 16:58:00 +02:00
Raul Ochoa
1ee0878631 Scheduler uses a red–black tree to decide on next job candidate 2016-10-19 16:55:49 +02:00
Raul Ochoa
71d32e003b Better debug 2016-10-19 11:46:02 +02:00
Raul Ochoa
4daa39bd2c Start scheduler from host-scheduler 2016-10-19 11:45:48 +02:00
Raul Ochoa
e26bed2e66 Move status close to entity 2016-10-19 10:45:37 +02:00
Raul Ochoa
ca3d71ea48 Tasks with their own entity 
- Use a list of tasks and keep an index per user.
- Removes WAITING status.

TODO: improve candidate selection.
 2016-10-19 10:43:24 +02:00
Raul Ochoa
6c232a1fd0 Discard numeric status 2016-10-19 10:40:03 +02:00
Raul Ochoa
51ac1a3ab7 Remove TODO as it is already done 2016-10-19 10:38:39 +02:00
Raul Ochoa
ac65c1c39a Rename 2016-10-19 10:36:13 +02:00
Raul Ochoa
3a57331a54 Delegate job scheduling 
There is a host scheduler managing the host locking.

When it can acquire a lock over the host it will delegate
all the tasks related to that host to the same scheduler.

This scheduler will take care of how many jobs it will submit,
and in which order. It's also responsible for guaranteeing the
execution order per user.

Capacity planner dictates how many jobs can be run at the
same time in a given host. There are two simple strategies:

1. Infinity: it will attempt to run as many jobs as different users.
2. One: it will run just one job at the same time.

Missing things:
 - Handle lock renewal failures.
 - Fair scheduling for pending/waiting users.
 - Capacity based on real resources.
 2016-10-18 20:43:15 +02:00
Raul Ochoa
dce051d52b Make leader locker to emit on renewal errors 2016-10-18 20:34:22 +02:00
Raul Ochoa
d1e3be2e22 Do not emit job:status from batch 2016-10-18 20:19:44 +02:00
Raul Ochoa
ef6cd24bf3 Correct debug 2016-10-18 11:18:11 +02:00
Raul Ochoa
ac7bad43a5 Lock by host instead of host + user 
- Host lock only released if there are no pending jobs.
- Will allow to schedule jobs by host.
 2016-10-17 19:03:55 +02:00
Raul Ochoa
761fbe5205 Separate job draining from processing 2016-10-17 18:44:47 +02:00
Raul Ochoa
a8e03f01c9 Add debug information in Jobs Queue 2016-10-17 18:44:37 +02:00
Raul Ochoa
c6e906d3ef Use same debug group 2016-10-17 18:44:28 +02:00
Raul Ochoa
3772b1c896 Log created at time and waiting time for fallback jobs 2016-10-17 16:12:02 +02:00
Raul Ochoa
803a4b533f Add some notes about redis data structures for batch queries 2016-10-17 16:00:30 +02:00
Raul Ochoa
66d1c18941 Default to 64 queued jobs as max 2016-10-17 15:23:53 +02:00
Raul Ochoa
cdde1be29e Re-use redis pool as much as possible 2016-10-17 15:02:34 +02:00
Raul Ochoa
431f72873a 250 queued jobs as default limit 2016-10-17 13:00:23 +02:00
Raul Ochoa
180ba19df5 Fix host queue seeking 2016-10-17 12:51:01 +02:00
Raul Ochoa
39bb7e6249 Lock resources by host+user 
This allows to run multiple jobs in parallel but guarantees order by user
 2016-10-17 12:34:52 +02:00
Raul Ochoa
8b9a30eb75 Queue seeker was not _finding_ queues when only one present 2016-10-17 12:27:06 +02:00
Raul Ochoa
c62fe29160 Load config on object creation 2016-10-17 10:51:50 +02:00
Raul Ochoa
6179327486 Rename 2016-10-14 13:10:27 +02:00
Raul Ochoa
b8c63f5ffc Rename 2016-10-14 12:56:41 +02:00
Raul Ochoa
5bb7d8fa1c Merge branch 'master' into batch-user-queues 2016-10-14 12:33:37 +02:00
Raul Ochoa
a8802d1163 redis-distlock acquires and releases redis clients by operation 2016-10-13 13:48:06 +02:00
Raul Ochoa
05eda290be Create one client for queue-seeker and share per seek cycle 2016-10-13 13:09:56 +02:00
Raul Ochoa
1e442b37ab Allow to set a max number of queued jobs per user 2016-10-12 22:40:35 +02:00
Raul Ochoa
1f038ac1f4 Moves from host queues to user queues 
- Existing jobs are moved before start processing them.
 - Uses a new queue prefix to avoid collisions.
 - Pub/Sub also changes communication channel.
 - Job subscriber emits user+host on new jobs.
 - Batch processor is faulty. See TODO in batch.js.
 2016-10-12 21:32:29 +02:00
Raul Ochoa
f7d1f9426c Use constants for queues 2016-10-12 17:53:03 +02:00
Raul Ochoa
189aff2aa9 Only log message on empty queue 2016-10-12 17:42:46 +02:00
Raul Ochoa
6bb2abde0d Only start lock renewal on lock acquisition 2016-10-12 17:01:24 +02:00
Raul Ochoa
b86f82d3ca Batch.stop removes all listeners 2016-10-12 16:43:18 +02:00
Raul Ochoa
75fc21241f Locker TTL is configured 2016-10-12 13:11:20 +02:00
Raul Ochoa
88f6d46d00 Reuse existing redlock 
Return not connected clients to pool
 2016-10-12 13:10:18 +02:00
Raul Ochoa
3f1b67993c Locker keep refreshing lock by itself 2016-10-12 12:30:13 +02:00
Raul Ochoa
67566c1d0e Callback in subscriber unsubscribe errors 2016-10-12 12:29:54 +02:00
Raul Ochoa
c74f9bcce0 More aggressive on seek interval 2016-10-12 12:29:18 +02:00
Raul Ochoa
98185e55cf Remove Job Queue Pool and use internal structure 
- We don't need to create a different job queue per host.
- Batch locks on message instead of dequeue.
 2016-10-12 12:26:50 +02:00
Raul Ochoa
e1d0ffc7dd Logger set to fatal on test environment 2016-10-12 01:40:35 +02:00
Raul Ochoa
22d8e48f53 Only lock on dequeue 2016-10-12 00:10:40 +02:00
Raul Ochoa
81393190f7 Add callback to jobseeker result from initial load 2016-10-11 19:59:11 +02:00
Raul Ochoa
8bc52b09cf Remove console call 2016-10-11 19:46:27 +02:00
Raul Ochoa
dc1a23e886 Add error handler for channel subscriber 2016-10-11 19:45:43 +02:00
Raul Ochoa
2822b68198 onJobHandler receives host with job 
Queue seeker only returns hosts, not mixing responsibilities
 2016-10-11 19:45:26 +02:00
Raul Ochoa
01cf6f244f Share redis pool for pubsub 2016-10-11 19:41:58 +02:00
Raul Ochoa
ecc6bf0400 Use real on message handler 2016-10-11 19:04:12 +02:00
Raul Ochoa
611508c654 Hide queue seeker behind job subscriber 2016-10-11 19:01:39 +02:00
Raul Ochoa
e7c4ee32df Share redis channel config 2016-10-11 18:41:59 +02:00
Raul Ochoa
d15c7ab0de Always return client to pool 2016-10-11 18:30:35 +02:00
Raul Ochoa
e4b1711e8e pub/sub package 2016-10-11 18:28:46 +02:00
Raul Ochoa
2c064041a1 Add dist lock to run all jobs by host in order 
It uses http://redis.io/topics/distlock
Which is not perfect: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
 2016-10-10 19:54:59 +02:00
Raul Ochoa
0de5d94617 Use debug with same params, no considering job status 2016-10-10 19:53:59 +02:00
Raul Ochoa
90c489119b Add distributed lock implementation with redis distlock 2016-10-10 19:51:11 +02:00
Raul Ochoa
56a632347b Inject publisher 2016-10-10 19:47:50 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Raul Ochoa
deb1ccf876 DRY job final statuses 2016-10-10 12:09:13 +02:00
Raul Ochoa
8a4f54bb87 Allow users to set max statement_timeout for their queries 2016-10-10 12:01:36 +02:00
Raul Ochoa
5401a7edff Timeout is passed into query runner 2016-10-10 12:00:54 +02:00
Raul Ochoa
51d4ff0698 Differentiate between statement timeout and user cancelled query 2016-10-10 11:58:44 +02:00
Raul Ochoa
1d20f11f0c Remove unused var 2016-10-06 18:44:21 +02:00
Raul Ochoa
578c43b1a8 Multiple queries jobs pushed as first job between queries 2016-10-06 18:27:38 +02:00
Raul Ochoa
e108d0df57 Debug query to run 2016-10-06 18:24:28 +02:00
Raul Ochoa
7c7320061f Merge branch 'master' into limit-batch-queries 2016-10-06 12:46:34 +02:00
Raul Ochoa
26fe6a1626 Add readme for batch queries feature 2016-10-06 12:27:38 +02:00
Raul Ochoa
eb2768c197 Add dbhost attribute to batch queries logs 2016-10-05 19:09:10 +02:00
Raul Ochoa
6c2db4385c Batch Queries: use path instead of stream to be able to reopen FD 2016-10-03 13:28:13 +02:00
Raul Ochoa
857ba747d0 Rename 2016-09-30 18:45:33 +02:00
Raul Ochoa
b269418db4 Tag logs 2016-09-30 18:45:15 +02:00
Raul Ochoa
72fb851db7 Fix typo 2016-09-30 16:55:33 +02:00
Raul Ochoa
20573a7f67 Always log queries from fallback jobs 
- Add query id if exists
- Only log analyses for expected format
- Log with query start and end times
 2016-09-30 16:54:10 +02:00
Raul Ochoa
ba34412ce3 Return on boolean on log 2016-09-30 16:51:34 +02:00
Raul Ochoa
9555a2cbde Bring back log logic for FallbackJob 2016-09-30 16:47:37 +02:00
Raul Ochoa
60546de147 Move log logic to each job 2016-09-30 16:44:39 +02:00
Daniel García Aubert
ed27b67cec Simplified batch logger construction 2016-09-29 15:17:25 +02:00
Daniel García Aubert
aa0ce62a85 Implement batch logger to log query times when queries are defined with id 2016-09-29 15:09:36 +02:00
Daniel García Aubert
4f3d361226 Fixes #356, set limit to batch queries to 12h 2016-09-13 12:32:41 +02:00
Raul Ochoa
461728d3e2 Remove user indexer 2016-08-30 19:08:06 +02:00
Raul Ochoa
d33fe5ac21 Stop indexing jobs per user 
Removes .list() from job backend
 2016-08-30 19:01:23 +02:00
Raul Ochoa
05ada98124 Remove .list() from job service 2016-08-30 18:51:51 +02:00
Raul Ochoa
4c8d734bbf Remove update method from job service 2016-08-30 17:46:32 +02:00
Daniel García Aubert
9f50475ad1 Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API 2016-08-30 13:50:03 +02:00
Daniel García Aubert
2932227e8b Improved naming for jobs TTL constant 2016-08-30 13:49:16 +02:00
Daniel García Aubert
02a252940a Improved naming for jobs TTL constant 2016-08-30 10:11:49 +02:00
Daniel García Aubert
0586f45413 Added callback to job subscriber to allow to batch service emit ready event 2016-07-22 13:47:14 +02:00
Daniel García Aubert
89c3681be0 Fix bug when checking if a job is found 2016-07-19 12:34:06 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
74d83a457e Now batch publisher sends a ping to server before publishing and create a new connection if error. 
Batch publisher and subscriber logs (if debug enabled) both outcoming and incoming messages to give more visibility.
 2016-07-07 10:44:17 +02:00
Raul Ochoa
be0f059f01 Add <%= job_id %> template support for onerror and onsuccess fallback queries 2016-06-30 17:41:02 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Raul Ochoa
85fa9d3c2b Merge branch 'master' into batch-onerror-template 
Conflicts:
	NEWS.md
 2016-06-29 16:25:05 +02:00