cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,013 Commits 1 Branch 118 Tags 9.7 MiB
df523960a2
Commit Graph

306 Commits

Author SHA1 Message Date
Daniel García Aubert
aa1527b371 Use clear work-in-progress after finishing jobs 2016-10-28 15:58:41 +02:00
Daniel García Aubert
eaa3315982 Add clear work in progress job to service 2016-10-28 15:19:49 +02:00
Daniel García Aubert
ea06581ddb Add clear work in progress job 2016-10-28 15:18:57 +02:00
Daniel García Aubert
64d0dc93bf Removed unnecessary debug 2016-10-28 15:06:07 +02:00
Daniel García Aubert
af75bbda50 Rename 2016-10-28 12:50:33 +02:00
Daniel García Aubert
c35b21407a Call list of work in progress job 2016-10-28 12:49:59 +02:00
Daniel García Aubert
f0de347b56 Rename 2016-10-28 12:30:33 +02:00
Daniel García Aubert
7b48e43d92 Rename 2016-10-28 12:29:28 +02:00
Daniel García Aubert
224a4c933a Rename 2016-10-28 12:26:24 +02:00
Daniel García Aubert
1f657a4f94 Ensure test is isolated 2016-10-28 12:24:23 +02:00
Daniel García Aubert
97836e62b9 Add debug message 2016-10-27 20:45:47 +02:00
Daniel García Aubert
72419072ea Improve var definitions 2016-10-27 20:43:42 +02:00
Daniel García Aubert
0085b8ee3d List users with work in progress jobs 2016-10-27 20:42:49 +02:00
Daniel García Aubert
f65208ba0d Add listWorkInProgressJobByUser function 2016-10-27 18:43:28 +02:00
Daniel García Aubert
5b8108d4a8 Use job service to add jobs to work-in-progress list 2016-10-27 18:00:56 +02:00
Daniel García Aubert
c1f2f9377d Change signature 2016-10-27 17:46:43 +02:00
Daniel García Aubert
ed5b2fb132 Implement proxy function to save work-in-progress jobs 2016-10-27 17:40:13 +02:00
Daniel García Aubert
869139260b Implement function to save work-in-progress jobs 2016-10-27 17:36:40 +02:00
Daniel García Aubert
f6dffb81cb Use default name 2016-10-27 16:55:41 +02:00
Raul Ochoa
594aba6179 Stop migrating old queues by default 2016-10-21 16:07:27 +02:00
Raul Ochoa
16e9e709b8 Cancel with user statement_timeout limit from redis 2016-10-21 14:35:24 +02:00
Raul Ochoa
26e4cb3196 Get timeout from async function 2016-10-21 13:09:17 +02:00
Raul Ochoa
7563868514 Re-insert into the tree if there was a user in done state that gets a new task 2016-10-21 11:42:27 +02:00
Raul Ochoa
66a1c33f96 Simplify listener subscription logic 
Always remove pending listeners on acquire call.
Always register add and release listeners on acquire.
 2016-10-20 23:47:39 +02:00
Raul Ochoa
58deb49972 Remove runAt property as it is not used 2016-10-20 23:45:30 +02:00
Raul Ochoa
e4d54e9ab7 Fix condition to pick next candidate 2016-10-20 20:16:57 +02:00
Raul Ochoa
aa69bcf34c Increase to 4 the default value for fixed capacity 2016-10-20 20:16:34 +02:00
Raul Ochoa
4e3bff9a70 Simplify scheduler to only consider task creation and number of queries 2016-10-20 12:21:41 +02:00
Raul Ochoa
5185c1e225 Cache valid responses for 500 ms 2016-10-20 12:06:51 +02:00
Raul Ochoa
d3f3d5ca36 Call parent with params 2016-10-20 12:06:32 +02:00
Raul Ochoa
75f1ddb049 Timeout for http capacity requests 2016-10-20 12:06:17 +02:00
Raul Ochoa
19def2f31e Default to 2 jobs in fixed capacity. 2016-10-20 11:12:27 +02:00
Raul Ochoa
66cc137d04 Split http capacity between simple and load 
- Simple will use 'available_cores' from response.
- Load will use 'cores' and 'relative_load'.
 2016-10-20 11:12:08 +02:00
Raul Ochoa
0af5cf703a Allow to configure capacity strategy 
- HTTP strategy: mechanism to compute load from db host.
- Fixed strategy: hardcoded number of queries to run at the same time, via configuration.
 2016-10-19 18:42:53 +02:00
Raul Ochoa
9596ac4730 Scheduler handles new tasks when there is free slots 2016-10-19 16:59:27 +02:00
Raul Ochoa
95b3a8adf1 Be explicit about queue status 2016-10-19 16:58:31 +02:00
Raul Ochoa
b164ec8c86 Better debugging 2016-10-19 16:58:00 +02:00
Raul Ochoa
1ee0878631 Scheduler uses a red–black tree to decide on next job candidate 2016-10-19 16:55:49 +02:00
Raul Ochoa
71d32e003b Better debug 2016-10-19 11:46:02 +02:00
Raul Ochoa
4daa39bd2c Start scheduler from host-scheduler 2016-10-19 11:45:48 +02:00
Raul Ochoa
e26bed2e66 Move status close to entity 2016-10-19 10:45:37 +02:00
Raul Ochoa
ca3d71ea48 Tasks with their own entity 
- Use a list of tasks and keep an index per user.
- Removes WAITING status.

TODO: improve candidate selection.
 2016-10-19 10:43:24 +02:00
Raul Ochoa
6c232a1fd0 Discard numeric status 2016-10-19 10:40:03 +02:00
Raul Ochoa
51ac1a3ab7 Remove TODO as it is already done 2016-10-19 10:38:39 +02:00
Raul Ochoa
ac65c1c39a Rename 2016-10-19 10:36:13 +02:00
Raul Ochoa
3a57331a54 Delegate job scheduling 
There is a host scheduler managing the host locking.

When it can acquire a lock over the host it will delegate
all the tasks related to that host to the same scheduler.

This scheduler will take care of how many jobs it will submit,
and in which order. It's also responsible for guaranteeing the
execution order per user.

Capacity planner dictates how many jobs can be run at the
same time in a given host. There are two simple strategies:

1. Infinity: it will attempt to run as many jobs as different users.
2. One: it will run just one job at the same time.

Missing things:
 - Handle lock renewal failures.
 - Fair scheduling for pending/waiting users.
 - Capacity based on real resources.
 2016-10-18 20:43:15 +02:00
Raul Ochoa
dce051d52b Make leader locker to emit on renewal errors 2016-10-18 20:34:22 +02:00
Raul Ochoa
d1e3be2e22 Do not emit job:status from batch 2016-10-18 20:19:44 +02:00
Raul Ochoa
ef6cd24bf3 Correct debug 2016-10-18 11:18:11 +02:00
Raul Ochoa
ac7bad43a5 Lock by host instead of host + user 
- Host lock only released if there are no pending jobs.
- Will allow to schedule jobs by host.
 2016-10-17 19:03:55 +02:00
Raul Ochoa
761fbe5205 Separate job draining from processing 2016-10-17 18:44:47 +02:00
Raul Ochoa
a8e03f01c9 Add debug information in Jobs Queue 2016-10-17 18:44:37 +02:00
Raul Ochoa
c6e906d3ef Use same debug group 2016-10-17 18:44:28 +02:00
Raul Ochoa
3772b1c896 Log created at time and waiting time for fallback jobs 2016-10-17 16:12:02 +02:00
Raul Ochoa
803a4b533f Add some notes about redis data structures for batch queries 2016-10-17 16:00:30 +02:00
Raul Ochoa
66d1c18941 Default to 64 queued jobs as max 2016-10-17 15:23:53 +02:00
Raul Ochoa
cdde1be29e Re-use redis pool as much as possible 2016-10-17 15:02:34 +02:00
Raul Ochoa
431f72873a 250 queued jobs as default limit 2016-10-17 13:00:23 +02:00
Raul Ochoa
180ba19df5 Fix host queue seeking 2016-10-17 12:51:01 +02:00
Raul Ochoa
39bb7e6249 Lock resources by host+user 
This allows to run multiple jobs in parallel but guarantees order by user
 2016-10-17 12:34:52 +02:00
Raul Ochoa
8b9a30eb75 Queue seeker was not _finding_ queues when only one present 2016-10-17 12:27:06 +02:00
Raul Ochoa
c62fe29160 Load config on object creation 2016-10-17 10:51:50 +02:00
Raul Ochoa
6179327486 Rename 2016-10-14 13:10:27 +02:00
Raul Ochoa
b8c63f5ffc Rename 2016-10-14 12:56:41 +02:00
Raul Ochoa
5bb7d8fa1c Merge branch 'master' into batch-user-queues 2016-10-14 12:33:37 +02:00
Raul Ochoa
a8802d1163 redis-distlock acquires and releases redis clients by operation 2016-10-13 13:48:06 +02:00
Raul Ochoa
05eda290be Create one client for queue-seeker and share per seek cycle 2016-10-13 13:09:56 +02:00
Raul Ochoa
1e442b37ab Allow to set a max number of queued jobs per user 2016-10-12 22:40:35 +02:00
Raul Ochoa
1f038ac1f4 Moves from host queues to user queues 
- Existing jobs are moved before start processing them.
 - Uses a new queue prefix to avoid collisions.
 - Pub/Sub also changes communication channel.
 - Job subscriber emits user+host on new jobs.
 - Batch processor is faulty. See TODO in batch.js.
 2016-10-12 21:32:29 +02:00
Raul Ochoa
f7d1f9426c Use constants for queues 2016-10-12 17:53:03 +02:00
Raul Ochoa
189aff2aa9 Only log message on empty queue 2016-10-12 17:42:46 +02:00
Raul Ochoa
6bb2abde0d Only start lock renewal on lock acquisition 2016-10-12 17:01:24 +02:00
Raul Ochoa
b86f82d3ca Batch.stop removes all listeners 2016-10-12 16:43:18 +02:00
Raul Ochoa
75fc21241f Locker TTL is configured 2016-10-12 13:11:20 +02:00
Raul Ochoa
88f6d46d00 Reuse existing redlock 
Return not connected clients to pool
 2016-10-12 13:10:18 +02:00
Raul Ochoa
3f1b67993c Locker keep refreshing lock by itself 2016-10-12 12:30:13 +02:00
Raul Ochoa
67566c1d0e Callback in subscriber unsubscribe errors 2016-10-12 12:29:54 +02:00
Raul Ochoa
c74f9bcce0 More aggressive on seek interval 2016-10-12 12:29:18 +02:00
Raul Ochoa
98185e55cf Remove Job Queue Pool and use internal structure 
- We don't need to create a different job queue per host.
- Batch locks on message instead of dequeue.
 2016-10-12 12:26:50 +02:00
Raul Ochoa
e1d0ffc7dd Logger set to fatal on test environment 2016-10-12 01:40:35 +02:00
Raul Ochoa
22d8e48f53 Only lock on dequeue 2016-10-12 00:10:40 +02:00
Raul Ochoa
81393190f7 Add callback to jobseeker result from initial load 2016-10-11 19:59:11 +02:00
Raul Ochoa
8bc52b09cf Remove console call 2016-10-11 19:46:27 +02:00
Raul Ochoa
dc1a23e886 Add error handler for channel subscriber 2016-10-11 19:45:43 +02:00
Raul Ochoa
2822b68198 onJobHandler receives host with job 
Queue seeker only returns hosts, not mixing responsibilities
 2016-10-11 19:45:26 +02:00
Raul Ochoa
01cf6f244f Share redis pool for pubsub 2016-10-11 19:41:58 +02:00
Raul Ochoa
ecc6bf0400 Use real on message handler 2016-10-11 19:04:12 +02:00
Raul Ochoa
611508c654 Hide queue seeker behind job subscriber 2016-10-11 19:01:39 +02:00
Raul Ochoa
e7c4ee32df Share redis channel config 2016-10-11 18:41:59 +02:00
Raul Ochoa
d15c7ab0de Always return client to pool 2016-10-11 18:30:35 +02:00
Raul Ochoa
e4b1711e8e pub/sub package 2016-10-11 18:28:46 +02:00
Raul Ochoa
2c064041a1 Add dist lock to run all jobs by host in order 
It uses http://redis.io/topics/distlock
Which is not perfect: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
 2016-10-10 19:54:59 +02:00
Raul Ochoa
0de5d94617 Use debug with same params, no considering job status 2016-10-10 19:53:59 +02:00
Raul Ochoa
90c489119b Add distributed lock implementation with redis distlock 2016-10-10 19:51:11 +02:00
Raul Ochoa
56a632347b Inject publisher 2016-10-10 19:47:50 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Raul Ochoa
deb1ccf876 DRY job final statuses 2016-10-10 12:09:13 +02:00
Raul Ochoa
8a4f54bb87 Allow users to set max statement_timeout for their queries 2016-10-10 12:01:36 +02:00
Raul Ochoa
5401a7edff Timeout is passed into query runner 2016-10-10 12:00:54 +02:00
Raul Ochoa
51d4ff0698 Differentiate between statement timeout and user cancelled query 2016-10-10 11:58:44 +02:00