cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,959 Commits 1 Branch 118 Tags 9.7 MiB
3cae34b62c
Commit Graph

1959 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Raul Ochoa
62cfe974b7 Use constants for users 2016-10-21 11:10:17 +02:00
Raul Ochoa
66a1c33f96 Simplify listener subscription logic 
Always remove pending listeners on acquire call.
Always register add and release listeners on acquire.
 2016-10-20 23:47:39 +02:00
Raul Ochoa
58deb49972 Remove runAt property as it is not used 2016-10-20 23:45:30 +02:00
Raul Ochoa
e4d54e9ab7 Fix condition to pick next candidate 2016-10-20 20:16:57 +02:00
Raul Ochoa
aa69bcf34c Increase to 4 the default value for fixed capacity 2016-10-20 20:16:34 +02:00
Raul Ochoa
30e353ddb9 Stubs next version 2016-10-20 12:43:20 +02:00
Raul Ochoa
4bfbf047b7 Release 1.40.0 2016-10-20 12:42:14 +02:00
Raul Ochoa
5ed8e655a8 Merge pull request #376 from CartoDB/queue-work-scheduling 
Queue work scheduling
 2016-10-20 12:37:13 +02:00
Raul Ochoa
4e3bff9a70 Simplify scheduler to only consider task creation and number of queries 2016-10-20 12:21:41 +02:00
Raul Ochoa
5185c1e225 Cache valid responses for 500 ms 2016-10-20 12:06:51 +02:00
Raul Ochoa
d3f3d5ca36 Call parent with params 2016-10-20 12:06:32 +02:00
Raul Ochoa
75f1ddb049 Timeout for http capacity requests 2016-10-20 12:06:17 +02:00
Raul Ochoa
19def2f31e Default to 2 jobs in fixed capacity. 2016-10-20 11:12:27 +02:00
Raul Ochoa
66cc137d04 Split http capacity between simple and load 
- Simple will use 'available_cores' from response.
- Load will use 'cores' and 'relative_load'.
 2016-10-20 11:12:08 +02:00
Raul Ochoa
4a57d641c7 Update news and bump version 2016-10-20 10:20:51 +02:00
csobier
232e69de39 Merge pull request #377 from CartoDB/docs-hyperlink-fixes 
fixed broken docs hyperlink
 2016-10-19 13:39:08 -04:00
csobier
a327b30128 fixed broken docs hyperlink 2016-10-19 13:14:31 -04:00
Raul Ochoa
80d2e190ad Fix test, use fixed to replace one and infinity 2016-10-19 18:47:55 +02:00
Raul Ochoa
0af5cf703a Allow to configure capacity strategy 
- HTTP strategy: mechanism to compute load from db host.
- Fixed strategy: hardcoded number of queries to run at the same time, via configuration.
 2016-10-19 18:42:53 +02:00
Raul Ochoa
9596ac4730 Scheduler handles new tasks when there is free slots 2016-10-19 16:59:27 +02:00
Raul Ochoa
604e28533c Fix and improve test 2016-10-19 16:58:57 +02:00
Raul Ochoa
95b3a8adf1 Be explicit about queue status 2016-10-19 16:58:31 +02:00
Raul Ochoa
b164ec8c86 Better debugging 2016-10-19 16:58:00 +02:00
Raul Ochoa
372c9f5511 Basic test to test scheduler happy case 2016-10-19 16:57:10 +02:00
Raul Ochoa
5030fddc9c Allow to override test client 2016-10-19 16:56:43 +02:00
Raul Ochoa
1ee0878631 Scheduler uses a red–black tree to decide on next job candidate 2016-10-19 16:55:49 +02:00
Raul Ochoa
71d32e003b Better debug 2016-10-19 11:46:02 +02:00
Raul Ochoa
4daa39bd2c Start scheduler from host-scheduler 2016-10-19 11:45:48 +02:00
Raul Ochoa
e26bed2e66 Move status close to entity 2016-10-19 10:45:37 +02:00
Raul Ochoa
ca3d71ea48 Tasks with their own entity 
- Use a list of tasks and keep an index per user.
- Removes WAITING status.

TODO: improve candidate selection.
 2016-10-19 10:43:24 +02:00
Raul Ochoa
6c232a1fd0 Discard numeric status 2016-10-19 10:40:03 +02:00
Raul Ochoa
51ac1a3ab7 Remove TODO as it is already done 2016-10-19 10:38:39 +02:00
Raul Ochoa
2853c7b0a7 Fix status check 2016-10-19 10:36:27 +02:00
Raul Ochoa
ac65c1c39a Rename 2016-10-19 10:36:13 +02:00
Raul Ochoa
3a57331a54 Delegate job scheduling 
There is a host scheduler managing the host locking.

When it can acquire a lock over the host it will delegate
all the tasks related to that host to the same scheduler.

This scheduler will take care of how many jobs it will submit,
and in which order. It's also responsible for guaranteeing the
execution order per user.

Capacity planner dictates how many jobs can be run at the
same time in a given host. There are two simple strategies:

1. Infinity: it will attempt to run as many jobs as different users.
2. One: it will run just one job at the same time.

Missing things:
 - Handle lock renewal failures.
 - Fair scheduling for pending/waiting users.
 - Capacity based on real resources.
 2016-10-18 20:43:15 +02:00
Raul Ochoa
dce051d52b Make leader locker to emit on renewal errors 2016-10-18 20:34:22 +02:00
Raul Ochoa
d1e3be2e22 Do not emit job:status from batch 2016-10-18 20:19:44 +02:00
Raul Ochoa
1e490be0a1 Don't rely on batch.on(job:status) 2016-10-18 20:18:49 +02:00
Raul Ochoa
a29f847767 Don't rely on batch.on(job:status) 2016-10-18 20:05:57 +02:00
Raul Ochoa
a1400e956d Don't rely on batch.on(job:status) 2016-10-18 20:01:11 +02:00
Raul Ochoa
ef6cd24bf3 Correct debug 2016-10-18 11:18:11 +02:00
Raul Ochoa
ac7bad43a5 Lock by host instead of host + user 
- Host lock only released if there are no pending jobs.
- Will allow to schedule jobs by host.
 2016-10-17 19:03:55 +02:00
Raul Ochoa
761fbe5205 Separate job draining from processing 2016-10-17 18:44:47 +02:00
Raul Ochoa
a8e03f01c9 Add debug information in Jobs Queue 2016-10-17 18:44:37 +02:00
Raul Ochoa
c6e906d3ef Use same debug group 2016-10-17 18:44:28 +02:00
Raul Ochoa
e51a4fc398 Stubs next version 2016-10-17 16:35:10 +02:00
Raul Ochoa
cb9aaef80a Release 1.39.1 2016-10-17 16:34:05 +02:00
Raul Ochoa
179fb4e1bc Update news 2016-10-17 16:33:49 +02:00
Raul Ochoa
3772b1c896 Log created at time and waiting time for fallback jobs 2016-10-17 16:12:02 +02:00
Raul Ochoa
803a4b533f Add some notes about redis data structures for batch queries 2016-10-17 16:00:30 +02:00