Daniel García Aubert
c1f2f9377d
Change signature
2016-10-27 17:46:43 +02:00
Daniel García Aubert
ed5b2fb132
Implement proxy function to save work-in-progress jobs
2016-10-27 17:40:13 +02:00
Daniel García Aubert
869139260b
Implement function to save work-in-progress jobs
2016-10-27 17:36:40 +02:00
Daniel García Aubert
f6dffb81cb
Use default name
2016-10-27 16:55:41 +02:00
Raul Ochoa
861b9bb037
Stubs next version
2016-10-21 16:08:44 +02:00
Raul Ochoa
17ab40feee
Release 1.41.0
2016-10-21 16:07:57 +02:00
Raul Ochoa
594aba6179
Stop migrating old queues by default
2016-10-21 16:07:27 +02:00
Raul Ochoa
506581a4a9
Update news
2016-10-21 15:29:26 +02:00
Raul Ochoa
16e9e709b8
Cancel with user statement_timeout limit from redis
2016-10-21 14:35:24 +02:00
Raul Ochoa
26e4cb3196
Get timeout from async function
2016-10-21 13:09:17 +02:00
Raul Ochoa
7563868514
Re-insert into the tree if there was a user in done state that gets a new task
2016-10-21 11:42:27 +02:00
Raul Ochoa
62cfe974b7
Use constants for users
2016-10-21 11:10:17 +02:00
Raul Ochoa
66a1c33f96
Simplify listener subscription logic
...
Always remove pending listeners on acquire call.
Always register add and release listeners on acquire.
2016-10-20 23:47:39 +02:00
Raul Ochoa
58deb49972
Remove runAt property as it is not used
2016-10-20 23:45:30 +02:00
Raul Ochoa
e4d54e9ab7
Fix condition to pick next candidate
2016-10-20 20:16:57 +02:00
Raul Ochoa
aa69bcf34c
Increase to 4 the default value for fixed capacity
2016-10-20 20:16:34 +02:00
Raul Ochoa
30e353ddb9
Stubs next version
2016-10-20 12:43:20 +02:00
Raul Ochoa
4bfbf047b7
Release 1.40.0
2016-10-20 12:42:14 +02:00
Raul Ochoa
5ed8e655a8
Merge pull request #376 from CartoDB/queue-work-scheduling
...
Queue work scheduling
2016-10-20 12:37:13 +02:00
Raul Ochoa
4e3bff9a70
Simplify scheduler to only consider task creation and number of queries
2016-10-20 12:21:41 +02:00
Raul Ochoa
5185c1e225
Cache valid responses for 500 ms
2016-10-20 12:06:51 +02:00
Raul Ochoa
d3f3d5ca36
Call parent with params
2016-10-20 12:06:32 +02:00
Raul Ochoa
75f1ddb049
Timeout for http capacity requests
2016-10-20 12:06:17 +02:00
Raul Ochoa
19def2f31e
Default to 2 jobs in fixed capacity.
2016-10-20 11:12:27 +02:00
Raul Ochoa
66cc137d04
Split http capacity between simple and load
...
- Simple will use 'available_cores' from response.
- Load will use 'cores' and 'relative_load'.
2016-10-20 11:12:08 +02:00
Raul Ochoa
4a57d641c7
Update news and bump version
2016-10-20 10:20:51 +02:00
csobier
232e69de39
Merge pull request #377 from CartoDB/docs-hyperlink-fixes
...
fixed broken docs hyperlink
2016-10-19 13:39:08 -04:00
csobier
a327b30128
fixed broken docs hyperlink
2016-10-19 13:14:31 -04:00
Raul Ochoa
80d2e190ad
Fix test, use fixed to replace one and infinity
2016-10-19 18:47:55 +02:00
Raul Ochoa
0af5cf703a
Allow to configure capacity strategy
...
- HTTP strategy: mechanism to compute load from db host.
- Fixed strategy: hardcoded number of queries to run at the same time, via configuration.
2016-10-19 18:42:53 +02:00
Raul Ochoa
9596ac4730
Scheduler handles new tasks when there is free slots
2016-10-19 16:59:27 +02:00
Raul Ochoa
604e28533c
Fix and improve test
2016-10-19 16:58:57 +02:00
Raul Ochoa
95b3a8adf1
Be explicit about queue status
2016-10-19 16:58:31 +02:00
Raul Ochoa
b164ec8c86
Better debugging
2016-10-19 16:58:00 +02:00
Raul Ochoa
372c9f5511
Basic test to test scheduler happy case
2016-10-19 16:57:10 +02:00
Raul Ochoa
5030fddc9c
Allow to override test client
2016-10-19 16:56:43 +02:00
Raul Ochoa
1ee0878631
Scheduler uses a red–black tree to decide on next job candidate
2016-10-19 16:55:49 +02:00
Raul Ochoa
71d32e003b
Better debug
2016-10-19 11:46:02 +02:00
Raul Ochoa
4daa39bd2c
Start scheduler from host-scheduler
2016-10-19 11:45:48 +02:00
Raul Ochoa
e26bed2e66
Move status close to entity
2016-10-19 10:45:37 +02:00
Raul Ochoa
ca3d71ea48
Tasks with their own entity
...
- Use a list of tasks and keep an index per user.
- Removes WAITING status.
TODO: improve candidate selection.
2016-10-19 10:43:24 +02:00
Raul Ochoa
6c232a1fd0
Discard numeric status
2016-10-19 10:40:03 +02:00
Raul Ochoa
51ac1a3ab7
Remove TODO as it is already done
2016-10-19 10:38:39 +02:00
Raul Ochoa
2853c7b0a7
Fix status check
2016-10-19 10:36:27 +02:00
Raul Ochoa
ac65c1c39a
Rename
2016-10-19 10:36:13 +02:00
Raul Ochoa
3a57331a54
Delegate job scheduling
...
There is a host scheduler managing the host locking.
When it can acquire a lock over the host it will delegate
all the tasks related to that host to the same scheduler.
This scheduler will take care of how many jobs it will submit,
and in which order. It's also responsible for guaranteeing the
execution order per user.
Capacity planner dictates how many jobs can be run at the
same time in a given host. There are two simple strategies:
1. Infinity: it will attempt to run as many jobs as different users.
2. One: it will run just one job at the same time.
Missing things:
- Handle lock renewal failures.
- Fair scheduling for pending/waiting users.
- Capacity based on real resources.
2016-10-18 20:43:15 +02:00
Raul Ochoa
dce051d52b
Make leader locker to emit on renewal errors
2016-10-18 20:34:22 +02:00
Raul Ochoa
d1e3be2e22
Do not emit job:status from batch
2016-10-18 20:19:44 +02:00
Raul Ochoa
1e490be0a1
Don't rely on batch.on(job:status)
2016-10-18 20:18:49 +02:00
Raul Ochoa
a29f847767
Don't rely on batch.on(job:status)
2016-10-18 20:05:57 +02:00
