Daniel García Aubert
7faa78c3f4
skip state dependant test
2016-11-21 15:31:43 +01:00
Daniel García Aubert
78f5706726
Avoid to use scan command to search work in progress queues
2016-11-03 14:22:43 +01:00
Daniel García Aubert
802c93dcee
Moved work in progress test
2016-11-02 13:03:22 +01:00
Daniel García Aubert
bd716867fa
Revert last env test changes
2016-11-02 13:02:29 +01:00
Daniel García Aubert
041b69a1f6
Isolate failing test in CI
2016-11-02 12:47:29 +01:00
Daniel García Aubert
e459494123
Be more accurate on query sleep times
2016-11-02 11:39:05 +01:00
Daniel García Aubert
aa1527b371
Use clear work-in-progress after finishing jobs
2016-10-28 15:58:41 +02:00
Daniel García Aubert
4a64d37c6c
Add work in progrees list endpoint
2016-10-28 15:08:42 +02:00
Daniel García Aubert
af75bbda50
Rename
2016-10-28 12:50:33 +02:00
Daniel García Aubert
1f657a4f94
Ensure test is isolated
2016-10-28 12:24:23 +02:00
Daniel García Aubert
0085b8ee3d
List users with work in progress jobs
2016-10-27 20:42:49 +02:00
Daniel García Aubert
f65208ba0d
Add listWorkInProgressJobByUser function
2016-10-27 18:43:28 +02:00
Daniel García Aubert
8f65e6b16c
Add test
2016-10-27 18:24:39 +02:00
Raul Ochoa
16e9e709b8
Cancel with user statement_timeout limit from redis
2016-10-21 14:35:24 +02:00
Raul Ochoa
7563868514
Re-insert into the tree if there was a user in done state that gets a new task
2016-10-21 11:42:27 +02:00
Raul Ochoa
62cfe974b7
Use constants for users
2016-10-21 11:10:17 +02:00
Raul Ochoa
66a1c33f96
Simplify listener subscription logic
...
Always remove pending listeners on acquire call.
Always register add and release listeners on acquire.
2016-10-20 23:47:39 +02:00
Raul Ochoa
e4d54e9ab7
Fix condition to pick next candidate
2016-10-20 20:16:57 +02:00
Raul Ochoa
80d2e190ad
Fix test, use fixed to replace one and infinity
2016-10-19 18:47:55 +02:00
Raul Ochoa
604e28533c
Fix and improve test
2016-10-19 16:58:57 +02:00
Raul Ochoa
372c9f5511
Basic test to test scheduler happy case
2016-10-19 16:57:10 +02:00
Raul Ochoa
5030fddc9c
Allow to override test client
2016-10-19 16:56:43 +02:00
Raul Ochoa
1ee0878631
Scheduler uses a red–black tree to decide on next job candidate
2016-10-19 16:55:49 +02:00
Raul Ochoa
2853c7b0a7
Fix status check
2016-10-19 10:36:27 +02:00
Raul Ochoa
1e490be0a1
Don't rely on batch.on(job:status)
2016-10-18 20:18:49 +02:00
Raul Ochoa
a29f847767
Don't rely on batch.on(job:status)
2016-10-18 20:05:57 +02:00
Raul Ochoa
a1400e956d
Don't rely on batch.on(job:status)
2016-10-18 20:01:11 +02:00
Raul Ochoa
66d1c18941
Default to 64 queued jobs as max
2016-10-17 15:23:53 +02:00
Raul Ochoa
cdde1be29e
Re-use redis pool as much as possible
2016-10-17 15:02:34 +02:00
Raul Ochoa
4dad54d004
Add test to validate query order by user
2016-10-17 12:33:49 +02:00
Raul Ochoa
b8a57460dc
Handle stop gracefully
2016-10-17 12:28:01 +02:00
Raul Ochoa
8b9a30eb75
Queue seeker was not _finding_ queues when only one present
2016-10-17 12:27:06 +02:00
Raul Ochoa
f6dc991ab5
Allow to override configuration per API call
2016-10-17 10:52:35 +02:00
Raul Ochoa
cb23b7f46c
Make test finish quicker
2016-10-17 10:52:07 +02:00
Raul Ochoa
5bb7d8fa1c
Merge branch 'master' into batch-user-queues
2016-10-14 12:33:37 +02:00
Raul Ochoa
b510a3684d
Add option to skip script downloading
2016-10-14 12:28:43 +02:00
Raul Ochoa
1e442b37ab
Allow to set a max number of queued jobs per user
2016-10-12 22:40:35 +02:00
Raul Ochoa
1f038ac1f4
Moves from host queues to user queues
...
- Existing jobs are moved before start processing them.
- Uses a new queue prefix to avoid collisions.
- Pub/Sub also changes communication channel.
- Job subscriber emits user+host on new jobs.
- Batch processor is faulty. See TODO in batch.js.
2016-10-12 21:32:29 +02:00
Raul Ochoa
aa68dff24b
Use constants for tests
2016-10-12 18:44:46 +02:00
Raul Ochoa
b86f82d3ca
Batch.stop removes all listeners
2016-10-12 16:43:18 +02:00
Raul Ochoa
c21f373291
Tests for locker
2016-10-12 13:15:55 +02:00
Raul Ochoa
167ecc6965
Set NODE_ENV environment var to test
2016-10-12 01:41:01 +02:00
Raul Ochoa
611508c654
Hide queue seeker behind job subscriber
2016-10-11 19:01:39 +02:00
Raul Ochoa
e4b1711e8e
pub/sub package
2016-10-11 18:28:46 +02:00
Raul Ochoa
f036e29b8f
Make test stable
2016-10-10 19:47:18 +02:00
Raul Ochoa
41d7823227
Drop table if exists
2016-10-10 19:46:53 +02:00
Raul Ochoa
a0ace8d034
Use name in test
2016-10-10 19:46:39 +02:00
Raul Ochoa
66820a67bb
Make possible to specify a name for batch
2016-10-10 19:46:07 +02:00
Raul Ochoa
f4950f73c0
Add test to validate multiple consumers query order scenario
2016-10-10 15:56:04 +02:00
Raul Ochoa
ece95afc95
Add test client to run plain queries
2016-10-10 15:37:46 +02:00
Raul Ochoa
46579093dd
Fix test scenario description
2016-10-10 15:37:19 +02:00
Raul Ochoa
8ac656492f
Move to support directory
2016-10-10 15:22:50 +02:00
Raul Ochoa
faa02a79b9
Test for 578c43b1a8
2016-10-10 15:17:05 +02:00
Raul Ochoa
8a4f54bb87
Allow users to set max statement_timeout for their queries
2016-10-10 12:01:36 +02:00
Raul Ochoa
2ac39b5748
Move all batch tests to their own directory
2016-10-06 11:53:56 +02:00
Raul Ochoa
b07e2d3de5
Merge pull request #346 from CartoDB/fix-batch-api-test
...
Removed duplicated config
2016-10-04 13:01:46 +02:00
Raul Ochoa
0e03bcd1d3
Delete table so suite ends clear
2016-09-26 18:28:15 +02:00
Raul Ochoa
2712d81d50
Remove unused server utils
2016-09-26 18:13:08 +02:00
Raul Ochoa
b29358a0ed
Remove old response from assert
2016-09-26 18:11:40 +02:00
Raul Ochoa
96fc32b24a
Concurrency test with assert.response
2016-09-26 18:10:09 +02:00
Raul Ochoa
1a20ca0bc3
Use listener for test
2016-09-26 18:09:50 +02:00
Raul Ochoa
abc2f130c9
Migrate to express 4.x series
...
- Remove express logger
- Error handler responds with application/[json|javascript]
- Fix all tests relying on res.headers
- assert.response based on request module
2016-09-26 18:09:27 +02:00
Raul Ochoa
a4aba62e9c
Fix name clash for err vars
2016-09-26 14:51:41 +02:00
Raul Ochoa
90e4600a6c
Rename vars to avoid name clash
2016-09-26 14:42:50 +02:00
Raul Ochoa
3dce6af20f
Remove unused param
2016-09-26 14:42:35 +02:00
Raul Ochoa
9bd240ef6c
Remove unused params
2016-09-26 14:42:24 +02:00
Raul Ochoa
26400342f3
Remove unused param
2016-09-26 14:42:14 +02:00
Raul Ochoa
c47dde1a29
Short hand method
2016-09-26 14:42:02 +02:00
Raul Ochoa
c9846a2437
Change callback signature in assert.response
2016-09-26 14:37:40 +02:00
Raul Ochoa
0d54604428
Remove setMaxListeners from tests
2016-09-14 21:24:22 +02:00
Raul Ochoa
84b80547c2
Remove settings' app_root
2016-09-14 21:02:56 +02:00
Raul Ochoa
f92d50cccf
Rename from app to server
...
Removes app_root dependency in requires
2016-09-14 20:54:53 +02:00
Daniel García Aubert
17411f2bb6
Avoids to delete empty keys
2016-09-12 16:00:33 +02:00
Raul Ochoa
461728d3e2
Remove user indexer
2016-08-30 19:08:06 +02:00
Raul Ochoa
20edddc721
Remove user indexer dependency from tests
2016-08-30 19:06:59 +02:00
Raul Ochoa
d33fe5ac21
Stop indexing jobs per user
...
Removes .list() from job backend
2016-08-30 19:01:23 +02:00
Raul Ochoa
05ada98124
Remove .list() from job service
2016-08-30 18:51:51 +02:00
Raul Ochoa
ba0f2f1066
Remove endpoint to retrieve jobs list
2016-08-30 18:49:01 +02:00
Raul Ochoa
4c8d734bbf
Remove update method from job service
2016-08-30 17:46:32 +02:00
Raul Ochoa
ad0d101bfd
Remove patch/put endpoints for jobs
2016-08-30 17:43:34 +02:00
Daniel García Aubert
085abb3446
Removed duplicated config
2016-08-01 11:48:11 +02:00
Daniel García Aubert
f3f45bb4e2
Fixed lint issues
2016-07-22 17:11:21 +02:00
Daniel García Aubert
587ab6a5d9
Extratcs redis clean method for batch keys
2016-07-22 17:05:01 +02:00
Daniel García Aubert
7fd5e42213
Removed timeouts for batch tests and skipped again a failing one in job callback template
2016-07-22 14:41:26 +02:00
Daniel García Aubert
2e8382eeff
Fixed issues in batch tests:
...
- Wait for batch is ready before testing it
- Remove batch keys in redis after testing every suit
- Fixed issue creating jobs with same id in job service integration test
2016-07-22 14:30:55 +02:00
Daniel García Aubert
438d61fb70
Set timeout to 30s
2016-07-20 19:50:28 +02:00
Daniel García Aubert
027e7b8768
Removed unused dependency
2016-07-20 19:42:53 +02:00
Daniel García Aubert
1625d8a556
Fixed timeout in job canceller test
2016-07-20 19:41:55 +02:00
Daniel García Aubert
457414c2fe
Skipping integration test
2016-07-20 18:27:54 +02:00
Daniel García Aubert
39a9f950b0
Added integration test for multiquery jobs
2016-07-20 16:38:57 +02:00
Daniel García Aubert
b33caf6126
Moved batch integration test to a separated folder
2016-07-19 19:56:12 +02:00
Daniel García Aubert
8d4dbd85df
Implmented integration tests for publisher, runner and canceller of batch service.
2016-07-19 19:42:49 +02:00
Daniel García Aubert
fe11828046
Implemented integration test for job service
2016-07-19 16:20:53 +02:00
Daniel García Aubert
2a4364142d
Added integration test for job backend
2016-07-19 13:08:52 +02:00
Daniel García Aubert
ddd5a0bd19
Added integration folder to tests
2016-07-18 14:49:57 +02:00
Daniel García Aubert
395635d246
Merge branch 'master' into fix-publisher-connection
2016-07-11 17:49:11 +02:00
Raul Ochoa
3ebbd9f7c4
Skip tables with no updated_at registered in cdb_tablemetadata
2016-07-11 16:39:12 +02:00
Daniel García Aubert
ccff602bbf
Merge branch 'master' into fix-publisher-connection
2016-07-07 16:07:41 +02:00
Raul Ochoa
368fe2403e
Allow to setup more than one domain to validate oauth against
2016-07-07 14:20:36 +02:00
Daniel García Aubert
5eaad4d5d9
Uses redis-mpool for pubsub in Batch API
2016-07-07 14:14:46 +02:00
Daniel García Aubert
74d83a457e
Now batch publisher sends a ping to server before publishing and create a new connection if error.
...
Batch publisher and subscriber logs (if debug enabled) both outcoming and incoming messages to give more visibility.
2016-07-07 10:44:17 +02:00
Raul Ochoa
63d3cd59e2
Skip test for travis
2016-06-30 20:02:43 +02:00
Raul Ochoa
b4ee2effbd
Fix line too long error
2016-06-30 19:49:34 +02:00
Raul Ochoa
b9af5e7846
Create table in before
2016-06-30 19:40:36 +02:00
Raul Ochoa
8a57a64961
Output actual response when it fails
2016-06-30 19:14:21 +02:00
Raul Ochoa
1f8f5e2c57
Use another name for test table
2016-06-30 19:12:45 +02:00
Raul Ochoa
dc5807b790
Do not validate expected output
2016-06-30 18:55:19 +02:00
Raul Ochoa
74964bc696
Be more clear about what is not matching
2016-06-30 18:31:58 +02:00
Raul Ochoa
51747879d6
Reorder tests
2016-06-30 18:16:52 +02:00
Raul Ochoa
be0f059f01
Add <%= job_id %> template support for onerror and onsuccess fallback queries
2016-06-30 17:41:02 +02:00
Daniel García Aubert
a1f31df92e
Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job
2016-06-29 18:29:53 +02:00
Raul Ochoa
85fa9d3c2b
Merge branch 'master' into batch-onerror-template
...
Conflicts:
NEWS.md
2016-06-29 16:25:05 +02:00
Raul Ochoa
a3117a2f01
Add <%= error_message %> template support for onerror fallback queries
2016-06-29 14:22:23 +02:00
Raul Ochoa
1d8f5539a7
Adds start and end time for batch queries with fallback
2016-06-29 13:56:45 +02:00
Daniel García Aubert
c7a9f59320
Added CDB_OverviewsSupport as remote sql script in fixtrues
2016-06-22 16:38:30 +02:00
Daniel
58055080c9
Merge pull request #311 from CartoDB/309-skipped-status
...
Fixes #309 , added skipped status to fallback-jobs
2016-06-08 11:12:59 +02:00
Daniel García Aubert
2a2127f4e1
Improved falback job to get the status of last finished query
2016-06-02 18:20:53 +02:00
Raul Ochoa
299490f46f
Put all remote sql script together
2016-06-01 10:52:54 +02:00
Daniel García Aubert
481a82500b
Refactored fallback jobs
2016-06-01 10:39:01 +02:00
Raul Ochoa
5710d83908
Output postgresql server version
2016-06-01 10:25:16 +02:00
Daniel García Aubert
1a0e2b681b
Fixes #309 , added skipped status to fallback-jobs
2016-05-26 17:37:37 +02:00
Daniel García Aubert
e079491a7e
Fixed issue with status transition fallback jobs
2016-05-25 17:00:27 +02:00
Raul Ochoa
23228b2d73
Payload size validates multiple queries and fallback queries
...
It uses a middleware to check the body size
2016-05-24 14:28:00 +02:00
Daniel García Aubert
2d86fc2ce3
In fallback jobs, used fallback_status property instead of an array of statuses
2016-05-23 15:05:14 +02:00
Daniel García Aubert
052cb0796d
Modified fallback tests in order to use our custom assert.response
2016-05-20 14:42:53 +02:00
Daniel García Aubert
e04787a070
Fixed missing failed_reason in failing jobs
2016-05-19 17:02:48 +02:00
Daniel García Aubert
cf7dff4696
Removed commented code
2016-05-19 14:13:54 +02:00
Daniel García Aubert
62c8a9d315
Refactor and fixed issue, when cancelling a fallback job, it is not cancelled properly
2016-05-19 14:10:18 +02:00
Daniel García Aubert
5c4ced34e6
Added acceptance test for job cancelling
2016-05-18 13:25:08 +02:00
Daniel García Aubert
b94a9c7c0a
Fixed conflicts
2016-05-18 11:48:39 +02:00
Daniel
8c8da8e39c
Merge pull request #301 from CartoDB/job-model-refactor
...
Job model refactor
2016-05-18 11:38:41 +02:00
Daniel García Aubert
6acc02e998
Improved fallback setStatus method and added acceptance test
2016-05-17 18:51:02 +02:00
Daniel García Aubert
b2eac6b880
Implemented test for fallback jobs
2016-05-17 01:00:27 +02:00
Daniel García Aubert
be3d9d2316
Fallback jobs WIP
2016-05-16 13:49:56 +02:00
Daniel García Aubert
d2d3ba8159
Passed tests
2016-05-16 01:22:47 +02:00
Raul Ochoa
4e3238c2e8
Fix control flow when using OGR with _needSRS=true and empty queries
...
Continue on empty results as srid and geom type are not critical
when there are no results
Fixes #299
2016-05-12 17:21:07 +02:00
Juan Ignacio Sánchez Lara
e979b79f49
gpkg format #291
2016-05-10 18:42:23 +02:00
Juan Ignacio Sánchez Lara
7b9911416b
geopackage export test #291
2016-05-10 18:01:50 +02:00
Juan Ignacio Sánchez Lara
c82cafb1ed
Geopackage format support #291
2016-04-27 16:56:06 +02:00
Daniel García Aubert
5de931daa5
Improver error message in job max payload limit.
2016-04-25 11:18:30 +02:00
Daniel García Aubert
832695031b
Improved tests for query size limit
2016-04-18 18:05:03 +02:00
Daniel García Aubert
1dcbb1afdf
Improved error message
2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124
Fixed typos
2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944
Implemented limit to query size in Batch API.
2016-04-18 15:30:16 +02:00
Daniel
423e729e11
Merge pull request #283 from CartoDB/batch-improve-startup
...
Improved initialization in Batch API
2016-04-05 15:46:21 +02:00
Daniel García Aubert
7121b3b800
Passed test
2016-04-04 20:05:58 +02:00
Daniel García Aubert
b62edcb64f
Refactored batch-api startup. Isolated code related to queue discovering, api much clear and concise.
2016-04-04 16:05:33 +02:00
dgaubert
e8a44136c5
Now Batch API looks for queues with jobs to consume at service initialization time
2016-03-31 17:37:35 +02:00
dgaubert
fd9bfe277e
Implemented multiquery jobs for Batch API
2016-03-31 12:39:03 +02:00
Daniel García Aubert
a32fa7b108
Implemented new test for multiquery jobs
2016-03-18 18:12:01 +01:00
