cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,197 Commits 1 Branch 118 Tags 9.7 MiB
c9bc11078d
Commit Graph

597 Commits

Author SHA1 Message Date
Simon Martín
27e43c8d91 handling python timeout error 2017-11-17 13:30:58 +01:00
Daniel García Aubert
b1112e2bb1 Remove unneeded header 2017-09-20 16:48:22 +02:00
Daniel García Aubert
980e762ab7 Skip user auth middlewares 2017-09-20 16:35:33 +02:00
Daniel García Aubert
0597a3b8a6 Change job work in progress endpoint 2017-09-20 15:46:29 +02:00
Raul Ochoa
32154b67c6 Merge branch 'master' into upgrade-cartodb-psql 2017-08-11 11:57:16 +02:00
Daniel García Aubert
f13c16e523 Typo 2017-08-10 11:22:08 +02:00
Daniel García Aubert
01a71ee60e Apply user timeout to ogr2ogr command 2017-08-09 12:50:16 +02:00
Raul Ochoa
3e536b822e Remove unnecessary destroyOnError option 2017-08-08 18:21:10 +02:00
Daniel García Aubert
fb477260ec Respond with 429 and proper message when database fires a timeout error 2017-08-03 17:19:08 +02:00
Rafa de la Torre
07890ec608 Change tableCache.get for tableCache.peek #244 
This will make it consider age of the key when it was initially set,
instead of renewing it with the time of the last access.
 2017-07-03 11:13:21 +02:00
Rafa de la Torre
1db935a24b Merge pull request #423 from CartoDB/422-disable-query-lru-cache 
422 disable query lru cache
 2017-06-27 17:40:33 +02:00
Rafa de la Torre
2d9a5e9655 Move cyclomatic complexity to the factory #422 2017-06-27 16:05:05 +02:00
Rafa de la Torre
c8e42f561d Reduce cyclomatic complexity (jshint 2/2) #422 2017-06-27 15:35:51 +02:00
Rafa de la Torre
34ccfc10df Please jshint (1/2) #422 2017-06-27 13:26:14 +02:00
Rafa de la Torre
bf478cabb2 Fix initialization of NoCache #422 2017-06-27 12:44:11 +02:00
Rafa de la Torre
bc71889f93 First attempt at removing tableCache #422 2017-06-27 12:10:24 +02:00
Daniel García Aubert
aa66000848 Going green: use replacer parameter in JSON.stringify method to not cast special number values to null 2017-06-08 12:42:37 +02:00
Mario de Frutos
3a99020632 Added regexp for the test to matches both 0.10 and 6.X node version 2017-04-18 12:15:47 +02:00
Mario de Frutos
da42daa536 Improved error message for zip spawn command 2017-04-18 12:15:47 +02:00
Daniel García Aubert
e4eeb9e1f6 Clean commented code 2017-04-18 12:15:47 +02:00
Daniel García Aubert
e0f9209017 Attach error handler while spawning zip command 2017-04-18 12:15:47 +02:00
Daniel García Aubert
884d3aea0a Dont call 'next' callback twice, include error message if zip don't exit normally 2017-04-18 12:15:47 +02:00
Daniel García Aubert
4b98b7056a Make zip command path configurable 2017-04-18 12:15:47 +02:00
Daniel García Aubert
6dba06130b Retur a proper error message when ogr2ogr command fails 2017-04-18 11:40:37 +02:00
Daniel García Aubert
d60e2107af Add error callback to ogr command while spawning 2017-04-11 09:43:25 +02:00
Raul Ochoa
db9bfacf07 Extract stats client creation 2017-03-30 16:13:17 +02:00
Raul Ochoa
d48a901533 Merge branch 'master' into node-v6 2017-01-12 19:49:56 +01:00
Javier Torres
62a67f148c Correctly accept custom options for formats in ogr.js 2017-01-11 19:07:50 +01:00
Javier Torres
c67a2e7d09 Use cartodb_id as FID column in gpkg 2017-01-11 19:07:50 +01:00
Daniel García Aubert
ef124d689e Handle query end in error listener since pg no longer emits end if and error occurs 2016-12-07 16:22:21 +01:00
Daniel García Aubert
99c7a6e4f9 Raise job query size to 16kb 2016-11-07 16:50:44 +01:00
Daniel García Aubert
4a64d37c6c Add work in progrees list endpoint 2016-10-28 15:08:42 +02:00
Raul Ochoa
cdde1be29e Re-use redis pool as much as possible 2016-10-17 15:02:34 +02:00
Raul Ochoa
e401c01d78 Only log on non-test environments 2016-10-12 01:40:14 +02:00
Raul Ochoa
e4b1711e8e pub/sub package 2016-10-11 18:28:46 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Raul Ochoa
42a1f3ad4c Accept multipart requests 2016-10-05 17:29:16 +02:00
Raul Ochoa
6309318534 Use body-parser from old connect module 2016-10-05 14:22:44 +02:00
Raul Ochoa
025b3f3cc7 Increase body limit to 20mb 2016-10-05 11:14:06 +02:00
Raul Ochoa
2a2a54a073 DRY in job response handler 2016-10-04 16:07:13 +02:00
Raul Ochoa
19c9bec633 Callback with job 2016-10-04 15:57:13 +02:00
Raul Ochoa
05cbd55b95 Use db host from request's context 2016-10-04 15:50:39 +02:00
Raul Ochoa
c32a2199fa Use request bootstrapper to add host header 2016-10-04 15:43:19 +02:00
Raul Ochoa
7b7d651d8f DRY while authenticating requests 2016-10-04 15:40:56 +02:00
Raul Ochoa
20f50d988e Use user middleware in job controller 2016-10-04 15:19:31 +02:00
Raul Ochoa
b139b9ab21 Add context object in all requests 2016-10-04 15:12:46 +02:00
Raul Ochoa
2edc7505e7 Do not condition req.profiler 2016-10-04 15:08:31 +02:00
Raul Ochoa
cb9db0c4cb Make request.profiler always available 2016-10-04 14:40:04 +02:00
Raul Ochoa
3b6bc14d17 Increment errors on err 2016-10-04 13:19:29 +02:00
Raul Ochoa
bf1a67780d Merge branch 'master' into express-4.x 2016-09-30 17:37:30 +02:00
Daniel García Aubert
59e5c10f85 Set batch queries log path 2016-09-29 15:37:14 +02:00
Raul Ochoa
2aa1f045ff Use .status() API 2016-09-26 18:10:20 +02:00
Raul Ochoa
abc2f130c9 Migrate to express 4.x series 
- Remove express logger
 - Error handler responds with application/[json|javascript]
 - Fix all tests relying on res.headers
 - assert.response based on request module
 2016-09-26 18:09:27 +02:00
Raul Ochoa
f92d50cccf Rename from app to server 
Removes app_root dependency in requires
 2016-09-14 20:54:53 +02:00
Raul Ochoa
0626d80e24 Removes support for optional rollbar logging 2016-09-14 19:22:31 +02:00
Raul Ochoa
6f4fb931f7 Remove user indexer from app 2016-08-30 19:04:36 +02:00
Raul Ochoa
ba0f2f1066 Remove endpoint to retrieve jobs list 2016-08-30 18:49:01 +02:00
Raul Ochoa
704ba110ba Log job creation 2016-08-30 18:43:09 +02:00
Raul Ochoa
ad0d101bfd Remove patch/put endpoints for jobs 2016-08-30 17:43:34 +02:00
Daniel García Aubert
395635d246 Merge branch 'master' into fix-publisher-connection 2016-07-11 17:49:11 +02:00
Raul Ochoa
3ebbd9f7c4 Skip tables with no updated_at registered in cdb_tablemetadata 2016-07-11 16:39:12 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Raul Ochoa
368fe2403e Allow to setup more than one domain to validate oauth against 2016-07-07 14:20:36 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Daniel García Aubert
282da58ffe Set default value for statsd-client in job-controller to avoid check it every time it's going to be used 2016-05-30 12:27:19 +02:00
Daniel García Aubert
976bf5b039 Implemented profiling for job-runner and job-controller 2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577 Merge branch 'master' into batch-add-profile 2016-05-26 17:46:52 +02:00
Raul Ochoa
a91c7a6a88 Increase job payload size to 8kb 2016-05-24 15:41:18 +02:00
Raul Ochoa
23228b2d73 Payload size validates multiple queries and fallback queries 
It uses a middleware to check the body size
 2016-05-24 14:28:00 +02:00
Daniel García Aubert
64ad284c9c WIP: adding metrics to Batch API 2016-05-24 11:19:00 +02:00
Daniel
8c8da8e39c Merge pull request #301 from CartoDB/job-model-refactor 
Job model refactor
 2016-05-18 11:38:41 +02:00
Daniel García Aubert
d2d3ba8159 Passed tests 2016-05-16 01:22:47 +02:00
Daniel García Aubert
cc7dd7a0d2 Job model refactor 2016-05-13 18:50:55 +02:00
Raul Ochoa
4e3238c2e8 Fix control flow when using OGR with _needSRS=true and empty queries 
Continue on empty results as srid and geom type are not critical
when there are no results

Fixes #299
 2016-05-12 17:21:07 +02:00
Juan Ignacio Sánchez Lara
e979b79f49 gpkg format #291 2016-05-10 18:42:23 +02:00
Juan Ignacio Sánchez Lara
c82cafb1ed Geopackage format support #291 2016-04-27 16:56:06 +02:00
Daniel García Aubert
5de931daa5 Improver error message in job max payload limit. 2016-04-25 11:18:30 +02:00
Daniel García Aubert
1dcbb1afdf Improved error message 2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124 Fixed typos 2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944 Implemented limit to query size in Batch API. 2016-04-18 15:30:16 +02:00
Daniel García Aubert
056f22b156 Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series 2016-03-18 14:57:18 +01:00
Raul Ochoa
66cd1f400f Skip query tables cache for authenticated requests 2016-03-10 19:20:56 +01:00
Raul Ochoa
5ebc8e43d5 Start hits=0 as first time is not a hit 2016-03-08 14:50:08 +01:00
Raul Ochoa
b8460d033e Add a cache decorator over QueryTables 2016-03-08 14:48:56 +01:00
Raul Ochoa
dbe033112d Rethrow error but do not fail when getting affected tables 2016-03-08 14:00:31 +01:00
Alejandro Martínez
cd593e12ce Join all Surrogate-Keys in one line 2016-02-24 11:43:43 +01:00
Alejandro Martínez
7d024dbf3f Fix include in query_controller 2016-02-22 19:10:59 +01:00
Alejandro Martínez
5411aa3a01 Use Date.now() fallback value for getLastUpdatedTime 2016-02-22 19:08:04 +01:00
Alejandro Martínez
2c4b659100 Properly discard errors fetching updated tables 2016-02-22 19:07:41 +01:00
Alejandro Martínez
5e06711b4b Use node-cartodb-querytables library 2016-02-22 12:24:05 +01:00
Raul Ochoa
15f90c1a78 QueryTablesApi only caches affected tables and always retrieve last modification 2016-02-02 01:16:24 +01:00
Raul Ochoa
9deab814c9 Skip query-tables-api for authenticated requests 2016-02-01 16:53:28 +01:00
Raul Ochoa
24ecf130a1 Namespace queries so two users don't share results 
For instance 'select * from table' should result in different affected
tables (schema+table) depending on the database.
 2016-01-28 15:38:08 +01:00
Raul Ochoa
65bcdad657 Fix cyclomatic complexity 2016-01-28 14:48:12 +01:00
Raul Ochoa
86ecc3ad0a Implements authDbParams interface 
Uses user and pass from redis and relies on rest of params prepared
for normal dbParams
 2016-01-28 14:38:02 +01:00
Raul Ochoa
e70a21a2b4 Adds documentation about values passed over callback in getConnectionParams 2016-01-28 14:25:03 +01:00
Raul Ochoa
a654c90b6d X-Cache-Channel generation only requires affected tables 
mayWrite is redundant, and it's not its responsibility
 2016-01-28 14:18:38 +01:00
Raul Ochoa
131d798653 Change interface to return also authenticated connection params 2016-01-28 14:18:10 +01:00
Raul Ochoa
235e7f9bc1 Rename method to match returning connection params behaviour 2016-01-28 14:14:31 +01:00
Raul Ochoa
023d06fd19 Affected tables and last modified behind QueryTablesApi 
It handles the internal cache and exposes an unified result
 2016-01-28 14:08:18 +01:00
Raul Ochoa
7dad90a108 Remove unused dependencies from job controller 2016-01-28 13:10:21 +01:00
Raul Ochoa
3be769a67e Fix style 2016-01-28 12:33:32 +01:00
Raul Ochoa
9881a6f7d4 Use readonly flag to run affected tables query 2016-01-28 12:33:00 +01:00
Daniel García Aubert
e9bd93cf3f Now batch service is linked to express app in order to make it accesible from main app module to expose drain mechanism 2016-01-25 16:28:19 +01:00
Daniel García Aubert
f2ea01f627 Removed unnecessary check in job controller 2016-01-25 14:50:43 +01:00
Daniel García Aubert
e6f714a51b Implemented some new unit test 2016-01-22 19:31:25 +01:00
Daniel García Aubert
968caae9b7 Removed void comment 2016-01-21 18:26:48 +01:00
Daniel García Aubert
79e626a71e Added cors middleware, decoupled query and job controllers from its dependencies 2016-01-21 17:24:48 +01:00
Daniel García Aubert
048d2cd0ea Refactored user database service ans updated controllers to use it. 2016-01-21 16:17:17 +01:00
Daniel García Aubert
6024791e18 Merge branch 'batch-api' of github.com:CartoDB/CartoDB-SQL-API into batch-api 2016-01-21 15:35:23 +01:00
Daniel García Aubert
ad8b610321 Removed unused code 2016-01-21 15:14:55 +01:00
Daniel García Aubert
f42727212b Removed unnecessary checkAborted function in job controller and changed signature for getUserDatabase. 2016-01-21 15:05:46 +01:00
Raul Ochoa
2923c2e083 Update max cyclomatic complexity as it has decreased 2016-01-21 11:18:11 +01:00
Daniel García Aubert
2913aed697 Placed job controller endpoit under /api/vX/sql 2016-01-18 20:12:44 +01:00
Daniel García Aubert
2d13bde301 Implemented uses cases test for Batch API 2016-01-11 20:14:15 +01:00
Daniel García Aubert
078eb7706b Now SQL API is able to disable Batch Service using argv param --no-batch 2016-01-08 18:29:36 +01:00
Daniel García Aubert
20f00d58d9 Refactored batch service to avoid event noise, doing in callback way 2016-01-08 15:47:59 +01:00
Daniel García Aubert
f9f52d2bd1 Improved object creation in batch service 2016-01-08 11:32:01 +01:00
Daniel García Aubert
e6a4e0f7eb Returned 201 HTTP code in job creation 
Fixed bug listing jobs, now user job index is cleaned before respond to client
 2016-01-07 12:07:30 +01:00
Daniel García Aubert
40d0161218 Fixed bug in authentication for PUT methods in Job Controller 2016-01-05 13:54:18 +01:00
Daniel García Aubert
246655de94 Changed redis data structure for users jobs 2016-01-04 19:08:13 +01:00
Daniel García Aubert
35650985db Implemented job modification whether job is pending 2015-12-31 15:42:31 +01:00
Daniel García Aubert
ada39d84b8 Implemented job cancelation in Batch API 2015-12-30 20:16:18 +01:00
Daniel García Aubert
851c393f75 Refactored batch manager avoiding concurrence issues 2015-12-29 10:19:10 +01:00
Daniel García Aubert
4dcd7c0a50 Fixed typo and prepared for listing user's job 2015-12-24 17:47:41 +01:00
Daniel García Aubert
c4f96b4a59 Implemented support for listing jobs for one user 2015-12-24 17:42:49 +01:00
Daniel García Aubert
ecb76ca1de Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs. 2015-12-23 17:29:11 +01:00
Daniel García Aubert
62db809476 Return 401 when no API KEY is provided 2015-12-23 14:55:49 +01:00
Daniel García Aubert
3c23bf12e7 Fixed bug whether a job consumer is empty. And refactored batch module, now is a common object intead of module function 2015-12-22 20:12:10 +01:00
Daniel García Aubert
30d0b2906b Implemented GET for jobs API 2015-12-22 11:45:25 +01:00
Daniel García Aubert
ef59a1d942 Fixed typo in batch test 2015-12-22 11:06:46 +01:00
Daniel García Aubert
328d4c6bfe Fixed bad assertion in batch acceptance test 2015-12-22 11:02:16 +01:00
Daniel García Aubert
3762ad7a39 Now jobs are stored in redis instead of user's database 2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964 Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout 2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it. 2015-12-16 15:57:58 +01:00
Daniel García Aubert
d0787d03f7 Reused matadata backend instance for batch service 2015-12-14 10:36:16 +01:00
Daniel García Aubert
29e4c3c4af Avoid launch batch service in test enviroment 2015-12-10 18:40:44 +01:00
Daniel García Aubert
f5ca879ce3 Added entry point for main app to batch api 2015-12-10 15:08:31 +01:00
Daniel García Aubert
78ee92dbe5 Renamed exposed function 2015-12-10 15:06:25 +01:00
Daniel García Aubert
6cc48bf9dd Implemented batch service 2015-12-09 20:17:45 +01:00
Daniel García Aubert
c7680722ca Passed tests 2015-12-09 12:35:20 +01:00
Daniel García Aubert
00721bcd02 Implementing batch service 2015-12-09 00:02:08 +01:00
Daniel García Aubert
6f741827cd Fixed jshint error 2015-12-07 12:19:16 +01:00
vagrant
35d6600f36 Implemented test for job manegement 2015-12-07 10:29:55 +00:00
Daniel García Aubert
ee41db99ad Returned job_id to client/user after job enqueue 2015-12-07 09:59:05 +01:00
Daniel García Aubert
a1243ad64d Implemented job controller to enqueue jobs 2015-12-07 09:40:51 +01:00
Daniel García Aubert
a0895d4310 Fixed jshint error 2015-12-03 18:45:12 +01:00
Daniel García Aubert
f7a3c6ac4e Renamed controllers routing method 2015-12-03 18:43:13 +01:00
Daniel García Aubert
96e9fcbe33 Moved requirements to the top of module 2015-12-03 18:35:49 +01:00