cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,475 Commits 1 Branch 118 Tags 9.7 MiB
7727a9d506
Commit Graph

425 Commits

Author SHA1 Message Date
Daniel García Aubert
ea310db62c Cosmetic chenges 2018-02-19 11:04:28 +01:00
Daniel García Aubert
edd382fb8f Check user is the same user that sends the request when basic-auth is provided 2018-02-16 18:21:06 +01:00
Daniel García Aubert
2cadfe6f18 Add support for Basic Auth 2018-02-16 10:46:58 +01:00
Daniel García Aubert
756fbe42c9 Use apikey midlleware 2018-02-15 17:24:04 +01:00
Daniel García Aubert
67f2a85abd Add database credentials to jobs 2018-02-15 12:30:04 +01:00
Daniel García Aubert
b10937a8fb Use res.locals instead of req.context to share object among middlewares 2017-11-24 15:49:25 +01:00
Daniel García Aubert
c2d1a4653e Improve naming for connection database params holder 2017-11-24 15:45:49 +01:00
Daniel García Aubert
b1112e2bb1 Remove unneeded header 2017-09-20 16:48:22 +02:00
Daniel García Aubert
980e762ab7 Skip user auth middlewares 2017-09-20 16:35:33 +02:00
Daniel García Aubert
0597a3b8a6 Change job work in progress endpoint 2017-09-20 15:46:29 +02:00
Raul Ochoa
32154b67c6 Merge branch 'master' into upgrade-cartodb-psql 2017-08-11 11:57:16 +02:00
Daniel García Aubert
01a71ee60e Apply user timeout to ogr2ogr command 2017-08-09 12:50:16 +02:00
Raul Ochoa
3e536b822e Remove unnecessary destroyOnError option 2017-08-08 18:21:10 +02:00
Daniel García Aubert
99c7a6e4f9 Raise job query size to 16kb 2016-11-07 16:50:44 +01:00
Daniel García Aubert
4a64d37c6c Add work in progrees list endpoint 2016-10-28 15:08:42 +02:00
Raul Ochoa
e401c01d78 Only log on non-test environments 2016-10-12 01:40:14 +02:00
Raul Ochoa
2a2a54a073 DRY in job response handler 2016-10-04 16:07:13 +02:00
Raul Ochoa
19c9bec633 Callback with job 2016-10-04 15:57:13 +02:00
Raul Ochoa
05cbd55b95 Use db host from request's context 2016-10-04 15:50:39 +02:00
Raul Ochoa
c32a2199fa Use request bootstrapper to add host header 2016-10-04 15:43:19 +02:00
Raul Ochoa
7b7d651d8f DRY while authenticating requests 2016-10-04 15:40:56 +02:00
Raul Ochoa
20f50d988e Use user middleware in job controller 2016-10-04 15:19:31 +02:00
Raul Ochoa
2edc7505e7 Do not condition req.profiler 2016-10-04 15:08:31 +02:00
Raul Ochoa
3b6bc14d17 Increment errors on err 2016-10-04 13:19:29 +02:00
Raul Ochoa
2aa1f045ff Use .status() API 2016-09-26 18:10:20 +02:00
Raul Ochoa
ba0f2f1066 Remove endpoint to retrieve jobs list 2016-08-30 18:49:01 +02:00
Raul Ochoa
704ba110ba Log job creation 2016-08-30 18:43:09 +02:00
Raul Ochoa
ad0d101bfd Remove patch/put endpoints for jobs 2016-08-30 17:43:34 +02:00
Raul Ochoa
3ebbd9f7c4 Skip tables with no updated_at registered in cdb_tablemetadata 2016-07-11 16:39:12 +02:00
Daniel García Aubert
282da58ffe Set default value for statsd-client in job-controller to avoid check it every time it's going to be used 2016-05-30 12:27:19 +02:00
Daniel García Aubert
976bf5b039 Implemented profiling for job-runner and job-controller 2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577 Merge branch 'master' into batch-add-profile 2016-05-26 17:46:52 +02:00
Raul Ochoa
a91c7a6a88 Increase job payload size to 8kb 2016-05-24 15:41:18 +02:00
Raul Ochoa
23228b2d73 Payload size validates multiple queries and fallback queries 
It uses a middleware to check the body size
 2016-05-24 14:28:00 +02:00
Daniel García Aubert
64ad284c9c WIP: adding metrics to Batch API 2016-05-24 11:19:00 +02:00
Daniel García Aubert
cc7dd7a0d2 Job model refactor 2016-05-13 18:50:55 +02:00
Daniel García Aubert
5de931daa5 Improver error message in job max payload limit. 2016-04-25 11:18:30 +02:00
Daniel García Aubert
1dcbb1afdf Improved error message 2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124 Fixed typos 2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944 Implemented limit to query size in Batch API. 2016-04-18 15:30:16 +02:00
Daniel García Aubert
056f22b156 Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series 2016-03-18 14:57:18 +01:00
Raul Ochoa
66cd1f400f Skip query tables cache for authenticated requests 2016-03-10 19:20:56 +01:00
Raul Ochoa
b8460d033e Add a cache decorator over QueryTables 2016-03-08 14:48:56 +01:00
Raul Ochoa
dbe033112d Rethrow error but do not fail when getting affected tables 2016-03-08 14:00:31 +01:00
Alejandro Martínez
cd593e12ce Join all Surrogate-Keys in one line 2016-02-24 11:43:43 +01:00
Alejandro Martínez
7d024dbf3f Fix include in query_controller 2016-02-22 19:10:59 +01:00
Alejandro Martínez
5411aa3a01 Use Date.now() fallback value for getLastUpdatedTime 2016-02-22 19:08:04 +01:00
Alejandro Martínez
2c4b659100 Properly discard errors fetching updated tables 2016-02-22 19:07:41 +01:00
Alejandro Martínez
5e06711b4b Use node-cartodb-querytables library 2016-02-22 12:24:05 +01:00
Raul Ochoa
15f90c1a78 QueryTablesApi only caches affected tables and always retrieve last modification 2016-02-02 01:16:24 +01:00
Raul Ochoa
9deab814c9 Skip query-tables-api for authenticated requests 2016-02-01 16:53:28 +01:00
Raul Ochoa
a654c90b6d X-Cache-Channel generation only requires affected tables 
mayWrite is redundant, and it's not its responsibility
 2016-01-28 14:18:38 +01:00
Raul Ochoa
131d798653 Change interface to return also authenticated connection params 2016-01-28 14:18:10 +01:00
Raul Ochoa
235e7f9bc1 Rename method to match returning connection params behaviour 2016-01-28 14:14:31 +01:00
Raul Ochoa
023d06fd19 Affected tables and last modified behind QueryTablesApi 
It handles the internal cache and exposes an unified result
 2016-01-28 14:08:18 +01:00
Raul Ochoa
7dad90a108 Remove unused dependencies from job controller 2016-01-28 13:10:21 +01:00
Raul Ochoa
9881a6f7d4 Use readonly flag to run affected tables query 2016-01-28 12:33:00 +01:00
Daniel García Aubert
f2ea01f627 Removed unnecessary check in job controller 2016-01-25 14:50:43 +01:00
Daniel García Aubert
79e626a71e Added cors middleware, decoupled query and job controllers from its dependencies 2016-01-21 17:24:48 +01:00
Daniel García Aubert
048d2cd0ea Refactored user database service ans updated controllers to use it. 2016-01-21 16:17:17 +01:00
Daniel García Aubert
f42727212b Removed unnecessary checkAborted function in job controller and changed signature for getUserDatabase. 2016-01-21 15:05:46 +01:00
Daniel García Aubert
2913aed697 Placed job controller endpoit under /api/vX/sql 2016-01-18 20:12:44 +01:00
Daniel García Aubert
2d13bde301 Implemented uses cases test for Batch API 2016-01-11 20:14:15 +01:00
Daniel García Aubert
20f00d58d9 Refactored batch service to avoid event noise, doing in callback way 2016-01-08 15:47:59 +01:00
Daniel García Aubert
e6a4e0f7eb Returned 201 HTTP code in job creation 
Fixed bug listing jobs, now user job index is cleaned before respond to client
 2016-01-07 12:07:30 +01:00
Daniel García Aubert
40d0161218 Fixed bug in authentication for PUT methods in Job Controller 2016-01-05 13:54:18 +01:00
Daniel García Aubert
246655de94 Changed redis data structure for users jobs 2016-01-04 19:08:13 +01:00
Daniel García Aubert
35650985db Implemented job modification whether job is pending 2015-12-31 15:42:31 +01:00
Daniel García Aubert
ada39d84b8 Implemented job cancelation in Batch API 2015-12-30 20:16:18 +01:00
Daniel García Aubert
851c393f75 Refactored batch manager avoiding concurrence issues 2015-12-29 10:19:10 +01:00
Daniel García Aubert
4dcd7c0a50 Fixed typo and prepared for listing user's job 2015-12-24 17:47:41 +01:00
Daniel García Aubert
c4f96b4a59 Implemented support for listing jobs for one user 2015-12-24 17:42:49 +01:00
Daniel García Aubert
ecb76ca1de Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs. 2015-12-23 17:29:11 +01:00
Daniel García Aubert
62db809476 Return 401 when no API KEY is provided 2015-12-23 14:55:49 +01:00
Daniel García Aubert
30d0b2906b Implemented GET for jobs API 2015-12-22 11:45:25 +01:00
Daniel García Aubert
ef59a1d942 Fixed typo in batch test 2015-12-22 11:06:46 +01:00
Daniel García Aubert
328d4c6bfe Fixed bad assertion in batch acceptance test 2015-12-22 11:02:16 +01:00
Daniel García Aubert
3762ad7a39 Now jobs are stored in redis instead of user's database 2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964 Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout 2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it. 2015-12-16 15:57:58 +01:00
Daniel García Aubert
6cc48bf9dd Implemented batch service 2015-12-09 20:17:45 +01:00
Daniel García Aubert
c7680722ca Passed tests 2015-12-09 12:35:20 +01:00
Daniel García Aubert
00721bcd02 Implementing batch service 2015-12-09 00:02:08 +01:00
Daniel García Aubert
6f741827cd Fixed jshint error 2015-12-07 12:19:16 +01:00
vagrant
35d6600f36 Implemented test for job manegement 2015-12-07 10:29:55 +00:00
Daniel García Aubert
ee41db99ad Returned job_id to client/user after job enqueue 2015-12-07 09:59:05 +01:00
Daniel García Aubert
a1243ad64d Implemented job controller to enqueue jobs 2015-12-07 09:40:51 +01:00
Daniel García Aubert
f7a3c6ac4e Renamed controllers routing method 2015-12-03 18:43:13 +01:00
Daniel García Aubert
c6bae6d951 app module moved to right place 2015-12-03 18:33:17 +01:00
Daniel García Aubert
d31d496a2e Add useful comment 2015-12-03 18:25:35 +01:00
Daniel García Aubert
a697fe6faa Making test pass 2015-12-03 18:19:39 +01:00
Daniel García Aubert
e5dedb6315 Split query controller and separated from app 2015-12-03 17:28:18 +01:00
Daniel García Aubert
df59405dca Split some functionallity from app to dedicated controllers 2015-12-03 15:00:35 +01:00
Raul Ochoa
b160f4051a Merge pull request #250 from CartoDB/issue-238 
Stop adding X-Cache-Channel header when no tables involved
 2015-09-08 12:30:26 +02:00
Raul Ochoa
5ec6d7c77b Remove no longer needed health check params 2015-09-07 18:34:50 +02:00
Raul Ochoa
875eccba62 Stop adding X-Cache-Channel header when no tables were identified in SQL query 
Fixes #238 and fixes #157
 2015-09-07 18:22:24 +02:00
Raul Ochoa
335d71bc04 Adds a custom log4js format function to log sql query from POST requests 
Tokens in log4js are not provided with request/response for context,
using a custom format function allow to access them

In the case of express logger token functions receive request/response
for context so it uses a custom token

Closes #206
 2015-09-04 12:29:20 +02:00
Raul Ochoa
f816093036 Set Last-Modified header based on affected tables 
Closes #101
 2015-09-02 16:25:56 +02:00
Raul Ochoa
c6c614b5bf Do not return results from health check 
It also removed old dependencies and takes disabled file path in ctor.
 2015-08-28 18:16:04 +02:00
Raul Ochoa
d42e5a6b75 Do not expose remove from pool after query error as a config option 2015-08-04 16:23:36 +02:00
Raul Ochoa
7c696317d0 New option to remove client from pool after error happens 
This help to avoid issues with transactions.

Closes #241
 2015-08-04 15:53:50 +02:00
Raul Ochoa
a297a14300 Revert "Uses tables ogr2ogr param to avoid heavy query on pg_class and friends" 
This reverts commit b6e53f7326.
 2015-06-16 16:58:22 +02:00
Francisco Dans
3a8f94641b Merge pull request #225 from CartoDB/log-addr 
Uses :remote-addr instead of :req[X-Real-IP]
 2015-06-01 16:17:47 +02:00
Francisco Dans
996bd971cd replaces default log format string to remote-addr 2015-05-26 19:18:54 +02:00
Raul Ochoa
debeb8ca3e Update news 2015-05-25 16:38:07 +02:00
Raul Ochoa
b6e53f7326 Uses tables ogr2ogr param to avoid heavy query on pg_class and friends 
Fixes #204
Newer versions of ogr2ogr don't use that heavy query so it should work
Ref 2e3eeefd0f
 2015-05-25 16:29:18 +02:00
Raul Ochoa
9dd887dc4d jshint fixes, not addressing complexity 2015-05-13 12:22:41 +02:00
Raul Ochoa
9dcb397737 Merge pull request #205 from CartoDB/url_rewrite 
added /u/:user routing
 2015-03-20 12:22:07 +01:00
Raul Ochoa
9b31df6793 Improve row size limit error message 2015-03-02 14:34:01 +01:00
Raul Ochoa
302a856d37 Move note about username extraction to cartodb_request 2015-02-02 12:15:53 +01:00
Raul Ochoa
e5ab4272eb Use a multiple params route to be able to extract the username from 
the path or default to host header.
 2015-02-02 12:09:34 +01:00
javi
824b7c084e added /u/:user routing 2015-01-30 11:37:29 +01:00
Seth Fitzsimmons
bcc1577ec4 Add 'refresh' to the list of keywords that write 2015-01-06 12:40:01 -08:00
Raul Ochoa
d50ddbb10f Add more fields to error responses with hint, detail and context for SQL errors. 2014-11-21 12:59:48 +01:00
Raul Ochoa
74429f82e1 Improve topojson output by streaming json 2014-11-12 11:36:59 +01:00
Raul Ochoa
dc601a5feb Improve statement timeout error messages 2014-11-11 13:57:15 +01:00
Raul Ochoa
25d2e64891 Removes query tables console.log 2014-09-17 15:54:46 +02:00
Raul Ochoa
2d232c20ef Returns 401 Unauthorized for queries without permission 2014-09-02 15:00:04 +02:00
Raul Ochoa
7c62632ea8 Health check endpoint 2014-08-26 18:40:58 +02:00
Raul Ochoa
3a26692448 New header for database host serving the request 2014-08-22 12:29:27 +02:00
Raul Ochoa
0035f04edd Metrics revamp: removes and adds some metrics 2014-08-18 19:45:17 +02:00
Raul Ochoa
77cb86154c Starts using cartodb-psql node module in SQL API 2014-08-11 20:15:55 +02:00
Raul Ochoa
951636892c Removes unused vars 2014-08-08 12:54:36 +02:00
Raul Ochoa
21b8e6947c Non authenticated request cannot use pg_ catalogs/functions 2014-08-08 12:48:29 +02:00
Raul Ochoa
f6c364b3b9 CDB-3032 Removes sql statements restriction on pg_ queries 2014-08-07 16:22:48 +02:00
Raul Ochoa
bd60f8f748 Using new method from cartodb-redis to retrieve oauth values so we 
can reuse the same redis-mpool
 2014-08-06 12:51:55 +02:00
Raul Ochoa
a7027f4b05 Removes console.log 2014-08-05 16:48:11 +02:00
Raul Ochoa
480a9f27b4 New authentication mechanism: checks in advance if credentials are provided 
in order to do a single request to redis to retrieve the required database
connection parameters.
 2014-08-05 16:20:06 +02:00
Raul Ochoa
49406c99fa Moves auth functionality to its own directory 2014-08-05 11:57:43 +02:00
Raul Ochoa
e16f278087 CDB-3629 Uses one request to redis to retrieve all connection params 2014-08-05 02:29:07 +02:00
Raul Ochoa
ce70e7252b Callback requests send 200 status error even if the query failed 2014-08-04 15:56:43 +02:00
Raul Ochoa
73a195a7fa Use streaming for json responses. Number of buffered rows can be specified by config. 2014-07-31 02:17:07 +02:00
Luis Bosque
6bb015bb14 [CDB-3678] Add X-Served-By-Host header 2014-07-30 12:39:03 +02:00
Raul Ochoa
2653314818 CDB-3657 Sends profiler data as json in X-SQLAPI-Profiler header 2014-07-29 19:29:16 +02:00
Raul Ochoa
0f14118235 CDB-3629 Pickup redis pool config values 2014-07-29 13:58:16 +02:00
Raul Ochoa
a7d444ca92 Changes authentication to start using public user if it is defined in redis. 2014-07-29 13:52:04 +02:00
Raul Ochoa
1a390a3d40 Some style changes 2014-07-29 13:52:04 +02:00
Raul Ochoa
21d6924088 Removes the call to set the search path as it should get the correct public user from redis 2014-07-29 13:52:04 +02:00
javi
ca7b87d81a fixed problem when a query raised a database error 2014-07-29 13:52:04 +02:00
Raul Ochoa
9c255c504b Updates CDB_QueryTables to match the one from the extension. Fixes some tests to show the schema presence. 
Skip some tests that we need to review because with the current implementation of CDB_QueryTables dont make sense.
CDB_QueryTables should be used from the extension as a dependency.
 2014-07-29 13:52:04 +02:00
javi
755ed362d4 set scheme to host user when public user is accessing 2014-07-29 13:52:04 +02:00
Raul Ochoa
b3609696a3 requests associated with formats based on postgres expose a cancel method that will be called on client request abortion/cancelling so postgres can cancel ongoing queries 2014-06-02 14:48:38 +02:00
Sandro Santilli
4b5a5921d5 Fix application_name to "cartodb_sqlapi" 2014-05-07 16:15:46 +02:00
Sandro Santilli
51b135c0ac Set default PostgreSQL application name to "cartodb_sqlapi" 2014-05-07 16:14:17 +02:00
Raul Ochoa
2d2c056d6b Removes spawn unused dependency 2014-04-22 11:18:52 +02:00
Raul Ochoa
537cb238c6 CDB-2081 Adds support for order_by through http query params. 2014-04-14 15:01:12 +02:00
Raul Ochoa
909530cfb3 documentation for sql parameter 2014-04-11 12:13:44 +02:00
Raul Ochoa
d10bd8e3b0 precompiled query may write regex 2014-04-11 12:03:43 +02:00
Sandro Santilli
766345595e Add statsd support 
Closes #133
 2014-03-20 11:22:13 +01:00
Sandro Santilli
2cbdc4c38c Add profiler support 
Closes #142
 2014-03-19 18:34:21 +01:00