cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,908 Commits 1 Branch 118 Tags 9.7 MiB
f77df67eba
Commit Graph

312 Commits

Author SHA1 Message Date
Raul Ochoa
d50ddbb10f Add more fields to error responses with hint, detail and context for SQL errors. 2014-11-21 12:59:48 +01:00
Raul Ochoa
74429f82e1 Improve topojson output by streaming json 2014-11-12 11:36:59 +01:00
Raul Ochoa
dc601a5feb Improve statement timeout error messages 2014-11-11 13:57:15 +01:00
Raul Ochoa
25d2e64891 Removes query tables console.log 2014-09-17 15:54:46 +02:00
Raul Ochoa
2d232c20ef Returns 401 Unauthorized for queries without permission 2014-09-02 15:00:04 +02:00
Raul Ochoa
7c62632ea8 Health check endpoint 2014-08-26 18:40:58 +02:00
Raul Ochoa
3a26692448 New header for database host serving the request 2014-08-22 12:29:27 +02:00
Raul Ochoa
0035f04edd Metrics revamp: removes and adds some metrics 2014-08-18 19:45:17 +02:00
Raul Ochoa
77cb86154c Starts using cartodb-psql node module in SQL API 2014-08-11 20:15:55 +02:00
Raul Ochoa
951636892c Removes unused vars 2014-08-08 12:54:36 +02:00
Raul Ochoa
21b8e6947c Non authenticated request cannot use pg_ catalogs/functions 2014-08-08 12:48:29 +02:00
Raul Ochoa
f6c364b3b9 CDB-3032 Removes sql statements restriction on pg_ queries 2014-08-07 16:22:48 +02:00
Raul Ochoa
bd60f8f748 Using new method from cartodb-redis to retrieve oauth values so we 
can reuse the same redis-mpool
 2014-08-06 12:51:55 +02:00
Raul Ochoa
a7027f4b05 Removes console.log 2014-08-05 16:48:11 +02:00
Raul Ochoa
480a9f27b4 New authentication mechanism: checks in advance if credentials are provided 
in order to do a single request to redis to retrieve the required database
connection parameters.
 2014-08-05 16:20:06 +02:00
Raul Ochoa
49406c99fa Moves auth functionality to its own directory 2014-08-05 11:57:43 +02:00
Raul Ochoa
e16f278087 CDB-3629 Uses one request to redis to retrieve all connection params 2014-08-05 02:29:07 +02:00
Raul Ochoa
ce70e7252b Callback requests send 200 status error even if the query failed 2014-08-04 15:56:43 +02:00
Raul Ochoa
73a195a7fa Use streaming for json responses. Number of buffered rows can be specified by config. 2014-07-31 02:17:07 +02:00
Luis Bosque
6bb015bb14 [CDB-3678] Add X-Served-By-Host header 2014-07-30 12:39:03 +02:00
Raul Ochoa
2653314818 CDB-3657 Sends profiler data as json in X-SQLAPI-Profiler header 2014-07-29 19:29:16 +02:00
Raul Ochoa
0f14118235 CDB-3629 Pickup redis pool config values 2014-07-29 13:58:16 +02:00
Raul Ochoa
a7d444ca92 Changes authentication to start using public user if it is defined in redis. 2014-07-29 13:52:04 +02:00
Raul Ochoa
1a390a3d40 Some style changes 2014-07-29 13:52:04 +02:00
Raul Ochoa
21d6924088 Removes the call to set the search path as it should get the correct public user from redis 2014-07-29 13:52:04 +02:00
javi
ca7b87d81a fixed problem when a query raised a database error 2014-07-29 13:52:04 +02:00
Raul Ochoa
9c255c504b Updates CDB_QueryTables to match the one from the extension. Fixes some tests to show the schema presence. 
Skip some tests that we need to review because with the current implementation of CDB_QueryTables dont make sense.
CDB_QueryTables should be used from the extension as a dependency.
 2014-07-29 13:52:04 +02:00
javi
755ed362d4 set scheme to host user when public user is accessing 2014-07-29 13:52:04 +02:00
Raul Ochoa
b3609696a3 requests associated with formats based on postgres expose a cancel method that will be called on client request abortion/cancelling so postgres can cancel ongoing queries 2014-06-02 14:48:38 +02:00
Sandro Santilli
4b5a5921d5 Fix application_name to "cartodb_sqlapi" 2014-05-07 16:15:46 +02:00
Sandro Santilli
51b135c0ac Set default PostgreSQL application name to "cartodb_sqlapi" 2014-05-07 16:14:17 +02:00
Raul Ochoa
2d2c056d6b Removes spawn unused dependency 2014-04-22 11:18:52 +02:00
Raul Ochoa
537cb238c6 CDB-2081 Adds support for order_by through http query params. 2014-04-14 15:01:12 +02:00
Raul Ochoa
909530cfb3 documentation for sql parameter 2014-04-11 12:13:44 +02:00
Raul Ochoa
d10bd8e3b0 precompiled query may write regex 2014-04-11 12:03:43 +02:00
Sandro Santilli
766345595e Add statsd support 
Closes #133
 2014-03-20 11:22:13 +01:00
Sandro Santilli
2cbdc4c38c Add profiler support 
Closes #142
 2014-03-19 18:34:21 +01:00
Sandro Santilli
b0d0d4d073 Reduce work on aborted requests 
Closes #129
Includes testcase
 2014-03-19 13:30:29 +01:00
Sandro Santilli
67cc05085f Always use INFO severity for express logging 
Avoids sending non-200 responses logs to rollbar, when active
 2014-03-13 17:13:53 +01:00
Sandro Santilli
622846b453 Add '/version' endpoint 
Closes #138
Includes testcase
 2014-03-13 13:40:56 +01:00
Sandro Santilli
ac4acf98ae Use console.error for logging errors 2014-03-13 12:14:11 +01:00
Sandro Santilli
1f3aca837b Use log4js for logging 
Closes #136
 2014-03-13 11:52:40 +01:00
Sandro Santilli
b565ce2959 Allow configuring log_format 
Closes #131
 2014-02-10 17:34:27 +01:00
Sandro Santilli
ec1d002812 Stop using ANSI colors in the logs 
Closes #130
 2014-02-10 17:10:34 +01:00
Sandro Santilli
6c47537f47 Honour the 'node_socket_timeout' configuration directive 
Closes #128
Includes testcase, which required enhancing assert.response to
handle errors.
Changes app controller to a function that only reads configuration
when executed (rather than when only require'd).
 2014-01-31 10:58:19 +01:00
Sandro Santilli
161a0bf5e5 Restore compatibility with 1.6.x configuration 2014-01-17 13:14:30 +01:00
Sandro Santilli
80c30858bd Enhance error message on unknown cartodb username 
Closes #126
 2013-12-18 12:02:51 +01:00
Sandro Santilli
87d35aa155 Generalize CartoDB username extraction, allowing for multiuser setups 
Closes #124
 2013-12-18 11:59:00 +01:00
Sandro Santilli
6ebeed04c2 Optionally read user-specific database_password from redis 
Follows CartoDB-2.5.0 model. Includes testcase.
Closes #120 -- Jira ref CDB-870
 2013-11-18 13:31:11 +01:00
Sandro Santilli
af1f3daa69 Optionally read user-specific database_host from redis 
Follows CartoDB-2.5.0 model. Includes testcase.
Closes #121 -- Jire ref CDB-870
 2013-11-18 12:21:30 +01:00
Sandro Santilli
7218c89b2d Refactor PSQL model to take full db config in the constructor 
Closes #58, stops reading configuration variables itself
 2013-11-18 11:42:43 +01:00
Sandro Santilli
03ea51a375 CartoDB redis interaction delegated to "cartodb-redis" module 2013-11-15 19:14:55 +01:00
Sandro Santilli
973c7181f0 Allow access to tables whose name contains "pg_" (but not at beginning) 
Closes #118
 2013-11-07 10:16:58 +01:00
Sandro Santilli
8d2347af99 Make public PostgreSQL user name a configuration parameter 
Closes #56.
Updates documentation and tests and example config files
 2013-11-05 17:49:10 +01:00
Sandro Santilli
1637610f66 Set a meaningful X-Cache-Channel with cache_policy=persist 
Closes #105
 2013-10-18 13:29:06 +02:00
Sandro Santilli
a0ff7ee10c Remove last bit of hard-coded format names from app controller 2013-09-05 17:10:48 +02:00
javi
d83a81bf32 merged body_param_fix 2013-08-02 15:50:48 +02:00
Sandro Santilli
7a07a25086 Add REINDEX to the list of uncacheable queries 2013-07-24 18:43:38 +02:00
Sandro Santilli
8506108aaa Do not cache results for queries you can't fetch source tables of 2013-07-22 18:20:26 +02:00
Sandro Santilli
2ee9411889 Review cache-control headers: 
* Raise max-age to one year for all cacheable queries
 * Set max-age to 0 for uncacheable (mutating) queries
 2013-07-22 17:21:38 +02:00
javi
1eb7b3c885 Merge branch 'develop' into body_param_fix 2013-07-16 11:21:05 +02:00
javi
8ba1464db1 keep req.query as it comes to not broke ouaht refs Vizzuality/cartodb-management/2853 2013-07-15 14:39:15 +02:00
Sandro Santilli
3b9bc8a9f1 Do not set Last-Modified in the remote past 
... or max-age will be always reached
 2013-07-15 13:13:19 +02:00
Sandro Santilli
dfff99224f Set Last-Modified header with cache_policy=persist 2013-07-15 12:12:07 +02:00
Luis Bosque
7b6b541083 Revert "Set max-age=0 when using no-cache" 2013-07-10 07:58:35 +02:00
Sandro Santilli
bc49aebd2a Don't assume SELECT "updated_at" may write to the database. 
Closes #99
 2013-07-08 17:36:53 +02:00
javi
69b5d825e3 reverted support for get params in post 2013-07-04 16:23:48 +02:00
javi
e114b4266d removed debug console.log 2013-07-04 16:17:51 +02:00
javi
838c0c93ba Add support for GET params in POST requests 2013-07-04 16:14:44 +02:00
Sandro Santilli
46e93355c4 Set max-age=0 when using no-cache (don't be self-contraddicting) 2013-06-28 11:17:52 +02:00
Sandro Santilli
7eff0cb107 Retain UTC offset in JSON output for dates 
Hopefully helps reducing confusion when using timezone-less
with postgresql, in that the time that comes out of JSON output
matches the one input by user, except it may be in an unexpected
timezone (the server's one)
 2013-06-19 12:25:48 +02:00
Sandro Santilli
5e379012a8 Add support for CTE in sql windowing, add unit tests 
This is still an undocumented feature, but as long as it's present
and used (by cartodb UI) better tested than broken...

NOTE: more tests are needed for CTE and RETURNING queries
 2013-06-17 16:36:36 +02:00
Sandro Santilli
339bc0b3a4 Fix windowing support for non-uppercased SELECT queries 2013-06-06 15:24:57 +02:00
Luis Bosque
fac2b1bb68 Trust forwarded proto header from proxy 2013-05-30 11:56:43 +02:00
javi
0be9e47930 Add arraybuffer format 2013-05-28 10:37:39 +02:00
Sandro Santilli
76705a3fd7 Refactor format objects to expose a common stream-able interface 
See app/models/formats/README for info
 2013-05-28 10:36:15 +02:00
Sandro Santilli
75717bfb8e Drop another unused function 2013-05-27 09:24:11 +02:00
Sandro Santilli
3f98cab09a Do not execute queries on OPTIONS. Closes #94 2013-05-24 14:21:13 +02:00
Sandro Santilli
cf6bed9bdb Bubble paging UI hack up from model to controller 
... one day we'll need to completely drop this hack!
 2013-05-24 10:22:17 +02:00
javi
c46e9786b5 refactor formats to external object 2013-05-24 08:50:03 +02:00
Sandro Santilli
b1e7bb8f8f Upgrade underscore dependencies 2013-05-24 08:26:23 +02:00
Sandro Santilli
81a83b93e4 Make base url configurable, use a wildcard in the example config 2013-05-23 11:49:23 +02:00
Luis Bosque
fcf95755b6 Fixed problem identifying OAuth request protocol 
The protocol of a OAuth request wasn't being identified correctly. It
always considered that the request protocol was https.
 2013-05-14 18:01:37 +02:00
Sandro Santilli
414f4b6c3e Do not request caching of TRUNCATE queries 2013-05-06 18:21:22 +02:00
Sandro Santilli
cc74244b33 Do not choke on multiple skipfields parameter 2013-05-06 12:30:32 +02:00
Sandro Santilli
0ec66c69a7 Allow sql queries to end with a semicolon. Closes #90 2013-04-10 17:21:42 +02:00
Sandro Santilli
e7437ba7cd Rework system catalogue prevention access check to use CDB_QueryTables 
This change reduces the chances of false positive
(forbidding legit queries). Doesn't solve the problem of false
negative (allowing illegit queries).
 2013-04-09 12:36:37 +02:00
Sandro Santilli
104ed355d0 Fix fd leak on export query cancel 2013-03-27 20:02:21 +01:00
Sandro Santilli
8f60735b5e Queue export requests for shapefile format. Closes #65 2013-03-27 16:43:12 +01:00
Sandro Santilli
b2e88145ff Comment out debugging line 2013-03-27 16:09:56 +01:00
Sandro Santilli
0261e8f687 Drop unused dependencies (csv and strftime) 2013-03-27 15:55:57 +01:00
Sandro Santilli
6d6dd16446 Draft delegating CSV output to OGR 
Adapt tests to new output
 2013-03-27 15:41:24 +01:00
Sandro Santilli
ff49346d4d Drop unused variable 2013-03-26 18:20:36 +01:00
Sandro Santilli
58b079e212 Fix reace condition in new KML output, test KML export of empty table 2013-03-26 18:18:13 +01:00
Sandro Santilli
38f99c586a Fix concurrent requests for KML export 2013-03-26 16:59:37 +01:00
Sandro Santilli
a0d9038e59 Auth / non-auth tests for SHP and KML export are in place 2013-03-26 16:14:48 +01:00
Sandro Santilli
c3ae7074cb Make temporary dir a configuration setting 2013-03-26 16:02:05 +01:00
Sandro Santilli
12553dafdd Don't forget to call .end() on the result (fixes Shapefile export) 2013-03-26 15:32:07 +01:00
Sandro Santilli
3f8a606432 Simplify steps for SHP and KML outputs 2013-03-26 12:59:46 +01:00
Sandro Santilli
41dd23b429 Do not run the query twice when exporting to KML 2013-03-26 12:47:03 +01:00
Sandro Santilli
dc87209033 Do not confuse warnings with errors on shapefile output 
Closes #87
 2013-02-25 18:32:28 +01:00
Sandro Santilli
d4b1f580ff Fix X-Cache-Channel computation with paging parameters. Closes #85. 2013-02-18 16:19:12 +01:00
Sandro Santilli
10ec5b9fda Do not attempt to compute an X-CacheChannel from a broken query info 
Also enhance debugging for broken query info results
 2013-02-18 12:20:18 +01:00
Sandro Santilli
19eb51b06d Use an LRU cache for query tables, and add expiration support 
Defaults to 8192 items each valid for no more than 10 minutes
Both settings can be changed via environment config.
See #83
 2013-02-13 18:59:09 +01:00
Sandro Santilli
7d2d585c54 Check CDB_QueryTable response before saving into cache 
Also add pid to /cachestatus (#83)
 2013-02-13 16:43:04 +01:00
Sandro Santilli
5959e6465a Fix Content-Disposition for error responses. Closes #82 2013-02-13 13:32:34 +01:00
Sandro Santilli
d667d64c78 Fix skipfields use with SHP output format. Closes #81 2013-02-13 13:02:37 +01:00
Sandro Santilli
977ecbeb29 Return an error when "the_geom" is in skipfield for SVG output 
Closes #73
 2013-01-21 10:06:51 +01:00
Sandro Santilli
e897cb07d2 Add test for null geoms in geojson 
Also drop an hard-coded "the_geom" in geojson output code
 2013-01-11 18:57:45 +01:00
Sandro Santilli
9bc79bded6 Re-add null geometry support in geojson output 
You never know if anyone is relying on them, and they are supported
by the specs:
https://github.com/mbostock/topojson/issues/18#issuecomment-12154398

We'll still skip those records for topojson
 2013-01-11 18:43:05 +01:00
Sandro Santilli
ca065e89b9 Omit NULL the_geom features from from geojson and topojson outputs 
Closes #80
 2013-01-11 18:20:25 +01:00
Sandro Santilli
3e7c8f4d22 Improve mixed geometry export error message. Closes #78 2013-01-11 17:09:22 +01:00
Sandro Santilli
a81b55e421 Include properties to TopoJSON output. Closes #79 
Includes support for the skipfields option
 2013-01-11 11:10:51 +01:00
Sandro Santilli
39669578b6 Initial support for TopoJSON (#79) 
Does not include any attributes in the format
 2013-01-09 17:43:23 +01:00
Sandro Santilli
2e85588d99 Format date fields in CSV output 2012-12-07 14:27:02 +01:00
Sandro Santilli
c00cf6d390 Revert "Upgrade node-csv module to version 0.2.4" 
This reverts commit 270e6280ed.
The performance issue must be fixed before we can upgrade:
https://github.com/wdavidw/node-csv-parser/issues/64
 2012-11-19 18:44:53 +01:00
Sandro Santilli
270e6280ed Upgrade node-csv module to version 0.2.4 
Will help with compressed-csv export format
 2012-11-19 16:20:40 +01:00
Sandro Santilli
3a641beda8 Fix UTF8 in shapefile export. Closes #66. 
Testcases count reaches 100 ! Happy post-post-gis day :)
 2012-11-16 12:50:21 +01:00
Sandro Santilli
da05e6f4ca Fix KML export truncation. Closes #947 
Also adds test for SHP truncation (isn't trucated)
 2012-11-16 12:22:06 +01:00
Sandro Santilli
75fcd5ae9c Do not write back to request.query as it breaks oAuth signature 
NOTE: this breakage misses a testcase (overcomplex at the moment)
 2012-11-14 18:04:38 +01:00
Sandro Santilli
bd08eb4add Use inline attachment also with POST, if format isn't given 2012-11-14 16:30:18 +01:00
Sandro Santilli
6b7cada97d Never dispose "inline" when using POST. 2012-11-13 19:27:25 +01:00
johnhackworth
e64c3f57f7 changes in the way the parameters are collected to support both POST and GET 2012-11-13 19:27:25 +01:00
Sandro Santilli
0ef13f08c2 Use inline disposition when no format and no filename are given 
See #61
 2012-11-12 19:44:16 +01:00
Sandro Santilli
120cf3f0c5 Add 'cache_policy' parameter. Closes #62 2012-11-12 19:14:20 +01:00
Sandro Santilli
005ae48e3a Support for specifying a list of fields to skip from output. 
Closes #63
 2012-11-12 17:11:17 +01:00
Sandro Santilli
46cec7a0e5 Add support for specifying a filename for exports. Closes #64 
Sets release target to 1.3.0, due to parameter addition
 2012-11-12 12:40:53 +01:00
javi
3fd3c5fabc added the needed headers for CORS 2012-11-01 13:17:45 +01:00
Sandro Santilli
110af3e9d9 Add "sqlapi" prefix to temporary dir for shapefile output 2012-10-31 11:57:20 +01:00
Sandro Santilli
9aa28c5bea Initial support for KML output format. Closes #54. 2012-10-25 18:10:56 +02:00
Sandro Santilli
978c0b4cbe Advertise header presence in CSV Content-Type 2012-10-25 13:40:21 +02:00
Sandro Santilli
a6837573c5 Use "attachment" Content-Disposition for all output formats. 
Closes #61, includes tests

NOTE: this includes the default "json" format.
 2012-10-25 13:34:06 +02:00
Sandro Santilli
5d8eccc81e Fix CSV output with no rows. Closes #60 2012-10-25 13:14:47 +02:00
Sandro Santilli
a560a37823 Generalize OGR output function 2012-10-23 17:45:56 +02:00
Sandro Santilli
26bdccf541 Remove debugging output 2012-10-18 13:19:08 +02:00
Sandro Santilli
d0ae7e08a6 Initial support for Shapefile output 2012-10-18 12:51:12 +02:00
Sandro Santilli
8574517ab8 Add a userid_to_dbuser function 2012-10-18 11:33:35 +02:00
Sandro Santilli
d9b733e5c6 Recognize ALTER as a writing query 2012-10-15 13:40:04 +02:00
Sandro Santilli
d23416cc60 Set X-Cache-Channel to NONE when the SQL may write to the database 
Note that "may write" allows for false positive, so there could be
less cache hits than possibly allowable. If this will be a problem
for any real use case we could still improve the regular expression
used to detect "writing" queries.

Automated tests are added to check for the X-Cache-Channel header
with both writing and read-only queries performed by authenticated
requests.

Closes #27
Closes #43
 2012-10-15 13:23:17 +02:00
Sandro Santilli
553146e6dd Add consistency checking in packageResult about format value 2012-10-12 12:57:03 +02:00
Sandro Santilli
ac83700810 Send a 404 on unsupported format requested 2012-10-12 12:17:35 +02:00
Sandro Santilli
0d91ab2c6a Survive multiple "format" parameters, only using last one 2012-10-12 11:42:03 +02:00
Sandro Santilli
6a2c0e9727 Initial support for SVG output (#49) 
This version only dumps the geometries with no identifier.
Scales geometries to fit in a 1024x768 pixels area, using
a circle radius of 5 pixels, a stroke-width of 1 pixel and
no fill.  Supports trimming number of decimals.
Adds a viewBox tag to fit the drawing to the output device window.

Includes an automated testcase.
 2012-10-04 12:04:50 +02:00
Sandro Santilli
4521942820 Test "CSV" format, drop "KML" from the list of supported formats 2012-10-04 12:04:50 +02:00
Sandro Santilli
eb645d298c More embedded comments 2012-10-04 12:04:49 +02:00
Sandro Santilli
8b824801cf Fix INSERT and UPDATE with RETURNING clause. Closes #50 
Includes regression test
 2012-09-17 11:50:19 +02:00
JM
c881d991a6 Support case insensitive in format parameter. Closes #30 
Also fixes https://github.com/Vizzuality/cartodb/issues/795
 2012-07-13 10:24:12 +02:00
strk
e6b4896338 Merge pull request #33 from jmnavarro/issue-14 
Allow using without redis (closes #14)

... as long as OAuth is not used and database name is given in URL
 2012-07-12 08:31:24 -07:00
JM
7e1847bca5 fixed #13 
from support: Different JSON result for non-query request
http://support.cartodb.com/discussions/suggestions/50-sql-api-different-json-object-for-non-query-request
 2012-06-30 02:34:43 +02:00
JM
b65d947b5a skip redis when database is hardcoded 2012-06-30 01:54:53 +02:00