cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
689 Commits 1 Branch 118 Tags 9.7 MiB
63d41e4843
Commit Graph

271 Commits

Author SHA1 Message Date
Luis Bosque
80452ef6f4 Merge branch 'release/staging' into develop 2013-05-30 11:17:10 +02:00
Luis Bosque
12afc3c88f In OAuth requests, get req.path 
Using req.route.path breaks OAuth whtn used with regex routes. Instead
of this we need to use req.path because the path is already expanded
 2013-05-29 17:36:41 +02:00
Sandro Santilli
ffafe814ef Do not hang up to queued request on ogr generation failure 2013-05-29 17:33:10 +02:00
Sandro Santilli
7f9463e7fd Fix filesystem access conflict among clustered processes 2013-05-29 16:48:20 +02:00
Sandro Santilli
510ac0533f Switch to using evented query model for postgresql 
This enables formats for processing rows as they arrive from the
database, thus possibly reducing memory use. For a start the skip
fields are immediately removed from the result, rather than only
at the end.
 2013-05-29 15:39:39 +02:00
javi
0be9e47930 Add arraybuffer format 2013-05-28 10:37:39 +02:00
Sandro Santilli
76705a3fd7 Refactor format objects to expose a common stream-able interface 
See app/models/formats/README for info
 2013-05-28 10:36:15 +02:00
Sandro Santilli
75717bfb8e Drop another unused function 2013-05-27 09:24:11 +02:00
Sandro Santilli
ab8494ab12 Survive postgresql connection losses. Closes #95. 2013-05-24 18:18:17 +02:00
Sandro Santilli
11a61d16c4 Drop unused code. Spring cleanups! \o/ 2013-05-24 17:13:05 +02:00
Sandro Santilli
3f98cab09a Do not execute queries on OPTIONS. Closes #94 2013-05-24 14:21:13 +02:00
Sandro Santilli
7712d6f2a5 Expose PostgreSQL client pooling settings in environment files 
Actually also moves to a real pooling model, managed by node-pg.
Closes #47
See .example files for new configurations
 2013-05-24 12:09:06 +02:00
Sandro Santilli
64c0494105 Hide dangerous methods of the PSQL model class 
These steps are prepatorial to recactoring to event-based model
 2013-05-24 10:38:27 +02:00
Sandro Santilli
cf6bed9bdb Bubble paging UI hack up from model to controller 
... one day we'll need to completely drop this hack!
 2013-05-24 10:22:17 +02:00
javi
c46e9786b5 refactor formats to external object 2013-05-24 08:50:03 +02:00
Sandro Santilli
b1e7bb8f8f Upgrade underscore dependencies 2013-05-24 08:26:23 +02:00
Sandro Santilli
81a83b93e4 Make base url configurable, use a wildcard in the example config 2013-05-23 11:49:23 +02:00
Sandro Santilli
e5ad8f0760 Fix test broken after oAuth interface change introduced in fcf95755 
Also improve error message on miscall
 2013-05-20 09:06:08 +02:00
Luis Bosque
fcf95755b6 Fixed problem identifying OAuth request protocol 
The protocol of a OAuth request wasn't being identified correctly. It
always considered that the request protocol was https.
 2013-05-14 18:01:37 +02:00
Sandro Santilli
414f4b6c3e Do not request caching of TRUNCATE queries 2013-05-06 18:21:22 +02:00
Sandro Santilli
cc74244b33 Do not choke on multiple skipfields parameter 2013-05-06 12:30:32 +02:00
Sandro Santilli
0ec66c69a7 Allow sql queries to end with a semicolon. Closes #90 2013-04-10 17:21:42 +02:00
Sandro Santilli
e7437ba7cd Rework system catalogue prevention access check to use CDB_QueryTables 
This change reduces the chances of false positive
(forbidding legit queries). Doesn't solve the problem of false
negative (allowing illegit queries).
 2013-04-09 12:36:37 +02:00
Sandro Santilli
d54d953e75 Another bit of improvement for the system-table query prevention 
This work is more important for the testcases than the actual code
 2013-04-09 12:20:27 +02:00
Sandro Santilli
1bcffbc68c Make using SET or querying system catalogues harder 
An hack to "prevent" querying system tables already existed but
was pretty weak. This commits makes that a bit stronger. The
filter for SET is new.
 2013-04-09 11:52:34 +02:00
Sandro Santilli
104ed355d0 Fix fd leak on export query cancel 2013-03-27 20:02:21 +01:00
Sandro Santilli
8f60735b5e Queue export requests for shapefile format. Closes #65 2013-03-27 16:43:12 +01:00
Sandro Santilli
b2e88145ff Comment out debugging line 2013-03-27 16:09:56 +01:00
Sandro Santilli
0261e8f687 Drop unused dependencies (csv and strftime) 2013-03-27 15:55:57 +01:00
Sandro Santilli
6d6dd16446 Draft delegating CSV output to OGR 
Adapt tests to new output
 2013-03-27 15:41:24 +01:00
Sandro Santilli
ff49346d4d Drop unused variable 2013-03-26 18:20:36 +01:00
Sandro Santilli
58b079e212 Fix reace condition in new KML output, test KML export of empty table 2013-03-26 18:18:13 +01:00
Sandro Santilli
38f99c586a Fix concurrent requests for KML export 2013-03-26 16:59:37 +01:00
Sandro Santilli
a0d9038e59 Auth / non-auth tests for SHP and KML export are in place 2013-03-26 16:14:48 +01:00
Sandro Santilli
c3ae7074cb Make temporary dir a configuration setting 2013-03-26 16:02:05 +01:00
Sandro Santilli
12553dafdd Don't forget to call .end() on the result (fixes Shapefile export) 2013-03-26 15:32:07 +01:00
Sandro Santilli
3f8a606432 Simplify steps for SHP and KML outputs 2013-03-26 12:59:46 +01:00
Sandro Santilli
41dd23b429 Do not run the query twice when exporting to KML 2013-03-26 12:47:03 +01:00
Sandro Santilli
6c0d7813f6 Upgrade generic-pool to ~2.0.2 2013-03-14 11:58:30 +01:00
Sandro Santilli
dc87209033 Do not confuse warnings with errors on shapefile output 
Closes #87
 2013-02-25 18:32:28 +01:00
Sandro Santilli
d4b1f580ff Fix X-Cache-Channel computation with paging parameters. Closes #85. 2013-02-18 16:19:12 +01:00
Sandro Santilli
10ec5b9fda Do not attempt to compute an X-CacheChannel from a broken query info 
Also enhance debugging for broken query info results
 2013-02-18 12:20:18 +01:00
Sandro Santilli
19eb51b06d Use an LRU cache for query tables, and add expiration support 
Defaults to 8192 items each valid for no more than 10 minutes
Both settings can be changed via environment config.
See #83
 2013-02-13 18:59:09 +01:00
Sandro Santilli
7d2d585c54 Check CDB_QueryTable response before saving into cache 
Also add pid to /cachestatus (#83)
 2013-02-13 16:43:04 +01:00
Sandro Santilli
5959e6465a Fix Content-Disposition for error responses. Closes #82 2013-02-13 13:32:34 +01:00
Sandro Santilli
d667d64c78 Fix skipfields use with SHP output format. Closes #81 2013-02-13 13:02:37 +01:00
Sandro Santilli
977ecbeb29 Return an error when "the_geom" is in skipfield for SVG output 
Closes #73
 2013-01-21 10:06:51 +01:00
Sandro Santilli
e897cb07d2 Add test for null geoms in geojson 
Also drop an hard-coded "the_geom" in geojson output code
 2013-01-11 18:57:45 +01:00
Sandro Santilli
9bc79bded6 Re-add null geometry support in geojson output 
You never know if anyone is relying on them, and they are supported
by the specs:
https://github.com/mbostock/topojson/issues/18#issuecomment-12154398

We'll still skip those records for topojson
 2013-01-11 18:43:05 +01:00
Sandro Santilli
ca065e89b9 Omit NULL the_geom features from from geojson and topojson outputs 
Closes #80
 2013-01-11 18:20:25 +01:00
Sandro Santilli
3e7c8f4d22 Improve mixed geometry export error message. Closes #78 2013-01-11 17:09:22 +01:00
Sandro Santilli
a81b55e421 Include properties to TopoJSON output. Closes #79 
Includes support for the skipfields option
 2013-01-11 11:10:51 +01:00
Sandro Santilli
39669578b6 Initial support for TopoJSON (#79) 
Does not include any attributes in the format
 2013-01-09 17:43:23 +01:00
Sandro Santilli
2e85588d99 Format date fields in CSV output 2012-12-07 14:27:02 +01:00
Sandro Santilli
c00cf6d390 Revert "Upgrade node-csv module to version 0.2.4" 
This reverts commit 270e6280ed.
The performance issue must be fixed before we can upgrade:
https://github.com/wdavidw/node-csv-parser/issues/64
 2012-11-19 18:44:53 +01:00
Sandro Santilli
270e6280ed Upgrade node-csv module to version 0.2.4 
Will help with compressed-csv export format
 2012-11-19 16:20:40 +01:00
Sandro Santilli
3a641beda8 Fix UTF8 in shapefile export. Closes #66. 
Testcases count reaches 100 ! Happy post-post-gis day :)
 2012-11-16 12:50:21 +01:00
Sandro Santilli
da05e6f4ca Fix KML export truncation. Closes #947 
Also adds test for SHP truncation (isn't trucated)
 2012-11-16 12:22:06 +01:00
Sandro Santilli
75fcd5ae9c Do not write back to request.query as it breaks oAuth signature 
NOTE: this breakage misses a testcase (overcomplex at the moment)
 2012-11-14 18:04:38 +01:00
Sandro Santilli
bd08eb4add Use inline attachment also with POST, if format isn't given 2012-11-14 16:30:18 +01:00
Sandro Santilli
6b7cada97d Never dispose "inline" when using POST. 2012-11-13 19:27:25 +01:00
johnhackworth
e64c3f57f7 changes in the way the parameters are collected to support both POST and GET 2012-11-13 19:27:25 +01:00
Sandro Santilli
0ef13f08c2 Use inline disposition when no format and no filename are given 
See #61
 2012-11-12 19:44:16 +01:00
Sandro Santilli
120cf3f0c5 Add 'cache_policy' parameter. Closes #62 2012-11-12 19:14:20 +01:00
Sandro Santilli
005ae48e3a Support for specifying a list of fields to skip from output. 
Closes #63
 2012-11-12 17:11:17 +01:00
Sandro Santilli
46cec7a0e5 Add support for specifying a filename for exports. Closes #64 
Sets release target to 1.3.0, due to parameter addition
 2012-11-12 12:40:53 +01:00
javi
3fd3c5fabc added the needed headers for CORS 2012-11-01 13:17:45 +01:00
Sandro Santilli
110af3e9d9 Add "sqlapi" prefix to temporary dir for shapefile output 2012-10-31 11:57:20 +01:00
Sandro Santilli
9aa28c5bea Initial support for KML output format. Closes #54. 2012-10-25 18:10:56 +02:00
Sandro Santilli
978c0b4cbe Advertise header presence in CSV Content-Type 2012-10-25 13:40:21 +02:00
Sandro Santilli
a6837573c5 Use "attachment" Content-Disposition for all output formats. 
Closes #61, includes tests

NOTE: this includes the default "json" format.
 2012-10-25 13:34:06 +02:00
Sandro Santilli
5d8eccc81e Fix CSV output with no rows. Closes #60 2012-10-25 13:14:47 +02:00
Sandro Santilli
a560a37823 Generalize OGR output function 2012-10-23 17:45:56 +02:00
Sandro Santilli
26bdccf541 Remove debugging output 2012-10-18 13:19:08 +02:00
Sandro Santilli
d0ae7e08a6 Initial support for Shapefile output 2012-10-18 12:51:12 +02:00
Sandro Santilli
b038419abd Add missing newline 2012-10-18 11:35:06 +02:00
Sandro Santilli
729462b500 Fix throw in async function (verifyRequest) 2012-10-18 11:34:39 +02:00
Sandro Santilli
8574517ab8 Add a userid_to_dbuser function 2012-10-18 11:33:35 +02:00
Sandro Santilli
d9b733e5c6 Recognize ALTER as a writing query 2012-10-15 13:40:04 +02:00
Sandro Santilli
d23416cc60 Set X-Cache-Channel to NONE when the SQL may write to the database 
Note that "may write" allows for false positive, so there could be
less cache hits than possibly allowable. If this will be a problem
for any real use case we could still improve the regular expression
used to detect "writing" queries.

Automated tests are added to check for the X-Cache-Channel header
with both writing and read-only queries performed by authenticated
requests.

Closes #27
Closes #43
 2012-10-15 13:23:17 +02:00
Sandro Santilli
553146e6dd Add consistency checking in packageResult about format value 2012-10-12 12:57:03 +02:00
Sandro Santilli
ac83700810 Send a 404 on unsupported format requested 2012-10-12 12:17:35 +02:00
Sandro Santilli
0d91ab2c6a Survive multiple "format" parameters, only using last one 2012-10-12 11:42:03 +02:00
Sandro Santilli
6a2c0e9727 Initial support for SVG output (#49) 
This version only dumps the geometries with no identifier.
Scales geometries to fit in a 1024x768 pixels area, using
a circle radius of 5 pixels, a stroke-width of 1 pixel and
no fill.  Supports trimming number of decimals.
Adds a viewBox tag to fit the drawing to the output device window.

Includes an automated testcase.
 2012-10-04 12:04:50 +02:00
Sandro Santilli
4521942820 Test "CSV" format, drop "KML" from the list of supported formats 2012-10-04 12:04:50 +02:00
Sandro Santilli
eb645d298c More embedded comments 2012-10-04 12:04:49 +02:00
Sandro Santilli
8b824801cf Fix INSERT and UPDATE with RETURNING clause. Closes #50 
Includes regression test
 2012-09-17 11:50:19 +02:00
Sandro Santilli
0a53d1cd17 Make PSQL.end a no-op, to avoid breaking pooled clients. Closes #48 2012-09-04 12:31:38 +02:00
Luis Bosque
86e1439c2c Re-enabled the psql connections pooler. It is 
essential.

Related to:
e36bfdef5b
 2012-08-29 20:52:17 +02:00
Sandro Santilli
4723c028c9 Use the correct redis variable for checking map key (closes #44) 
Includes testcase
 2012-08-02 12:18:54 +02:00
JM
c881d991a6 Support case insensitive in format parameter. Closes #30 
Also fixes https://github.com/Vizzuality/cartodb/issues/795
 2012-07-13 10:24:12 +02:00
Sandro Santilli
e36bfdef5b Block while connecting, fixing #37 
See https://github.com/brianc/node-postgres/issues/81
 2012-07-12 20:13:47 +02:00
strk
e6b4896338 Merge pull request #33 from jmnavarro/issue-14 
Allow using without redis (closes #14)

... as long as OAuth is not used and database name is given in URL
 2012-07-12 08:31:24 -07:00
JM
7e1847bca5 fixed #13 
from support: Different JSON result for non-query request
http://support.cartodb.com/discussions/suggestions/50-sql-api-different-json-object-for-non-query-request
 2012-06-30 02:34:43 +02:00
JM
b65d947b5a skip redis when database is hardcoded 2012-06-30 01:54:53 +02:00
JM
907b0b0df5 skip redis with non-oauth requests 2012-06-30 01:54:21 +02:00
Simon Tokumine
374583e655 add tests and refactor validations for input parameters. closes #29 2012-06-06 19:47:21 +01:00
Javier de la Torre
72c3911453 Fix issue #723 to return better error 
https://github.com/Vizzuality/cartodb/issues/723
 2012-05-11 16:46:45 -04:00
Javier de la Torre
bba4ba6192 Change default number of decimals on geojson to 6 2012-05-11 15:48:50 -04:00
Simon Tokumine
1834d6e245 update tests with md5 fix 2012-05-08 15:28:22 +01:00
Simon Tokumine
d7df8b1449 fix api key tests 2012-05-08 15:25:19 +01:00
Simon Tokumine
b3fac461fb allow dollar quoted sql to be passed 2012-05-01 20:55:12 +01:00
Simon Tokumine
7e074bbc98 add explain cache and cache status route 2012-05-01 16:46:30 +01:00
Simon Tokumine
977f7d0e79 dollar quote sql 2012-04-20 16:09:31 -04:00
Simon Tokumine
9bb7afb7f5 add specific cache key 2012-04-20 14:34:18 -04:00
Simon Tokumine
d704dc39d0 red herring 2012-04-19 11:38:27 -04:00
Simon Tokumine
5f27cb896e update close 2012-04-19 11:25:09 -04:00
Simon Tokumine
f359cd3ed3 close connections 2012-04-19 11:14:29 -04:00
Simon Tokumine
b399fa3460 fix oauth signatures. thanks @javisantana 2012-04-13 19:34:59 +01:00
Simon Tokumine
4cbdf62cc2 make wellformed CSV and stream back to browser. #9 2012-04-13 15:37:09 +01:00
Simon Tokumine
ffc1fa3fe1 closes #11. closes #9. Pending reorganisation. 2012-04-13 00:30:45 +01:00
javi santana
59739483e8 added auth using api token 2011-12-26 19:16:41 +01:00
javi santana
6c003626b4 fixed oauth problems with POST requests fixes #10 2011-12-26 12:51:15 +01:00
Simon Tokumine
4a9e00e476 update logging format 2011-12-14 00:50:56 +00:00
Simon Tokumine
54a7aee5e5 better sql-api logging 2011-12-13 23:59:00 +00:00
Simon Tokumine
807dbce04c better sql-api logging 2011-12-13 23:55:35 +00:00
Simon Tokumine
00bec44720 add host to log 2011-12-13 10:18:37 +00:00
Simon Tokumine
c59e7964d0 add host to log 2011-12-13 10:13:55 +00:00
Simon Tokumine
9c6619f3e0 organise headers 2011-12-08 12:34:07 +00:00
Simon Tokumine
2bd532bc20 add experimental varnish headers 2011-12-08 02:22:44 +00:00
Simon Tokumine
025f201ea8 add system table sanitizer 2011-11-22 00:06:14 +00:00
Simon Tokumine
8ebdd0474c Add test for system tables 2011-11-21 23:06:38 +00:00
Simon Tokumine
de191ed33b Add decimal place limiting to geojson call 2011-11-08 23:35:59 +00:00
Simon Tokumine
f66022a63d add cross site post 2011-11-07 14:24:08 -05:00
Simon Tokumine
555e072195 tidy geojson, add content-disposition and remove KML for now. Closes #7 2011-10-28 12:11:18 +01:00
Simon Tokumine
d4ca767753 update readme 2011-10-07 17:19:02 +01:00
Andrew W. Hill
1c3fb9b072 allows either s sql.format or format=format 2011-09-20 20:04:43 -07:00
Andrew W. Hill
19b2829d73 added simple kml support to SQL API, format=kml will output 2011-09-20 19:15:43 -07:00
Andrew W. Hill
fd4bab78e1 moved sql.fogeojson? to sql?format=geojson 2011-09-20 18:04:26 -07:00
Andrew W. Hill
bc2c828c9a removed some console.log remnants 2011-09-19 05:16:53 -07:00
vizadmin
1a0c2d3eaf working sql.ext option, sql.geojson formats the json as a geojson spec. also rewrites the query to perform a ST_AsGeoJSON query 2011-09-19 05:15:14 -07:00
javi
4874568c5c added staging enviorment 2011-09-16 13:43:10 +02:00
Simon Tokumine
5360a41ccf add instructions for a mapuser 2011-09-08 15:40:27 +01:00
Simon Tokumine
4547ae932a Reference bug 2011-09-07 16:58:53 +01:00
Simon Tokumine
47db0644c9 reverted to non-native bindings while we bug check the PQsendQuery error 2011-09-07 16:50:35 +01:00
Simon Tokumine
ec74953e78 updated logging format 2011-09-07 15:38:48 +01:00
Simon Tokumine
d9870305da updated to post 2011-09-07 12:05:10 +01:00
Simon Tokumine
082807695c updated error messages 2011-08-25 11:15:52 +01:00
Simon Tokumine
f10cd72e0b updated acceptance tests for host name based public api and made work :) 2011-08-24 20:47:10 +01:00
Simon Tokumine
e19240c2dc metadata tests and added getID 2011-08-24 13:46:36 +01:00
Simon Tokumine
fe93684b85 tests pass 2011-08-24 12:54:26 +01:00
Simon Tokumine
e189aec262 added ability to query user metadata Redis data for database name from the request host 2011-08-23 21:42:27 +01:00
Simon Tokumine
c4f2195ed0 remove debug 2011-08-22 14:18:42 +01:00
Simon Tokumine
1d5444fa20 Added logging 2011-08-22 13:52:11 +01:00
Simon Tokumine
ce4413cbda added body_hash to oauth check and stopped firing exception if incomplete oauth variables sent 2011-08-22 13:33:12 +01:00
Alvaro Bautista
164f3725a7 Check that err object is defined 2011-08-18 13:35:55 +02:00
Simon Tokumine
5771cdc8a8 update jsonp 2011-08-17 21:54:18 +01:00
Simon Tokumine
5683394417 added jsonp 2011-08-17 21:46:47 +01:00
Simon Tokumine
94478ec775 added jsonp 2011-08-17 21:44:30 +01:00
Simon Tokumine
d4d1f3b223 remove logging 2011-08-17 21:26:15 +01:00