cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,332 Commits 1 Branch 118 Tags 9.7 MiB
58b91275f0
Commit Graph

167 Commits

Author SHA1 Message Date
Raul Ochoa
3625c07250 Fallback to nextTick for node v0.8, resolves #182 2014-11-13 11:18:00 +01:00
Raul Ochoa
74429f82e1 Improve topojson output by streaming json 2014-11-12 11:36:59 +01:00
Raul Ochoa
878c62e04e Order by dimension so it's possible to stream the rows as they come 2014-10-24 16:23:08 +02:00
Raul Ochoa
2e9a16d56d Don't loop twice over svg rows 2014-10-23 16:49:04 +02:00
Raul Ochoa
2e3eeefd0f Removes tables=fake wadus param in ogr2ogr command so it can go to 
geometry_colums view to retrieve the column data type. This requires
to grant permissions on geometry_columns and geography_columns to the
public user.
 2014-09-22 13:23:30 +02:00
Raul Ochoa
0035f04edd Metrics revamp: removes and adds some metrics 2014-08-18 19:45:17 +02:00
Raul Ochoa
77cb86154c Starts using cartodb-psql node module in SQL API 2014-08-11 20:15:55 +02:00
Raul Ochoa
49406c99fa Moves auth functionality to its own directory 2014-08-05 11:57:43 +02:00
Raul Ochoa
e16f278087 CDB-3629 Uses one request to redis to retrieve all connection params 2014-08-05 02:29:07 +02:00
Raul Ochoa
bbb82c2ceb CDB-3780 Adds stream start in case there were no rows. Adds regression test. 2014-08-05 00:47:49 +02:00
Raul Ochoa
15a3f4cc88 CDB-3780 Handle error at query end. Adds regression test. 2014-08-05 00:47:14 +02:00
Raul Ochoa
919110570c Unified naming for formats 2014-08-02 20:27:05 +02:00
Raul Ochoa
a747818cfa Unifying require for formats 2014-08-02 20:26:01 +02:00
Raul Ochoa
0bc88f1687 Style changes 2014-08-02 20:23:47 +02:00
Raul Ochoa
4830b69a2f Use streaming for geojson format 2014-07-31 03:17:14 +02:00
Raul Ochoa
73a195a7fa Use streaming for json responses. Number of buffered rows can be specified by config. 2014-07-31 02:17:07 +02:00
Raul Ochoa
f5f89606af Use separate method when there are skip fields 2014-07-30 19:48:57 +02:00
Raul Ochoa
3c6b5f1376 Internal renaming 2014-07-30 19:43:43 +02:00
Raul Ochoa
c55dabf5b4 Cached skipFields, better approach to call different methods if there are skip fields 2014-07-30 19:36:25 +02:00
Raul Ochoa
7906707130 Revert "Stream JSON responses" 
This reverts commit 49ef1bc0c7.

Conflicts:
	NEWS.md
 2014-06-04 20:57:11 +02:00
Raul Ochoa
23254513e6 removes unneeded comma 2014-06-02 18:57:45 +02:00
Raul Ochoa
b3609696a3 requests associated with formats based on postgres expose a cancel method that will be called on client request abortion/cancelling so postgres can cancel ongoing queries 2014-06-02 14:48:38 +02:00
Raul Ochoa
537cb238c6 CDB-2081 Adds support for order_by through http query params. 2014-04-14 15:01:12 +02:00
Sandro Santilli
49ef1bc0c7 Stream JSON responses 
Reduces memory usage for big datasets.
JIRA CDB-2600 #resolve
 2014-04-03 15:44:53 +02:00
Sandro Santilli
8a953c116c Fix paging with queries starting with comments 
Closes #144
Includes testcase
 2014-03-27 12:48:39 +01:00
Sandro Santilli
2cbdc4c38c Add profiler support 
Closes #142
 2014-03-19 18:34:21 +01:00
Sandro Santilli
b0d0d4d073 Reduce work on aborted requests 
Closes #129
Includes testcase
 2014-03-19 13:30:29 +01:00
Sandro Santilli
65f31fd7c1 Add optional support for rollbar 
Closes #150
Logs messages of severity ERROR or higher
 2014-03-13 12:09:39 +01:00
Sandro Santilli
5c22b8ddaa Drop unused code 2014-01-31 15:47:33 +01:00
Luis Bosque
973b800fca Use db_port in ogr2ogr 2014-01-20 12:28:28 +01:00
Sandro Santilli
61cec80d4d Fix use of "SELECT .. INTO" with windowing params 
Includes testcase. Closes #127
 2014-01-09 17:56:09 +01:00
Sandro Santilli
87d35aa155 Generalize CartoDB username extraction, allowing for multiuser setups 
Closes #124
 2013-12-18 11:59:00 +01:00
Sandro Santilli
3397f2f3f9 Fix field types names lookup after PSQL model refactoring 
Fixes missing .prj in shapefile export regression
Closes #122
 2013-12-02 11:47:52 +01:00
Sandro Santilli
9389a04030 Add warnings and notices to JSON response. Closes #104. 2013-11-18 17:01:06 +01:00
Sandro Santilli
6ebeed04c2 Optionally read user-specific database_password from redis 
Follows CartoDB-2.5.0 model. Includes testcase.
Closes #120 -- Jira ref CDB-870
 2013-11-18 13:31:11 +01:00
Sandro Santilli
0d44139d8c Do not choke if ogr2ogr fails without on stderr 2013-11-18 11:54:00 +01:00
Sandro Santilli
7218c89b2d Refactor PSQL model to take full db config in the constructor 
Closes #58, stops reading configuration variables itself
 2013-11-18 11:42:43 +01:00
Sandro Santilli
03ea51a375 CartoDB redis interaction delegated to "cartodb-redis" module 2013-11-15 19:14:55 +01:00
Sandro Santilli
4a5c9b0eed Refactor OGR class interface to improve flexibility 
The toOGR and toOGR_SingleFile function now take an "options"
object rather than a long list of arguments. This allows for
easier extension toward per-user database host and password
(see tickets #120 and #121)
 2013-11-15 13:24:48 +01:00
javi
93da1c110e removed console.log refs #119 2013-11-09 16:56:58 +01:00
javi
370d45d8b6 fixed #119 2013-11-09 16:55:40 +01:00
Sandro Santilli
0139c8fb16 JSON format: correctly recognize "date" type columns 
Closes #117 -- includes testcase
 2013-11-06 11:43:56 +01:00
Sandro Santilli
45d54c47cc Add note about which GDAL version fixes the SRID detection bug 2013-11-06 08:06:29 +01:00
Sandro Santilli
8d2347af99 Make public PostgreSQL user name a configuration parameter 
Closes #56.
Updates documentation and tests and example config files
 2013-11-05 17:49:10 +01:00
Sandro Santilli
2abb136258 Set KML folder name to the requested filename. 
Closes #115. Includes testcase.
 2013-11-05 17:29:02 +01:00
Sandro Santilli
3c03099e8d Do not trust GDAL SRID detection for KML output. 
Closes #116. Includes testcase.
 2013-11-05 16:57:23 +01:00
Sandro Santilli
75bc36039d Remove done TODOs 2013-10-11 17:25:38 +02:00
Sandro Santilli
82f965fa57 Upgrade node-pg dependency to 2.6.2 
Simplifies PSQL.quoteIdentifier and exposes PSQL.quoteLiteral
 2013-10-02 12:32:24 +02:00
Sandro Santilli
e92530d6e5 Fix shapefile export for non-linestring results starting with NULL 
Includes tests. Closes #111.
 2013-10-02 12:12:09 +02:00
Sandro Santilli
0d84a704df Fix missing .prj file in shapefile format 
Finds srid, when needed, with an additional query.
Closes #110.
Includes testcases.
 2013-10-02 11:44:06 +02:00
Sandro Santilli
7f7ef682ac Improve recognition of non-standard field types names by db lookup 
Closes #112.
Only looks up "geometry" and "raster" types for now, can be improved
over time.
 2013-10-02 10:22:13 +02:00
Sandro Santilli
1ddec5fdd1 Add note about why we want to skip querying geometry_columns 2013-09-30 10:43:32 +02:00
Sandro Santilli
742936b2b8 Honour skipfields in JSON schema response. Closes #109 2013-09-26 13:26:45 +02:00
Sandro Santilli
a0ff7ee10c Remove last bit of hard-coded format names from app controller 2013-09-05 17:10:48 +02:00
Sandro Santilli
f7bee2345d Report boolean values as of boolean type, not string. Closes #106 2013-08-22 13:45:20 +02:00
Sandro Santilli
23d092f768 Use float parsing for int8 too 
This makes us ready to upgrade to node-pg 2.x
 2013-07-22 18:22:30 +02:00
Sandro Santilli
46b7f7a309 Parse all numbers as floats. Closes #100 2013-07-09 21:51:40 +02:00
Sandro Santilli
ae502335c3 Set "numeric" type to "number" 2013-07-09 21:51:08 +02:00
Sandro Santilli
ba28496127 Add support for "text" datatypes in json output format 2013-07-04 16:04:13 +02:00
javi
78a80e6c82 added string type 2013-07-04 15:47:09 +02:00
Sandro Santilli
5e379012a8 Add support for CTE in sql windowing, add unit tests 
This is still an undocumented feature, but as long as it's present
and used (by cartodb UI) better tested than broken...

NOTE: more tests are needed for CTE and RETURNING queries
 2013-06-17 16:36:36 +02:00
Sandro Santilli
88f1d33b42 Add "fields" member in JSON return. Closes #97 2013-06-14 18:36:05 +02:00
Luis Bosque
80452ef6f4 Merge branch 'release/staging' into develop 2013-05-30 11:17:10 +02:00
Luis Bosque
12afc3c88f In OAuth requests, get req.path 
Using req.route.path breaks OAuth whtn used with regex routes. Instead
of this we need to use req.path because the path is already expanded
 2013-05-29 17:36:41 +02:00
Sandro Santilli
ffafe814ef Do not hang up to queued request on ogr generation failure 2013-05-29 17:33:10 +02:00
Sandro Santilli
7f9463e7fd Fix filesystem access conflict among clustered processes 2013-05-29 16:48:20 +02:00
Sandro Santilli
510ac0533f Switch to using evented query model for postgresql 
This enables formats for processing rows as they arrive from the
database, thus possibly reducing memory use. For a start the skip
fields are immediately removed from the result, rather than only
at the end.
 2013-05-29 15:39:39 +02:00
javi
0be9e47930 Add arraybuffer format 2013-05-28 10:37:39 +02:00
Sandro Santilli
76705a3fd7 Refactor format objects to expose a common stream-able interface 
See app/models/formats/README for info
 2013-05-28 10:36:15 +02:00
Sandro Santilli
ab8494ab12 Survive postgresql connection losses. Closes #95. 2013-05-24 18:18:17 +02:00
Sandro Santilli
11a61d16c4 Drop unused code. Spring cleanups! \o/ 2013-05-24 17:13:05 +02:00
Sandro Santilli
7712d6f2a5 Expose PostgreSQL client pooling settings in environment files 
Actually also moves to a real pooling model, managed by node-pg.
Closes #47
See .example files for new configurations
 2013-05-24 12:09:06 +02:00
Sandro Santilli
64c0494105 Hide dangerous methods of the PSQL model class 
These steps are prepatorial to recactoring to event-based model
 2013-05-24 10:38:27 +02:00
Sandro Santilli
cf6bed9bdb Bubble paging UI hack up from model to controller 
... one day we'll need to completely drop this hack!
 2013-05-24 10:22:17 +02:00
javi
c46e9786b5 refactor formats to external object 2013-05-24 08:50:03 +02:00
Sandro Santilli
e5ad8f0760 Fix test broken after oAuth interface change introduced in fcf95755 
Also improve error message on miscall
 2013-05-20 09:06:08 +02:00
Luis Bosque
fcf95755b6 Fixed problem identifying OAuth request protocol 
The protocol of a OAuth request wasn't being identified correctly. It
always considered that the request protocol was https.
 2013-05-14 18:01:37 +02:00
Sandro Santilli
e7437ba7cd Rework system catalogue prevention access check to use CDB_QueryTables 
This change reduces the chances of false positive
(forbidding legit queries). Doesn't solve the problem of false
negative (allowing illegit queries).
 2013-04-09 12:36:37 +02:00
Sandro Santilli
d54d953e75 Another bit of improvement for the system-table query prevention 
This work is more important for the testcases than the actual code
 2013-04-09 12:20:27 +02:00
Sandro Santilli
1bcffbc68c Make using SET or querying system catalogues harder 
An hack to "prevent" querying system tables already existed but
was pretty weak. This commits makes that a bit stronger. The
filter for SET is new.
 2013-04-09 11:52:34 +02:00
Sandro Santilli
6c0d7813f6 Upgrade generic-pool to ~2.0.2 2013-03-14 11:58:30 +01:00
Sandro Santilli
d4b1f580ff Fix X-Cache-Channel computation with paging parameters. Closes #85. 2013-02-18 16:19:12 +01:00
Sandro Santilli
b038419abd Add missing newline 2012-10-18 11:35:06 +02:00
Sandro Santilli
729462b500 Fix throw in async function (verifyRequest) 2012-10-18 11:34:39 +02:00
Sandro Santilli
0a53d1cd17 Make PSQL.end a no-op, to avoid breaking pooled clients. Closes #48 2012-09-04 12:31:38 +02:00
Luis Bosque
86e1439c2c Re-enabled the psql connections pooler. It is 
essential.

Related to:
e36bfdef5b
 2012-08-29 20:52:17 +02:00
Sandro Santilli
4723c028c9 Use the correct redis variable for checking map key (closes #44) 
Includes testcase
 2012-08-02 12:18:54 +02:00
Sandro Santilli
e36bfdef5b Block while connecting, fixing #37 
See https://github.com/brianc/node-postgres/issues/81
 2012-07-12 20:13:47 +02:00
JM
907b0b0df5 skip redis with non-oauth requests 2012-06-30 01:54:21 +02:00
Simon Tokumine
b399fa3460 fix oauth signatures. thanks @javisantana 2012-04-13 19:34:59 +01:00
javi santana
59739483e8 added auth using api token 2011-12-26 19:16:41 +01:00
javi santana
6c003626b4 fixed oauth problems with POST requests fixes #10 2011-12-26 12:51:15 +01:00
Simon Tokumine
025f201ea8 add system table sanitizer 2011-11-22 00:06:14 +00:00
Simon Tokumine
8ebdd0474c Add test for system tables 2011-11-21 23:06:38 +00:00
javi
4874568c5c added staging enviorment 2011-09-16 13:43:10 +02:00
Simon Tokumine
4547ae932a Reference bug 2011-09-07 16:58:53 +01:00
Simon Tokumine
47db0644c9 reverted to non-native bindings while we bug check the PQsendQuery error 2011-09-07 16:50:35 +01:00
Simon Tokumine
082807695c updated error messages 2011-08-25 11:15:52 +01:00
Simon Tokumine
e19240c2dc metadata tests and added getID 2011-08-24 13:46:36 +01:00
Simon Tokumine
e189aec262 added ability to query user metadata Redis data for database name from the request host 2011-08-23 21:42:27 +01:00