cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,683 Commits 1 Branch 118 Tags 9.7 MiB
64d0dc93bf
Commit Graph

465 Commits

Author SHA1 Message Date
Daniel García Aubert
328d4c6bfe Fixed bad assertion in batch acceptance test 2015-12-22 11:02:16 +01:00
Daniel García Aubert
3762ad7a39 Now jobs are stored in redis instead of user's database 2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964 Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout 2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it. 2015-12-16 15:57:58 +01:00
Daniel García Aubert
d0787d03f7 Reused matadata backend instance for batch service 2015-12-14 10:36:16 +01:00
Daniel García Aubert
29e4c3c4af Avoid launch batch service in test enviroment 2015-12-10 18:40:44 +01:00
Daniel García Aubert
f5ca879ce3 Added entry point for main app to batch api 2015-12-10 15:08:31 +01:00
Daniel García Aubert
78ee92dbe5 Renamed exposed function 2015-12-10 15:06:25 +01:00
Daniel García Aubert
6cc48bf9dd Implemented batch service 2015-12-09 20:17:45 +01:00
Daniel García Aubert
c7680722ca Passed tests 2015-12-09 12:35:20 +01:00
Daniel García Aubert
00721bcd02 Implementing batch service 2015-12-09 00:02:08 +01:00
Daniel García Aubert
6f741827cd Fixed jshint error 2015-12-07 12:19:16 +01:00
vagrant
35d6600f36 Implemented test for job manegement 2015-12-07 10:29:55 +00:00
Daniel García Aubert
ee41db99ad Returned job_id to client/user after job enqueue 2015-12-07 09:59:05 +01:00
Daniel García Aubert
a1243ad64d Implemented job controller to enqueue jobs 2015-12-07 09:40:51 +01:00
Daniel García Aubert
a0895d4310 Fixed jshint error 2015-12-03 18:45:12 +01:00
Daniel García Aubert
f7a3c6ac4e Renamed controllers routing method 2015-12-03 18:43:13 +01:00
Daniel García Aubert
96e9fcbe33 Moved requirements to the top of module 2015-12-03 18:35:49 +01:00
Daniel García Aubert
c6bae6d951 app module moved to right place 2015-12-03 18:33:17 +01:00
Daniel García Aubert
d31d496a2e Add useful comment 2015-12-03 18:25:35 +01:00
Daniel García Aubert
a697fe6faa Making test pass 2015-12-03 18:19:39 +01:00
Daniel García Aubert
e5dedb6315 Split query controller and separated from app 2015-12-03 17:28:18 +01:00
Daniel García Aubert
df59405dca Split some functionallity from app to dedicated controllers 2015-12-03 15:00:35 +01:00
Raul Ochoa
b160f4051a Merge pull request #250 from CartoDB/issue-238 
Stop adding X-Cache-Channel header when no tables involved
 2015-09-08 12:30:26 +02:00
Raul Ochoa
5ec6d7c77b Remove no longer needed health check params 2015-09-07 18:34:50 +02:00
Raul Ochoa
875eccba62 Stop adding X-Cache-Channel header when no tables were identified in SQL query 
Fixes #238 and fixes #157
 2015-09-07 18:22:24 +02:00
Raul Ochoa
335d71bc04 Adds a custom log4js format function to log sql query from POST requests 
Tokens in log4js are not provided with request/response for context,
using a custom format function allow to access them

In the case of express logger token functions receive request/response
for context so it uses a custom token

Closes #206
 2015-09-04 12:29:20 +02:00
Raul Ochoa
f816093036 Set Last-Modified header based on affected tables 
Closes #101
 2015-09-02 16:25:56 +02:00
Raul Ochoa
c6c614b5bf Do not return results from health check 
It also removed old dependencies and takes disabled file path in ctor.
 2015-08-28 18:16:04 +02:00
Raul Ochoa
d42e5a6b75 Do not expose remove from pool after query error as a config option 2015-08-04 16:23:36 +02:00
Raul Ochoa
7c696317d0 New option to remove client from pool after error happens 
This help to avoid issues with transactions.

Closes #241
 2015-08-04 15:53:50 +02:00
Raul Ochoa
a297a14300 Revert "Uses tables ogr2ogr param to avoid heavy query on pg_class and friends" 
This reverts commit b6e53f7326.
 2015-06-16 16:58:22 +02:00
Raul Ochoa
93e965af29 Makes ogr2ogr command configurable so it's possible to change path/bin 2015-06-16 15:53:33 +02:00
Francisco Dans
d1db897699 Merge pull request #228 from CartoDB/spatialite 
Adds SpatiaLite as export format
 2015-06-03 16:02:16 +02:00
Francisco Dans
f34e010785 uses concat instead of looping 2015-06-03 15:32:20 +02:00
Francisco Dans
e57f2e7219 specifies spatialite spec in the formatter 2015-06-03 12:10:57 +02:00
Francisco Dans
dfa18f7c11 checks if there are command params provided by the specific exporter 2015-06-03 12:09:48 +02:00
Francisco Dans
5140aaa132 actually uses variable... 2015-06-01 16:48:27 +02:00
Francisco Dans
61ef29a02d puts SPATIALITE command option in SpatialiteFormat model 2015-06-01 16:33:04 +02:00
Francisco Dans
3a8f94641b Merge pull request #225 from CartoDB/log-addr 
Uses :remote-addr instead of :req[X-Real-IP]
 2015-06-01 16:17:47 +02:00
Francisco Dans
f3bdddf123 forces spatialite if sqlite is passed as format 2015-05-29 15:25:19 +02:00
Francisco Dans
44be0ea1b7 adds spatialite export model 2015-05-29 15:24:50 +02:00
Francisco Dans
996bd971cd replaces default log format string to remote-addr 2015-05-26 19:18:54 +02:00
Raul Ochoa
debeb8ca3e Update news 2015-05-25 16:38:07 +02:00
Raul Ochoa
b6e53f7326 Uses tables ogr2ogr param to avoid heavy query on pg_class and friends 
Fixes #204
Newer versions of ogr2ogr don't use that heavy query so it should work
Ref 2e3eeefd0f
 2015-05-25 16:29:18 +02:00
Raul Ochoa
b29933f96e Closes stream responses on error 
fixes #219
 2015-05-13 18:15:38 +02:00
Raul Ochoa
6bb3ccbe1e Format files split into pg and ogr directories 2015-05-13 15:15:53 +02:00
Raul Ochoa
8ae1f1c976 jshint fixes, does not deal with integers as strings 2015-05-13 13:00:38 +02:00
Raul Ochoa
07cff25d38 jshint fixes 2015-05-13 13:00:12 +02:00
Raul Ochoa
09b95c9846 jshint fixes 2015-05-13 13:00:01 +02:00
Raul Ochoa
6e0acbda50 jshint fixes 2015-05-13 12:23:27 +02:00
Raul Ochoa
9dd887dc4d jshint fixes, not addressing complexity 2015-05-13 12:22:41 +02:00
Raul Ochoa
d108225001 jshint: fix monitoring directory 2015-05-12 18:02:23 +02:00
Raul Ochoa
703479b7de jshint: fix auth directory 2015-05-12 18:00:30 +02:00
Luis Bosque
d3409bdf3c Disable per user health check 2015-04-06 16:08:29 +02:00
Luis Bosque
8b7dc6f058 Removed unnecessary variable in health check 2015-03-25 18:22:13 +01:00
Luis Bosque
897df7634b Return failed health checks with disabled file 2015-03-25 17:15:41 +01:00
Raul Ochoa
9dcb397737 Merge pull request #205 from CartoDB/url_rewrite 
added /u/:user routing
 2015-03-20 12:22:07 +01:00
Raul Ochoa
9b31df6793 Improve row size limit error message 2015-03-02 14:34:01 +01:00
Raul Ochoa
45f5b398a0 Logs with console.error too large row erros 2015-03-02 12:31:11 +01:00
Raul Ochoa
302a856d37 Move note about username extraction to cartodb_request 2015-02-02 12:15:53 +01:00
Raul Ochoa
e5ab4272eb Use a multiple params route to be able to extract the username from 
the path or default to host header.
 2015-02-02 12:09:34 +01:00
javi
824b7c084e added /u/:user routing 2015-01-30 11:37:29 +01:00
Seth Fitzsimmons
bcc1577ec4 Add 'refresh' to the list of keywords that write 2015-01-06 12:40:01 -08:00
Raul Ochoa
8477d4569a Removes buffering for every single row 2014-12-02 12:32:54 +01:00
Raul Ochoa
025a6abdaa Closes stream responses on error. Fixes #188 2014-12-01 16:55:59 +01:00
Raul Ochoa
d50ddbb10f Add more fields to error responses with hint, detail and context for SQL errors. 2014-11-21 12:59:48 +01:00
Raul Ochoa
3625c07250 Fallback to nextTick for node v0.8, resolves #182 2014-11-13 11:18:00 +01:00
Raul Ochoa
74429f82e1 Improve topojson output by streaming json 2014-11-12 11:36:59 +01:00
Raul Ochoa
3093e813fa Merge branch 'master' into formats-svg-topojson-improvements 
Conflicts:
	NEWS.md
 2014-11-11 14:32:38 +01:00
Raul Ochoa
dc601a5feb Improve statement timeout error messages 2014-11-11 13:57:15 +01:00
Raul Ochoa
878c62e04e Order by dimension so it's possible to stream the rows as they come 2014-10-24 16:23:08 +02:00
Raul Ochoa
2e9a16d56d Don't loop twice over svg rows 2014-10-23 16:49:04 +02:00
Raul Ochoa
2e3eeefd0f Removes tables=fake wadus param in ogr2ogr command so it can go to 
geometry_colums view to retrieve the column data type. This requires
to grant permissions on geometry_columns and geography_columns to the
public user.
 2014-09-22 13:23:30 +02:00
Raul Ochoa
25d2e64891 Removes query tables console.log 2014-09-17 15:54:46 +02:00
Raul Ochoa
2d232c20ef Returns 401 Unauthorized for queries without permission 2014-09-02 15:00:04 +02:00
Raul Ochoa
5f2fb65095 Proper db params object 2014-08-27 17:43:28 +02:00
Raul Ochoa
7c62632ea8 Health check endpoint 2014-08-26 18:40:58 +02:00
Raul Ochoa
3a26692448 New header for database host serving the request 2014-08-22 12:29:27 +02:00
Raul Ochoa
0035f04edd Metrics revamp: removes and adds some metrics 2014-08-18 19:45:17 +02:00
Raul Ochoa
2fc16ce2c5 Updates news 2014-08-14 14:23:41 +02:00
Raul Ochoa
77cb86154c Starts using cartodb-psql node module in SQL API 2014-08-11 20:15:55 +02:00
Raul Ochoa
951636892c Removes unused vars 2014-08-08 12:54:36 +02:00
Raul Ochoa
21b8e6947c Non authenticated request cannot use pg_ catalogs/functions 2014-08-08 12:48:29 +02:00
Raul Ochoa
f6c364b3b9 CDB-3032 Removes sql statements restriction on pg_ queries 2014-08-07 16:22:48 +02:00
Raul Ochoa
bd60f8f748 Using new method from cartodb-redis to retrieve oauth values so we 
can reuse the same redis-mpool
 2014-08-06 12:51:55 +02:00
Raul Ochoa
81e191bcf3 Fixes redis-mpool dependency 2014-08-06 11:32:12 +02:00
Raul Ochoa
a7027f4b05 Removes console.log 2014-08-05 16:48:11 +02:00
Raul Ochoa
480a9f27b4 New authentication mechanism: checks in advance if credentials are provided 
in order to do a single request to redis to retrieve the required database
connection parameters.
 2014-08-05 16:20:06 +02:00
Raul Ochoa
49406c99fa Moves auth functionality to its own directory 2014-08-05 11:57:43 +02:00
Raul Ochoa
e16f278087 CDB-3629 Uses one request to redis to retrieve all connection params 2014-08-05 02:29:07 +02:00
Raul Ochoa
41fe1f4cb5 Merge pull request #164 from CartoDB/CDB-3780 
Fixes for GeoJSON stream responses
 2014-08-05 00:57:35 +02:00
Raul Ochoa
bbb82c2ceb CDB-3780 Adds stream start in case there were no rows. Adds regression test. 2014-08-05 00:47:49 +02:00
Raul Ochoa
15a3f4cc88 CDB-3780 Handle error at query end. Adds regression test. 2014-08-05 00:47:14 +02:00
Raul Ochoa
ce70e7252b Callback requests send 200 status error even if the query failed 2014-08-04 15:56:43 +02:00
Raul Ochoa
919110570c Unified naming for formats 2014-08-02 20:27:05 +02:00
Raul Ochoa
a747818cfa Unifying require for formats 2014-08-02 20:26:01 +02:00
Raul Ochoa
0bc88f1687 Style changes 2014-08-02 20:23:47 +02:00
Raul Ochoa
4830b69a2f Use streaming for geojson format 2014-07-31 03:17:14 +02:00
Raul Ochoa
73a195a7fa Use streaming for json responses. Number of buffered rows can be specified by config. 2014-07-31 02:17:07 +02:00
Raul Ochoa
f5f89606af Use separate method when there are skip fields 2014-07-30 19:48:57 +02:00
Raul Ochoa
3c6b5f1376 Internal renaming 2014-07-30 19:43:43 +02:00
Raul Ochoa
c55dabf5b4 Cached skipFields, better approach to call different methods if there are skip fields 2014-07-30 19:36:25 +02:00
Luis Bosque
6bb015bb14 [CDB-3678] Add X-Served-By-Host header 2014-07-30 12:39:03 +02:00
Raul Ochoa
2653314818 CDB-3657 Sends profiler data as json in X-SQLAPI-Profiler header 2014-07-29 19:29:16 +02:00
Raul Ochoa
0f14118235 CDB-3629 Pickup redis pool config values 2014-07-29 13:58:16 +02:00
Raul Ochoa
a7d444ca92 Changes authentication to start using public user if it is defined in redis. 2014-07-29 13:52:04 +02:00
Raul Ochoa
1a390a3d40 Some style changes 2014-07-29 13:52:04 +02:00
Raul Ochoa
21d6924088 Removes the call to set the search path as it should get the correct public user from redis 2014-07-29 13:52:04 +02:00
javi
ca7b87d81a fixed problem when a query raised a database error 2014-07-29 13:52:04 +02:00
Raul Ochoa
9c255c504b Updates CDB_QueryTables to match the one from the extension. Fixes some tests to show the schema presence. 
Skip some tests that we need to review because with the current implementation of CDB_QueryTables dont make sense.
CDB_QueryTables should be used from the extension as a dependency.
 2014-07-29 13:52:04 +02:00
javi
755ed362d4 set scheme to host user when public user is accessing 2014-07-29 13:52:04 +02:00
Raul Ochoa
7906707130 Revert "Stream JSON responses" 
This reverts commit 49ef1bc0c7.

Conflicts:
	NEWS.md
 2014-06-04 20:57:11 +02:00
Raul Ochoa
23254513e6 removes unneeded comma 2014-06-02 18:57:45 +02:00
Raul Ochoa
b3609696a3 requests associated with formats based on postgres expose a cancel method that will be called on client request abortion/cancelling so postgres can cancel ongoing queries 2014-06-02 14:48:38 +02:00
Sandro Santilli
4b5a5921d5 Fix application_name to "cartodb_sqlapi" 2014-05-07 16:15:46 +02:00
Sandro Santilli
51b135c0ac Set default PostgreSQL application name to "cartodb_sqlapi" 2014-05-07 16:14:17 +02:00
Raul Ochoa
2d2c056d6b Removes spawn unused dependency 2014-04-22 11:18:52 +02:00
Raul Ochoa
8dbf1d9c45 Removes ending semicolons from SQL queries. Fixes #147 2014-04-15 15:29:03 +02:00
Raul Ochoa
3e6b63c9d6 CDB-2081 Returns original query in case there are no clauses to apply. 2014-04-14 15:44:43 +02:00
Raul Ochoa
537cb238c6 CDB-2081 Adds support for order_by through http query params. 2014-04-14 15:01:12 +02:00
Raul Ochoa
909530cfb3 documentation for sql parameter 2014-04-11 12:13:44 +02:00
Raul Ochoa
d10bd8e3b0 precompiled query may write regex 2014-04-11 12:03:43 +02:00
Sandro Santilli
49ef1bc0c7 Stream JSON responses 
Reduces memory usage for big datasets.
JIRA CDB-2600 #resolve
 2014-04-03 15:44:53 +02:00
Sandro Santilli
8a953c116c Fix paging with queries starting with comments 
Closes #144
Includes testcase
 2014-03-27 12:48:39 +01:00
Sandro Santilli
766345595e Add statsd support 
Closes #133
 2014-03-20 11:22:13 +01:00
Sandro Santilli
2cbdc4c38c Add profiler support 
Closes #142
 2014-03-19 18:34:21 +01:00
Sandro Santilli
b0d0d4d073 Reduce work on aborted requests 
Closes #129
Includes testcase
 2014-03-19 13:30:29 +01:00
Sandro Santilli
67cc05085f Always use INFO severity for express logging 
Avoids sending non-200 responses logs to rollbar, when active
 2014-03-13 17:13:53 +01:00
Sandro Santilli
622846b453 Add '/version' endpoint 
Closes #138
Includes testcase
 2014-03-13 13:40:56 +01:00
Sandro Santilli
ac4acf98ae Use console.error for logging errors 2014-03-13 12:14:11 +01:00
Sandro Santilli
65f31fd7c1 Add optional support for rollbar 
Closes #150
Logs messages of severity ERROR or higher
 2014-03-13 12:09:39 +01:00
Sandro Santilli
1f3aca837b Use log4js for logging 
Closes #136
 2014-03-13 11:52:40 +01:00
Sandro Santilli
b565ce2959 Allow configuring log_format 
Closes #131
 2014-02-10 17:34:27 +01:00
Sandro Santilli
ec1d002812 Stop using ANSI colors in the logs 
Closes #130
 2014-02-10 17:10:34 +01:00
Sandro Santilli
5c22b8ddaa Drop unused code 2014-01-31 15:47:33 +01:00
Sandro Santilli
6c47537f47 Honour the 'node_socket_timeout' configuration directive 
Closes #128
Includes testcase, which required enhancing assert.response to
handle errors.
Changes app controller to a function that only reads configuration
when executed (rather than when only require'd).
 2014-01-31 10:58:19 +01:00
Luis Bosque
973b800fca Use db_port in ogr2ogr 2014-01-20 12:28:28 +01:00
Sandro Santilli
161a0bf5e5 Restore compatibility with 1.6.x configuration 2014-01-17 13:14:30 +01:00
Sandro Santilli
61cec80d4d Fix use of "SELECT .. INTO" with windowing params 
Includes testcase. Closes #127
 2014-01-09 17:56:09 +01:00
Sandro Santilli
80c30858bd Enhance error message on unknown cartodb username 
Closes #126
 2013-12-18 12:02:51 +01:00
Sandro Santilli
87d35aa155 Generalize CartoDB username extraction, allowing for multiuser setups 
Closes #124
 2013-12-18 11:59:00 +01:00
Sandro Santilli
3397f2f3f9 Fix field types names lookup after PSQL model refactoring 
Fixes missing .prj in shapefile export regression
Closes #122
 2013-12-02 11:47:52 +01:00
Sandro Santilli
9389a04030 Add warnings and notices to JSON response. Closes #104. 2013-11-18 17:01:06 +01:00
Sandro Santilli
6ebeed04c2 Optionally read user-specific database_password from redis 
Follows CartoDB-2.5.0 model. Includes testcase.
Closes #120 -- Jira ref CDB-870
 2013-11-18 13:31:11 +01:00
Sandro Santilli
af1f3daa69 Optionally read user-specific database_host from redis 
Follows CartoDB-2.5.0 model. Includes testcase.
Closes #121 -- Jire ref CDB-870
 2013-11-18 12:21:30 +01:00
Sandro Santilli
0d44139d8c Do not choke if ogr2ogr fails without on stderr 2013-11-18 11:54:00 +01:00
Sandro Santilli
7218c89b2d Refactor PSQL model to take full db config in the constructor 
Closes #58, stops reading configuration variables itself
 2013-11-18 11:42:43 +01:00
Sandro Santilli
03ea51a375 CartoDB redis interaction delegated to "cartodb-redis" module 2013-11-15 19:14:55 +01:00
Sandro Santilli
4a5c9b0eed Refactor OGR class interface to improve flexibility 
The toOGR and toOGR_SingleFile function now take an "options"
object rather than a long list of arguments. This allows for
easier extension toward per-user database host and password
(see tickets #120 and #121)
 2013-11-15 13:24:48 +01:00