cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,998 Commits 1 Branch 118 Tags 9.7 MiB
298297f2c5
Commit Graph

934 Commits

Author SHA1 Message Date
Daniel García Aubert
e04529c501 Rename middleware 2018-02-19 14:20:09 +01:00
Daniel García Aubert
75c2d85dbb Use authenticated middleware in query controller 2018-02-19 13:24:44 +01:00
Daniel García Aubert
114070ef96 Rename middleware 2018-02-19 12:37:19 +01:00
Daniel García Aubert
adfe3dd52b Remove comment 2018-02-19 11:10:39 +01:00
Daniel García Aubert
ea310db62c Cosmetic chenges 2018-02-19 11:04:28 +01:00
Daniel García Aubert
edd382fb8f Check user is the same user that sends the request when basic-auth is provided 2018-02-16 18:21:06 +01:00
Daniel García Aubert
bcf88093e9 Support basic-auth for batch api 2018-02-16 11:52:57 +01:00
Daniel García Aubert
2cadfe6f18 Add support for Basic Auth 2018-02-16 10:46:58 +01:00
Daniel García Aubert
756fbe42c9 Use apikey midlleware 2018-02-15 17:24:04 +01:00
Daniel García Aubert
ad772246d0 Get api_key from specific middleware and save it into res.locals 2018-02-15 17:23:35 +01:00
Daniel García Aubert
67f2a85abd Add database credentials to jobs 2018-02-15 12:30:04 +01:00
Daniel García Aubert
ea6e8b5315 Implement fallback mechanism to be able to authenticate as usual in case of apikey is not found 2018-02-14 16:22:36 +01:00
Daniel García Aubert
7c859f0d7b Use actual apikey properties 2018-02-12 17:49:50 +01:00
Daniel García Aubert
c003a491c0 Use current implementation of cartodb-redis 2018-02-12 16:41:35 +01:00
Daniel García Aubert
5c7f5fbaa5 Merge branch 'master' into auth-api 2018-02-12 16:30:09 +01:00
Simon Martín
bd410fb5f6 using same way to search a string 2018-01-09 17:16:38 +01:00
Simon Martín
df86cc16e4 method to detect timeout error 2018-01-09 17:14:11 +01:00
Simon Martín
6e45c39b1f Merge branch 'master' into pythonTimeout 2018-01-08 11:29:17 +01:00
Simon Martín
ee511d3762 stringifyForLogs more usual case first 2017-12-18 14:54:23 +01:00
Simon Martín
b531922459 add line at EOF 2017-12-18 12:59:20 +01:00
Simon Martín
485d9adddd escape chars function 2017-12-18 12:45:44 +01:00
Simon Martín
3fa09a007e line at EOF and better comment 2017-12-12 17:16:40 +01:00
Simon Martín
6481d14192 escape quotes for logs 2017-12-12 12:58:31 +01:00
Simon Martín
c478e8e62e removing default error log value 2017-12-12 10:55:28 +01:00
Simon Martín
61b351535c change funcion name 2017-11-30 15:06:25 +01:00
Simon Martín
0ab87df644 extract error message from array 2017-11-28 17:19:10 +01:00
Simon Martín
4d598eacd7 dont modify err, works with errorsLog 2017-11-28 16:11:04 +01:00
Simon Martín
a17e1fc5ec logErrors function 2017-11-28 15:58:14 +01:00
Simon Martín
3e23be2087 add errors header with default value 2017-11-28 15:57:58 +01:00
Daniel García Aubert
88015aaf0f Implement new auth system based on api-key tokens with scoped permissions 2017-11-24 17:57:54 +01:00
Simon Martín
670cb6eb8b Merge branch 'master' into pythonTimeout 2017-11-24 16:02:13 +01:00
Daniel García Aubert
15949d2fc2 Fix wrong function name 2017-11-24 15:55:16 +01:00
Daniel García Aubert
b10937a8fb Use res.locals instead of req.context to share object among middlewares 2017-11-24 15:49:25 +01:00
Daniel García Aubert
c2d1a4653e Improve naming for connection database params holder 2017-11-24 15:45:49 +01:00
Simon Martín
11e0ac89d2 undo test of postgres9.5 error 2017-11-24 12:22:30 +01:00
Rafa de la Torre
296d1878fd Trying regexp instead of exact match 2017-11-24 10:55:02 +01:00
Daniel García Aubert
833b7b962c Order geo-columns to export 'the_geom' by default while exporting SHP format 2017-11-17 14:45:58 +01:00
Simon Martín
27e43c8d91 handling python timeout error 2017-11-17 13:30:58 +01:00
Daniel García Aubert
b1112e2bb1 Remove unneeded header 2017-09-20 16:48:22 +02:00
Daniel García Aubert
980e762ab7 Skip user auth middlewares 2017-09-20 16:35:33 +02:00
Daniel García Aubert
0597a3b8a6 Change job work in progress endpoint 2017-09-20 15:46:29 +02:00
Raul Ochoa
32154b67c6 Merge branch 'master' into upgrade-cartodb-psql 2017-08-11 11:57:16 +02:00
Daniel García Aubert
f13c16e523 Typo 2017-08-10 11:22:08 +02:00
Daniel García Aubert
01a71ee60e Apply user timeout to ogr2ogr command 2017-08-09 12:50:16 +02:00
Raul Ochoa
3e536b822e Remove unnecessary destroyOnError option 2017-08-08 18:21:10 +02:00
Daniel García Aubert
fb477260ec Respond with 429 and proper message when database fires a timeout error 2017-08-03 17:19:08 +02:00
Rafa de la Torre
07890ec608 Change tableCache.get for tableCache.peek #244 
This will make it consider age of the key when it was initially set,
instead of renewing it with the time of the last access.
 2017-07-03 11:13:21 +02:00
Rafa de la Torre
1db935a24b Merge pull request #423 from CartoDB/422-disable-query-lru-cache 
422 disable query lru cache
 2017-06-27 17:40:33 +02:00
Rafa de la Torre
2d9a5e9655 Move cyclomatic complexity to the factory #422 2017-06-27 16:05:05 +02:00
Rafa de la Torre
c8e42f561d Reduce cyclomatic complexity (jshint 2/2) #422 2017-06-27 15:35:51 +02:00
Rafa de la Torre
34ccfc10df Please jshint (1/2) #422 2017-06-27 13:26:14 +02:00
Rafa de la Torre
bf478cabb2 Fix initialization of NoCache #422 2017-06-27 12:44:11 +02:00
Rafa de la Torre
bc71889f93 First attempt at removing tableCache #422 2017-06-27 12:10:24 +02:00
Daniel García Aubert
aa66000848 Going green: use replacer parameter in JSON.stringify method to not cast special number values to null 2017-06-08 12:42:37 +02:00
Mario de Frutos
3a99020632 Added regexp for the test to matches both 0.10 and 6.X node version 2017-04-18 12:15:47 +02:00
Mario de Frutos
da42daa536 Improved error message for zip spawn command 2017-04-18 12:15:47 +02:00
Daniel García Aubert
e4eeb9e1f6 Clean commented code 2017-04-18 12:15:47 +02:00
Daniel García Aubert
e0f9209017 Attach error handler while spawning zip command 2017-04-18 12:15:47 +02:00
Daniel García Aubert
884d3aea0a Dont call 'next' callback twice, include error message if zip don't exit normally 2017-04-18 12:15:47 +02:00
Daniel García Aubert
4b98b7056a Make zip command path configurable 2017-04-18 12:15:47 +02:00
Daniel García Aubert
6dba06130b Retur a proper error message when ogr2ogr command fails 2017-04-18 11:40:37 +02:00
Daniel García Aubert
d60e2107af Add error callback to ogr command while spawning 2017-04-11 09:43:25 +02:00
Raul Ochoa
db9bfacf07 Extract stats client creation 2017-03-30 16:13:17 +02:00
Raul Ochoa
d48a901533 Merge branch 'master' into node-v6 2017-01-12 19:49:56 +01:00
Javier Torres
62a67f148c Correctly accept custom options for formats in ogr.js 2017-01-11 19:07:50 +01:00
Javier Torres
c67a2e7d09 Use cartodb_id as FID column in gpkg 2017-01-11 19:07:50 +01:00
Daniel García Aubert
ef124d689e Handle query end in error listener since pg no longer emits end if and error occurs 2016-12-07 16:22:21 +01:00
Daniel García Aubert
99c7a6e4f9 Raise job query size to 16kb 2016-11-07 16:50:44 +01:00
Daniel García Aubert
4a64d37c6c Add work in progrees list endpoint 2016-10-28 15:08:42 +02:00
Raul Ochoa
cdde1be29e Re-use redis pool as much as possible 2016-10-17 15:02:34 +02:00
Raul Ochoa
e401c01d78 Only log on non-test environments 2016-10-12 01:40:14 +02:00
Raul Ochoa
e4b1711e8e pub/sub package 2016-10-11 18:28:46 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Raul Ochoa
42a1f3ad4c Accept multipart requests 2016-10-05 17:29:16 +02:00
Raul Ochoa
6309318534 Use body-parser from old connect module 2016-10-05 14:22:44 +02:00
Raul Ochoa
025b3f3cc7 Increase body limit to 20mb 2016-10-05 11:14:06 +02:00
Raul Ochoa
2a2a54a073 DRY in job response handler 2016-10-04 16:07:13 +02:00
Raul Ochoa
19c9bec633 Callback with job 2016-10-04 15:57:13 +02:00
Raul Ochoa
05cbd55b95 Use db host from request's context 2016-10-04 15:50:39 +02:00
Raul Ochoa
c32a2199fa Use request bootstrapper to add host header 2016-10-04 15:43:19 +02:00
Raul Ochoa
7b7d651d8f DRY while authenticating requests 2016-10-04 15:40:56 +02:00
Raul Ochoa
20f50d988e Use user middleware in job controller 2016-10-04 15:19:31 +02:00
Raul Ochoa
b139b9ab21 Add context object in all requests 2016-10-04 15:12:46 +02:00
Raul Ochoa
2edc7505e7 Do not condition req.profiler 2016-10-04 15:08:31 +02:00
Raul Ochoa
cb9db0c4cb Make request.profiler always available 2016-10-04 14:40:04 +02:00
Raul Ochoa
3b6bc14d17 Increment errors on err 2016-10-04 13:19:29 +02:00
Raul Ochoa
bf1a67780d Merge branch 'master' into express-4.x 2016-09-30 17:37:30 +02:00
Daniel García Aubert
59e5c10f85 Set batch queries log path 2016-09-29 15:37:14 +02:00
Raul Ochoa
2aa1f045ff Use .status() API 2016-09-26 18:10:20 +02:00
Raul Ochoa
abc2f130c9 Migrate to express 4.x series 
- Remove express logger
 - Error handler responds with application/[json|javascript]
 - Fix all tests relying on res.headers
 - assert.response based on request module
 2016-09-26 18:09:27 +02:00
Raul Ochoa
f92d50cccf Rename from app to server 
Removes app_root dependency in requires
 2016-09-14 20:54:53 +02:00
Raul Ochoa
0626d80e24 Removes support for optional rollbar logging 2016-09-14 19:22:31 +02:00
Raul Ochoa
6f4fb931f7 Remove user indexer from app 2016-08-30 19:04:36 +02:00
Raul Ochoa
ba0f2f1066 Remove endpoint to retrieve jobs list 2016-08-30 18:49:01 +02:00
Raul Ochoa
704ba110ba Log job creation 2016-08-30 18:43:09 +02:00
Raul Ochoa
ad0d101bfd Remove patch/put endpoints for jobs 2016-08-30 17:43:34 +02:00
Daniel García Aubert
395635d246 Merge branch 'master' into fix-publisher-connection 2016-07-11 17:49:11 +02:00
Raul Ochoa
3ebbd9f7c4 Skip tables with no updated_at registered in cdb_tablemetadata 2016-07-11 16:39:12 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Raul Ochoa
368fe2403e Allow to setup more than one domain to validate oauth against 2016-07-07 14:20:36 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Daniel García Aubert
282da58ffe Set default value for statsd-client in job-controller to avoid check it every time it's going to be used 2016-05-30 12:27:19 +02:00
Daniel García Aubert
976bf5b039 Implemented profiling for job-runner and job-controller 2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577 Merge branch 'master' into batch-add-profile 2016-05-26 17:46:52 +02:00
Raul Ochoa
a91c7a6a88 Increase job payload size to 8kb 2016-05-24 15:41:18 +02:00
Raul Ochoa
23228b2d73 Payload size validates multiple queries and fallback queries 
It uses a middleware to check the body size
 2016-05-24 14:28:00 +02:00
Daniel García Aubert
64ad284c9c WIP: adding metrics to Batch API 2016-05-24 11:19:00 +02:00
Daniel
8c8da8e39c Merge pull request #301 from CartoDB/job-model-refactor 
Job model refactor
 2016-05-18 11:38:41 +02:00
Daniel García Aubert
d2d3ba8159 Passed tests 2016-05-16 01:22:47 +02:00
Daniel García Aubert
cc7dd7a0d2 Job model refactor 2016-05-13 18:50:55 +02:00
Raul Ochoa
4e3238c2e8 Fix control flow when using OGR with _needSRS=true and empty queries 
Continue on empty results as srid and geom type are not critical
when there are no results

Fixes #299
 2016-05-12 17:21:07 +02:00
Juan Ignacio Sánchez Lara
e979b79f49 gpkg format #291 2016-05-10 18:42:23 +02:00
Juan Ignacio Sánchez Lara
c82cafb1ed Geopackage format support #291 2016-04-27 16:56:06 +02:00
Daniel García Aubert
5de931daa5 Improver error message in job max payload limit. 2016-04-25 11:18:30 +02:00
Daniel García Aubert
1dcbb1afdf Improved error message 2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124 Fixed typos 2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944 Implemented limit to query size in Batch API. 2016-04-18 15:30:16 +02:00
Daniel García Aubert
056f22b156 Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series 2016-03-18 14:57:18 +01:00
Raul Ochoa
66cd1f400f Skip query tables cache for authenticated requests 2016-03-10 19:20:56 +01:00
Raul Ochoa
5ebc8e43d5 Start hits=0 as first time is not a hit 2016-03-08 14:50:08 +01:00
Raul Ochoa
b8460d033e Add a cache decorator over QueryTables 2016-03-08 14:48:56 +01:00
Raul Ochoa
dbe033112d Rethrow error but do not fail when getting affected tables 2016-03-08 14:00:31 +01:00
Alejandro Martínez
cd593e12ce Join all Surrogate-Keys in one line 2016-02-24 11:43:43 +01:00
Alejandro Martínez
7d024dbf3f Fix include in query_controller 2016-02-22 19:10:59 +01:00
Alejandro Martínez
5411aa3a01 Use Date.now() fallback value for getLastUpdatedTime 2016-02-22 19:08:04 +01:00
Alejandro Martínez
2c4b659100 Properly discard errors fetching updated tables 2016-02-22 19:07:41 +01:00
Alejandro Martínez
5e06711b4b Use node-cartodb-querytables library 2016-02-22 12:24:05 +01:00
Raul Ochoa
15f90c1a78 QueryTablesApi only caches affected tables and always retrieve last modification 2016-02-02 01:16:24 +01:00
Raul Ochoa
9deab814c9 Skip query-tables-api for authenticated requests 2016-02-01 16:53:28 +01:00
Raul Ochoa
24ecf130a1 Namespace queries so two users don't share results 
For instance 'select * from table' should result in different affected
tables (schema+table) depending on the database.
 2016-01-28 15:38:08 +01:00
Raul Ochoa
65bcdad657 Fix cyclomatic complexity 2016-01-28 14:48:12 +01:00
Raul Ochoa
86ecc3ad0a Implements authDbParams interface 
Uses user and pass from redis and relies on rest of params prepared
for normal dbParams
 2016-01-28 14:38:02 +01:00
Raul Ochoa
e70a21a2b4 Adds documentation about values passed over callback in getConnectionParams 2016-01-28 14:25:03 +01:00
Raul Ochoa
a654c90b6d X-Cache-Channel generation only requires affected tables 
mayWrite is redundant, and it's not its responsibility
 2016-01-28 14:18:38 +01:00
Raul Ochoa
131d798653 Change interface to return also authenticated connection params 2016-01-28 14:18:10 +01:00
Raul Ochoa
235e7f9bc1 Rename method to match returning connection params behaviour 2016-01-28 14:14:31 +01:00
Raul Ochoa
023d06fd19 Affected tables and last modified behind QueryTablesApi 
It handles the internal cache and exposes an unified result
 2016-01-28 14:08:18 +01:00
Raul Ochoa
7dad90a108 Remove unused dependencies from job controller 2016-01-28 13:10:21 +01:00
Raul Ochoa
3be769a67e Fix style 2016-01-28 12:33:32 +01:00
Raul Ochoa
9881a6f7d4 Use readonly flag to run affected tables query 2016-01-28 12:33:00 +01:00
Daniel García Aubert
e9bd93cf3f Now batch service is linked to express app in order to make it accesible from main app module to expose drain mechanism 2016-01-25 16:28:19 +01:00
Daniel García Aubert
f2ea01f627 Removed unnecessary check in job controller 2016-01-25 14:50:43 +01:00
Daniel García Aubert
e6f714a51b Implemented some new unit test 2016-01-22 19:31:25 +01:00
Daniel García Aubert
968caae9b7 Removed void comment 2016-01-21 18:26:48 +01:00
Daniel García Aubert
79e626a71e Added cors middleware, decoupled query and job controllers from its dependencies 2016-01-21 17:24:48 +01:00
Daniel García Aubert
048d2cd0ea Refactored user database service ans updated controllers to use it. 2016-01-21 16:17:17 +01:00
Daniel García Aubert
6024791e18 Merge branch 'batch-api' of github.com:CartoDB/CartoDB-SQL-API into batch-api 2016-01-21 15:35:23 +01:00
Daniel García Aubert
ad8b610321 Removed unused code 2016-01-21 15:14:55 +01:00
Daniel García Aubert
f42727212b Removed unnecessary checkAborted function in job controller and changed signature for getUserDatabase. 2016-01-21 15:05:46 +01:00
Raul Ochoa
2923c2e083 Update max cyclomatic complexity as it has decreased 2016-01-21 11:18:11 +01:00
Daniel García Aubert
2913aed697 Placed job controller endpoit under /api/vX/sql 2016-01-18 20:12:44 +01:00
Daniel García Aubert
2d13bde301 Implemented uses cases test for Batch API 2016-01-11 20:14:15 +01:00
Daniel García Aubert
078eb7706b Now SQL API is able to disable Batch Service using argv param --no-batch 2016-01-08 18:29:36 +01:00
Daniel García Aubert
20f00d58d9 Refactored batch service to avoid event noise, doing in callback way 2016-01-08 15:47:59 +01:00
Daniel García Aubert
f9f52d2bd1 Improved object creation in batch service 2016-01-08 11:32:01 +01:00
Daniel García Aubert
e6a4e0f7eb Returned 201 HTTP code in job creation 
Fixed bug listing jobs, now user job index is cleaned before respond to client
 2016-01-07 12:07:30 +01:00
Daniel García Aubert
40d0161218 Fixed bug in authentication for PUT methods in Job Controller 2016-01-05 13:54:18 +01:00
Daniel García Aubert
246655de94 Changed redis data structure for users jobs 2016-01-04 19:08:13 +01:00
Daniel García Aubert
35650985db Implemented job modification whether job is pending 2015-12-31 15:42:31 +01:00
Daniel García Aubert
ada39d84b8 Implemented job cancelation in Batch API 2015-12-30 20:16:18 +01:00
Daniel García Aubert
851c393f75 Refactored batch manager avoiding concurrence issues 2015-12-29 10:19:10 +01:00
Daniel García Aubert
4dcd7c0a50 Fixed typo and prepared for listing user's job 2015-12-24 17:47:41 +01:00
Daniel García Aubert
c4f96b4a59 Implemented support for listing jobs for one user 2015-12-24 17:42:49 +01:00
Daniel García Aubert
ecb76ca1de Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs. 2015-12-23 17:29:11 +01:00
Daniel García Aubert
62db809476 Return 401 when no API KEY is provided 2015-12-23 14:55:49 +01:00
Daniel García Aubert
3c23bf12e7 Fixed bug whether a job consumer is empty. And refactored batch module, now is a common object intead of module function 2015-12-22 20:12:10 +01:00
Daniel García Aubert
30d0b2906b Implemented GET for jobs API 2015-12-22 11:45:25 +01:00
Daniel García Aubert
ef59a1d942 Fixed typo in batch test 2015-12-22 11:06:46 +01:00
Daniel García Aubert
328d4c6bfe Fixed bad assertion in batch acceptance test 2015-12-22 11:02:16 +01:00
Daniel García Aubert
3762ad7a39 Now jobs are stored in redis instead of user's database 2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964 Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout 2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it. 2015-12-16 15:57:58 +01:00
Daniel García Aubert
d0787d03f7 Reused matadata backend instance for batch service 2015-12-14 10:36:16 +01:00
Daniel García Aubert
29e4c3c4af Avoid launch batch service in test enviroment 2015-12-10 18:40:44 +01:00
Daniel García Aubert
f5ca879ce3 Added entry point for main app to batch api 2015-12-10 15:08:31 +01:00
Daniel García Aubert
78ee92dbe5 Renamed exposed function 2015-12-10 15:06:25 +01:00
Daniel García Aubert
6cc48bf9dd Implemented batch service 2015-12-09 20:17:45 +01:00
Daniel García Aubert
c7680722ca Passed tests 2015-12-09 12:35:20 +01:00
Daniel García Aubert
00721bcd02 Implementing batch service 2015-12-09 00:02:08 +01:00
Daniel García Aubert
6f741827cd Fixed jshint error 2015-12-07 12:19:16 +01:00
vagrant
35d6600f36 Implemented test for job manegement 2015-12-07 10:29:55 +00:00
Daniel García Aubert
ee41db99ad Returned job_id to client/user after job enqueue 2015-12-07 09:59:05 +01:00
Daniel García Aubert
a1243ad64d Implemented job controller to enqueue jobs 2015-12-07 09:40:51 +01:00
Daniel García Aubert
a0895d4310 Fixed jshint error 2015-12-03 18:45:12 +01:00
Daniel García Aubert
f7a3c6ac4e Renamed controllers routing method 2015-12-03 18:43:13 +01:00
Daniel García Aubert
96e9fcbe33 Moved requirements to the top of module 2015-12-03 18:35:49 +01:00
Daniel García Aubert
c6bae6d951 app module moved to right place 2015-12-03 18:33:17 +01:00
Daniel García Aubert
d31d496a2e Add useful comment 2015-12-03 18:25:35 +01:00
Daniel García Aubert
a697fe6faa Making test pass 2015-12-03 18:19:39 +01:00
Daniel García Aubert
e5dedb6315 Split query controller and separated from app 2015-12-03 17:28:18 +01:00
Daniel García Aubert
df59405dca Split some functionallity from app to dedicated controllers 2015-12-03 15:00:35 +01:00
Raul Ochoa
b160f4051a Merge pull request #250 from CartoDB/issue-238 
Stop adding X-Cache-Channel header when no tables involved
 2015-09-08 12:30:26 +02:00
Raul Ochoa
5ec6d7c77b Remove no longer needed health check params 2015-09-07 18:34:50 +02:00
Raul Ochoa
875eccba62 Stop adding X-Cache-Channel header when no tables were identified in SQL query 
Fixes #238 and fixes #157
 2015-09-07 18:22:24 +02:00
Raul Ochoa
335d71bc04 Adds a custom log4js format function to log sql query from POST requests 
Tokens in log4js are not provided with request/response for context,
using a custom format function allow to access them

In the case of express logger token functions receive request/response
for context so it uses a custom token

Closes #206
 2015-09-04 12:29:20 +02:00
Raul Ochoa
f816093036 Set Last-Modified header based on affected tables 
Closes #101
 2015-09-02 16:25:56 +02:00
Raul Ochoa
c6c614b5bf Do not return results from health check 
It also removed old dependencies and takes disabled file path in ctor.
 2015-08-28 18:16:04 +02:00
Raul Ochoa
d42e5a6b75 Do not expose remove from pool after query error as a config option 2015-08-04 16:23:36 +02:00
Raul Ochoa
7c696317d0 New option to remove client from pool after error happens 
This help to avoid issues with transactions.

Closes #241
 2015-08-04 15:53:50 +02:00
Raul Ochoa
a297a14300 Revert "Uses tables ogr2ogr param to avoid heavy query on pg_class and friends" 
This reverts commit b6e53f7326.
 2015-06-16 16:58:22 +02:00
Raul Ochoa
93e965af29 Makes ogr2ogr command configurable so it's possible to change path/bin 2015-06-16 15:53:33 +02:00
Francisco Dans
d1db897699 Merge pull request #228 from CartoDB/spatialite 
Adds SpatiaLite as export format
 2015-06-03 16:02:16 +02:00
Francisco Dans
f34e010785 uses concat instead of looping 2015-06-03 15:32:20 +02:00
Francisco Dans
e57f2e7219 specifies spatialite spec in the formatter 2015-06-03 12:10:57 +02:00
Francisco Dans
dfa18f7c11 checks if there are command params provided by the specific exporter 2015-06-03 12:09:48 +02:00
Francisco Dans
5140aaa132 actually uses variable... 2015-06-01 16:48:27 +02:00
Francisco Dans
61ef29a02d puts SPATIALITE command option in SpatialiteFormat model 2015-06-01 16:33:04 +02:00
Francisco Dans
3a8f94641b Merge pull request #225 from CartoDB/log-addr 
Uses :remote-addr instead of :req[X-Real-IP]
 2015-06-01 16:17:47 +02:00
Francisco Dans
f3bdddf123 forces spatialite if sqlite is passed as format 2015-05-29 15:25:19 +02:00
Francisco Dans
44be0ea1b7 adds spatialite export model 2015-05-29 15:24:50 +02:00
Francisco Dans
996bd971cd replaces default log format string to remote-addr 2015-05-26 19:18:54 +02:00
Raul Ochoa
debeb8ca3e Update news 2015-05-25 16:38:07 +02:00
Raul Ochoa
b6e53f7326 Uses tables ogr2ogr param to avoid heavy query on pg_class and friends 
Fixes #204
Newer versions of ogr2ogr don't use that heavy query so it should work
Ref 2e3eeefd0f
 2015-05-25 16:29:18 +02:00
Raul Ochoa
b29933f96e Closes stream responses on error 
fixes #219
 2015-05-13 18:15:38 +02:00
Raul Ochoa
6bb3ccbe1e Format files split into pg and ogr directories 2015-05-13 15:15:53 +02:00
Raul Ochoa
8ae1f1c976 jshint fixes, does not deal with integers as strings 2015-05-13 13:00:38 +02:00
Raul Ochoa
07cff25d38 jshint fixes 2015-05-13 13:00:12 +02:00
Raul Ochoa
09b95c9846 jshint fixes 2015-05-13 13:00:01 +02:00
Raul Ochoa
6e0acbda50 jshint fixes 2015-05-13 12:23:27 +02:00
Raul Ochoa
9dd887dc4d jshint fixes, not addressing complexity 2015-05-13 12:22:41 +02:00
Raul Ochoa
d108225001 jshint: fix monitoring directory 2015-05-12 18:02:23 +02:00
Raul Ochoa
703479b7de jshint: fix auth directory 2015-05-12 18:00:30 +02:00
Luis Bosque
d3409bdf3c Disable per user health check 2015-04-06 16:08:29 +02:00
Luis Bosque
8b7dc6f058 Removed unnecessary variable in health check 2015-03-25 18:22:13 +01:00
Luis Bosque
897df7634b Return failed health checks with disabled file 2015-03-25 17:15:41 +01:00
Raul Ochoa
9dcb397737 Merge pull request #205 from CartoDB/url_rewrite 
added /u/:user routing
 2015-03-20 12:22:07 +01:00
Raul Ochoa
9b31df6793 Improve row size limit error message 2015-03-02 14:34:01 +01:00
Raul Ochoa
45f5b398a0 Logs with console.error too large row erros 2015-03-02 12:31:11 +01:00
Raul Ochoa
302a856d37 Move note about username extraction to cartodb_request 2015-02-02 12:15:53 +01:00
Raul Ochoa
e5ab4272eb Use a multiple params route to be able to extract the username from 
the path or default to host header.
 2015-02-02 12:09:34 +01:00
javi
824b7c084e added /u/:user routing 2015-01-30 11:37:29 +01:00
Seth Fitzsimmons
bcc1577ec4 Add 'refresh' to the list of keywords that write 2015-01-06 12:40:01 -08:00
Raul Ochoa
8477d4569a Removes buffering for every single row 2014-12-02 12:32:54 +01:00
Raul Ochoa
025a6abdaa Closes stream responses on error. Fixes #188 2014-12-01 16:55:59 +01:00
Raul Ochoa
d50ddbb10f Add more fields to error responses with hint, detail and context for SQL errors. 2014-11-21 12:59:48 +01:00
Raul Ochoa
3625c07250 Fallback to nextTick for node v0.8, resolves #182 2014-11-13 11:18:00 +01:00
Raul Ochoa
74429f82e1 Improve topojson output by streaming json 2014-11-12 11:36:59 +01:00
Raul Ochoa
3093e813fa Merge branch 'master' into formats-svg-topojson-improvements 
Conflicts:
	NEWS.md
 2014-11-11 14:32:38 +01:00
Raul Ochoa
dc601a5feb Improve statement timeout error messages 2014-11-11 13:57:15 +01:00
Raul Ochoa
878c62e04e Order by dimension so it's possible to stream the rows as they come 2014-10-24 16:23:08 +02:00
Raul Ochoa
2e9a16d56d Don't loop twice over svg rows 2014-10-23 16:49:04 +02:00
Raul Ochoa
2e3eeefd0f Removes tables=fake wadus param in ogr2ogr command so it can go to 
geometry_colums view to retrieve the column data type. This requires
to grant permissions on geometry_columns and geography_columns to the
public user.
 2014-09-22 13:23:30 +02:00
Raul Ochoa
25d2e64891 Removes query tables console.log 2014-09-17 15:54:46 +02:00
Raul Ochoa
2d232c20ef Returns 401 Unauthorized for queries without permission 2014-09-02 15:00:04 +02:00
Raul Ochoa
5f2fb65095 Proper db params object 2014-08-27 17:43:28 +02:00
Raul Ochoa
7c62632ea8 Health check endpoint 2014-08-26 18:40:58 +02:00
Raul Ochoa
3a26692448 New header for database host serving the request 2014-08-22 12:29:27 +02:00
Raul Ochoa
0035f04edd Metrics revamp: removes and adds some metrics 2014-08-18 19:45:17 +02:00
Raul Ochoa
2fc16ce2c5 Updates news 2014-08-14 14:23:41 +02:00
Raul Ochoa
77cb86154c Starts using cartodb-psql node module in SQL API 2014-08-11 20:15:55 +02:00
Raul Ochoa
951636892c Removes unused vars 2014-08-08 12:54:36 +02:00
Raul Ochoa
21b8e6947c Non authenticated request cannot use pg_ catalogs/functions 2014-08-08 12:48:29 +02:00
Raul Ochoa
f6c364b3b9 CDB-3032 Removes sql statements restriction on pg_ queries 2014-08-07 16:22:48 +02:00
Raul Ochoa
bd60f8f748 Using new method from cartodb-redis to retrieve oauth values so we 
can reuse the same redis-mpool
 2014-08-06 12:51:55 +02:00
Raul Ochoa
81e191bcf3 Fixes redis-mpool dependency 2014-08-06 11:32:12 +02:00
Raul Ochoa
a7027f4b05 Removes console.log 2014-08-05 16:48:11 +02:00
Raul Ochoa
480a9f27b4 New authentication mechanism: checks in advance if credentials are provided 
in order to do a single request to redis to retrieve the required database
connection parameters.
 2014-08-05 16:20:06 +02:00
Raul Ochoa
49406c99fa Moves auth functionality to its own directory 2014-08-05 11:57:43 +02:00
Raul Ochoa
e16f278087 CDB-3629 Uses one request to redis to retrieve all connection params 2014-08-05 02:29:07 +02:00
Raul Ochoa
41fe1f4cb5 Merge pull request #164 from CartoDB/CDB-3780 
Fixes for GeoJSON stream responses
 2014-08-05 00:57:35 +02:00
Raul Ochoa
bbb82c2ceb CDB-3780 Adds stream start in case there were no rows. Adds regression test. 2014-08-05 00:47:49 +02:00
Raul Ochoa
15a3f4cc88 CDB-3780 Handle error at query end. Adds regression test. 2014-08-05 00:47:14 +02:00
Raul Ochoa
ce70e7252b Callback requests send 200 status error even if the query failed 2014-08-04 15:56:43 +02:00
Raul Ochoa
919110570c Unified naming for formats 2014-08-02 20:27:05 +02:00
Raul Ochoa
a747818cfa Unifying require for formats 2014-08-02 20:26:01 +02:00
Raul Ochoa
0bc88f1687 Style changes 2014-08-02 20:23:47 +02:00
Raul Ochoa
4830b69a2f Use streaming for geojson format 2014-07-31 03:17:14 +02:00
Raul Ochoa
73a195a7fa Use streaming for json responses. Number of buffered rows can be specified by config. 2014-07-31 02:17:07 +02:00
Raul Ochoa
f5f89606af Use separate method when there are skip fields 2014-07-30 19:48:57 +02:00
Raul Ochoa
3c6b5f1376 Internal renaming 2014-07-30 19:43:43 +02:00
Raul Ochoa
c55dabf5b4 Cached skipFields, better approach to call different methods if there are skip fields 2014-07-30 19:36:25 +02:00
Luis Bosque
6bb015bb14 [CDB-3678] Add X-Served-By-Host header 2014-07-30 12:39:03 +02:00
Raul Ochoa
2653314818 CDB-3657 Sends profiler data as json in X-SQLAPI-Profiler header 2014-07-29 19:29:16 +02:00
Raul Ochoa
0f14118235 CDB-3629 Pickup redis pool config values 2014-07-29 13:58:16 +02:00
Raul Ochoa
a7d444ca92 Changes authentication to start using public user if it is defined in redis. 2014-07-29 13:52:04 +02:00
Raul Ochoa
1a390a3d40 Some style changes 2014-07-29 13:52:04 +02:00
Raul Ochoa
21d6924088 Removes the call to set the search path as it should get the correct public user from redis 2014-07-29 13:52:04 +02:00
javi
ca7b87d81a fixed problem when a query raised a database error 2014-07-29 13:52:04 +02:00
Raul Ochoa
9c255c504b Updates CDB_QueryTables to match the one from the extension. Fixes some tests to show the schema presence. 
Skip some tests that we need to review because with the current implementation of CDB_QueryTables dont make sense.
CDB_QueryTables should be used from the extension as a dependency.
 2014-07-29 13:52:04 +02:00
javi
755ed362d4 set scheme to host user when public user is accessing 2014-07-29 13:52:04 +02:00
Raul Ochoa
7906707130 Revert "Stream JSON responses" 
This reverts commit 49ef1bc0c7.

Conflicts:
	NEWS.md
 2014-06-04 20:57:11 +02:00
Raul Ochoa
23254513e6 removes unneeded comma 2014-06-02 18:57:45 +02:00
Raul Ochoa
b3609696a3 requests associated with formats based on postgres expose a cancel method that will be called on client request abortion/cancelling so postgres can cancel ongoing queries 2014-06-02 14:48:38 +02:00
Sandro Santilli
4b5a5921d5 Fix application_name to "cartodb_sqlapi" 2014-05-07 16:15:46 +02:00
Sandro Santilli
51b135c0ac Set default PostgreSQL application name to "cartodb_sqlapi" 2014-05-07 16:14:17 +02:00
Raul Ochoa
2d2c056d6b Removes spawn unused dependency 2014-04-22 11:18:52 +02:00
Raul Ochoa
8dbf1d9c45 Removes ending semicolons from SQL queries. Fixes #147 2014-04-15 15:29:03 +02:00
Raul Ochoa
3e6b63c9d6 CDB-2081 Returns original query in case there are no clauses to apply. 2014-04-14 15:44:43 +02:00
Raul Ochoa
537cb238c6 CDB-2081 Adds support for order_by through http query params. 2014-04-14 15:01:12 +02:00
Raul Ochoa
909530cfb3 documentation for sql parameter 2014-04-11 12:13:44 +02:00
Raul Ochoa
d10bd8e3b0 precompiled query may write regex 2014-04-11 12:03:43 +02:00
Sandro Santilli
49ef1bc0c7 Stream JSON responses 
Reduces memory usage for big datasets.
JIRA CDB-2600 #resolve
 2014-04-03 15:44:53 +02:00
Sandro Santilli
8a953c116c Fix paging with queries starting with comments 
Closes #144
Includes testcase
 2014-03-27 12:48:39 +01:00
Sandro Santilli
766345595e Add statsd support 
Closes #133
 2014-03-20 11:22:13 +01:00
Sandro Santilli
2cbdc4c38c Add profiler support 
Closes #142
 2014-03-19 18:34:21 +01:00
Sandro Santilli
b0d0d4d073 Reduce work on aborted requests 
Closes #129
Includes testcase
 2014-03-19 13:30:29 +01:00
Sandro Santilli
67cc05085f Always use INFO severity for express logging 
Avoids sending non-200 responses logs to rollbar, when active
 2014-03-13 17:13:53 +01:00
Sandro Santilli
622846b453 Add '/version' endpoint 
Closes #138
Includes testcase
 2014-03-13 13:40:56 +01:00
Sandro Santilli
ac4acf98ae Use console.error for logging errors 2014-03-13 12:14:11 +01:00
Sandro Santilli
65f31fd7c1 Add optional support for rollbar 
Closes #150
Logs messages of severity ERROR or higher
 2014-03-13 12:09:39 +01:00
Sandro Santilli
1f3aca837b Use log4js for logging 
Closes #136
 2014-03-13 11:52:40 +01:00
Sandro Santilli
b565ce2959 Allow configuring log_format 
Closes #131
 2014-02-10 17:34:27 +01:00
Sandro Santilli
ec1d002812 Stop using ANSI colors in the logs 
Closes #130
 2014-02-10 17:10:34 +01:00
Sandro Santilli
5c22b8ddaa Drop unused code 2014-01-31 15:47:33 +01:00
Sandro Santilli
6c47537f47 Honour the 'node_socket_timeout' configuration directive 
Closes #128
Includes testcase, which required enhancing assert.response to
handle errors.
Changes app controller to a function that only reads configuration
when executed (rather than when only require'd).
 2014-01-31 10:58:19 +01:00
Luis Bosque
973b800fca Use db_port in ogr2ogr 2014-01-20 12:28:28 +01:00
Sandro Santilli
161a0bf5e5 Restore compatibility with 1.6.x configuration 2014-01-17 13:14:30 +01:00
Sandro Santilli
61cec80d4d Fix use of "SELECT .. INTO" with windowing params 
Includes testcase. Closes #127
 2014-01-09 17:56:09 +01:00
Sandro Santilli
80c30858bd Enhance error message on unknown cartodb username 
Closes #126
 2013-12-18 12:02:51 +01:00
Sandro Santilli
87d35aa155 Generalize CartoDB username extraction, allowing for multiuser setups 
Closes #124
 2013-12-18 11:59:00 +01:00
Sandro Santilli
3397f2f3f9 Fix field types names lookup after PSQL model refactoring 
Fixes missing .prj in shapefile export regression
Closes #122
 2013-12-02 11:47:52 +01:00
Sandro Santilli
9389a04030 Add warnings and notices to JSON response. Closes #104. 2013-11-18 17:01:06 +01:00
Sandro Santilli
6ebeed04c2 Optionally read user-specific database_password from redis 
Follows CartoDB-2.5.0 model. Includes testcase.
Closes #120 -- Jira ref CDB-870
 2013-11-18 13:31:11 +01:00
Sandro Santilli
af1f3daa69 Optionally read user-specific database_host from redis 
Follows CartoDB-2.5.0 model. Includes testcase.
Closes #121 -- Jire ref CDB-870
 2013-11-18 12:21:30 +01:00
Sandro Santilli
0d44139d8c Do not choke if ogr2ogr fails without on stderr 2013-11-18 11:54:00 +01:00
Sandro Santilli
7218c89b2d Refactor PSQL model to take full db config in the constructor 
Closes #58, stops reading configuration variables itself
 2013-11-18 11:42:43 +01:00
Sandro Santilli
03ea51a375 CartoDB redis interaction delegated to "cartodb-redis" module 2013-11-15 19:14:55 +01:00
Sandro Santilli
4a5c9b0eed Refactor OGR class interface to improve flexibility 
The toOGR and toOGR_SingleFile function now take an "options"
object rather than a long list of arguments. This allows for
easier extension toward per-user database host and password
(see tickets #120 and #121)
 2013-11-15 13:24:48 +01:00
javi
93da1c110e removed console.log refs #119 2013-11-09 16:56:58 +01:00
javi
370d45d8b6 fixed #119 2013-11-09 16:55:40 +01:00
Sandro Santilli
973c7181f0 Allow access to tables whose name contains "pg_" (but not at beginning) 
Closes #118
 2013-11-07 10:16:58 +01:00
Sandro Santilli
0139c8fb16 JSON format: correctly recognize "date" type columns 
Closes #117 -- includes testcase
 2013-11-06 11:43:56 +01:00
Sandro Santilli
45d54c47cc Add note about which GDAL version fixes the SRID detection bug 2013-11-06 08:06:29 +01:00
Sandro Santilli
8d2347af99 Make public PostgreSQL user name a configuration parameter 
Closes #56.
Updates documentation and tests and example config files
 2013-11-05 17:49:10 +01:00
Sandro Santilli
2abb136258 Set KML folder name to the requested filename. 
Closes #115. Includes testcase.
 2013-11-05 17:29:02 +01:00
Sandro Santilli
3c03099e8d Do not trust GDAL SRID detection for KML output. 
Closes #116. Includes testcase.
 2013-11-05 16:57:23 +01:00
Sandro Santilli
1637610f66 Set a meaningful X-Cache-Channel with cache_policy=persist 
Closes #105
 2013-10-18 13:29:06 +02:00
Sandro Santilli
75bc36039d Remove done TODOs 2013-10-11 17:25:38 +02:00
Sandro Santilli
82f965fa57 Upgrade node-pg dependency to 2.6.2 
Simplifies PSQL.quoteIdentifier and exposes PSQL.quoteLiteral
 2013-10-02 12:32:24 +02:00
Sandro Santilli
e92530d6e5 Fix shapefile export for non-linestring results starting with NULL 
Includes tests. Closes #111.
 2013-10-02 12:12:09 +02:00
Sandro Santilli
0d84a704df Fix missing .prj file in shapefile format 
Finds srid, when needed, with an additional query.
Closes #110.
Includes testcases.
 2013-10-02 11:44:06 +02:00
Sandro Santilli
7f7ef682ac Improve recognition of non-standard field types names by db lookup 
Closes #112.
Only looks up "geometry" and "raster" types for now, can be improved
over time.
 2013-10-02 10:22:13 +02:00
Sandro Santilli
1ddec5fdd1 Add note about why we want to skip querying geometry_columns 2013-09-30 10:43:32 +02:00
Sandro Santilli
742936b2b8 Honour skipfields in JSON schema response. Closes #109 2013-09-26 13:26:45 +02:00
Sandro Santilli
a0ff7ee10c Remove last bit of hard-coded format names from app controller 2013-09-05 17:10:48 +02:00
Sandro Santilli
f7bee2345d Report boolean values as of boolean type, not string. Closes #106 2013-08-22 13:45:20 +02:00
javi
d83a81bf32 merged body_param_fix 2013-08-02 15:50:48 +02:00
Sandro Santilli
7a07a25086 Add REINDEX to the list of uncacheable queries 2013-07-24 18:43:38 +02:00
Sandro Santilli
23d092f768 Use float parsing for int8 too 
This makes us ready to upgrade to node-pg 2.x
 2013-07-22 18:22:30 +02:00
Sandro Santilli
8506108aaa Do not cache results for queries you can't fetch source tables of 2013-07-22 18:20:26 +02:00
Sandro Santilli
2ee9411889 Review cache-control headers: 
* Raise max-age to one year for all cacheable queries
 * Set max-age to 0 for uncacheable (mutating) queries
 2013-07-22 17:21:38 +02:00
javi
1eb7b3c885 Merge branch 'develop' into body_param_fix 2013-07-16 11:21:05 +02:00
javi
8ba1464db1 keep req.query as it comes to not broke ouaht refs Vizzuality/cartodb-management/2853 2013-07-15 14:39:15 +02:00
Sandro Santilli
3b9bc8a9f1 Do not set Last-Modified in the remote past 
... or max-age will be always reached
 2013-07-15 13:13:19 +02:00
Sandro Santilli
dfff99224f Set Last-Modified header with cache_policy=persist 2013-07-15 12:12:07 +02:00
Luis Bosque
7b6b541083 Revert "Set max-age=0 when using no-cache" 2013-07-10 07:58:35 +02:00
Sandro Santilli
46b7f7a309 Parse all numbers as floats. Closes #100 2013-07-09 21:51:40 +02:00
Sandro Santilli
ae502335c3 Set "numeric" type to "number" 2013-07-09 21:51:08 +02:00
Sandro Santilli
bc49aebd2a Don't assume SELECT "updated_at" may write to the database. 
Closes #99
 2013-07-08 17:36:53 +02:00
javi
69b5d825e3 reverted support for get params in post 2013-07-04 16:23:48 +02:00
javi
e114b4266d removed debug console.log 2013-07-04 16:17:51 +02:00
javi
320e0b97a8 Merge branch 'develop' of github.com:Vizzuality/CartoDB-SQL-API into develop 2013-07-04 16:14:51 +02:00
javi
838c0c93ba Add support for GET params in POST requests 2013-07-04 16:14:44 +02:00
Sandro Santilli
ba28496127 Add support for "text" datatypes in json output format 2013-07-04 16:04:13 +02:00
javi
78a80e6c82 added string type 2013-07-04 15:47:09 +02:00
Sandro Santilli
46e93355c4 Set max-age=0 when using no-cache (don't be self-contraddicting) 2013-06-28 11:17:52 +02:00
Sandro Santilli
7eff0cb107 Retain UTC offset in JSON output for dates 
Hopefully helps reducing confusion when using timezone-less
with postgresql, in that the time that comes out of JSON output
matches the one input by user, except it may be in an unexpected
timezone (the server's one)
 2013-06-19 12:25:48 +02:00
Sandro Santilli
5e379012a8 Add support for CTE in sql windowing, add unit tests 
This is still an undocumented feature, but as long as it's present
and used (by cartodb UI) better tested than broken...

NOTE: more tests are needed for CTE and RETURNING queries
 2013-06-17 16:36:36 +02:00
Sandro Santilli
88f1d33b42 Add "fields" member in JSON return. Closes #97 2013-06-14 18:36:05 +02:00
Sandro Santilli
339bc0b3a4 Fix windowing support for non-uppercased SELECT queries 2013-06-06 15:24:57 +02:00
Luis Bosque
d476f74a29 Merge branch 'release/staging' into develop 2013-05-30 11:57:29 +02:00
Luis Bosque
fac2b1bb68 Trust forwarded proto header from proxy 2013-05-30 11:56:43 +02:00
Luis Bosque
80452ef6f4 Merge branch 'release/staging' into develop 2013-05-30 11:17:10 +02:00
Luis Bosque
12afc3c88f In OAuth requests, get req.path 
Using req.route.path breaks OAuth whtn used with regex routes. Instead
of this we need to use req.path because the path is already expanded
 2013-05-29 17:36:41 +02:00
Sandro Santilli
ffafe814ef Do not hang up to queued request on ogr generation failure 2013-05-29 17:33:10 +02:00
Sandro Santilli
7f9463e7fd Fix filesystem access conflict among clustered processes 2013-05-29 16:48:20 +02:00
Sandro Santilli
510ac0533f Switch to using evented query model for postgresql 
This enables formats for processing rows as they arrive from the
database, thus possibly reducing memory use. For a start the skip
fields are immediately removed from the result, rather than only
at the end.
 2013-05-29 15:39:39 +02:00
javi
0be9e47930 Add arraybuffer format 2013-05-28 10:37:39 +02:00
Sandro Santilli
76705a3fd7 Refactor format objects to expose a common stream-able interface 
See app/models/formats/README for info
 2013-05-28 10:36:15 +02:00
Sandro Santilli
75717bfb8e Drop another unused function 2013-05-27 09:24:11 +02:00
Sandro Santilli
ab8494ab12 Survive postgresql connection losses. Closes #95. 2013-05-24 18:18:17 +02:00
Sandro Santilli
11a61d16c4 Drop unused code. Spring cleanups! \o/ 2013-05-24 17:13:05 +02:00
Sandro Santilli
3f98cab09a Do not execute queries on OPTIONS. Closes #94 2013-05-24 14:21:13 +02:00
Sandro Santilli
7712d6f2a5 Expose PostgreSQL client pooling settings in environment files 
Actually also moves to a real pooling model, managed by node-pg.
Closes #47
See .example files for new configurations
 2013-05-24 12:09:06 +02:00
Sandro Santilli
64c0494105 Hide dangerous methods of the PSQL model class 
These steps are prepatorial to recactoring to event-based model
 2013-05-24 10:38:27 +02:00
Sandro Santilli
cf6bed9bdb Bubble paging UI hack up from model to controller 
... one day we'll need to completely drop this hack!
 2013-05-24 10:22:17 +02:00
javi
c46e9786b5 refactor formats to external object 2013-05-24 08:50:03 +02:00
Sandro Santilli
b1e7bb8f8f Upgrade underscore dependencies 2013-05-24 08:26:23 +02:00
Sandro Santilli
81a83b93e4 Make base url configurable, use a wildcard in the example config 2013-05-23 11:49:23 +02:00
Sandro Santilli
e5ad8f0760 Fix test broken after oAuth interface change introduced in fcf95755 
Also improve error message on miscall
 2013-05-20 09:06:08 +02:00
Luis Bosque
fcf95755b6 Fixed problem identifying OAuth request protocol 
The protocol of a OAuth request wasn't being identified correctly. It
always considered that the request protocol was https.
 2013-05-14 18:01:37 +02:00
Sandro Santilli
414f4b6c3e Do not request caching of TRUNCATE queries 2013-05-06 18:21:22 +02:00
Sandro Santilli
cc74244b33 Do not choke on multiple skipfields parameter 2013-05-06 12:30:32 +02:00
Sandro Santilli
0ec66c69a7 Allow sql queries to end with a semicolon. Closes #90 2013-04-10 17:21:42 +02:00
Sandro Santilli
e7437ba7cd Rework system catalogue prevention access check to use CDB_QueryTables 
This change reduces the chances of false positive
(forbidding legit queries). Doesn't solve the problem of false
negative (allowing illegit queries).
 2013-04-09 12:36:37 +02:00
Sandro Santilli
d54d953e75 Another bit of improvement for the system-table query prevention 
This work is more important for the testcases than the actual code
 2013-04-09 12:20:27 +02:00
Sandro Santilli
1bcffbc68c Make using SET or querying system catalogues harder 
An hack to "prevent" querying system tables already existed but
was pretty weak. This commits makes that a bit stronger. The
filter for SET is new.
 2013-04-09 11:52:34 +02:00
Sandro Santilli
104ed355d0 Fix fd leak on export query cancel 2013-03-27 20:02:21 +01:00
Sandro Santilli
8f60735b5e Queue export requests for shapefile format. Closes #65 2013-03-27 16:43:12 +01:00
Sandro Santilli
b2e88145ff Comment out debugging line 2013-03-27 16:09:56 +01:00
Sandro Santilli
0261e8f687 Drop unused dependencies (csv and strftime) 2013-03-27 15:55:57 +01:00
Sandro Santilli
6d6dd16446 Draft delegating CSV output to OGR 
Adapt tests to new output
 2013-03-27 15:41:24 +01:00
Sandro Santilli
ff49346d4d Drop unused variable 2013-03-26 18:20:36 +01:00
Sandro Santilli
58b079e212 Fix reace condition in new KML output, test KML export of empty table 2013-03-26 18:18:13 +01:00
Sandro Santilli
38f99c586a Fix concurrent requests for KML export 2013-03-26 16:59:37 +01:00
Sandro Santilli
a0d9038e59 Auth / non-auth tests for SHP and KML export are in place 2013-03-26 16:14:48 +01:00
Sandro Santilli
c3ae7074cb Make temporary dir a configuration setting 2013-03-26 16:02:05 +01:00
Sandro Santilli
12553dafdd Don't forget to call .end() on the result (fixes Shapefile export) 2013-03-26 15:32:07 +01:00
Sandro Santilli
3f8a606432 Simplify steps for SHP and KML outputs 2013-03-26 12:59:46 +01:00
Sandro Santilli
41dd23b429 Do not run the query twice when exporting to KML 2013-03-26 12:47:03 +01:00
Sandro Santilli
6c0d7813f6 Upgrade generic-pool to ~2.0.2 2013-03-14 11:58:30 +01:00
Sandro Santilli
dc87209033 Do not confuse warnings with errors on shapefile output 
Closes #87
 2013-02-25 18:32:28 +01:00
Sandro Santilli
d4b1f580ff Fix X-Cache-Channel computation with paging parameters. Closes #85. 2013-02-18 16:19:12 +01:00
Sandro Santilli
10ec5b9fda Do not attempt to compute an X-CacheChannel from a broken query info 
Also enhance debugging for broken query info results
 2013-02-18 12:20:18 +01:00
Sandro Santilli
19eb51b06d Use an LRU cache for query tables, and add expiration support 
Defaults to 8192 items each valid for no more than 10 minutes
Both settings can be changed via environment config.
See #83
 2013-02-13 18:59:09 +01:00
Sandro Santilli
7d2d585c54 Check CDB_QueryTable response before saving into cache 
Also add pid to /cachestatus (#83)
 2013-02-13 16:43:04 +01:00
Sandro Santilli
5959e6465a Fix Content-Disposition for error responses. Closes #82 2013-02-13 13:32:34 +01:00
Sandro Santilli
d667d64c78 Fix skipfields use with SHP output format. Closes #81 2013-02-13 13:02:37 +01:00
Sandro Santilli
977ecbeb29 Return an error when "the_geom" is in skipfield for SVG output 
Closes #73
 2013-01-21 10:06:51 +01:00
Sandro Santilli
e897cb07d2 Add test for null geoms in geojson 
Also drop an hard-coded "the_geom" in geojson output code
 2013-01-11 18:57:45 +01:00
Sandro Santilli
9bc79bded6 Re-add null geometry support in geojson output 
You never know if anyone is relying on them, and they are supported
by the specs:
https://github.com/mbostock/topojson/issues/18#issuecomment-12154398

We'll still skip those records for topojson
 2013-01-11 18:43:05 +01:00
Sandro Santilli
ca065e89b9 Omit NULL the_geom features from from geojson and topojson outputs 
Closes #80
 2013-01-11 18:20:25 +01:00
Sandro Santilli
3e7c8f4d22 Improve mixed geometry export error message. Closes #78 2013-01-11 17:09:22 +01:00
Sandro Santilli
a81b55e421 Include properties to TopoJSON output. Closes #79 
Includes support for the skipfields option
 2013-01-11 11:10:51 +01:00
Sandro Santilli
39669578b6 Initial support for TopoJSON (#79) 
Does not include any attributes in the format
 2013-01-09 17:43:23 +01:00
Sandro Santilli
2e85588d99 Format date fields in CSV output 2012-12-07 14:27:02 +01:00
Sandro Santilli
c00cf6d390 Revert "Upgrade node-csv module to version 0.2.4" 
This reverts commit 270e6280ed.
The performance issue must be fixed before we can upgrade:
https://github.com/wdavidw/node-csv-parser/issues/64
 2012-11-19 18:44:53 +01:00
Sandro Santilli
270e6280ed Upgrade node-csv module to version 0.2.4 
Will help with compressed-csv export format
 2012-11-19 16:20:40 +01:00
Sandro Santilli
3a641beda8 Fix UTF8 in shapefile export. Closes #66. 
Testcases count reaches 100 ! Happy post-post-gis day :)
 2012-11-16 12:50:21 +01:00
Sandro Santilli
da05e6f4ca Fix KML export truncation. Closes #947 
Also adds test for SHP truncation (isn't trucated)
 2012-11-16 12:22:06 +01:00
Sandro Santilli
75fcd5ae9c Do not write back to request.query as it breaks oAuth signature 
NOTE: this breakage misses a testcase (overcomplex at the moment)
 2012-11-14 18:04:38 +01:00
Sandro Santilli
bd08eb4add Use inline attachment also with POST, if format isn't given 2012-11-14 16:30:18 +01:00
Sandro Santilli
6b7cada97d Never dispose "inline" when using POST. 2012-11-13 19:27:25 +01:00
johnhackworth
e64c3f57f7 changes in the way the parameters are collected to support both POST and GET 2012-11-13 19:27:25 +01:00
Sandro Santilli
0ef13f08c2 Use inline disposition when no format and no filename are given 
See #61
 2012-11-12 19:44:16 +01:00
Sandro Santilli
120cf3f0c5 Add 'cache_policy' parameter. Closes #62 2012-11-12 19:14:20 +01:00
Sandro Santilli
005ae48e3a Support for specifying a list of fields to skip from output. 
Closes #63
 2012-11-12 17:11:17 +01:00
Sandro Santilli
46cec7a0e5 Add support for specifying a filename for exports. Closes #64 
Sets release target to 1.3.0, due to parameter addition
 2012-11-12 12:40:53 +01:00
javi
3fd3c5fabc added the needed headers for CORS 2012-11-01 13:17:45 +01:00
Sandro Santilli
110af3e9d9 Add "sqlapi" prefix to temporary dir for shapefile output 2012-10-31 11:57:20 +01:00
Sandro Santilli
9aa28c5bea Initial support for KML output format. Closes #54. 2012-10-25 18:10:56 +02:00
Sandro Santilli
978c0b4cbe Advertise header presence in CSV Content-Type 2012-10-25 13:40:21 +02:00
Sandro Santilli
a6837573c5 Use "attachment" Content-Disposition for all output formats. 
Closes #61, includes tests

NOTE: this includes the default "json" format.
 2012-10-25 13:34:06 +02:00
Sandro Santilli
5d8eccc81e Fix CSV output with no rows. Closes #60 2012-10-25 13:14:47 +02:00
Sandro Santilli
a560a37823 Generalize OGR output function 2012-10-23 17:45:56 +02:00
Sandro Santilli
26bdccf541 Remove debugging output 2012-10-18 13:19:08 +02:00
Sandro Santilli
d0ae7e08a6 Initial support for Shapefile output 2012-10-18 12:51:12 +02:00
Sandro Santilli
b038419abd Add missing newline 2012-10-18 11:35:06 +02:00
Sandro Santilli
729462b500 Fix throw in async function (verifyRequest) 2012-10-18 11:34:39 +02:00
Sandro Santilli
8574517ab8 Add a userid_to_dbuser function 2012-10-18 11:33:35 +02:00
Sandro Santilli
d9b733e5c6 Recognize ALTER as a writing query 2012-10-15 13:40:04 +02:00
Sandro Santilli
d23416cc60 Set X-Cache-Channel to NONE when the SQL may write to the database 
Note that "may write" allows for false positive, so there could be
less cache hits than possibly allowable. If this will be a problem
for any real use case we could still improve the regular expression
used to detect "writing" queries.

Automated tests are added to check for the X-Cache-Channel header
with both writing and read-only queries performed by authenticated
requests.

Closes #27
Closes #43
 2012-10-15 13:23:17 +02:00
Sandro Santilli
553146e6dd Add consistency checking in packageResult about format value 2012-10-12 12:57:03 +02:00
Sandro Santilli
ac83700810 Send a 404 on unsupported format requested 2012-10-12 12:17:35 +02:00
Sandro Santilli
0d91ab2c6a Survive multiple "format" parameters, only using last one 2012-10-12 11:42:03 +02:00
Sandro Santilli
6a2c0e9727 Initial support for SVG output (#49) 
This version only dumps the geometries with no identifier.
Scales geometries to fit in a 1024x768 pixels area, using
a circle radius of 5 pixels, a stroke-width of 1 pixel and
no fill.  Supports trimming number of decimals.
Adds a viewBox tag to fit the drawing to the output device window.

Includes an automated testcase.
 2012-10-04 12:04:50 +02:00
Sandro Santilli
4521942820 Test "CSV" format, drop "KML" from the list of supported formats 2012-10-04 12:04:50 +02:00
Sandro Santilli
eb645d298c More embedded comments 2012-10-04 12:04:49 +02:00
Sandro Santilli
8b824801cf Fix INSERT and UPDATE with RETURNING clause. Closes #50 
Includes regression test
 2012-09-17 11:50:19 +02:00