cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,030 Commits 1 Branch 118 Tags 9.7 MiB
1a9081135d
Commit Graph

953 Commits

Author SHA1 Message Date
Eneko Lakasta
ebe04d04ad refactor user database service. Get proper DB config based on auth type: oauth vs apikey 
Oauth uses only master api key configuration
Api key uses master and the provided api key configurations

Also move default api key fallback to this service
 2018-05-28 15:53:51 +02:00
Simon Martín
e092647246 ensuring data ingestion log 2018-05-28 11:25:09 +02:00
Simon Martín
7d32ae293f rename buyan_logger file name 2018-05-28 11:18:30 +02:00
Simon Martín
3cf28bb7ff jshint happy 2018-05-25 18:50:56 +02:00
Simon Martín
22caa049ee StreamCopyMetrics 2018-05-25 18:47:41 +02:00
Simon Martín
b31984cbc6 simplify response 2018-05-25 17:57:40 +02:00
Simon Martín
1fa5afd759 unify error handler 2018-05-25 17:50:59 +02:00
Simon Martín
6c3f9cf1d3 simplify controllers 2018-05-25 17:42:30 +02:00
Eneko Lakasta
d3b6ebd260 Fallback to default api key if no api key is provided 
What happens with oauth authorization? This way we will always have an api_key and oauth will never trigger....
 2018-05-25 17:28:56 +02:00
Eneko Lakasta
ec75227aa2 use apikeys instead of metadata for the database connections parameters 2018-05-25 17:23:24 +02:00
Simon Martín
fd70673d88 more details 2018-05-25 16:46:25 +02:00
Simon Martín
8a2c777577 details 2018-05-25 16:00:27 +02:00
Simon Martín
9b0b92fb6b removing try catch 2018-05-25 15:46:12 +02:00
Simon Martín
5bed04a38c fix gzip problem with req close event 2018-05-25 15:34:30 +02:00
Simon Martín
fe3bd4fd37 handling errors from request in COPYfrom 2018-05-25 14:33:54 +02:00
Simon Martín
e2bba967f9 handling copyto errors 2018-05-24 19:48:24 +02:00
Eneko Lakasta
6bc90eb9b3 fix typo: Backed -> Backend 2018-05-24 11:04:14 +02:00
Simon Martín
fca6ee8232 changing name of dataIngestionLogPath 2018-05-23 17:32:44 +02:00
Simon Martín
b2a36eb556 copy metrics with BunyanLogger 2018-05-23 17:25:46 +02:00
Simon Martín
fdc542f7b5 BunyanLogger 2018-05-23 17:24:48 +02:00
Simon Martín
5ba7dca79c copyfrom metrics to kibana 2018-05-23 10:30:37 +02:00
Paul Ramsey
f77707f6da Fix reponse copy from 2018-05-22 10:48:11 -04:00
Paul Ramsey
a4654f82b2 Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q 2018-05-22 10:37:50 -04:00
Paul Ramsey
ed696a96ec Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q 2018-05-22 10:23:04 -04:00
Simon Martín
6d73d97ae2 copyFrom metrics to statsd 2018-05-22 16:02:14 +02:00
Simon Martín
e347985465 changing query parameter name from sql to q 
unify query validatrion
 2018-05-22 15:42:57 +02:00
Simon Martín
b9474e7fc3 total_rows in copyto metrics 2018-05-22 15:29:08 +02:00
Simon Martín
0bac7a484c improving query info 2018-05-22 14:16:22 +02:00
Simon Martín
185f708d5b small style refactor 2018-05-22 14:15:55 +02:00
Simon Martín
86e0ebc148 small refactor in copy format regex 2018-05-22 12:02:41 +02:00
Simon Martín
f01191472b refactoring copy controller middlewares 2018-05-22 11:56:50 +02:00
Simon Martín
433bd01c27 copyto metrics 2018-05-22 11:54:10 +02:00
Simon Martín
81be15fbc3 adding format to copy metrics 2018-05-21 19:13:44 +02:00
Paul Ramsey
64fc0c32e3 Change from 'sql' as query parameter to 'q', aping existing sql api 2018-05-21 07:47:16 -07:00
Simon Martín
816779fefd erratum 2018-05-18 17:32:59 +02:00
Simon Martín
82c9fec18a copy from metrics 2018-05-18 17:31:29 +02:00
Eneko Lakasta
fc7e246704 please jshint 2018-05-18 11:35:54 +02:00
Simon Martín
bf4d12868b Merge branch 'master' into pgcopy-stream 2018-05-18 10:57:32 +02:00
Eneko Lakasta
1e8c6e198c remove api key fallback 2018-05-17 17:13:59 +02:00
Simon Martín
9016647a6f reverting use of busboy instead of multer 2018-05-14 18:31:44 +02:00
Simon Martín
60b48df4d6 default rate limits params to server.js 2018-05-11 15:41:35 +02:00
Simon Martín
d70013b0f4 object assing for default rate limits params 2018-05-11 15:36:59 +02:00
Simon Martín
bc8a867334 jshinty happy 2018-05-11 14:51:43 +02:00
Simon Martín
f2acc8f653 forgotten log 2018-05-11 14:16:26 +02:00
Simon Martín
eee8bc2674 moving copyto headers to correct position 2018-05-11 14:12:23 +02:00
Simon Martín
791967877c addining gzip support to copyfrom 2018-05-11 13:33:54 +02:00
Simon Martín
f749798ca5 validating inputs 2018-05-11 10:38:33 +02:00
Paul Ramsey
8f211f905c Remove wrong examples 2018-05-10 12:21:33 -07:00
Paul Ramsey
f0021264d2 Change from form-multipart to POST with chunked upload 2018-05-10 12:11:55 -07:00
Simon Martín
0bb7cec9f4 params existence responsability for UserLimits 2018-05-10 11:51:58 +02:00
Simon Martín
d400926387 using busboy instead of multer 2018-05-08 18:39:16 +02:00
Simon Martín
a997881048 ensuring existence of rate limits params in code 2018-05-08 16:00:16 +02:00
Simon Martín
82d14ab98f busboy configuration 2018-05-08 13:15:19 +02:00
Simon Martín
7c4409bcf5 from var to const/let 2018-05-08 13:10:25 +02:00
Simon Martín
c40ab801b1 refactoring handleCopyTo 2018-05-08 13:08:29 +02:00
Simon Martín
b036b876ff using Busboy instead of multer 2018-05-08 12:52:33 +02:00
Simon Martín
6b96032fcf ensuring return error 2018-05-04 16:50:17 +02:00
Simon Martín
383b86cf9b simplify sql paramter handler and removing underscore 2018-05-04 16:47:50 +02:00
Simon Martín
2b066eadfa removing unused dependencies 2018-05-04 15:35:23 +02:00
Simon Martín
02238fefe1 moving body-parser from global to routes of query and job 2018-05-04 15:15:37 +02:00
Simon Martín
0346cf11d8 enabling multipart/form-data again 2018-05-04 15:11:45 +02:00
Simon Martín
4322ccdd09 adding new endpoints to rate limits 2018-05-03 18:50:13 +02:00
Simon Martín
af4c5906c8 adding /sql tro the endpoints 2018-05-03 18:31:49 +02:00
Paul Ramsey
36bdf84f71 Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream 2018-04-27 15:56:13 +02:00
Paul Ramsey
e2d8896307 Typo in curl example 2018-04-27 12:32:47 +02:00
Paul Ramsey
5f9cc37dba Jshint change, DRY the content-disposition handler 2018-04-27 12:19:53 +02:00
Paul Ramsey
fe52af71ac Add documentation SQL API copyfrom/copyto end points, 
and make small modifications arising from that task
 2018-04-27 11:55:06 +02:00
Paul Ramsey
c78cac9bd6 Use the correct database credentials, so we obey security 2018-04-26 10:45:52 +02:00
Paul Ramsey
f13028b497 More jshint changes 2018-04-26 10:20:21 +02:00
Simon Martín
5f8e1d8f53 fix validatePGEntitiesAccess access 2018-04-26 10:13:03 +02:00
Paul Ramsey
cc5fa5c6ce Changes suggested by jshint 2018-04-26 10:04:38 +02:00
Paul Ramsey
ea66076255 Remove 'console.debug', try and get travis tests clean 2018-04-26 09:40:13 +02:00
Paul Ramsey
a98781d335 Working code for /copyfrom (streaming) and /copyto (streaming) 2018-04-25 18:37:04 +02:00
Paul Ramsey
b59bda5780 Get working copyto implementation in place. Many test cases 
needed still and thoughts on how to communicate errors back
that are maybe nicer than stack messages?
 2018-04-24 15:55:20 +02:00
Simon Martín
e12e16507a create object and export 2018-04-24 13:21:43 +02:00
Paul Ramsey
4914100205 CopyFrom works, but still needs a decent return payload and 
a lot of work on returning useful information for error cases (post empty input,
db errors returned more nicely? etc)
 2018-04-24 13:07:57 +02:00
Paul Ramsey
72bce5732b WIP on /copyfrom 2018-04-24 11:26:15 +02:00
Simon Martín
3b228ac17b solving jshint crazy error 2018-04-24 10:10:37 +02:00
Simon Martín
985f9ef7f5 pgEntitiesAccessValidator 2018-04-23 18:17:44 +02:00
Rafa de la Torre
2229d0ee57 Fix jshint issues 2018-04-13 16:36:43 +02:00
Paul Ramsey
0161696627
Update copy_controller.js 2018-04-13 05:43:23 -07:00
Paul Ramsey
d54e2f5a07 Implementation including multer, custom storage engine, and pg-copy, 
but without turning over pg-copy, and demonstrating the missing 'sql'
parameter in the custom storage engine.
 2018-04-12 12:25:28 -07:00
Simon Martín
2a729b7da8 destructuring error_handler parameters 2018-04-03 13:43:17 +02:00
Simon Martín
3f6b6d3cff middleware naming 2018-04-03 13:41:18 +02:00
Simon Martín
db6ab4a591 let & const vars 2018-03-28 13:52:45 +02:00
Simon Martín
adab7f21d8 setErrorHeader refactor 2018-03-28 13:50:45 +02:00
Simon Martín
720c3eb8cb changing msg var name for errorResponse 2018-03-28 13:35:19 +02:00
Simon Martín
4301b63189 fix error 2018-03-28 13:31:31 +02:00
Simon Martín
616f3f3c07 errorHandlerFactory 2018-03-28 13:06:39 +02:00
Simon Martín
16ea727fc3 move ErrorHandler logic from error middleware 2018-03-28 12:58:10 +02:00
Simon Martín
5b6f876cd6 change pgErrorHandler for more generic errorHandler 2018-03-28 12:50:07 +02:00
Simon Martín
b45f5fe31e move ErrorHandler to services folder 2018-03-28 12:49:33 +02:00
Simon Martín
287d4634c6 rename createPgError to createGenericError 2018-03-28 12:43:34 +02:00
Simon Martín
e611d21029 getName refactor 2018-03-28 12:38:45 +02:00
Simon Martín
2bfa00ab57 getStatus refactor 2018-03-28 12:35:01 +02:00
Simon Martín
d727a50407 jshint 2018-03-28 12:17:42 +02:00
Simon Martín
96435d28e0 removing underscore dependency 2018-03-28 12:15:55 +02:00
Simon Martín
e2641200f7 extracting error type check 2018-03-28 12:13:31 +02:00
Simon Martín
575f1b3a05 simplify error response on models 2018-03-28 11:55:52 +02:00
Simon Martín
050c598e90 using new error handler as a Error 2018-03-28 11:30:11 +02:00
Simon Martín
2722302a89 using error handler as a Error 2018-03-28 11:29:38 +02:00
Simon Martín
60021a70dc extends Error 2018-03-28 10:56:06 +02:00
Simon Martín
522363c419 removing semicolons 2018-03-28 10:49:50 +02:00
Simon Martín
eefea03b80 error handler to class 2018-03-28 10:48:25 +02:00
Simon Martín
67e1c04efd indentation details 2018-03-28 10:48:08 +02:00
Simon Martín
c6eb0ad4c7 error middleware name 2018-03-28 09:58:05 +02:00
Simon Martín
0998e50014 refactor getFields method 2018-03-27 15:56:53 +02:00
Simon Martín
b75334f1e7 Merge branch 'master' into limits-error-refactor 2018-03-27 15:32:24 +02:00
Simon Martín
483c263f61 adding and testing limits error response with params 2018-03-26 19:10:23 +02:00
Simon Martín
54876fa203 addind properties to limits errors 2018-03-26 15:56:50 +02:00
Simon Martín
50b8a9ed17 remove unused parameter 2018-03-23 18:02:44 +01:00
Simon Martín
0b708165b3 simplify timeout error catch 2018-03-23 17:48:21 +01:00
Simon Martín
8c86c2b3f8 correct error message in rate limit 2018-03-23 15:35:06 +01:00
Simon Martín
26baaf6081 removing retry after when no necessary 2018-03-23 15:09:16 +01:00
Simon Martín
80818c3cbb fix userLimits reference 2018-03-20 16:20:56 +01:00
Simon Martín
581bd0dd89 Merge branch 'master' into rateLimits 2018-03-14 12:47:45 +01:00
Simon Martín
1ffeb456e8 Headers following rfc6648 2018-03-14 12:17:26 +01:00
Daniel García Aubert
8dea7615c4 Use user's port instead of the generic one 2018-03-13 19:08:51 +01:00
Daniel García Aubert
328d6f253e Honor batch api port 2018-03-13 18:49:01 +01:00
Eneko Lakasta
2ec767c1fc Set Vary Header to honor Authorization header when caching (fastly) 2018-03-13 11:59:07 +01:00
Simon Martín
9cd4e85768 using cartodb-redis in rate limits 2018-03-03 15:01:45 +01:00
Simon Martín
b6ff37d33f changing param name 2018-03-02 19:57:33 +01:00
Simon Martín
967cdf8a06 refactoring user_limits 2018-03-02 19:54:22 +01:00
Simon Martín
5cd7c24776 refactoring middleware and updating tests 2018-03-02 16:42:45 +01:00
Simon Martín
a11d9870a3 exporting RATE_LIMIT_REDIS_DB 2018-03-02 13:17:50 +01:00
Simon Martín
25d1e5442c new redis keys 2018-03-01 15:51:19 +01:00
Simon Martín
7a6ef637ae make jshint happy 2018-03-01 14:47:34 +01:00
Simon Martín
f3adad57aa fix assignment 2018-03-01 13:15:32 +01:00
Simon Martín
779ab3b83f adding rate limit middleware to controllers 2018-03-01 12:19:57 +01:00
Simon Martín
a3dfd2767b rate limit middleware 2018-03-01 12:19:28 +01:00
Simon Martín
2834975d14 controllers using user limits service 2018-03-01 11:31:35 +01:00
Simon Martín
3fdb7abeaf user limits service 2018-03-01 11:30:34 +01:00
Daniel García Aubert
5802b2182e Improve profiling 2018-02-27 13:56:00 +01:00
Daniel García Aubert
78f8c430ea Merge branch 'master' into auth-api 2018-02-27 12:24:40 +01:00
Daniel García Aubert
6f4c79eaef Fix common interface for auth-backends 2018-02-26 19:02:05 +01:00
Daniel García Aubert
418ec1304f Fix issue with auth fallback, it wasn't authenticated when apikey master was provided 2018-02-26 18:13:49 +01:00
Daniel García Aubert
cf0214f5c3 Respond with 403 intead of 401 when the request has no permissions to the specific resource 2018-02-23 15:50:23 +01:00
Daniel García Aubert
f85bdf53a5 Missing middleware 2018-02-23 13:19:26 +01:00
Daniel García Aubert
897d38c3f4 Do not set undefined step 2018-02-23 13:05:41 +01:00
Daniel García Aubert
47b54612c7 Place send-response-middleware to the very end of middleware stack 2018-02-23 13:03:56 +01:00
Rafa de la Torre
90b0ec61c2 Add RESIZE=yes param to shapefiles export 
Add the parameter `RESIZE=yes` to gdal shapefile driver. This resizes
fields to their optimal size. See
http://www.gdal.org/drv_shapefile.html for details.

This is needed because some customers are having trouble exporting
shapefiles, then reimporting into ArcGIS.
 2018-02-23 10:34:44 +01:00
Daniel García Aubert
3807d9f94d Template for User not found error 2018-02-22 17:49:02 +01:00
Daniel García Aubert
ec24019cc0 Apply some ES6 goodies 2018-02-22 17:35:03 +01:00
Daniel García Aubert
868dc798b2 Cosmetic line break 2018-02-22 15:59:13 +01:00
Daniel García Aubert
095c34358e Remove step 2018-02-22 15:58:52 +01:00
Daniel García Aubert
8c3b9fa05b Remove bad assignment 2018-02-22 13:27:50 +01:00
Daniel García Aubert
e4067132b0 Add profile to user limit middleware 2018-02-22 12:48:39 +01:00
Daniel García Aubert
700c64bba3 Extract to a middleware user timeout limit from user-database-services 2018-02-22 12:45:55 +01:00
Daniel García Aubert
afceac7369 Improve profile step name 2018-02-22 12:23:37 +01:00
Daniel García Aubert
94c5bd11df Split authorization middleware, it was actually doing two things: authorize and get database connection params 2018-02-22 12:22:39 +01:00
Daniel García Aubert
1252f32579 Get api_key token from res.locals 2018-02-22 12:07:27 +01:00
Daniel García Aubert
8730b5c517 Read if the request is authenticated through res.locals 2018-02-22 12:04:05 +01:00
Daniel García Aubert
b399abee18 Move authorization to auth-api and extract it from user-database-service 2018-02-22 11:46:34 +01:00
Daniel García Aubert
6c92781600 Remove empty line 2018-02-21 16:19:12 +01:00
Daniel García Aubert
9088cbf150 Improve naming 2018-02-21 13:51:28 +01:00
Daniel García Aubert
28a7cf9259 Don't authenticate when default_public api-key is provided 2018-02-21 11:14:31 +01:00
Daniel García Aubert
ba113d8628 Improve naming 2018-02-20 16:43:43 +01:00
Daniel García Aubert
33baa399ae Create function to compose middlewares to perfom job operations 2018-02-20 16:25:16 +01:00
Daniel García Aubert
864ec30058 Remove duplication 2018-02-20 13:22:33 +01:00
Daniel García Aubert
d4d54648d4 Improve naming 2018-02-20 13:14:28 +01:00
Daniel García Aubert
03e9b8006b Actually pass just apikey instead of the entire "res" object 2018-02-19 19:16:33 +01:00
Daniel García Aubert
d6776189bd Remove unused param 2018-02-19 16:04:57 +01:00
Daniel García Aubert
4af751ff62 Rename param 2018-02-19 15:58:48 +01:00
Daniel García Aubert
b6a666a8b4 Miss rename middleware 2018-02-19 15:54:05 +01:00
Daniel García Aubert
8c13add7a8 Rename middleware 2018-02-19 15:49:17 +01:00
Daniel García Aubert
939443ef81 Unify credentials and authenticated request midllewares 2018-02-19 15:44:28 +01:00
Daniel García Aubert
7898b49e59 Extract profiler middleware to used in query and job controllers 2018-02-19 15:13:36 +01:00
Daniel García Aubert
85fbd7c6b2 Create middlewares to initialize and finish profiling 2018-02-19 14:42:52 +01:00
Daniel García Aubert
633d06bf2d Do not use underscore 2018-02-19 14:41:06 +01:00
Daniel García Aubert
e04529c501 Rename middleware 2018-02-19 14:20:09 +01:00
Daniel García Aubert
75c2d85dbb Use authenticated middleware in query controller 2018-02-19 13:24:44 +01:00
Daniel García Aubert
114070ef96 Rename middleware 2018-02-19 12:37:19 +01:00
Daniel García Aubert
adfe3dd52b Remove comment 2018-02-19 11:10:39 +01:00
Daniel García Aubert
ea310db62c Cosmetic chenges 2018-02-19 11:04:28 +01:00
Daniel García Aubert
edd382fb8f Check user is the same user that sends the request when basic-auth is provided 2018-02-16 18:21:06 +01:00
Daniel García Aubert
bcf88093e9 Support basic-auth for batch api 2018-02-16 11:52:57 +01:00
Daniel García Aubert
2cadfe6f18 Add support for Basic Auth 2018-02-16 10:46:58 +01:00
Daniel García Aubert
756fbe42c9 Use apikey midlleware 2018-02-15 17:24:04 +01:00
Daniel García Aubert
ad772246d0 Get api_key from specific middleware and save it into res.locals 2018-02-15 17:23:35 +01:00
Daniel García Aubert
67f2a85abd Add database credentials to jobs 2018-02-15 12:30:04 +01:00
Daniel García Aubert
ea6e8b5315 Implement fallback mechanism to be able to authenticate as usual in case of apikey is not found 2018-02-14 16:22:36 +01:00
Daniel García Aubert
7c859f0d7b Use actual apikey properties 2018-02-12 17:49:50 +01:00
Daniel García Aubert
c003a491c0 Use current implementation of cartodb-redis 2018-02-12 16:41:35 +01:00
Daniel García Aubert
5c7f5fbaa5 Merge branch 'master' into auth-api 2018-02-12 16:30:09 +01:00
Simon Martín
bd410fb5f6 using same way to search a string 2018-01-09 17:16:38 +01:00
Simon Martín
df86cc16e4 method to detect timeout error 2018-01-09 17:14:11 +01:00
Simon Martín
6e45c39b1f Merge branch 'master' into pythonTimeout 2018-01-08 11:29:17 +01:00
Simon Martín
ee511d3762 stringifyForLogs more usual case first 2017-12-18 14:54:23 +01:00
Simon Martín
b531922459 add line at EOF 2017-12-18 12:59:20 +01:00
Simon Martín
485d9adddd escape chars function 2017-12-18 12:45:44 +01:00
Simon Martín
3fa09a007e line at EOF and better comment 2017-12-12 17:16:40 +01:00
Simon Martín
6481d14192 escape quotes for logs 2017-12-12 12:58:31 +01:00
Simon Martín
c478e8e62e removing default error log value 2017-12-12 10:55:28 +01:00
Simon Martín
61b351535c change funcion name 2017-11-30 15:06:25 +01:00
Simon Martín
0ab87df644 extract error message from array 2017-11-28 17:19:10 +01:00
Simon Martín
4d598eacd7 dont modify err, works with errorsLog 2017-11-28 16:11:04 +01:00
Simon Martín
a17e1fc5ec logErrors function 2017-11-28 15:58:14 +01:00
Simon Martín
3e23be2087 add errors header with default value 2017-11-28 15:57:58 +01:00
Daniel García Aubert
88015aaf0f Implement new auth system based on api-key tokens with scoped permissions 2017-11-24 17:57:54 +01:00
Simon Martín
670cb6eb8b Merge branch 'master' into pythonTimeout 2017-11-24 16:02:13 +01:00
Daniel García Aubert
15949d2fc2 Fix wrong function name 2017-11-24 15:55:16 +01:00
Daniel García Aubert
b10937a8fb Use res.locals instead of req.context to share object among middlewares 2017-11-24 15:49:25 +01:00
Daniel García Aubert
c2d1a4653e Improve naming for connection database params holder 2017-11-24 15:45:49 +01:00
Simon Martín
11e0ac89d2 undo test of postgres9.5 error 2017-11-24 12:22:30 +01:00
Rafa de la Torre
296d1878fd Trying regexp instead of exact match 2017-11-24 10:55:02 +01:00
Daniel García Aubert
833b7b962c Order geo-columns to export 'the_geom' by default while exporting SHP format 2017-11-17 14:45:58 +01:00
Simon Martín
27e43c8d91 handling python timeout error 2017-11-17 13:30:58 +01:00
Daniel García Aubert
b1112e2bb1 Remove unneeded header 2017-09-20 16:48:22 +02:00
Daniel García Aubert
980e762ab7 Skip user auth middlewares 2017-09-20 16:35:33 +02:00
Daniel García Aubert
0597a3b8a6 Change job work in progress endpoint 2017-09-20 15:46:29 +02:00
Raul Ochoa
32154b67c6 Merge branch 'master' into upgrade-cartodb-psql 2017-08-11 11:57:16 +02:00
Daniel García Aubert
f13c16e523 Typo 2017-08-10 11:22:08 +02:00
Daniel García Aubert
01a71ee60e Apply user timeout to ogr2ogr command 2017-08-09 12:50:16 +02:00
Raul Ochoa
3e536b822e Remove unnecessary destroyOnError option 2017-08-08 18:21:10 +02:00
Daniel García Aubert
fb477260ec Respond with 429 and proper message when database fires a timeout error 2017-08-03 17:19:08 +02:00
Rafa de la Torre
07890ec608 Change tableCache.get for tableCache.peek #244 
This will make it consider age of the key when it was initially set,
instead of renewing it with the time of the last access.
 2017-07-03 11:13:21 +02:00
Rafa de la Torre
1db935a24b Merge pull request #423 from CartoDB/422-disable-query-lru-cache 
422 disable query lru cache
 2017-06-27 17:40:33 +02:00
Rafa de la Torre
2d9a5e9655 Move cyclomatic complexity to the factory #422 2017-06-27 16:05:05 +02:00
Rafa de la Torre
c8e42f561d Reduce cyclomatic complexity (jshint 2/2) #422 2017-06-27 15:35:51 +02:00
Rafa de la Torre
34ccfc10df Please jshint (1/2) #422 2017-06-27 13:26:14 +02:00
Rafa de la Torre
bf478cabb2 Fix initialization of NoCache #422 2017-06-27 12:44:11 +02:00
Rafa de la Torre
bc71889f93 First attempt at removing tableCache #422 2017-06-27 12:10:24 +02:00
Daniel García Aubert
aa66000848 Going green: use replacer parameter in JSON.stringify method to not cast special number values to null 2017-06-08 12:42:37 +02:00
Mario de Frutos
3a99020632 Added regexp for the test to matches both 0.10 and 6.X node version 2017-04-18 12:15:47 +02:00
Mario de Frutos
da42daa536 Improved error message for zip spawn command 2017-04-18 12:15:47 +02:00
Daniel García Aubert
e4eeb9e1f6 Clean commented code 2017-04-18 12:15:47 +02:00
Daniel García Aubert
e0f9209017 Attach error handler while spawning zip command 2017-04-18 12:15:47 +02:00
Daniel García Aubert
884d3aea0a Dont call 'next' callback twice, include error message if zip don't exit normally 2017-04-18 12:15:47 +02:00
Daniel García Aubert
4b98b7056a Make zip command path configurable 2017-04-18 12:15:47 +02:00
Daniel García Aubert
6dba06130b Retur a proper error message when ogr2ogr command fails 2017-04-18 11:40:37 +02:00
Daniel García Aubert
d60e2107af Add error callback to ogr command while spawning 2017-04-11 09:43:25 +02:00
Raul Ochoa
db9bfacf07 Extract stats client creation 2017-03-30 16:13:17 +02:00
Raul Ochoa
d48a901533 Merge branch 'master' into node-v6 2017-01-12 19:49:56 +01:00
Javier Torres
62a67f148c Correctly accept custom options for formats in ogr.js 2017-01-11 19:07:50 +01:00
Javier Torres
c67a2e7d09 Use cartodb_id as FID column in gpkg 2017-01-11 19:07:50 +01:00
Daniel García Aubert
ef124d689e Handle query end in error listener since pg no longer emits end if and error occurs 2016-12-07 16:22:21 +01:00
Daniel García Aubert
99c7a6e4f9 Raise job query size to 16kb 2016-11-07 16:50:44 +01:00
Daniel García Aubert
4a64d37c6c Add work in progrees list endpoint 2016-10-28 15:08:42 +02:00
Raul Ochoa
cdde1be29e Re-use redis pool as much as possible 2016-10-17 15:02:34 +02:00
Raul Ochoa
e401c01d78 Only log on non-test environments 2016-10-12 01:40:14 +02:00
Raul Ochoa
e4b1711e8e pub/sub package 2016-10-11 18:28:46 +02:00
Raul Ochoa
66820a67bb Make possible to specify a name for batch 2016-10-10 19:46:07 +02:00
Raul Ochoa
42a1f3ad4c Accept multipart requests 2016-10-05 17:29:16 +02:00
Raul Ochoa
6309318534 Use body-parser from old connect module 2016-10-05 14:22:44 +02:00
Raul Ochoa
025b3f3cc7 Increase body limit to 20mb 2016-10-05 11:14:06 +02:00
Raul Ochoa
2a2a54a073 DRY in job response handler 2016-10-04 16:07:13 +02:00
Raul Ochoa
19c9bec633 Callback with job 2016-10-04 15:57:13 +02:00
Raul Ochoa
05cbd55b95 Use db host from request's context 2016-10-04 15:50:39 +02:00
Raul Ochoa
c32a2199fa Use request bootstrapper to add host header 2016-10-04 15:43:19 +02:00
Raul Ochoa
7b7d651d8f DRY while authenticating requests 2016-10-04 15:40:56 +02:00
Raul Ochoa
20f50d988e Use user middleware in job controller 2016-10-04 15:19:31 +02:00
Raul Ochoa
b139b9ab21 Add context object in all requests 2016-10-04 15:12:46 +02:00
Raul Ochoa
2edc7505e7 Do not condition req.profiler 2016-10-04 15:08:31 +02:00
Raul Ochoa
cb9db0c4cb Make request.profiler always available 2016-10-04 14:40:04 +02:00
Raul Ochoa
3b6bc14d17 Increment errors on err 2016-10-04 13:19:29 +02:00
Raul Ochoa
bf1a67780d Merge branch 'master' into express-4.x 2016-09-30 17:37:30 +02:00
Daniel García Aubert
59e5c10f85 Set batch queries log path 2016-09-29 15:37:14 +02:00
Raul Ochoa
2aa1f045ff Use .status() API 2016-09-26 18:10:20 +02:00
Raul Ochoa
abc2f130c9 Migrate to express 4.x series 
- Remove express logger
 - Error handler responds with application/[json|javascript]
 - Fix all tests relying on res.headers
 - assert.response based on request module
 2016-09-26 18:09:27 +02:00
Raul Ochoa
f92d50cccf Rename from app to server 
Removes app_root dependency in requires
 2016-09-14 20:54:53 +02:00
Raul Ochoa
0626d80e24 Removes support for optional rollbar logging 2016-09-14 19:22:31 +02:00
Raul Ochoa
6f4fb931f7 Remove user indexer from app 2016-08-30 19:04:36 +02:00
Raul Ochoa
ba0f2f1066 Remove endpoint to retrieve jobs list 2016-08-30 18:49:01 +02:00
Raul Ochoa
704ba110ba Log job creation 2016-08-30 18:43:09 +02:00
Raul Ochoa
ad0d101bfd Remove patch/put endpoints for jobs 2016-08-30 17:43:34 +02:00
Daniel García Aubert
395635d246 Merge branch 'master' into fix-publisher-connection 2016-07-11 17:49:11 +02:00
Raul Ochoa
3ebbd9f7c4 Skip tables with no updated_at registered in cdb_tablemetadata 2016-07-11 16:39:12 +02:00
Daniel García Aubert
ccff602bbf Merge branch 'master' into fix-publisher-connection 2016-07-07 16:07:41 +02:00
Raul Ochoa
368fe2403e Allow to setup more than one domain to validate oauth against 2016-07-07 14:20:36 +02:00
Daniel García Aubert
5eaad4d5d9 Uses redis-mpool for pubsub in Batch API 2016-07-07 14:14:46 +02:00
Daniel García Aubert
a1f31df92e Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job 2016-06-29 18:29:53 +02:00
Daniel García Aubert
282da58ffe Set default value for statsd-client in job-controller to avoid check it every time it's going to be used 2016-05-30 12:27:19 +02:00
Daniel García Aubert
976bf5b039 Implemented profiling for job-runner and job-controller 2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577 Merge branch 'master' into batch-add-profile 2016-05-26 17:46:52 +02:00
Raul Ochoa
a91c7a6a88 Increase job payload size to 8kb 2016-05-24 15:41:18 +02:00
Raul Ochoa
23228b2d73 Payload size validates multiple queries and fallback queries 
It uses a middleware to check the body size
 2016-05-24 14:28:00 +02:00
Daniel García Aubert
64ad284c9c WIP: adding metrics to Batch API 2016-05-24 11:19:00 +02:00
Daniel
8c8da8e39c Merge pull request #301 from CartoDB/job-model-refactor 
Job model refactor
 2016-05-18 11:38:41 +02:00
Daniel García Aubert
d2d3ba8159 Passed tests 2016-05-16 01:22:47 +02:00
Daniel García Aubert
cc7dd7a0d2 Job model refactor 2016-05-13 18:50:55 +02:00
Raul Ochoa
4e3238c2e8 Fix control flow when using OGR with _needSRS=true and empty queries 
Continue on empty results as srid and geom type are not critical
when there are no results

Fixes #299
 2016-05-12 17:21:07 +02:00
Juan Ignacio Sánchez Lara
e979b79f49 gpkg format #291 2016-05-10 18:42:23 +02:00
Juan Ignacio Sánchez Lara
c82cafb1ed Geopackage format support #291 2016-04-27 16:56:06 +02:00
Daniel García Aubert
5de931daa5 Improver error message in job max payload limit. 2016-04-25 11:18:30 +02:00
Daniel García Aubert
1dcbb1afdf Improved error message 2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124 Fixed typos 2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944 Implemented limit to query size in Batch API. 2016-04-18 15:30:16 +02:00
Daniel García Aubert
056f22b156 Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series 2016-03-18 14:57:18 +01:00
Raul Ochoa
66cd1f400f Skip query tables cache for authenticated requests 2016-03-10 19:20:56 +01:00
Raul Ochoa
5ebc8e43d5 Start hits=0 as first time is not a hit 2016-03-08 14:50:08 +01:00
Raul Ochoa
b8460d033e Add a cache decorator over QueryTables 2016-03-08 14:48:56 +01:00
Raul Ochoa
dbe033112d Rethrow error but do not fail when getting affected tables 2016-03-08 14:00:31 +01:00
Alejandro Martínez
cd593e12ce Join all Surrogate-Keys in one line 2016-02-24 11:43:43 +01:00
Alejandro Martínez
7d024dbf3f Fix include in query_controller 2016-02-22 19:10:59 +01:00
Alejandro Martínez
5411aa3a01 Use Date.now() fallback value for getLastUpdatedTime 2016-02-22 19:08:04 +01:00
Alejandro Martínez
2c4b659100 Properly discard errors fetching updated tables 2016-02-22 19:07:41 +01:00
Alejandro Martínez
5e06711b4b Use node-cartodb-querytables library 2016-02-22 12:24:05 +01:00
Raul Ochoa
15f90c1a78 QueryTablesApi only caches affected tables and always retrieve last modification 2016-02-02 01:16:24 +01:00
Raul Ochoa
9deab814c9 Skip query-tables-api for authenticated requests 2016-02-01 16:53:28 +01:00
Raul Ochoa
24ecf130a1 Namespace queries so two users don't share results 
For instance 'select * from table' should result in different affected
tables (schema+table) depending on the database.
 2016-01-28 15:38:08 +01:00