Simon Martín
|
5bed04a38c
|
fix gzip problem with req close event
|
2018-05-25 15:34:30 +02:00 |
|
Simon Martín
|
fe3bd4fd37
|
handling errors from request in COPYfrom
|
2018-05-25 14:33:54 +02:00 |
|
Simon Martín
|
e2bba967f9
|
handling copyto errors
|
2018-05-24 19:48:24 +02:00 |
|
Simon Martín
|
fca6ee8232
|
changing name of dataIngestionLogPath
|
2018-05-23 17:32:44 +02:00 |
|
Simon Martín
|
b2a36eb556
|
copy metrics with BunyanLogger
|
2018-05-23 17:25:46 +02:00 |
|
Simon Martín
|
5ba7dca79c
|
copyfrom metrics to kibana
|
2018-05-23 10:30:37 +02:00 |
|
Paul Ramsey
|
f77707f6da
|
Fix reponse copy from
|
2018-05-22 10:48:11 -04:00 |
|
Paul Ramsey
|
a4654f82b2
|
Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q
|
2018-05-22 10:37:50 -04:00 |
|
Paul Ramsey
|
ed696a96ec
|
Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q
|
2018-05-22 10:23:04 -04:00 |
|
Simon Martín
|
6d73d97ae2
|
copyFrom metrics to statsd
|
2018-05-22 16:02:14 +02:00 |
|
Simon Martín
|
e347985465
|
changing query parameter name from sql to q
unify query validatrion
|
2018-05-22 15:42:57 +02:00 |
|
Simon Martín
|
b9474e7fc3
|
total_rows in copyto metrics
|
2018-05-22 15:29:08 +02:00 |
|
Simon Martín
|
185f708d5b
|
small style refactor
|
2018-05-22 14:15:55 +02:00 |
|
Simon Martín
|
f01191472b
|
refactoring copy controller middlewares
|
2018-05-22 11:56:50 +02:00 |
|
Simon Martín
|
433bd01c27
|
copyto metrics
|
2018-05-22 11:54:10 +02:00 |
|
Simon Martín
|
81be15fbc3
|
adding format to copy metrics
|
2018-05-21 19:13:44 +02:00 |
|
Paul Ramsey
|
64fc0c32e3
|
Change from 'sql' as query parameter to 'q', aping existing sql api
|
2018-05-21 07:47:16 -07:00 |
|
Simon Martín
|
816779fefd
|
erratum
|
2018-05-18 17:32:59 +02:00 |
|
Simon Martín
|
82c9fec18a
|
copy from metrics
|
2018-05-18 17:31:29 +02:00 |
|
Simon Martín
|
bc8a867334
|
jshinty happy
|
2018-05-11 14:51:43 +02:00 |
|
Simon Martín
|
f2acc8f653
|
forgotten log
|
2018-05-11 14:16:26 +02:00 |
|
Simon Martín
|
eee8bc2674
|
moving copyto headers to correct position
|
2018-05-11 14:12:23 +02:00 |
|
Simon Martín
|
791967877c
|
addining gzip support to copyfrom
|
2018-05-11 13:33:54 +02:00 |
|
Simon Martín
|
f749798ca5
|
validating inputs
|
2018-05-11 10:38:33 +02:00 |
|
Paul Ramsey
|
8f211f905c
|
Remove wrong examples
|
2018-05-10 12:21:33 -07:00 |
|
Paul Ramsey
|
f0021264d2
|
Change from form-multipart to POST with chunked upload
|
2018-05-10 12:11:55 -07:00 |
|
Simon Martín
|
82d14ab98f
|
busboy configuration
|
2018-05-08 13:15:19 +02:00 |
|
Simon Martín
|
7c4409bcf5
|
from var to const/let
|
2018-05-08 13:10:25 +02:00 |
|
Simon Martín
|
c40ab801b1
|
refactoring handleCopyTo
|
2018-05-08 13:08:29 +02:00 |
|
Simon Martín
|
b036b876ff
|
using Busboy instead of multer
|
2018-05-08 12:52:33 +02:00 |
|
Simon Martín
|
2b066eadfa
|
removing unused dependencies
|
2018-05-04 15:35:23 +02:00 |
|
Simon Martín
|
02238fefe1
|
moving body-parser from global to routes of query and job
|
2018-05-04 15:15:37 +02:00 |
|
Simon Martín
|
4322ccdd09
|
adding new endpoints to rate limits
|
2018-05-03 18:50:13 +02:00 |
|
Simon Martín
|
af4c5906c8
|
adding /sql tro the endpoints
|
2018-05-03 18:31:49 +02:00 |
|
Paul Ramsey
|
36bdf84f71
|
Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream
|
2018-04-27 15:56:13 +02:00 |
|
Paul Ramsey
|
e2d8896307
|
Typo in curl example
|
2018-04-27 12:32:47 +02:00 |
|
Paul Ramsey
|
5f9cc37dba
|
Jshint change, DRY the content-disposition handler
|
2018-04-27 12:19:53 +02:00 |
|
Paul Ramsey
|
fe52af71ac
|
Add documentation SQL API copyfrom/copyto end points,
and make small modifications arising from that task
|
2018-04-27 11:55:06 +02:00 |
|
Paul Ramsey
|
c78cac9bd6
|
Use the correct database credentials, so we obey security
|
2018-04-26 10:45:52 +02:00 |
|
Paul Ramsey
|
f13028b497
|
More jshint changes
|
2018-04-26 10:20:21 +02:00 |
|
Paul Ramsey
|
cc5fa5c6ce
|
Changes suggested by jshint
|
2018-04-26 10:04:38 +02:00 |
|
Paul Ramsey
|
ea66076255
|
Remove 'console.debug', try and get travis tests clean
|
2018-04-26 09:40:13 +02:00 |
|
Paul Ramsey
|
a98781d335
|
Working code for /copyfrom (streaming) and /copyto (streaming)
|
2018-04-25 18:37:04 +02:00 |
|
Paul Ramsey
|
b59bda5780
|
Get working copyto implementation in place. Many test cases
needed still and thoughts on how to communicate errors back
that are maybe nicer than stack messages?
|
2018-04-24 15:55:20 +02:00 |
|
Paul Ramsey
|
4914100205
|
CopyFrom works, but still needs a decent return payload and
a lot of work on returning useful information for error cases (post empty input,
db errors returned more nicely? etc)
|
2018-04-24 13:07:57 +02:00 |
|
Paul Ramsey
|
72bce5732b
|
WIP on /copyfrom
|
2018-04-24 11:26:15 +02:00 |
|
Simon Martín
|
985f9ef7f5
|
pgEntitiesAccessValidator
|
2018-04-23 18:17:44 +02:00 |
|
Rafa de la Torre
|
2229d0ee57
|
Fix jshint issues
|
2018-04-13 16:36:43 +02:00 |
|
Paul Ramsey
|
0161696627
|
Update copy_controller.js
|
2018-04-13 05:43:23 -07:00 |
|
Paul Ramsey
|
d54e2f5a07
|
Implementation including multer, custom storage engine, and pg-copy,
but without turning over pg-copy, and demonstrating the missing 'sql'
parameter in the custom storage engine.
|
2018-04-12 12:25:28 -07:00 |
|
Simon Martín
|
80818c3cbb
|
fix userLimits reference
|
2018-03-20 16:20:56 +01:00 |
|
Simon Martín
|
581bd0dd89
|
Merge branch 'master' into rateLimits
|
2018-03-14 12:47:45 +01:00 |
|
Daniel García Aubert
|
8dea7615c4
|
Use user's port instead of the generic one
|
2018-03-13 19:08:51 +01:00 |
|
Daniel García Aubert
|
328d6f253e
|
Honor batch api port
|
2018-03-13 18:49:01 +01:00 |
|
Simon Martín
|
b6ff37d33f
|
changing param name
|
2018-03-02 19:57:33 +01:00 |
|
Simon Martín
|
7a6ef637ae
|
make jshint happy
|
2018-03-01 14:47:34 +01:00 |
|
Simon Martín
|
f3adad57aa
|
fix assignment
|
2018-03-01 13:15:32 +01:00 |
|
Simon Martín
|
779ab3b83f
|
adding rate limit middleware to controllers
|
2018-03-01 12:19:57 +01:00 |
|
Simon Martín
|
2834975d14
|
controllers using user limits service
|
2018-03-01 11:31:35 +01:00 |
|
Daniel García Aubert
|
f85bdf53a5
|
Missing middleware
|
2018-02-23 13:19:26 +01:00 |
|
Daniel García Aubert
|
47b54612c7
|
Place send-response-middleware to the very end of middleware stack
|
2018-02-23 13:03:56 +01:00 |
|
Daniel García Aubert
|
700c64bba3
|
Extract to a middleware user timeout limit from user-database-services
|
2018-02-22 12:45:55 +01:00 |
|
Daniel García Aubert
|
94c5bd11df
|
Split authorization middleware, it was actually doing two things: authorize and get database connection params
|
2018-02-22 12:22:39 +01:00 |
|
Daniel García Aubert
|
8730b5c517
|
Read if the request is authenticated through res.locals
|
2018-02-22 12:04:05 +01:00 |
|
Daniel García Aubert
|
b399abee18
|
Move authorization to auth-api and extract it from user-database-service
|
2018-02-22 11:46:34 +01:00 |
|
Daniel García Aubert
|
9088cbf150
|
Improve naming
|
2018-02-21 13:51:28 +01:00 |
|
Daniel García Aubert
|
ba113d8628
|
Improve naming
|
2018-02-20 16:43:43 +01:00 |
|
Daniel García Aubert
|
33baa399ae
|
Create function to compose middlewares to perfom job operations
|
2018-02-20 16:25:16 +01:00 |
|
Daniel García Aubert
|
864ec30058
|
Remove duplication
|
2018-02-20 13:22:33 +01:00 |
|
Daniel García Aubert
|
d4d54648d4
|
Improve naming
|
2018-02-20 13:14:28 +01:00 |
|
Daniel García Aubert
|
8c13add7a8
|
Rename middleware
|
2018-02-19 15:49:17 +01:00 |
|
Daniel García Aubert
|
939443ef81
|
Unify credentials and authenticated request midllewares
|
2018-02-19 15:44:28 +01:00 |
|
Daniel García Aubert
|
7898b49e59
|
Extract profiler middleware to used in query and job controllers
|
2018-02-19 15:13:36 +01:00 |
|
Daniel García Aubert
|
85fbd7c6b2
|
Create middlewares to initialize and finish profiling
|
2018-02-19 14:42:52 +01:00 |
|
Daniel García Aubert
|
75c2d85dbb
|
Use authenticated middleware in query controller
|
2018-02-19 13:24:44 +01:00 |
|
Daniel García Aubert
|
114070ef96
|
Rename middleware
|
2018-02-19 12:37:19 +01:00 |
|
Daniel García Aubert
|
adfe3dd52b
|
Remove comment
|
2018-02-19 11:10:39 +01:00 |
|
Daniel García Aubert
|
ea310db62c
|
Cosmetic chenges
|
2018-02-19 11:04:28 +01:00 |
|
Daniel García Aubert
|
edd382fb8f
|
Check user is the same user that sends the request when basic-auth is provided
|
2018-02-16 18:21:06 +01:00 |
|
Daniel García Aubert
|
2cadfe6f18
|
Add support for Basic Auth
|
2018-02-16 10:46:58 +01:00 |
|
Daniel García Aubert
|
756fbe42c9
|
Use apikey midlleware
|
2018-02-15 17:24:04 +01:00 |
|
Daniel García Aubert
|
67f2a85abd
|
Add database credentials to jobs
|
2018-02-15 12:30:04 +01:00 |
|
Daniel García Aubert
|
b10937a8fb
|
Use res.locals instead of req.context to share object among middlewares
|
2017-11-24 15:49:25 +01:00 |
|
Daniel García Aubert
|
c2d1a4653e
|
Improve naming for connection database params holder
|
2017-11-24 15:45:49 +01:00 |
|
Daniel García Aubert
|
b1112e2bb1
|
Remove unneeded header
|
2017-09-20 16:48:22 +02:00 |
|
Daniel García Aubert
|
980e762ab7
|
Skip user auth middlewares
|
2017-09-20 16:35:33 +02:00 |
|
Daniel García Aubert
|
0597a3b8a6
|
Change job work in progress endpoint
|
2017-09-20 15:46:29 +02:00 |
|
Raul Ochoa
|
32154b67c6
|
Merge branch 'master' into upgrade-cartodb-psql
|
2017-08-11 11:57:16 +02:00 |
|
Daniel García Aubert
|
01a71ee60e
|
Apply user timeout to ogr2ogr command
|
2017-08-09 12:50:16 +02:00 |
|
Raul Ochoa
|
3e536b822e
|
Remove unnecessary destroyOnError option
|
2017-08-08 18:21:10 +02:00 |
|
Daniel García Aubert
|
99c7a6e4f9
|
Raise job query size to 16kb
|
2016-11-07 16:50:44 +01:00 |
|
Daniel García Aubert
|
4a64d37c6c
|
Add work in progrees list endpoint
|
2016-10-28 15:08:42 +02:00 |
|
Raul Ochoa
|
e401c01d78
|
Only log on non-test environments
|
2016-10-12 01:40:14 +02:00 |
|
Raul Ochoa
|
2a2a54a073
|
DRY in job response handler
|
2016-10-04 16:07:13 +02:00 |
|
Raul Ochoa
|
19c9bec633
|
Callback with job
|
2016-10-04 15:57:13 +02:00 |
|
Raul Ochoa
|
05cbd55b95
|
Use db host from request's context
|
2016-10-04 15:50:39 +02:00 |
|
Raul Ochoa
|
c32a2199fa
|
Use request bootstrapper to add host header
|
2016-10-04 15:43:19 +02:00 |
|
Raul Ochoa
|
7b7d651d8f
|
DRY while authenticating requests
|
2016-10-04 15:40:56 +02:00 |
|
Raul Ochoa
|
20f50d988e
|
Use user middleware in job controller
|
2016-10-04 15:19:31 +02:00 |
|
Raul Ochoa
|
2edc7505e7
|
Do not condition req.profiler
|
2016-10-04 15:08:31 +02:00 |
|
Raul Ochoa
|
3b6bc14d17
|
Increment errors on err
|
2016-10-04 13:19:29 +02:00 |
|
Raul Ochoa
|
2aa1f045ff
|
Use .status() API
|
2016-09-26 18:10:20 +02:00 |
|
Raul Ochoa
|
ba0f2f1066
|
Remove endpoint to retrieve jobs list
|
2016-08-30 18:49:01 +02:00 |
|
Raul Ochoa
|
704ba110ba
|
Log job creation
|
2016-08-30 18:43:09 +02:00 |
|
Raul Ochoa
|
ad0d101bfd
|
Remove patch/put endpoints for jobs
|
2016-08-30 17:43:34 +02:00 |
|
Raul Ochoa
|
3ebbd9f7c4
|
Skip tables with no updated_at registered in cdb_tablemetadata
|
2016-07-11 16:39:12 +02:00 |
|
Daniel García Aubert
|
282da58ffe
|
Set default value for statsd-client in job-controller to avoid check it every time it's going to be used
|
2016-05-30 12:27:19 +02:00 |
|
Daniel García Aubert
|
976bf5b039
|
Implemented profiling for job-runner and job-controller
|
2016-05-26 19:44:59 +02:00 |
|
Daniel García Aubert
|
ea00c22577
|
Merge branch 'master' into batch-add-profile
|
2016-05-26 17:46:52 +02:00 |
|
Raul Ochoa
|
a91c7a6a88
|
Increase job payload size to 8kb
|
2016-05-24 15:41:18 +02:00 |
|
Raul Ochoa
|
23228b2d73
|
Payload size validates multiple queries and fallback queries
It uses a middleware to check the body size
|
2016-05-24 14:28:00 +02:00 |
|
Daniel García Aubert
|
64ad284c9c
|
WIP: adding metrics to Batch API
|
2016-05-24 11:19:00 +02:00 |
|
Daniel García Aubert
|
cc7dd7a0d2
|
Job model refactor
|
2016-05-13 18:50:55 +02:00 |
|
Daniel García Aubert
|
5de931daa5
|
Improver error message in job max payload limit.
|
2016-04-25 11:18:30 +02:00 |
|
Daniel García Aubert
|
1dcbb1afdf
|
Improved error message
|
2016-04-18 16:24:52 +02:00 |
|
Daniel García Aubert
|
790b9c6124
|
Fixed typos
|
2016-04-18 15:44:48 +02:00 |
|
Daniel García Aubert
|
7cc7482944
|
Implemented limit to query size in Batch API.
|
2016-04-18 15:30:16 +02:00 |
|
Daniel García Aubert
|
056f22b156
|
Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series
|
2016-03-18 14:57:18 +01:00 |
|
Raul Ochoa
|
66cd1f400f
|
Skip query tables cache for authenticated requests
|
2016-03-10 19:20:56 +01:00 |
|
Raul Ochoa
|
b8460d033e
|
Add a cache decorator over QueryTables
|
2016-03-08 14:48:56 +01:00 |
|
Raul Ochoa
|
dbe033112d
|
Rethrow error but do not fail when getting affected tables
|
2016-03-08 14:00:31 +01:00 |
|
Alejandro Martínez
|
cd593e12ce
|
Join all Surrogate-Keys in one line
|
2016-02-24 11:43:43 +01:00 |
|
Alejandro Martínez
|
7d024dbf3f
|
Fix include in query_controller
|
2016-02-22 19:10:59 +01:00 |
|
Alejandro Martínez
|
5411aa3a01
|
Use Date.now() fallback value for getLastUpdatedTime
|
2016-02-22 19:08:04 +01:00 |
|
Alejandro Martínez
|
2c4b659100
|
Properly discard errors fetching updated tables
|
2016-02-22 19:07:41 +01:00 |
|
Alejandro Martínez
|
5e06711b4b
|
Use node-cartodb-querytables library
|
2016-02-22 12:24:05 +01:00 |
|
Raul Ochoa
|
15f90c1a78
|
QueryTablesApi only caches affected tables and always retrieve last modification
|
2016-02-02 01:16:24 +01:00 |
|
Raul Ochoa
|
9deab814c9
|
Skip query-tables-api for authenticated requests
|
2016-02-01 16:53:28 +01:00 |
|
Raul Ochoa
|
a654c90b6d
|
X-Cache-Channel generation only requires affected tables
mayWrite is redundant, and it's not its responsibility
|
2016-01-28 14:18:38 +01:00 |
|
Raul Ochoa
|
131d798653
|
Change interface to return also authenticated connection params
|
2016-01-28 14:18:10 +01:00 |
|
Raul Ochoa
|
235e7f9bc1
|
Rename method to match returning connection params behaviour
|
2016-01-28 14:14:31 +01:00 |
|
Raul Ochoa
|
023d06fd19
|
Affected tables and last modified behind QueryTablesApi
It handles the internal cache and exposes an unified result
|
2016-01-28 14:08:18 +01:00 |
|
Raul Ochoa
|
7dad90a108
|
Remove unused dependencies from job controller
|
2016-01-28 13:10:21 +01:00 |
|
Raul Ochoa
|
9881a6f7d4
|
Use readonly flag to run affected tables query
|
2016-01-28 12:33:00 +01:00 |
|
Daniel García Aubert
|
f2ea01f627
|
Removed unnecessary check in job controller
|
2016-01-25 14:50:43 +01:00 |
|
Daniel García Aubert
|
79e626a71e
|
Added cors middleware, decoupled query and job controllers from its dependencies
|
2016-01-21 17:24:48 +01:00 |
|
Daniel García Aubert
|
048d2cd0ea
|
Refactored user database service ans updated controllers to use it.
|
2016-01-21 16:17:17 +01:00 |
|
Daniel García Aubert
|
f42727212b
|
Removed unnecessary checkAborted function in job controller and changed signature for getUserDatabase.
|
2016-01-21 15:05:46 +01:00 |
|
Daniel García Aubert
|
2913aed697
|
Placed job controller endpoit under /api/vX/sql
|
2016-01-18 20:12:44 +01:00 |
|
Daniel García Aubert
|
2d13bde301
|
Implemented uses cases test for Batch API
|
2016-01-11 20:14:15 +01:00 |
|
Daniel García Aubert
|
20f00d58d9
|
Refactored batch service to avoid event noise, doing in callback way
|
2016-01-08 15:47:59 +01:00 |
|
Daniel García Aubert
|
e6a4e0f7eb
|
Returned 201 HTTP code in job creation
Fixed bug listing jobs, now user job index is cleaned before respond to client
|
2016-01-07 12:07:30 +01:00 |
|
Daniel García Aubert
|
40d0161218
|
Fixed bug in authentication for PUT methods in Job Controller
|
2016-01-05 13:54:18 +01:00 |
|
Daniel García Aubert
|
246655de94
|
Changed redis data structure for users jobs
|
2016-01-04 19:08:13 +01:00 |
|
Daniel García Aubert
|
35650985db
|
Implemented job modification whether job is pending
|
2015-12-31 15:42:31 +01:00 |
|
Daniel García Aubert
|
ada39d84b8
|
Implemented job cancelation in Batch API
|
2015-12-30 20:16:18 +01:00 |
|
Daniel García Aubert
|
851c393f75
|
Refactored batch manager avoiding concurrence issues
|
2015-12-29 10:19:10 +01:00 |
|
Daniel García Aubert
|
4dcd7c0a50
|
Fixed typo and prepared for listing user's job
|
2015-12-24 17:47:41 +01:00 |
|
Daniel García Aubert
|
c4f96b4a59
|
Implemented support for listing jobs for one user
|
2015-12-24 17:42:49 +01:00 |
|
Daniel García Aubert
|
ecb76ca1de
|
Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs.
|
2015-12-23 17:29:11 +01:00 |
|