cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,044 Commits 1 Branch 118 Tags 9.7 MiB
f71c7bc3a8
Commit Graph

962 Commits

Author SHA1 Message Date
Eneko Lakasta
1bf8a06ec2 Rephrase error message 2018-06-29 13:19:32 +02:00
Eneko Lakasta
99058e68af add space to message 2018-06-27 15:53:51 +02:00
Eneko Lakasta
2bb28b5578 remove line break in the error message 2018-06-27 15:43:06 +02:00
Eneko Lakasta
879e85d07f add 'SQL query timeout expired error.' to DB timeout limit error message. 2018-06-27 15:17:38 +02:00
Simon Martín
59dd495a87 small style details 2018-06-22 10:50:39 +02:00
Simon Martín
06c40a396b copy from: events is series 2018-06-21 15:58:23 +02:00
Simon Martín
195a4866c3 managing gzip pipe with PassThrough 2018-06-21 15:54:42 +02:00
Simon Martín
f5641a1873 remove unneeded this.connectionClosedByClient 2018-06-21 15:18:26 +02:00
Simon Martín
fae18fb242 using getPGStream 2018-06-21 15:17:32 +02:00
Simon Martín
823e3abcfe unify to and from method in only one getPGStream 2018-06-21 15:16:47 +02:00
Simon Martín
3ede1ea9f1 unify streams 2018-06-21 14:53:07 +02:00
Simon Martín
f4651cadae using this.copyFromStream 2018-06-21 14:49:38 +02:00
Simon Martín
9970052e1b copy from: removing copyFromStream dependency 2018-06-21 14:38:22 +02:00
Simon Martín
01eff0a5a2 copy from: removing done dependecy 2018-06-21 14:36:40 +02:00
Simon Martín
a15c2e58aa copy from: manage error in 1 point 2018-06-21 14:36:03 +02:00
Simon Martín
d17a5608fb copy from: unneeded pgstream.end in req error 2018-06-21 14:32:36 +02:00
Simon Martín
5a28d36d69 copy from: pgstream events in series 2018-06-21 14:29:19 +02:00
Simon Martín
925115e8c1 copy from: remove client dependecy 2018-06-21 14:25:54 +02:00
Simon Martín
327aaf3450 jshint happy 2018-06-21 13:41:13 +02:00
Simon Martín
bafcb8051f StreamCopy getResults refactor 
- changing get Result to getRowCount
- ensuring to and from independecy in StreamCopy
 2018-06-21 13:39:33 +02:00
Simon Martín
2cecc54cb2 copy to: removing copyToStream dependecy 2018-06-21 13:21:23 +02:00
Simon Martín
e39a9b2bf7 copy to: manage error in 1 point 2018-06-21 13:13:37 +02:00
Simon Martín
1dac5355d1 copy to: remove unneeded done and removing it as a dependency 2018-06-21 13:13:16 +02:00
Simon Martín
42a94a3b6c copy to: removing unnedeed client dependecy 2018-06-21 13:01:59 +02:00
Simon Martín
e6448a3255 copy to: cancelQuery to streamCopy 2018-06-21 13:00:57 +02:00
Simon Martín
39dfcb1f9b copy to events is series 2018-06-21 12:50:50 +02:00
Simon Martín
5fbd6e8e3a req on error is neccesary when connection is canceled 2018-06-21 12:49:06 +02:00
Simon Martín
3da32308ce jshint happy 2018-06-21 12:30:49 +02:00
Simon Martín
6955d254f8 Merge branch 'master' into pgcopy-stream-refactor2-dgaubert 2018-06-21 12:19:16 +02:00
Simon Martín
3d30f1f762 trailing white spaces 2018-06-19 10:34:52 +02:00
Simon Martín
30402f2e89 remove unused statsClient 2018-06-19 10:04:17 +02:00
Simon Martín
7727a9d506 data ingestion logger created in controller and added to app 2018-06-18 18:48:11 +02:00
Simon Martín
32dca8dfdb reopenFileStreams to parent class 2018-06-18 18:47:12 +02:00
Simon Martín
8f727fc15a cdb-user in logs 2018-06-15 13:30:42 +02:00
Simon Martín
8a450a862c recover the metrics header (needed for tests purposes) 2018-06-15 13:25:47 +02:00
Simon Martín
c7e01f2ed6 success in copy log 2018-06-14 18:40:06 +02:00
Simon Martín
9097cbe16e adding log of the error when headers are already sent 2018-06-14 18:29:50 +02:00
Simon Martín
39e5395edc change gzip to isGzip 2018-06-14 18:29:03 +02:00
Simon Martín
4cb7de7318 changing let for const 2018-06-14 10:51:05 +02:00
Simon Martín
b091a375fc fix ensuring right validation error 2018-06-14 10:49:36 +02:00
Simon Martín
1f31b8e2ae removing old copy logs 2018-06-14 10:45:23 +02:00
Daniel García Aubert
9299fe51ce Add todo 2018-06-14 01:01:05 +02:00
Daniel García Aubert
8eab164e4e Be more one liner 2018-06-14 00:54:03 +02:00
Daniel García Aubert
9c8d802f75 Chain response event callbacks after .pipe() in order to improve readability in copyto controller 2018-06-14 00:31:08 +02:00
Daniel García Aubert
508d688b9d Use request 'close' event when the request is closed unexpectedly by the client 2018-06-14 00:16:06 +02:00
Daniel García Aubert
37a079363a Do not extend from EventEmitter 2018-06-13 20:12:56 +02:00
Daniel García Aubert
bd8e4be596 Do not emit to regist the number of rows processed in copyfrom 2018-06-13 20:11:39 +02:00
Daniel García Aubert
5655f26fbd Do not emit to regist the number of rows processed in copyto 2018-06-13 20:06:35 +02:00
Daniel García Aubert
eb4ba60ba0 Pass the error to the pg connection's callback 2018-06-13 19:56:56 +02:00
Daniel García Aubert
e6a331433a Handle error when client disconnects in copyto requests 2018-06-13 19:54:34 +02:00
Daniel García Aubert
6aba1ea307 Handle error when sql injected in pgstream fails (e.g: unexisting table) 2018-06-13 19:52:53 +02:00
Daniel García Aubert
2792d95c41 Do not handle stream unpiping when it wasn't even created 2018-06-13 18:30:05 +02:00
Daniel García Aubert
b9971d94dd Do not use the regular callback in streams events 2018-06-13 18:28:34 +02:00
Daniel García Aubert
7d1a0abadd Preventing callbacks from accidentally being called twice 2018-06-13 18:26:58 +02:00
Eneko Lakasta
f5d56a2253
FIX add return to callback call 2018-06-13 13:17:01 +02:00
Simon Martín
270daae579 removing forgotten callback 2018-06-12 18:40:51 +02:00
Simon Martín
409273bafe removing 2nd calback using events 2018-06-12 18:39:50 +02:00
Simon Martín
19aae3c40f manage pgstrem error on stream copy class 2018-06-12 17:04:44 +02:00
Simon Martín
ba2f85421d StreamCopy to class 2018-06-12 16:56:18 +02:00
Simon Martín
8a15b2f8d1 Merge branch 'master' into pgcopy-stream-refactor2 2018-06-12 15:21:28 +02:00
Simon Martín
4d99872975 return in next err 2018-06-12 15:18:28 +02:00
Simon Martín
b9a0fa78d2 adding metadataBackend to user middleware in copycontroller 2018-06-11 14:55:37 +02:00
Simon Martín
1174499dcf Merge branch 'master' into pgcopy-stream 2018-06-11 14:38:52 +02:00
Simon Martín
df63e0eedb adding log of the error when headers are already sent 2018-06-11 13:19:12 +02:00
Simon Martín
2995676adb let to const 2018-06-11 12:56:16 +02:00
Simon Martín
4fab96cb3a rename gzip to isGzip 2018-06-11 12:55:30 +02:00
Simon Martín
87fd9db1fe remove uneeded comment 2018-06-11 12:43:35 +02:00
Simon Martín
2581ddbfb4 better formatting 2018-06-08 17:09:28 +02:00
Simon Martín
b63806f8de cleaning stream copy 2018-06-08 17:04:28 +02:00
Simon Martín
6a9e7d7c14 forgotten ) 2018-06-08 17:04:10 +02:00
Simon Martín
7266b3c708 one logger 2018-06-08 17:02:31 +02:00
Simon Martín
014f0a1491 refactor of copy to PoC 2018-06-08 16:58:32 +02:00
Simon Martín
f7454228c6 refactor of copy from PoC 2018-06-08 16:50:12 +02:00
Simon Martín
e259a51623 unifiyng to see 'the forest' 2018-06-08 15:03:21 +02:00
Simon Martín
8e8ccc9c67 removing old metrics in copy 2018-06-08 13:33:17 +02:00
Simon Martín
c62ef15dda using StreamCopy in controller 2018-06-08 11:15:23 +02:00
Simon Martín
732c229e63 creating StreamCopy class 2018-06-08 11:09:51 +02:00
Simon Martín
2bcea0484a rename BunyanLogger to Logger 2018-06-08 10:59:34 +02:00
Eneko Lakasta
440ba8c840 make function checkUserExists node callback pattern compilant 2018-06-06 15:48:22 +02:00
Eneko Lakasta
340e55ea46 reduce error info 2018-06-06 15:23:53 +02:00
Eneko Lakasta
ca8ae503ac Merge branch 'master' into remove-auth-fallback 2018-06-05 16:19:36 +02:00
Eneko Lakasta
eab3d289b6 refactor authenticated to authorizationLevel 2018-06-05 13:21:56 +02:00
Eneko Lakasta
da08e42921 refactor forceToBeAuthenticated to forceToBeMaster 2018-06-05 13:16:36 +02:00
Simon Martín
66af518deb jshint 2018-06-05 13:16:14 +02:00
Eneko Lakasta
0207b67d5f refactor forceToBeAuthenticated to forceToBeMaster 2018-06-05 13:14:50 +02:00
Eneko Lakasta
a4292f08cf refactor apikey to apikeyToken 2018-06-05 13:13:09 +02:00
Simon Martín
29d1fb1274 logging GzipSize 2018-06-05 13:02:14 +02:00
Simon Martín
02cc606be1 gzipSize support in metrics 2018-06-05 13:01:15 +02:00
Simon Martín
dd689d3568 log metrics only once 2018-06-05 12:36:36 +02:00
Simon Martín
550c736032 metrics in errors 2018-06-05 12:35:52 +02:00
Rafa de la Torre
d2c0e68a78
Merge pull request #497 from CartoDB/cancel-copy-to-client-disconnect 
Cancel copy to upon client disconnect
 2018-06-05 09:59:49 +02:00
Paul Ramsey
d6d9022c7f Coerce format string to upper for better log consistency 2018-06-04 13:12:06 -07:00
Simon Martín
66f7ab45fe release connection with error 2018-06-04 20:51:21 +02:00
Simon Martín
7b6056b799 using the correct errors in done 2018-06-04 20:36:16 +02:00
Rafa de la Torre
014ea8142b A cleaner approach to the cancel command 2018-06-04 19:06:11 +02:00
Simon Martín
2f2dcfd762 fix copy format case 2018-06-04 18:15:28 +02:00
Simon Martín
a844b5d31d jshint happy 2018-06-04 18:15:00 +02:00
Simon Martín
1cf7032c9a adding user and date in copy logs 2018-06-04 18:08:34 +02:00
Rafa de la Torre
994e8a702b Add callbacks to pg.connect 
And call them to return connections to the pool.
 2018-05-31 18:59:19 +02:00
Rafa de la Torre
b59ae1d057 Make sure the COPY TO query is cancelled 
Issue a CancelRequest upon client disconnection, to make sure the COPY
TO query is cancelled and the connection/session is put back to the pg
pool.
 2018-05-31 17:27:35 +02:00
Rafa de la Torre
332f7096d3 Listen to response events (on behalf of @oleurud) 2018-05-31 17:06:19 +02:00
Rafa de la Torre
0eab03a7e7 Add a more informative message 2018-05-31 16:41:22 +02:00
Eneko Lakasta
46e3a87f41 add some auth TODOs 2018-05-30 18:15:35 +02:00
Eneko Lakasta
cf8bf6e5e6 get DB configuration from job instead from metadata in Job Canceller 2018-05-30 17:36:55 +02:00
Simon Martín
310f652ae4 send CopyFail when user close connection 2018-05-30 12:59:49 +02:00
Eneko Lakasta
7764975c09 please jshint: remove unnecessary require (underscore) 2018-05-30 12:08:04 +02:00
Simon Martín
a083eb909c fix req events 2018-05-29 16:19:53 +02:00
Eneko Lakasta
fa5a99211c check user exists in user middleware 
This way, we keep sending a 404 error if the user does not exist.
 2018-05-29 13:23:50 +02:00
Simon Martín
008fad3d13 undo 'ensuring data ingestion log' 2018-05-29 12:45:57 +02:00
Eneko Lakasta
ebe04d04ad refactor user database service. Get proper DB config based on auth type: oauth vs apikey 
Oauth uses only master api key configuration
Api key uses master and the provided api key configurations

Also move default api key fallback to this service
 2018-05-28 15:53:51 +02:00
Simon Martín
e092647246 ensuring data ingestion log 2018-05-28 11:25:09 +02:00
Simon Martín
7d32ae293f rename buyan_logger file name 2018-05-28 11:18:30 +02:00
Simon Martín
3cf28bb7ff jshint happy 2018-05-25 18:50:56 +02:00
Simon Martín
22caa049ee StreamCopyMetrics 2018-05-25 18:47:41 +02:00
Simon Martín
b31984cbc6 simplify response 2018-05-25 17:57:40 +02:00
Simon Martín
1fa5afd759 unify error handler 2018-05-25 17:50:59 +02:00
Simon Martín
6c3f9cf1d3 simplify controllers 2018-05-25 17:42:30 +02:00
Eneko Lakasta
d3b6ebd260 Fallback to default api key if no api key is provided 
What happens with oauth authorization? This way we will always have an api_key and oauth will never trigger....
 2018-05-25 17:28:56 +02:00
Eneko Lakasta
ec75227aa2 use apikeys instead of metadata for the database connections parameters 2018-05-25 17:23:24 +02:00
Simon Martín
fd70673d88 more details 2018-05-25 16:46:25 +02:00
Simon Martín
8a2c777577 details 2018-05-25 16:00:27 +02:00
Simon Martín
9b0b92fb6b removing try catch 2018-05-25 15:46:12 +02:00
Simon Martín
5bed04a38c fix gzip problem with req close event 2018-05-25 15:34:30 +02:00
Simon Martín
fe3bd4fd37 handling errors from request in COPYfrom 2018-05-25 14:33:54 +02:00
Simon Martín
e2bba967f9 handling copyto errors 2018-05-24 19:48:24 +02:00
Eneko Lakasta
6bc90eb9b3 fix typo: Backed -> Backend 2018-05-24 11:04:14 +02:00
Simon Martín
fca6ee8232 changing name of dataIngestionLogPath 2018-05-23 17:32:44 +02:00
Simon Martín
b2a36eb556 copy metrics with BunyanLogger 2018-05-23 17:25:46 +02:00
Simon Martín
fdc542f7b5 BunyanLogger 2018-05-23 17:24:48 +02:00
Simon Martín
5ba7dca79c copyfrom metrics to kibana 2018-05-23 10:30:37 +02:00
Paul Ramsey
f77707f6da Fix reponse copy from 2018-05-22 10:48:11 -04:00
Paul Ramsey
a4654f82b2 Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q 2018-05-22 10:37:50 -04:00
Paul Ramsey
ed696a96ec Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q 2018-05-22 10:23:04 -04:00
Simon Martín
6d73d97ae2 copyFrom metrics to statsd 2018-05-22 16:02:14 +02:00
Simon Martín
e347985465 changing query parameter name from sql to q 
unify query validatrion
 2018-05-22 15:42:57 +02:00
Simon Martín
b9474e7fc3 total_rows in copyto metrics 2018-05-22 15:29:08 +02:00
Simon Martín
0bac7a484c improving query info 2018-05-22 14:16:22 +02:00
Simon Martín
185f708d5b small style refactor 2018-05-22 14:15:55 +02:00
Simon Martín
86e0ebc148 small refactor in copy format regex 2018-05-22 12:02:41 +02:00
Simon Martín
f01191472b refactoring copy controller middlewares 2018-05-22 11:56:50 +02:00
Simon Martín
433bd01c27 copyto metrics 2018-05-22 11:54:10 +02:00
Simon Martín
81be15fbc3 adding format to copy metrics 2018-05-21 19:13:44 +02:00
Paul Ramsey
64fc0c32e3 Change from 'sql' as query parameter to 'q', aping existing sql api 2018-05-21 07:47:16 -07:00
Simon Martín
816779fefd erratum 2018-05-18 17:32:59 +02:00
Simon Martín
82c9fec18a copy from metrics 2018-05-18 17:31:29 +02:00
Eneko Lakasta
fc7e246704 please jshint 2018-05-18 11:35:54 +02:00
Simon Martín
bf4d12868b Merge branch 'master' into pgcopy-stream 2018-05-18 10:57:32 +02:00
Eneko Lakasta
1e8c6e198c remove api key fallback 2018-05-17 17:13:59 +02:00
Simon Martín
9016647a6f reverting use of busboy instead of multer 2018-05-14 18:31:44 +02:00
Simon Martín
60b48df4d6 default rate limits params to server.js 2018-05-11 15:41:35 +02:00
Simon Martín
d70013b0f4 object assing for default rate limits params 2018-05-11 15:36:59 +02:00
Simon Martín
bc8a867334 jshinty happy 2018-05-11 14:51:43 +02:00
Simon Martín
f2acc8f653 forgotten log 2018-05-11 14:16:26 +02:00
Simon Martín
eee8bc2674 moving copyto headers to correct position 2018-05-11 14:12:23 +02:00
Simon Martín
791967877c addining gzip support to copyfrom 2018-05-11 13:33:54 +02:00
Simon Martín
f749798ca5 validating inputs 2018-05-11 10:38:33 +02:00
Paul Ramsey
8f211f905c Remove wrong examples 2018-05-10 12:21:33 -07:00
Paul Ramsey
f0021264d2 Change from form-multipart to POST with chunked upload 2018-05-10 12:11:55 -07:00
Simon Martín
0bb7cec9f4 params existence responsability for UserLimits 2018-05-10 11:51:58 +02:00
Simon Martín
d400926387 using busboy instead of multer 2018-05-08 18:39:16 +02:00
Simon Martín
a997881048 ensuring existence of rate limits params in code 2018-05-08 16:00:16 +02:00
Simon Martín
82d14ab98f busboy configuration 2018-05-08 13:15:19 +02:00
Simon Martín
7c4409bcf5 from var to const/let 2018-05-08 13:10:25 +02:00
Simon Martín
c40ab801b1 refactoring handleCopyTo 2018-05-08 13:08:29 +02:00
Simon Martín
b036b876ff using Busboy instead of multer 2018-05-08 12:52:33 +02:00
Simon Martín
6b96032fcf ensuring return error 2018-05-04 16:50:17 +02:00
Simon Martín
383b86cf9b simplify sql paramter handler and removing underscore 2018-05-04 16:47:50 +02:00
Simon Martín
2b066eadfa removing unused dependencies 2018-05-04 15:35:23 +02:00
Simon Martín
02238fefe1 moving body-parser from global to routes of query and job 2018-05-04 15:15:37 +02:00
Simon Martín
0346cf11d8 enabling multipart/form-data again 2018-05-04 15:11:45 +02:00
Simon Martín
4322ccdd09 adding new endpoints to rate limits 2018-05-03 18:50:13 +02:00
Simon Martín
af4c5906c8 adding /sql tro the endpoints 2018-05-03 18:31:49 +02:00
Paul Ramsey
36bdf84f71 Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream 2018-04-27 15:56:13 +02:00
Paul Ramsey
e2d8896307 Typo in curl example 2018-04-27 12:32:47 +02:00
Paul Ramsey
5f9cc37dba Jshint change, DRY the content-disposition handler 2018-04-27 12:19:53 +02:00
Paul Ramsey
fe52af71ac Add documentation SQL API copyfrom/copyto end points, 
and make small modifications arising from that task
 2018-04-27 11:55:06 +02:00
Paul Ramsey
c78cac9bd6 Use the correct database credentials, so we obey security 2018-04-26 10:45:52 +02:00
Paul Ramsey
f13028b497 More jshint changes 2018-04-26 10:20:21 +02:00
Simon Martín
5f8e1d8f53 fix validatePGEntitiesAccess access 2018-04-26 10:13:03 +02:00
Paul Ramsey
cc5fa5c6ce Changes suggested by jshint 2018-04-26 10:04:38 +02:00
Paul Ramsey
ea66076255 Remove 'console.debug', try and get travis tests clean 2018-04-26 09:40:13 +02:00
Paul Ramsey
a98781d335 Working code for /copyfrom (streaming) and /copyto (streaming) 2018-04-25 18:37:04 +02:00
Paul Ramsey
b59bda5780 Get working copyto implementation in place. Many test cases 
needed still and thoughts on how to communicate errors back
that are maybe nicer than stack messages?
 2018-04-24 15:55:20 +02:00
Simon Martín
e12e16507a create object and export 2018-04-24 13:21:43 +02:00
Paul Ramsey
4914100205 CopyFrom works, but still needs a decent return payload and 
a lot of work on returning useful information for error cases (post empty input,
db errors returned more nicely? etc)
 2018-04-24 13:07:57 +02:00
Paul Ramsey
72bce5732b WIP on /copyfrom 2018-04-24 11:26:15 +02:00
Simon Martín
3b228ac17b solving jshint crazy error 2018-04-24 10:10:37 +02:00
Simon Martín
985f9ef7f5 pgEntitiesAccessValidator 2018-04-23 18:17:44 +02:00
Rafa de la Torre
2229d0ee57 Fix jshint issues 2018-04-13 16:36:43 +02:00
Paul Ramsey
0161696627
Update copy_controller.js 2018-04-13 05:43:23 -07:00
Paul Ramsey
d54e2f5a07 Implementation including multer, custom storage engine, and pg-copy, 
but without turning over pg-copy, and demonstrating the missing 'sql'
parameter in the custom storage engine.
 2018-04-12 12:25:28 -07:00
Simon Martín
2a729b7da8 destructuring error_handler parameters 2018-04-03 13:43:17 +02:00
Simon Martín
3f6b6d3cff middleware naming 2018-04-03 13:41:18 +02:00
Simon Martín
db6ab4a591 let & const vars 2018-03-28 13:52:45 +02:00
Simon Martín
adab7f21d8 setErrorHeader refactor 2018-03-28 13:50:45 +02:00
Simon Martín
720c3eb8cb changing msg var name for errorResponse 2018-03-28 13:35:19 +02:00
Simon Martín
4301b63189 fix error 2018-03-28 13:31:31 +02:00
Simon Martín
616f3f3c07 errorHandlerFactory 2018-03-28 13:06:39 +02:00
Simon Martín
16ea727fc3 move ErrorHandler logic from error middleware 2018-03-28 12:58:10 +02:00
Simon Martín
5b6f876cd6 change pgErrorHandler for more generic errorHandler 2018-03-28 12:50:07 +02:00