cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,044 Commits 1 Branch 118 Tags 9.7 MiB
f71c7bc3a8
Commit Graph

962 Commits

Author SHA1 Message Date
Simon Martín
254d804f0c fix log middleware folder 2019-02-27 09:04:43 +01:00
Simon Martín
09dcb6f2fa log copy endpoint 2019-02-27 09:02:49 +01:00
Simon Martín
2066115002 log query and job endpoints 2019-02-27 09:02:31 +01:00
Simon Martín
2c1ce33e34 log middleware 2019-02-27 09:01:51 +01:00
Simon Martín
6768baa0ae ensuring req.body 2019-02-26 17:11:11 +01:00
Simon Martín
b618691af9 splitting behavior batch & not batch handling query 2019-02-26 17:09:36 +01:00
Simon Martín
f925863534 using handle query middleware in copy endpoint 2019-02-26 16:43:18 +01:00
Simon Martín
90b693f050 handling job queries with middleware 2019-02-26 16:40:20 +01:00
Simon Martín
7b5a118c7e renaming from q to sql in middleware 2019-02-26 16:34:26 +01:00
Simon Martín
a6eddef252 same things together 2019-02-26 15:05:14 +01:00
Simon Martín
69e2aaf748 using handleQueryMiddleware in query controller 2019-02-26 14:47:23 +01:00
Simon Martín
143af82b8b semicolon 2019-02-26 14:40:41 +01:00
Simon Martín
0a5fc471c1 handleQueryMiddleware 2019-02-26 14:36:11 +01:00
Simon Martín
799b767505 remove old query log system 2019-02-26 13:58:01 +01:00
Daniel García Aubert
23d76393cd Lint 2019-01-16 15:58:11 +01:00
Simon Martín
60702faa57 adding Authorization to Access-Control-Allow-Headers 2019-01-10 15:55:08 +01:00
Simon Martín
f947fd7cd5 Merge branch 'master' into unique-query-ratelimit 2019-01-09 08:50:41 +01:00
Raul Marin
df4732ff94 Consider cancelled queries as platform limits 2019-01-03 15:17:47 +01:00
Raul Marin
03701ae695 Set platform limits message on streaming responses too 2019-01-02 16:21:33 +01:00
Simon Martín
254896e666 remove query_format rate limit 2018-12-10 18:20:10 +01:00
Simon Martín
498ddc58b7 unique rate limit fot both query endpoints 2018-12-10 18:19:56 +01:00
Rafa de la Torre
aeec1922ee Fix test and add const #543 2018-12-05 19:15:57 +01:00
Simon Martín
1293acca48 cancel query when Quota exceeded or max POST size 2018-11-29 18:15:14 +01:00
Daniel García Aubert
e0e011e806 Do not use assert to throw erros as in Node.js > 6 wraps the original error, the keyword 'throw' does the trick and it's backwards compatible 2018-11-07 18:05:39 +01:00
Daniel García Aubert
121a19eafb Prevent: Octal literals are not allowed in strict mode. 2018-10-24 15:42:54 +02:00
Daniel García Aubert
a616982ec3 Use strict mode 2018-10-24 15:42:33 +02:00
Rafa de la Torre
d5a7f6d2d6 Fix paused stream (DB socket) after copy to #590 
Ideally, the CopyToQueryStream would take care o piping, unpiping and
resuming the connection with the DB.

For some reason it does not always work when trying that from the
_transform callback.

Since there's no better callback (at least in node6), this patch is
required to make sure that such connection can be returned to the pool
and reused by some other query.
 2018-09-17 16:54:51 +02:00
Simon Martín
32ff1e7a8e ensure tmpDir folder 2018-08-30 12:27:34 +02:00
Simon Martín
9365986272 better message 2018-08-14 15:30:51 +02:00
Simon Martín
0a1f7c6ba5 copyto adding gzip to log 2018-08-14 15:14:22 +02:00
Simon Martín
735033faee jshint 2018-08-10 14:39:32 +02:00
Simon Martín
c0ea5be5d3 Improve gzip error message 2018-08-10 14:38:50 +02:00
Simon Martín
9eeeadbc07 maintain req error handler 2018-08-10 14:38:24 +02:00
Rafa de la Torre
8a76df8884 Fix for unhandled gunzip/zlib errors #528 2018-08-10 12:37:49 +02:00
Rafa de la Torre
a709e8f26f Set the default copy_from_max_post_size to 2 GB 
Set the default copy_from_max_post_size to exactly 2 GB, as the nginx
limitation has been removed for that endpoint. So, the application is
now in control and this keeps the code clear and clean.
 2018-07-26 12:52:20 +02:00
Rafa de la Torre
82348a79a5 Remove unnecessary metrics.end(null, err) 
As per comment in PR
 2018-07-24 14:36:39 +02:00
Rafa de la Torre
19c9c15969 Replace let by const in pseudo-constant conditional assignment 2018-07-23 18:41:27 +02:00
Rafa de la Torre
151bffaad4 return to avoid further errors 
As per a PR comment: `return done(err);` should prevent further errors
and make sure the request is over.
 2018-07-23 18:39:30 +02:00
Rafa de la Torre
21295a2d93 Use pgstream.emit('error', myError) 
As suggested in PR, emit our own error to deal with the quota and max
POST size checks, to do proper error handling.
 2018-07-23 18:29:41 +02:00
Rafa de la Torre
a35a519702 Use CAPITALS for constants (as suggested in PR) 2018-07-23 18:24:48 +02:00
Rafa de la Torre
bac9e94428 Please jshint 2018-07-23 17:15:21 +02:00
Rafa de la Torre
774c144837 Implementation of the max POST size check 2018-07-23 16:59:05 +02:00
Rafa de la Torre
8ec7e0e309 Fix for "Can't set headers after they are sent" 2018-07-23 16:15:11 +02:00
Rafa de la Torre
a5046f58de Add dbRemainingQuota check to COPY FROM 2018-07-23 15:05:22 +02:00
Rafa de la Torre
84d88f91fb Better implementation of dbQuotaMiddleware 2018-07-23 15:02:20 +02:00
Rafa de la Torre
7e54912ee4 First version of the dbQuotaMiddleware 2018-07-23 14:38:56 +02:00
Rafa de la Torre
ea46db5923 Remove uneeded timeout middleware 2018-07-20 17:48:56 +02:00
Rafa de la Torre
cc348f3725 Make the SET statement_timeout async 2018-07-20 16:25:47 +02:00
Rafa de la Torre
17c8cbc723 Add a timeout setting for COPY commands 2018-07-20 15:32:32 +02:00
Rafa de la Torre
763134437b Use the db_batch_port for copy 2018-07-20 10:22:54 +02:00
Eneko Lakasta
1bf8a06ec2 Rephrase error message 2018-06-29 13:19:32 +02:00
Eneko Lakasta
99058e68af add space to message 2018-06-27 15:53:51 +02:00
Eneko Lakasta
2bb28b5578 remove line break in the error message 2018-06-27 15:43:06 +02:00
Eneko Lakasta
879e85d07f add 'SQL query timeout expired error.' to DB timeout limit error message. 2018-06-27 15:17:38 +02:00
Simon Martín
59dd495a87 small style details 2018-06-22 10:50:39 +02:00
Simon Martín
06c40a396b copy from: events is series 2018-06-21 15:58:23 +02:00
Simon Martín
195a4866c3 managing gzip pipe with PassThrough 2018-06-21 15:54:42 +02:00
Simon Martín
f5641a1873 remove unneeded this.connectionClosedByClient 2018-06-21 15:18:26 +02:00
Simon Martín
fae18fb242 using getPGStream 2018-06-21 15:17:32 +02:00
Simon Martín
823e3abcfe unify to and from method in only one getPGStream 2018-06-21 15:16:47 +02:00
Simon Martín
3ede1ea9f1 unify streams 2018-06-21 14:53:07 +02:00
Simon Martín
f4651cadae using this.copyFromStream 2018-06-21 14:49:38 +02:00
Simon Martín
9970052e1b copy from: removing copyFromStream dependency 2018-06-21 14:38:22 +02:00
Simon Martín
01eff0a5a2 copy from: removing done dependecy 2018-06-21 14:36:40 +02:00
Simon Martín
a15c2e58aa copy from: manage error in 1 point 2018-06-21 14:36:03 +02:00
Simon Martín
d17a5608fb copy from: unneeded pgstream.end in req error 2018-06-21 14:32:36 +02:00
Simon Martín
5a28d36d69 copy from: pgstream events in series 2018-06-21 14:29:19 +02:00
Simon Martín
925115e8c1 copy from: remove client dependecy 2018-06-21 14:25:54 +02:00
Simon Martín
327aaf3450 jshint happy 2018-06-21 13:41:13 +02:00
Simon Martín
bafcb8051f StreamCopy getResults refactor 
- changing get Result to getRowCount
- ensuring to and from independecy in StreamCopy
 2018-06-21 13:39:33 +02:00
Simon Martín
2cecc54cb2 copy to: removing copyToStream dependecy 2018-06-21 13:21:23 +02:00
Simon Martín
e39a9b2bf7 copy to: manage error in 1 point 2018-06-21 13:13:37 +02:00
Simon Martín
1dac5355d1 copy to: remove unneeded done and removing it as a dependency 2018-06-21 13:13:16 +02:00
Simon Martín
42a94a3b6c copy to: removing unnedeed client dependecy 2018-06-21 13:01:59 +02:00
Simon Martín
e6448a3255 copy to: cancelQuery to streamCopy 2018-06-21 13:00:57 +02:00
Simon Martín
39dfcb1f9b copy to events is series 2018-06-21 12:50:50 +02:00
Simon Martín
5fbd6e8e3a req on error is neccesary when connection is canceled 2018-06-21 12:49:06 +02:00
Simon Martín
3da32308ce jshint happy 2018-06-21 12:30:49 +02:00
Simon Martín
6955d254f8 Merge branch 'master' into pgcopy-stream-refactor2-dgaubert 2018-06-21 12:19:16 +02:00
Simon Martín
3d30f1f762 trailing white spaces 2018-06-19 10:34:52 +02:00
Simon Martín
30402f2e89 remove unused statsClient 2018-06-19 10:04:17 +02:00
Simon Martín
7727a9d506 data ingestion logger created in controller and added to app 2018-06-18 18:48:11 +02:00
Simon Martín
32dca8dfdb reopenFileStreams to parent class 2018-06-18 18:47:12 +02:00
Simon Martín
8f727fc15a cdb-user in logs 2018-06-15 13:30:42 +02:00
Simon Martín
8a450a862c recover the metrics header (needed for tests purposes) 2018-06-15 13:25:47 +02:00
Simon Martín
c7e01f2ed6 success in copy log 2018-06-14 18:40:06 +02:00
Simon Martín
9097cbe16e adding log of the error when headers are already sent 2018-06-14 18:29:50 +02:00
Simon Martín
39e5395edc change gzip to isGzip 2018-06-14 18:29:03 +02:00
Simon Martín
4cb7de7318 changing let for const 2018-06-14 10:51:05 +02:00
Simon Martín
b091a375fc fix ensuring right validation error 2018-06-14 10:49:36 +02:00
Simon Martín
1f31b8e2ae removing old copy logs 2018-06-14 10:45:23 +02:00
Daniel García Aubert
9299fe51ce Add todo 2018-06-14 01:01:05 +02:00
Daniel García Aubert
8eab164e4e Be more one liner 2018-06-14 00:54:03 +02:00
Daniel García Aubert
9c8d802f75 Chain response event callbacks after .pipe() in order to improve readability in copyto controller 2018-06-14 00:31:08 +02:00
Daniel García Aubert
508d688b9d Use request 'close' event when the request is closed unexpectedly by the client 2018-06-14 00:16:06 +02:00
Daniel García Aubert
37a079363a Do not extend from EventEmitter 2018-06-13 20:12:56 +02:00
Daniel García Aubert
bd8e4be596 Do not emit to regist the number of rows processed in copyfrom 2018-06-13 20:11:39 +02:00
Daniel García Aubert
5655f26fbd Do not emit to regist the number of rows processed in copyto 2018-06-13 20:06:35 +02:00
Daniel García Aubert
eb4ba60ba0 Pass the error to the pg connection's callback 2018-06-13 19:56:56 +02:00
Daniel García Aubert
e6a331433a Handle error when client disconnects in copyto requests 2018-06-13 19:54:34 +02:00
Daniel García Aubert
6aba1ea307 Handle error when sql injected in pgstream fails (e.g: unexisting table) 2018-06-13 19:52:53 +02:00
Daniel García Aubert
2792d95c41 Do not handle stream unpiping when it wasn't even created 2018-06-13 18:30:05 +02:00
Daniel García Aubert
b9971d94dd Do not use the regular callback in streams events 2018-06-13 18:28:34 +02:00
Daniel García Aubert
7d1a0abadd Preventing callbacks from accidentally being called twice 2018-06-13 18:26:58 +02:00
Eneko Lakasta
f5d56a2253
FIX add return to callback call 2018-06-13 13:17:01 +02:00
Simon Martín
270daae579 removing forgotten callback 2018-06-12 18:40:51 +02:00
Simon Martín
409273bafe removing 2nd calback using events 2018-06-12 18:39:50 +02:00
Simon Martín
19aae3c40f manage pgstrem error on stream copy class 2018-06-12 17:04:44 +02:00
Simon Martín
ba2f85421d StreamCopy to class 2018-06-12 16:56:18 +02:00
Simon Martín
8a15b2f8d1 Merge branch 'master' into pgcopy-stream-refactor2 2018-06-12 15:21:28 +02:00
Simon Martín
4d99872975 return in next err 2018-06-12 15:18:28 +02:00
Simon Martín
b9a0fa78d2 adding metadataBackend to user middleware in copycontroller 2018-06-11 14:55:37 +02:00
Simon Martín
1174499dcf Merge branch 'master' into pgcopy-stream 2018-06-11 14:38:52 +02:00
Simon Martín
df63e0eedb adding log of the error when headers are already sent 2018-06-11 13:19:12 +02:00
Simon Martín
2995676adb let to const 2018-06-11 12:56:16 +02:00
Simon Martín
4fab96cb3a rename gzip to isGzip 2018-06-11 12:55:30 +02:00
Simon Martín
87fd9db1fe remove uneeded comment 2018-06-11 12:43:35 +02:00
Simon Martín
2581ddbfb4 better formatting 2018-06-08 17:09:28 +02:00
Simon Martín
b63806f8de cleaning stream copy 2018-06-08 17:04:28 +02:00
Simon Martín
6a9e7d7c14 forgotten ) 2018-06-08 17:04:10 +02:00
Simon Martín
7266b3c708 one logger 2018-06-08 17:02:31 +02:00
Simon Martín
014f0a1491 refactor of copy to PoC 2018-06-08 16:58:32 +02:00
Simon Martín
f7454228c6 refactor of copy from PoC 2018-06-08 16:50:12 +02:00
Simon Martín
e259a51623 unifiyng to see 'the forest' 2018-06-08 15:03:21 +02:00
Simon Martín
8e8ccc9c67 removing old metrics in copy 2018-06-08 13:33:17 +02:00
Simon Martín
c62ef15dda using StreamCopy in controller 2018-06-08 11:15:23 +02:00
Simon Martín
732c229e63 creating StreamCopy class 2018-06-08 11:09:51 +02:00
Simon Martín
2bcea0484a rename BunyanLogger to Logger 2018-06-08 10:59:34 +02:00
Eneko Lakasta
440ba8c840 make function checkUserExists node callback pattern compilant 2018-06-06 15:48:22 +02:00
Eneko Lakasta
340e55ea46 reduce error info 2018-06-06 15:23:53 +02:00
Eneko Lakasta
ca8ae503ac Merge branch 'master' into remove-auth-fallback 2018-06-05 16:19:36 +02:00
Eneko Lakasta
eab3d289b6 refactor authenticated to authorizationLevel 2018-06-05 13:21:56 +02:00
Eneko Lakasta
da08e42921 refactor forceToBeAuthenticated to forceToBeMaster 2018-06-05 13:16:36 +02:00
Simon Martín
66af518deb jshint 2018-06-05 13:16:14 +02:00
Eneko Lakasta
0207b67d5f refactor forceToBeAuthenticated to forceToBeMaster 2018-06-05 13:14:50 +02:00
Eneko Lakasta
a4292f08cf refactor apikey to apikeyToken 2018-06-05 13:13:09 +02:00
Simon Martín
29d1fb1274 logging GzipSize 2018-06-05 13:02:14 +02:00
Simon Martín
02cc606be1 gzipSize support in metrics 2018-06-05 13:01:15 +02:00
Simon Martín
dd689d3568 log metrics only once 2018-06-05 12:36:36 +02:00
Simon Martín
550c736032 metrics in errors 2018-06-05 12:35:52 +02:00
Rafa de la Torre
d2c0e68a78
Merge pull request #497 from CartoDB/cancel-copy-to-client-disconnect 
Cancel copy to upon client disconnect
 2018-06-05 09:59:49 +02:00
Paul Ramsey
d6d9022c7f Coerce format string to upper for better log consistency 2018-06-04 13:12:06 -07:00
Simon Martín
66f7ab45fe release connection with error 2018-06-04 20:51:21 +02:00
Simon Martín
7b6056b799 using the correct errors in done 2018-06-04 20:36:16 +02:00
Rafa de la Torre
014ea8142b A cleaner approach to the cancel command 2018-06-04 19:06:11 +02:00
Simon Martín
2f2dcfd762 fix copy format case 2018-06-04 18:15:28 +02:00
Simon Martín
a844b5d31d jshint happy 2018-06-04 18:15:00 +02:00
Simon Martín
1cf7032c9a adding user and date in copy logs 2018-06-04 18:08:34 +02:00
Rafa de la Torre
994e8a702b Add callbacks to pg.connect 
And call them to return connections to the pool.
 2018-05-31 18:59:19 +02:00
Rafa de la Torre
b59ae1d057 Make sure the COPY TO query is cancelled 
Issue a CancelRequest upon client disconnection, to make sure the COPY
TO query is cancelled and the connection/session is put back to the pg
pool.
 2018-05-31 17:27:35 +02:00