cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,903 Commits 1 Branch 118 Tags 9.7 MiB
b297005ebb
Commit Graph

887 Commits

Author SHA1 Message Date
Daniel García Aubert
a616982ec3 Use strict mode 2018-10-24 15:42:33 +02:00
Rafa de la Torre
d5a7f6d2d6 Fix paused stream (DB socket) after copy to #590 
Ideally, the CopyToQueryStream would take care o piping, unpiping and
resuming the connection with the DB.

For some reason it does not always work when trying that from the
_transform callback.

Since there's no better callback (at least in node6), this patch is
required to make sure that such connection can be returned to the pool
and reused by some other query.
 2018-09-17 16:54:51 +02:00
Simon Martín
32ff1e7a8e ensure tmpDir folder 2018-08-30 12:27:34 +02:00
Simon Martín
9365986272 better message 2018-08-14 15:30:51 +02:00
Simon Martín
0a1f7c6ba5 copyto adding gzip to log 2018-08-14 15:14:22 +02:00
Simon Martín
735033faee jshint 2018-08-10 14:39:32 +02:00
Simon Martín
c0ea5be5d3 Improve gzip error message 2018-08-10 14:38:50 +02:00
Simon Martín
9eeeadbc07 maintain req error handler 2018-08-10 14:38:24 +02:00
Rafa de la Torre
8a76df8884 Fix for unhandled gunzip/zlib errors #528 2018-08-10 12:37:49 +02:00
Rafa de la Torre
a709e8f26f Set the default copy_from_max_post_size to 2 GB 
Set the default copy_from_max_post_size to exactly 2 GB, as the nginx
limitation has been removed for that endpoint. So, the application is
now in control and this keeps the code clear and clean.
 2018-07-26 12:52:20 +02:00
Rafa de la Torre
82348a79a5 Remove unnecessary metrics.end(null, err) 
As per comment in PR
 2018-07-24 14:36:39 +02:00
Rafa de la Torre
19c9c15969 Replace let by const in pseudo-constant conditional assignment 2018-07-23 18:41:27 +02:00
Rafa de la Torre
151bffaad4 return to avoid further errors 
As per a PR comment: `return done(err);` should prevent further errors
and make sure the request is over.
 2018-07-23 18:39:30 +02:00
Rafa de la Torre
21295a2d93 Use pgstream.emit('error', myError) 
As suggested in PR, emit our own error to deal with the quota and max
POST size checks, to do proper error handling.
 2018-07-23 18:29:41 +02:00
Rafa de la Torre
a35a519702 Use CAPITALS for constants (as suggested in PR) 2018-07-23 18:24:48 +02:00
Rafa de la Torre
bac9e94428 Please jshint 2018-07-23 17:15:21 +02:00
Rafa de la Torre
774c144837 Implementation of the max POST size check 2018-07-23 16:59:05 +02:00
Rafa de la Torre
8ec7e0e309 Fix for "Can't set headers after they are sent" 2018-07-23 16:15:11 +02:00
Rafa de la Torre
a5046f58de Add dbRemainingQuota check to COPY FROM 2018-07-23 15:05:22 +02:00
Rafa de la Torre
84d88f91fb Better implementation of dbQuotaMiddleware 2018-07-23 15:02:20 +02:00
Rafa de la Torre
7e54912ee4 First version of the dbQuotaMiddleware 2018-07-23 14:38:56 +02:00
Rafa de la Torre
ea46db5923 Remove uneeded timeout middleware 2018-07-20 17:48:56 +02:00
Rafa de la Torre
cc348f3725 Make the SET statement_timeout async 2018-07-20 16:25:47 +02:00
Rafa de la Torre
17c8cbc723 Add a timeout setting for COPY commands 2018-07-20 15:32:32 +02:00
Rafa de la Torre
763134437b Use the db_batch_port for copy 2018-07-20 10:22:54 +02:00
Eneko Lakasta
1bf8a06ec2 Rephrase error message 2018-06-29 13:19:32 +02:00
Eneko Lakasta
99058e68af add space to message 2018-06-27 15:53:51 +02:00
Eneko Lakasta
2bb28b5578 remove line break in the error message 2018-06-27 15:43:06 +02:00
Eneko Lakasta
879e85d07f add 'SQL query timeout expired error.' to DB timeout limit error message. 2018-06-27 15:17:38 +02:00
Simon Martín
59dd495a87 small style details 2018-06-22 10:50:39 +02:00
Simon Martín
06c40a396b copy from: events is series 2018-06-21 15:58:23 +02:00
Simon Martín
195a4866c3 managing gzip pipe with PassThrough 2018-06-21 15:54:42 +02:00
Simon Martín
f5641a1873 remove unneeded this.connectionClosedByClient 2018-06-21 15:18:26 +02:00
Simon Martín
fae18fb242 using getPGStream 2018-06-21 15:17:32 +02:00
Simon Martín
823e3abcfe unify to and from method in only one getPGStream 2018-06-21 15:16:47 +02:00
Simon Martín
3ede1ea9f1 unify streams 2018-06-21 14:53:07 +02:00
Simon Martín
f4651cadae using this.copyFromStream 2018-06-21 14:49:38 +02:00
Simon Martín
9970052e1b copy from: removing copyFromStream dependency 2018-06-21 14:38:22 +02:00
Simon Martín
01eff0a5a2 copy from: removing done dependecy 2018-06-21 14:36:40 +02:00
Simon Martín
a15c2e58aa copy from: manage error in 1 point 2018-06-21 14:36:03 +02:00
Simon Martín
d17a5608fb copy from: unneeded pgstream.end in req error 2018-06-21 14:32:36 +02:00
Simon Martín
5a28d36d69 copy from: pgstream events in series 2018-06-21 14:29:19 +02:00
Simon Martín
925115e8c1 copy from: remove client dependecy 2018-06-21 14:25:54 +02:00
Simon Martín
327aaf3450 jshint happy 2018-06-21 13:41:13 +02:00
Simon Martín
bafcb8051f StreamCopy getResults refactor 
- changing get Result to getRowCount
- ensuring to and from independecy in StreamCopy
 2018-06-21 13:39:33 +02:00
Simon Martín
2cecc54cb2 copy to: removing copyToStream dependecy 2018-06-21 13:21:23 +02:00
Simon Martín
e39a9b2bf7 copy to: manage error in 1 point 2018-06-21 13:13:37 +02:00
Simon Martín
1dac5355d1 copy to: remove unneeded done and removing it as a dependency 2018-06-21 13:13:16 +02:00
Simon Martín
42a94a3b6c copy to: removing unnedeed client dependecy 2018-06-21 13:01:59 +02:00
Simon Martín
e6448a3255 copy to: cancelQuery to streamCopy 2018-06-21 13:00:57 +02:00
Simon Martín
39dfcb1f9b copy to events is series 2018-06-21 12:50:50 +02:00
Simon Martín
5fbd6e8e3a req on error is neccesary when connection is canceled 2018-06-21 12:49:06 +02:00
Simon Martín
3da32308ce jshint happy 2018-06-21 12:30:49 +02:00
Simon Martín
6955d254f8 Merge branch 'master' into pgcopy-stream-refactor2-dgaubert 2018-06-21 12:19:16 +02:00
Simon Martín
3d30f1f762 trailing white spaces 2018-06-19 10:34:52 +02:00
Simon Martín
30402f2e89 remove unused statsClient 2018-06-19 10:04:17 +02:00
Simon Martín
7727a9d506 data ingestion logger created in controller and added to app 2018-06-18 18:48:11 +02:00
Simon Martín
32dca8dfdb reopenFileStreams to parent class 2018-06-18 18:47:12 +02:00
Simon Martín
8f727fc15a cdb-user in logs 2018-06-15 13:30:42 +02:00
Simon Martín
8a450a862c recover the metrics header (needed for tests purposes) 2018-06-15 13:25:47 +02:00
Simon Martín
c7e01f2ed6 success in copy log 2018-06-14 18:40:06 +02:00
Simon Martín
9097cbe16e adding log of the error when headers are already sent 2018-06-14 18:29:50 +02:00
Simon Martín
39e5395edc change gzip to isGzip 2018-06-14 18:29:03 +02:00
Simon Martín
4cb7de7318 changing let for const 2018-06-14 10:51:05 +02:00
Simon Martín
b091a375fc fix ensuring right validation error 2018-06-14 10:49:36 +02:00
Simon Martín
1f31b8e2ae removing old copy logs 2018-06-14 10:45:23 +02:00
Daniel García Aubert
9299fe51ce Add todo 2018-06-14 01:01:05 +02:00
Daniel García Aubert
8eab164e4e Be more one liner 2018-06-14 00:54:03 +02:00
Daniel García Aubert
9c8d802f75 Chain response event callbacks after .pipe() in order to improve readability in copyto controller 2018-06-14 00:31:08 +02:00
Daniel García Aubert
508d688b9d Use request 'close' event when the request is closed unexpectedly by the client 2018-06-14 00:16:06 +02:00
Daniel García Aubert
37a079363a Do not extend from EventEmitter 2018-06-13 20:12:56 +02:00
Daniel García Aubert
bd8e4be596 Do not emit to regist the number of rows processed in copyfrom 2018-06-13 20:11:39 +02:00
Daniel García Aubert
5655f26fbd Do not emit to regist the number of rows processed in copyto 2018-06-13 20:06:35 +02:00
Daniel García Aubert
eb4ba60ba0 Pass the error to the pg connection's callback 2018-06-13 19:56:56 +02:00
Daniel García Aubert
e6a331433a Handle error when client disconnects in copyto requests 2018-06-13 19:54:34 +02:00
Daniel García Aubert
6aba1ea307 Handle error when sql injected in pgstream fails (e.g: unexisting table) 2018-06-13 19:52:53 +02:00
Daniel García Aubert
2792d95c41 Do not handle stream unpiping when it wasn't even created 2018-06-13 18:30:05 +02:00
Daniel García Aubert
b9971d94dd Do not use the regular callback in streams events 2018-06-13 18:28:34 +02:00
Daniel García Aubert
7d1a0abadd Preventing callbacks from accidentally being called twice 2018-06-13 18:26:58 +02:00
Eneko Lakasta
f5d56a2253
FIX add return to callback call 2018-06-13 13:17:01 +02:00
Simon Martín
270daae579 removing forgotten callback 2018-06-12 18:40:51 +02:00
Simon Martín
409273bafe removing 2nd calback using events 2018-06-12 18:39:50 +02:00
Simon Martín
19aae3c40f manage pgstrem error on stream copy class 2018-06-12 17:04:44 +02:00
Simon Martín
ba2f85421d StreamCopy to class 2018-06-12 16:56:18 +02:00
Simon Martín
8a15b2f8d1 Merge branch 'master' into pgcopy-stream-refactor2 2018-06-12 15:21:28 +02:00
Simon Martín
4d99872975 return in next err 2018-06-12 15:18:28 +02:00
Simon Martín
b9a0fa78d2 adding metadataBackend to user middleware in copycontroller 2018-06-11 14:55:37 +02:00
Simon Martín
1174499dcf Merge branch 'master' into pgcopy-stream 2018-06-11 14:38:52 +02:00
Simon Martín
df63e0eedb adding log of the error when headers are already sent 2018-06-11 13:19:12 +02:00
Simon Martín
2995676adb let to const 2018-06-11 12:56:16 +02:00
Simon Martín
4fab96cb3a rename gzip to isGzip 2018-06-11 12:55:30 +02:00
Simon Martín
87fd9db1fe remove uneeded comment 2018-06-11 12:43:35 +02:00
Simon Martín
2581ddbfb4 better formatting 2018-06-08 17:09:28 +02:00
Simon Martín
b63806f8de cleaning stream copy 2018-06-08 17:04:28 +02:00
Simon Martín
6a9e7d7c14 forgotten ) 2018-06-08 17:04:10 +02:00
Simon Martín
7266b3c708 one logger 2018-06-08 17:02:31 +02:00
Simon Martín
014f0a1491 refactor of copy to PoC 2018-06-08 16:58:32 +02:00
Simon Martín
f7454228c6 refactor of copy from PoC 2018-06-08 16:50:12 +02:00
Simon Martín
e259a51623 unifiyng to see 'the forest' 2018-06-08 15:03:21 +02:00
Simon Martín
8e8ccc9c67 removing old metrics in copy 2018-06-08 13:33:17 +02:00