Simon Martín
a6eddef252
same things together
2019-02-26 15:05:14 +01:00
Simon Martín
69e2aaf748
using handleQueryMiddleware in query controller
2019-02-26 14:47:23 +01:00
Simon Martín
143af82b8b
semicolon
2019-02-26 14:40:41 +01:00
Simon Martín
0a5fc471c1
handleQueryMiddleware
2019-02-26 14:36:11 +01:00
Simon Martín
799b767505
remove old query log system
2019-02-26 13:58:01 +01:00
Daniel García Aubert
23d76393cd
Lint
2019-01-16 15:58:11 +01:00
Simon Martín
60702faa57
adding Authorization to Access-Control-Allow-Headers
2019-01-10 15:55:08 +01:00
Simon Martín
f947fd7cd5
Merge branch 'master' into unique-query-ratelimit
2019-01-09 08:50:41 +01:00
Raul Marin
df4732ff94
Consider cancelled queries as platform limits
2019-01-03 15:17:47 +01:00
Raul Marin
03701ae695
Set platform limits message on streaming responses too
2019-01-02 16:21:33 +01:00
Simon Martín
254896e666
remove query_format rate limit
2018-12-10 18:20:10 +01:00
Simon Martín
498ddc58b7
unique rate limit fot both query endpoints
2018-12-10 18:19:56 +01:00
Rafa de la Torre
aeec1922ee
Fix test and add const #543
2018-12-05 19:15:57 +01:00
Simon Martín
1293acca48
cancel query when Quota exceeded or max POST size
2018-11-29 18:15:14 +01:00
Daniel García Aubert
e0e011e806
Do not use assert to throw erros as in Node.js > 6 wraps the original error, the keyword 'throw' does the trick and it's backwards compatible
2018-11-07 18:05:39 +01:00
Daniel García Aubert
121a19eafb
Prevent: Octal literals are not allowed in strict mode.
2018-10-24 15:42:54 +02:00
Daniel García Aubert
a616982ec3
Use strict mode
2018-10-24 15:42:33 +02:00
Rafa de la Torre
d5a7f6d2d6
Fix paused stream (DB socket) after copy to #590
...
Ideally, the CopyToQueryStream would take care o piping, unpiping and
resuming the connection with the DB.
For some reason it does not always work when trying that from the
_transform callback.
Since there's no better callback (at least in node6), this patch is
required to make sure that such connection can be returned to the pool
and reused by some other query.
2018-09-17 16:54:51 +02:00
Simon Martín
32ff1e7a8e
ensure tmpDir folder
2018-08-30 12:27:34 +02:00
Simon Martín
9365986272
better message
2018-08-14 15:30:51 +02:00
Simon Martín
0a1f7c6ba5
copyto adding gzip to log
2018-08-14 15:14:22 +02:00
Simon Martín
735033faee
jshint
2018-08-10 14:39:32 +02:00
Simon Martín
c0ea5be5d3
Improve gzip error message
2018-08-10 14:38:50 +02:00
Simon Martín
9eeeadbc07
maintain req error handler
2018-08-10 14:38:24 +02:00
Rafa de la Torre
8a76df8884
Fix for unhandled gunzip/zlib errors #528
2018-08-10 12:37:49 +02:00
Rafa de la Torre
a709e8f26f
Set the default copy_from_max_post_size to 2 GB
...
Set the default copy_from_max_post_size to exactly 2 GB, as the nginx
limitation has been removed for that endpoint. So, the application is
now in control and this keeps the code clear and clean.
2018-07-26 12:52:20 +02:00
Rafa de la Torre
82348a79a5
Remove unnecessary metrics.end(null, err)
...
As per comment in PR
2018-07-24 14:36:39 +02:00
Rafa de la Torre
19c9c15969
Replace let by const in pseudo-constant conditional assignment
2018-07-23 18:41:27 +02:00
Rafa de la Torre
151bffaad4
return to avoid further errors
...
As per a PR comment: `return done(err);` should prevent further errors
and make sure the request is over.
2018-07-23 18:39:30 +02:00
Rafa de la Torre
21295a2d93
Use pgstream.emit('error', myError)
...
As suggested in PR, emit our own error to deal with the quota and max
POST size checks, to do proper error handling.
2018-07-23 18:29:41 +02:00
Rafa de la Torre
a35a519702
Use CAPITALS for constants (as suggested in PR)
2018-07-23 18:24:48 +02:00
Rafa de la Torre
bac9e94428
Please jshint
2018-07-23 17:15:21 +02:00
Rafa de la Torre
774c144837
Implementation of the max POST size check
2018-07-23 16:59:05 +02:00
Rafa de la Torre
8ec7e0e309
Fix for "Can't set headers after they are sent"
2018-07-23 16:15:11 +02:00
Rafa de la Torre
a5046f58de
Add dbRemainingQuota check to COPY FROM
2018-07-23 15:05:22 +02:00
Rafa de la Torre
84d88f91fb
Better implementation of dbQuotaMiddleware
2018-07-23 15:02:20 +02:00
Rafa de la Torre
7e54912ee4
First version of the dbQuotaMiddleware
2018-07-23 14:38:56 +02:00
Rafa de la Torre
ea46db5923
Remove uneeded timeout middleware
2018-07-20 17:48:56 +02:00
Rafa de la Torre
cc348f3725
Make the SET statement_timeout async
2018-07-20 16:25:47 +02:00
Rafa de la Torre
17c8cbc723
Add a timeout setting for COPY commands
2018-07-20 15:32:32 +02:00
Rafa de la Torre
763134437b
Use the db_batch_port for copy
2018-07-20 10:22:54 +02:00
Eneko Lakasta
1bf8a06ec2
Rephrase error message
2018-06-29 13:19:32 +02:00
Eneko Lakasta
99058e68af
add space to message
2018-06-27 15:53:51 +02:00
Eneko Lakasta
2bb28b5578
remove line break in the error message
2018-06-27 15:43:06 +02:00
Eneko Lakasta
879e85d07f
add 'SQL query timeout expired error.' to DB timeout limit error message.
2018-06-27 15:17:38 +02:00
Simon Martín
59dd495a87
small style details
2018-06-22 10:50:39 +02:00
Simon Martín
06c40a396b
copy from: events is series
2018-06-21 15:58:23 +02:00
Simon Martín
195a4866c3
managing gzip pipe with PassThrough
2018-06-21 15:54:42 +02:00
Simon Martín
f5641a1873
remove unneeded this.connectionClosedByClient
2018-06-21 15:18:26 +02:00
Simon Martín
fae18fb242
using getPGStream
2018-06-21 15:17:32 +02:00
Simon Martín
823e3abcfe
unify to and from method in only one getPGStream
2018-06-21 15:16:47 +02:00
Simon Martín
3ede1ea9f1
unify streams
2018-06-21 14:53:07 +02:00
Simon Martín
f4651cadae
using this.copyFromStream
2018-06-21 14:49:38 +02:00
Simon Martín
9970052e1b
copy from: removing copyFromStream dependency
2018-06-21 14:38:22 +02:00
Simon Martín
01eff0a5a2
copy from: removing done dependecy
2018-06-21 14:36:40 +02:00
Simon Martín
a15c2e58aa
copy from: manage error in 1 point
2018-06-21 14:36:03 +02:00
Simon Martín
d17a5608fb
copy from: unneeded pgstream.end in req error
2018-06-21 14:32:36 +02:00
Simon Martín
5a28d36d69
copy from: pgstream events in series
2018-06-21 14:29:19 +02:00
Simon Martín
925115e8c1
copy from: remove client dependecy
2018-06-21 14:25:54 +02:00
Simon Martín
327aaf3450
jshint happy
2018-06-21 13:41:13 +02:00
Simon Martín
bafcb8051f
StreamCopy getResults refactor
...
- changing get Result to getRowCount
- ensuring to and from independecy in StreamCopy
2018-06-21 13:39:33 +02:00
Simon Martín
2cecc54cb2
copy to: removing copyToStream dependecy
2018-06-21 13:21:23 +02:00
Simon Martín
e39a9b2bf7
copy to: manage error in 1 point
2018-06-21 13:13:37 +02:00
Simon Martín
1dac5355d1
copy to: remove unneeded done and removing it as a dependency
2018-06-21 13:13:16 +02:00
Simon Martín
42a94a3b6c
copy to: removing unnedeed client dependecy
2018-06-21 13:01:59 +02:00
Simon Martín
e6448a3255
copy to: cancelQuery to streamCopy
2018-06-21 13:00:57 +02:00
Simon Martín
39dfcb1f9b
copy to events is series
2018-06-21 12:50:50 +02:00
Simon Martín
5fbd6e8e3a
req on error is neccesary when connection is canceled
2018-06-21 12:49:06 +02:00
Simon Martín
3da32308ce
jshint happy
2018-06-21 12:30:49 +02:00
Simon Martín
6955d254f8
Merge branch 'master' into pgcopy-stream-refactor2-dgaubert
2018-06-21 12:19:16 +02:00
Simon Martín
3d30f1f762
trailing white spaces
2018-06-19 10:34:52 +02:00
Simon Martín
30402f2e89
remove unused statsClient
2018-06-19 10:04:17 +02:00
Simon Martín
7727a9d506
data ingestion logger created in controller and added to app
2018-06-18 18:48:11 +02:00
Simon Martín
32dca8dfdb
reopenFileStreams to parent class
2018-06-18 18:47:12 +02:00
Simon Martín
8f727fc15a
cdb-user in logs
2018-06-15 13:30:42 +02:00
Simon Martín
8a450a862c
recover the metrics header (needed for tests purposes)
2018-06-15 13:25:47 +02:00
Simon Martín
c7e01f2ed6
success in copy log
2018-06-14 18:40:06 +02:00
Simon Martín
9097cbe16e
adding log of the error when headers are already sent
2018-06-14 18:29:50 +02:00
Simon Martín
39e5395edc
change gzip to isGzip
2018-06-14 18:29:03 +02:00
Simon Martín
4cb7de7318
changing let for const
2018-06-14 10:51:05 +02:00
Simon Martín
b091a375fc
fix ensuring right validation error
2018-06-14 10:49:36 +02:00
Simon Martín
1f31b8e2ae
removing old copy logs
2018-06-14 10:45:23 +02:00
Daniel García Aubert
9299fe51ce
Add todo
2018-06-14 01:01:05 +02:00
Daniel García Aubert
8eab164e4e
Be more one liner
2018-06-14 00:54:03 +02:00
Daniel García Aubert
9c8d802f75
Chain response event callbacks after .pipe() in order to improve readability in copyto controller
2018-06-14 00:31:08 +02:00
Daniel García Aubert
508d688b9d
Use request 'close' event when the request is closed unexpectedly by the client
2018-06-14 00:16:06 +02:00
Daniel García Aubert
37a079363a
Do not extend from EventEmitter
2018-06-13 20:12:56 +02:00
Daniel García Aubert
bd8e4be596
Do not emit to regist the number of rows processed in copyfrom
2018-06-13 20:11:39 +02:00
Daniel García Aubert
5655f26fbd
Do not emit to regist the number of rows processed in copyto
2018-06-13 20:06:35 +02:00
Daniel García Aubert
eb4ba60ba0
Pass the error to the pg connection's callback
2018-06-13 19:56:56 +02:00
Daniel García Aubert
e6a331433a
Handle error when client disconnects in copyto requests
2018-06-13 19:54:34 +02:00
Daniel García Aubert
6aba1ea307
Handle error when sql injected in pgstream fails (e.g: unexisting table)
2018-06-13 19:52:53 +02:00
Daniel García Aubert
2792d95c41
Do not handle stream unpiping when it wasn't even created
2018-06-13 18:30:05 +02:00
Daniel García Aubert
b9971d94dd
Do not use the regular callback in streams events
2018-06-13 18:28:34 +02:00
Daniel García Aubert
7d1a0abadd
Preventing callbacks from accidentally being called twice
2018-06-13 18:26:58 +02:00
Eneko Lakasta
f5d56a2253
FIX add return to callback call
2018-06-13 13:17:01 +02:00
Simon Martín
270daae579
removing forgotten callback
2018-06-12 18:40:51 +02:00
Simon Martín
409273bafe
removing 2nd calback using events
2018-06-12 18:39:50 +02:00
Simon Martín
19aae3c40f
manage pgstrem error on stream copy class
2018-06-12 17:04:44 +02:00
Simon Martín
ba2f85421d
StreamCopy to class
2018-06-12 16:56:18 +02:00
Simon Martín
8a15b2f8d1
Merge branch 'master' into pgcopy-stream-refactor2
2018-06-12 15:21:28 +02:00
Simon Martín
4d99872975
return in next err
2018-06-12 15:18:28 +02:00
Simon Martín
b9a0fa78d2
adding metadataBackend to user middleware in copycontroller
2018-06-11 14:55:37 +02:00
Simon Martín
1174499dcf
Merge branch 'master' into pgcopy-stream
2018-06-11 14:38:52 +02:00
Simon Martín
df63e0eedb
adding log of the error when headers are already sent
2018-06-11 13:19:12 +02:00
Simon Martín
2995676adb
let to const
2018-06-11 12:56:16 +02:00
Simon Martín
4fab96cb3a
rename gzip to isGzip
2018-06-11 12:55:30 +02:00
Simon Martín
87fd9db1fe
remove uneeded comment
2018-06-11 12:43:35 +02:00
Simon Martín
2581ddbfb4
better formatting
2018-06-08 17:09:28 +02:00
Simon Martín
b63806f8de
cleaning stream copy
2018-06-08 17:04:28 +02:00
Simon Martín
6a9e7d7c14
forgotten )
2018-06-08 17:04:10 +02:00
Simon Martín
7266b3c708
one logger
2018-06-08 17:02:31 +02:00
Simon Martín
014f0a1491
refactor of copy to PoC
2018-06-08 16:58:32 +02:00
Simon Martín
f7454228c6
refactor of copy from PoC
2018-06-08 16:50:12 +02:00
Simon Martín
e259a51623
unifiyng to see 'the forest'
2018-06-08 15:03:21 +02:00
Simon Martín
8e8ccc9c67
removing old metrics in copy
2018-06-08 13:33:17 +02:00
Simon Martín
c62ef15dda
using StreamCopy in controller
2018-06-08 11:15:23 +02:00
Simon Martín
732c229e63
creating StreamCopy class
2018-06-08 11:09:51 +02:00
Simon Martín
2bcea0484a
rename BunyanLogger to Logger
2018-06-08 10:59:34 +02:00
Eneko Lakasta
440ba8c840
make function checkUserExists node callback pattern compilant
2018-06-06 15:48:22 +02:00
Eneko Lakasta
340e55ea46
reduce error info
2018-06-06 15:23:53 +02:00
Eneko Lakasta
ca8ae503ac
Merge branch 'master' into remove-auth-fallback
2018-06-05 16:19:36 +02:00
Eneko Lakasta
eab3d289b6
refactor authenticated to authorizationLevel
2018-06-05 13:21:56 +02:00
Eneko Lakasta
da08e42921
refactor forceToBeAuthenticated to forceToBeMaster
2018-06-05 13:16:36 +02:00
Simon Martín
66af518deb
jshint
2018-06-05 13:16:14 +02:00
Eneko Lakasta
0207b67d5f
refactor forceToBeAuthenticated to forceToBeMaster
2018-06-05 13:14:50 +02:00
Eneko Lakasta
a4292f08cf
refactor apikey to apikeyToken
2018-06-05 13:13:09 +02:00
Simon Martín
29d1fb1274
logging GzipSize
2018-06-05 13:02:14 +02:00
Simon Martín
02cc606be1
gzipSize support in metrics
2018-06-05 13:01:15 +02:00
Simon Martín
dd689d3568
log metrics only once
2018-06-05 12:36:36 +02:00
Simon Martín
550c736032
metrics in errors
2018-06-05 12:35:52 +02:00
Rafa de la Torre
d2c0e68a78
Merge pull request #497 from CartoDB/cancel-copy-to-client-disconnect
...
Cancel copy to upon client disconnect
2018-06-05 09:59:49 +02:00
Paul Ramsey
d6d9022c7f
Coerce format string to upper for better log consistency
2018-06-04 13:12:06 -07:00
Simon Martín
66f7ab45fe
release connection with error
2018-06-04 20:51:21 +02:00
Simon Martín
7b6056b799
using the correct errors in done
2018-06-04 20:36:16 +02:00
Rafa de la Torre
014ea8142b
A cleaner approach to the cancel command
2018-06-04 19:06:11 +02:00
Simon Martín
2f2dcfd762
fix copy format case
2018-06-04 18:15:28 +02:00
Simon Martín
a844b5d31d
jshint happy
2018-06-04 18:15:00 +02:00
Simon Martín
1cf7032c9a
adding user and date in copy logs
2018-06-04 18:08:34 +02:00
Rafa de la Torre
994e8a702b
Add callbacks to pg.connect
...
And call them to return connections to the pool.
2018-05-31 18:59:19 +02:00
Rafa de la Torre
b59ae1d057
Make sure the COPY TO query is cancelled
...
Issue a CancelRequest upon client disconnection, to make sure the COPY
TO query is cancelled and the connection/session is put back to the pg
pool.
2018-05-31 17:27:35 +02:00
Rafa de la Torre
332f7096d3
Listen to response events (on behalf of @oleurud)
2018-05-31 17:06:19 +02:00
Rafa de la Torre
0eab03a7e7
Add a more informative message
2018-05-31 16:41:22 +02:00
Eneko Lakasta
46e3a87f41
add some auth TODOs
2018-05-30 18:15:35 +02:00
Eneko Lakasta
cf8bf6e5e6
get DB configuration from job instead from metadata in Job Canceller
2018-05-30 17:36:55 +02:00
Simon Martín
310f652ae4
send CopyFail when user close connection
2018-05-30 12:59:49 +02:00
Eneko Lakasta
7764975c09
please jshint: remove unnecessary require (underscore)
2018-05-30 12:08:04 +02:00
Simon Martín
a083eb909c
fix req events
2018-05-29 16:19:53 +02:00
Eneko Lakasta
fa5a99211c
check user exists in user middleware
...
This way, we keep sending a 404 error if the user does not exist.
2018-05-29 13:23:50 +02:00
Simon Martín
008fad3d13
undo 'ensuring data ingestion log'
2018-05-29 12:45:57 +02:00
