Daniel García Aubert
2792d95c41
Do not handle stream unpiping when it wasn't even created
2018-06-13 18:30:05 +02:00
Daniel García Aubert
b9971d94dd
Do not use the regular callback in streams events
2018-06-13 18:28:34 +02:00
Daniel García Aubert
7d1a0abadd
Preventing callbacks from accidentally being called twice
2018-06-13 18:26:58 +02:00
Eneko Lakasta
f5d56a2253
FIX add return to callback call
2018-06-13 13:17:01 +02:00
Simon Martín
270daae579
removing forgotten callback
2018-06-12 18:40:51 +02:00
Simon Martín
409273bafe
removing 2nd calback using events
2018-06-12 18:39:50 +02:00
Simon Martín
19aae3c40f
manage pgstrem error on stream copy class
2018-06-12 17:04:44 +02:00
Simon Martín
ba2f85421d
StreamCopy to class
2018-06-12 16:56:18 +02:00
Simon Martín
8a15b2f8d1
Merge branch 'master' into pgcopy-stream-refactor2
2018-06-12 15:21:28 +02:00
Simon Martín
4d99872975
return in next err
2018-06-12 15:18:28 +02:00
Simon Martín
b9a0fa78d2
adding metadataBackend to user middleware in copycontroller
2018-06-11 14:55:37 +02:00
Simon Martín
1174499dcf
Merge branch 'master' into pgcopy-stream
2018-06-11 14:38:52 +02:00
Simon Martín
df63e0eedb
adding log of the error when headers are already sent
2018-06-11 13:19:12 +02:00
Simon Martín
2995676adb
let to const
2018-06-11 12:56:16 +02:00
Simon Martín
4fab96cb3a
rename gzip to isGzip
2018-06-11 12:55:30 +02:00
Simon Martín
87fd9db1fe
remove uneeded comment
2018-06-11 12:43:35 +02:00
Simon Martín
2581ddbfb4
better formatting
2018-06-08 17:09:28 +02:00
Simon Martín
b63806f8de
cleaning stream copy
2018-06-08 17:04:28 +02:00
Simon Martín
6a9e7d7c14
forgotten )
2018-06-08 17:04:10 +02:00
Simon Martín
7266b3c708
one logger
2018-06-08 17:02:31 +02:00
Simon Martín
014f0a1491
refactor of copy to PoC
2018-06-08 16:58:32 +02:00
Simon Martín
f7454228c6
refactor of copy from PoC
2018-06-08 16:50:12 +02:00
Simon Martín
e259a51623
unifiyng to see 'the forest'
2018-06-08 15:03:21 +02:00
Simon Martín
8e8ccc9c67
removing old metrics in copy
2018-06-08 13:33:17 +02:00
Simon Martín
c62ef15dda
using StreamCopy in controller
2018-06-08 11:15:23 +02:00
Simon Martín
732c229e63
creating StreamCopy class
2018-06-08 11:09:51 +02:00
Simon Martín
2bcea0484a
rename BunyanLogger to Logger
2018-06-08 10:59:34 +02:00
Eneko Lakasta
440ba8c840
make function checkUserExists node callback pattern compilant
2018-06-06 15:48:22 +02:00
Eneko Lakasta
340e55ea46
reduce error info
2018-06-06 15:23:53 +02:00
Eneko Lakasta
ca8ae503ac
Merge branch 'master' into remove-auth-fallback
2018-06-05 16:19:36 +02:00
Eneko Lakasta
eab3d289b6
refactor authenticated to authorizationLevel
2018-06-05 13:21:56 +02:00
Eneko Lakasta
da08e42921
refactor forceToBeAuthenticated to forceToBeMaster
2018-06-05 13:16:36 +02:00
Simon Martín
66af518deb
jshint
2018-06-05 13:16:14 +02:00
Eneko Lakasta
0207b67d5f
refactor forceToBeAuthenticated to forceToBeMaster
2018-06-05 13:14:50 +02:00
Eneko Lakasta
a4292f08cf
refactor apikey to apikeyToken
2018-06-05 13:13:09 +02:00
Simon Martín
29d1fb1274
logging GzipSize
2018-06-05 13:02:14 +02:00
Simon Martín
02cc606be1
gzipSize support in metrics
2018-06-05 13:01:15 +02:00
Simon Martín
dd689d3568
log metrics only once
2018-06-05 12:36:36 +02:00
Simon Martín
550c736032
metrics in errors
2018-06-05 12:35:52 +02:00
Rafa de la Torre
d2c0e68a78
Merge pull request #497 from CartoDB/cancel-copy-to-client-disconnect
...
Cancel copy to upon client disconnect
2018-06-05 09:59:49 +02:00
Paul Ramsey
d6d9022c7f
Coerce format string to upper for better log consistency
2018-06-04 13:12:06 -07:00
Simon Martín
66f7ab45fe
release connection with error
2018-06-04 20:51:21 +02:00
Simon Martín
7b6056b799
using the correct errors in done
2018-06-04 20:36:16 +02:00
Rafa de la Torre
014ea8142b
A cleaner approach to the cancel command
2018-06-04 19:06:11 +02:00
Simon Martín
2f2dcfd762
fix copy format case
2018-06-04 18:15:28 +02:00
Simon Martín
a844b5d31d
jshint happy
2018-06-04 18:15:00 +02:00
Simon Martín
1cf7032c9a
adding user and date in copy logs
2018-06-04 18:08:34 +02:00
Rafa de la Torre
994e8a702b
Add callbacks to pg.connect
...
And call them to return connections to the pool.
2018-05-31 18:59:19 +02:00
Rafa de la Torre
b59ae1d057
Make sure the COPY TO query is cancelled
...
Issue a CancelRequest upon client disconnection, to make sure the COPY
TO query is cancelled and the connection/session is put back to the pg
pool.
2018-05-31 17:27:35 +02:00
Rafa de la Torre
332f7096d3
Listen to response events (on behalf of @oleurud)
2018-05-31 17:06:19 +02:00
Rafa de la Torre
0eab03a7e7
Add a more informative message
2018-05-31 16:41:22 +02:00
Eneko Lakasta
46e3a87f41
add some auth TODOs
2018-05-30 18:15:35 +02:00
Eneko Lakasta
cf8bf6e5e6
get DB configuration from job instead from metadata in Job Canceller
2018-05-30 17:36:55 +02:00
Simon Martín
310f652ae4
send CopyFail when user close connection
2018-05-30 12:59:49 +02:00
Eneko Lakasta
7764975c09
please jshint: remove unnecessary require (underscore)
2018-05-30 12:08:04 +02:00
Simon Martín
a083eb909c
fix req events
2018-05-29 16:19:53 +02:00
Eneko Lakasta
fa5a99211c
check user exists in user middleware
...
This way, we keep sending a 404 error if the user does not exist.
2018-05-29 13:23:50 +02:00
Simon Martín
008fad3d13
undo 'ensuring data ingestion log'
2018-05-29 12:45:57 +02:00
Eneko Lakasta
ebe04d04ad
refactor user database service. Get proper DB config based on auth type: oauth vs apikey
...
Oauth uses only master api key configuration
Api key uses master and the provided api key configurations
Also move default api key fallback to this service
2018-05-28 15:53:51 +02:00
Simon Martín
e092647246
ensuring data ingestion log
2018-05-28 11:25:09 +02:00
Simon Martín
7d32ae293f
rename buyan_logger file name
2018-05-28 11:18:30 +02:00
Simon Martín
3cf28bb7ff
jshint happy
2018-05-25 18:50:56 +02:00
Simon Martín
22caa049ee
StreamCopyMetrics
2018-05-25 18:47:41 +02:00
Simon Martín
b31984cbc6
simplify response
2018-05-25 17:57:40 +02:00
Simon Martín
1fa5afd759
unify error handler
2018-05-25 17:50:59 +02:00
Simon Martín
6c3f9cf1d3
simplify controllers
2018-05-25 17:42:30 +02:00
Eneko Lakasta
d3b6ebd260
Fallback to default api key if no api key is provided
...
What happens with oauth authorization? This way we will always have an api_key and oauth will never trigger....
2018-05-25 17:28:56 +02:00
Eneko Lakasta
ec75227aa2
use apikeys instead of metadata for the database connections parameters
2018-05-25 17:23:24 +02:00
Simon Martín
fd70673d88
more details
2018-05-25 16:46:25 +02:00
Simon Martín
8a2c777577
details
2018-05-25 16:00:27 +02:00
Simon Martín
9b0b92fb6b
removing try catch
2018-05-25 15:46:12 +02:00
Simon Martín
5bed04a38c
fix gzip problem with req close event
2018-05-25 15:34:30 +02:00
Simon Martín
fe3bd4fd37
handling errors from request in COPYfrom
2018-05-25 14:33:54 +02:00
Simon Martín
e2bba967f9
handling copyto errors
2018-05-24 19:48:24 +02:00
Eneko Lakasta
6bc90eb9b3
fix typo: Backed -> Backend
2018-05-24 11:04:14 +02:00
Simon Martín
fca6ee8232
changing name of dataIngestionLogPath
2018-05-23 17:32:44 +02:00
Simon Martín
b2a36eb556
copy metrics with BunyanLogger
2018-05-23 17:25:46 +02:00
Simon Martín
fdc542f7b5
BunyanLogger
2018-05-23 17:24:48 +02:00
Simon Martín
5ba7dca79c
copyfrom metrics to kibana
2018-05-23 10:30:37 +02:00
Paul Ramsey
f77707f6da
Fix reponse copy from
2018-05-22 10:48:11 -04:00
Paul Ramsey
a4654f82b2
Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q
2018-05-22 10:37:50 -04:00
Paul Ramsey
ed696a96ec
Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q
2018-05-22 10:23:04 -04:00
Simon Martín
6d73d97ae2
copyFrom metrics to statsd
2018-05-22 16:02:14 +02:00
Simon Martín
e347985465
changing query parameter name from sql to q
...
unify query validatrion
2018-05-22 15:42:57 +02:00
Simon Martín
b9474e7fc3
total_rows in copyto metrics
2018-05-22 15:29:08 +02:00
Simon Martín
0bac7a484c
improving query info
2018-05-22 14:16:22 +02:00
Simon Martín
185f708d5b
small style refactor
2018-05-22 14:15:55 +02:00
Simon Martín
86e0ebc148
small refactor in copy format regex
2018-05-22 12:02:41 +02:00
Simon Martín
f01191472b
refactoring copy controller middlewares
2018-05-22 11:56:50 +02:00
Simon Martín
433bd01c27
copyto metrics
2018-05-22 11:54:10 +02:00
Simon Martín
81be15fbc3
adding format to copy metrics
2018-05-21 19:13:44 +02:00
Paul Ramsey
64fc0c32e3
Change from 'sql' as query parameter to 'q', aping existing sql api
2018-05-21 07:47:16 -07:00
Simon Martín
816779fefd
erratum
2018-05-18 17:32:59 +02:00
Simon Martín
82c9fec18a
copy from metrics
2018-05-18 17:31:29 +02:00
Eneko Lakasta
fc7e246704
please jshint
2018-05-18 11:35:54 +02:00
Simon Martín
bf4d12868b
Merge branch 'master' into pgcopy-stream
2018-05-18 10:57:32 +02:00
Eneko Lakasta
1e8c6e198c
remove api key fallback
2018-05-17 17:13:59 +02:00
Simon Martín
9016647a6f
reverting use of busboy instead of multer
2018-05-14 18:31:44 +02:00
Simon Martín
60b48df4d6
default rate limits params to server.js
2018-05-11 15:41:35 +02:00
Simon Martín
d70013b0f4
object assing for default rate limits params
2018-05-11 15:36:59 +02:00
Simon Martín
bc8a867334
jshinty happy
2018-05-11 14:51:43 +02:00
Simon Martín
f2acc8f653
forgotten log
2018-05-11 14:16:26 +02:00
Simon Martín
eee8bc2674
moving copyto headers to correct position
2018-05-11 14:12:23 +02:00
Simon Martín
791967877c
addining gzip support to copyfrom
2018-05-11 13:33:54 +02:00
Simon Martín
f749798ca5
validating inputs
2018-05-11 10:38:33 +02:00
Paul Ramsey
8f211f905c
Remove wrong examples
2018-05-10 12:21:33 -07:00
Paul Ramsey
f0021264d2
Change from form-multipart to POST with chunked upload
2018-05-10 12:11:55 -07:00
Simon Martín
0bb7cec9f4
params existence responsability for UserLimits
2018-05-10 11:51:58 +02:00
Simon Martín
d400926387
using busboy instead of multer
2018-05-08 18:39:16 +02:00
Simon Martín
a997881048
ensuring existence of rate limits params in code
2018-05-08 16:00:16 +02:00
Simon Martín
82d14ab98f
busboy configuration
2018-05-08 13:15:19 +02:00
Simon Martín
7c4409bcf5
from var to const/let
2018-05-08 13:10:25 +02:00
Simon Martín
c40ab801b1
refactoring handleCopyTo
2018-05-08 13:08:29 +02:00
Simon Martín
b036b876ff
using Busboy instead of multer
2018-05-08 12:52:33 +02:00
Simon Martín
6b96032fcf
ensuring return error
2018-05-04 16:50:17 +02:00
Simon Martín
383b86cf9b
simplify sql paramter handler and removing underscore
2018-05-04 16:47:50 +02:00
Simon Martín
2b066eadfa
removing unused dependencies
2018-05-04 15:35:23 +02:00
Simon Martín
02238fefe1
moving body-parser from global to routes of query and job
2018-05-04 15:15:37 +02:00
Simon Martín
0346cf11d8
enabling multipart/form-data again
2018-05-04 15:11:45 +02:00
Simon Martín
4322ccdd09
adding new endpoints to rate limits
2018-05-03 18:50:13 +02:00
Simon Martín
af4c5906c8
adding /sql tro the endpoints
2018-05-03 18:31:49 +02:00
Paul Ramsey
36bdf84f71
Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream
2018-04-27 15:56:13 +02:00
Paul Ramsey
e2d8896307
Typo in curl example
2018-04-27 12:32:47 +02:00
Paul Ramsey
5f9cc37dba
Jshint change, DRY the content-disposition handler
2018-04-27 12:19:53 +02:00
Paul Ramsey
fe52af71ac
Add documentation SQL API copyfrom/copyto end points,
...
and make small modifications arising from that task
2018-04-27 11:55:06 +02:00
Paul Ramsey
c78cac9bd6
Use the correct database credentials, so we obey security
2018-04-26 10:45:52 +02:00
Paul Ramsey
f13028b497
More jshint changes
2018-04-26 10:20:21 +02:00
Simon Martín
5f8e1d8f53
fix validatePGEntitiesAccess access
2018-04-26 10:13:03 +02:00
Paul Ramsey
cc5fa5c6ce
Changes suggested by jshint
2018-04-26 10:04:38 +02:00
Paul Ramsey
ea66076255
Remove 'console.debug', try and get travis tests clean
2018-04-26 09:40:13 +02:00
Paul Ramsey
a98781d335
Working code for /copyfrom (streaming) and /copyto (streaming)
2018-04-25 18:37:04 +02:00
Paul Ramsey
b59bda5780
Get working copyto implementation in place. Many test cases
...
needed still and thoughts on how to communicate errors back
that are maybe nicer than stack messages?
2018-04-24 15:55:20 +02:00
Simon Martín
e12e16507a
create object and export
2018-04-24 13:21:43 +02:00
Paul Ramsey
4914100205
CopyFrom works, but still needs a decent return payload and
...
a lot of work on returning useful information for error cases (post empty input,
db errors returned more nicely? etc)
2018-04-24 13:07:57 +02:00
Paul Ramsey
72bce5732b
WIP on /copyfrom
2018-04-24 11:26:15 +02:00
Simon Martín
3b228ac17b
solving jshint crazy error
2018-04-24 10:10:37 +02:00
Simon Martín
985f9ef7f5
pgEntitiesAccessValidator
2018-04-23 18:17:44 +02:00
Rafa de la Torre
2229d0ee57
Fix jshint issues
2018-04-13 16:36:43 +02:00
Paul Ramsey
0161696627
Update copy_controller.js
2018-04-13 05:43:23 -07:00
Paul Ramsey
d54e2f5a07
Implementation including multer, custom storage engine, and pg-copy,
...
but without turning over pg-copy, and demonstrating the missing 'sql'
parameter in the custom storage engine.
2018-04-12 12:25:28 -07:00
Simon Martín
2a729b7da8
destructuring error_handler parameters
2018-04-03 13:43:17 +02:00
Simon Martín
3f6b6d3cff
middleware naming
2018-04-03 13:41:18 +02:00
Simon Martín
db6ab4a591
let & const vars
2018-03-28 13:52:45 +02:00
Simon Martín
adab7f21d8
setErrorHeader refactor
2018-03-28 13:50:45 +02:00
Simon Martín
720c3eb8cb
changing msg var name for errorResponse
2018-03-28 13:35:19 +02:00
Simon Martín
4301b63189
fix error
2018-03-28 13:31:31 +02:00
Simon Martín
616f3f3c07
errorHandlerFactory
2018-03-28 13:06:39 +02:00
Simon Martín
16ea727fc3
move ErrorHandler logic from error middleware
2018-03-28 12:58:10 +02:00
Simon Martín
5b6f876cd6
change pgErrorHandler for more generic errorHandler
2018-03-28 12:50:07 +02:00
Simon Martín
b45f5fe31e
move ErrorHandler to services folder
2018-03-28 12:49:33 +02:00
Simon Martín
287d4634c6
rename createPgError to createGenericError
2018-03-28 12:43:34 +02:00
Simon Martín
e611d21029
getName refactor
2018-03-28 12:38:45 +02:00
Simon Martín
2bfa00ab57
getStatus refactor
2018-03-28 12:35:01 +02:00
Simon Martín
d727a50407
jshint
2018-03-28 12:17:42 +02:00
Simon Martín
96435d28e0
removing underscore dependency
2018-03-28 12:15:55 +02:00
Simon Martín
e2641200f7
extracting error type check
2018-03-28 12:13:31 +02:00
Simon Martín
575f1b3a05
simplify error response on models
2018-03-28 11:55:52 +02:00
Simon Martín
050c598e90
using new error handler as a Error
2018-03-28 11:30:11 +02:00
Simon Martín
2722302a89
using error handler as a Error
2018-03-28 11:29:38 +02:00
Simon Martín
60021a70dc
extends Error
2018-03-28 10:56:06 +02:00
Simon Martín
522363c419
removing semicolons
2018-03-28 10:49:50 +02:00
Simon Martín
eefea03b80
error handler to class
2018-03-28 10:48:25 +02:00
Simon Martín
67e1c04efd
indentation details
2018-03-28 10:48:08 +02:00
Simon Martín
c6eb0ad4c7
error middleware name
2018-03-28 09:58:05 +02:00
Simon Martín
0998e50014
refactor getFields method
2018-03-27 15:56:53 +02:00
Simon Martín
b75334f1e7
Merge branch 'master' into limits-error-refactor
2018-03-27 15:32:24 +02:00
Simon Martín
483c263f61
adding and testing limits error response with params
2018-03-26 19:10:23 +02:00
Simon Martín
54876fa203
addind properties to limits errors
2018-03-26 15:56:50 +02:00
Simon Martín
50b8a9ed17
remove unused parameter
2018-03-23 18:02:44 +01:00
Simon Martín
0b708165b3
simplify timeout error catch
2018-03-23 17:48:21 +01:00
Simon Martín
8c86c2b3f8
correct error message in rate limit
2018-03-23 15:35:06 +01:00
Simon Martín
26baaf6081
removing retry after when no necessary
2018-03-23 15:09:16 +01:00
Simon Martín
80818c3cbb
fix userLimits reference
2018-03-20 16:20:56 +01:00
Simon Martín
581bd0dd89
Merge branch 'master' into rateLimits
2018-03-14 12:47:45 +01:00
Simon Martín
1ffeb456e8
Headers following rfc6648
2018-03-14 12:17:26 +01:00
Daniel García Aubert
8dea7615c4
Use user's port instead of the generic one
2018-03-13 19:08:51 +01:00
Daniel García Aubert
328d6f253e
Honor batch api port
2018-03-13 18:49:01 +01:00
Eneko Lakasta
2ec767c1fc
Set Vary Header to honor Authorization header when caching (fastly)
2018-03-13 11:59:07 +01:00
Simon Martín
9cd4e85768
using cartodb-redis in rate limits
2018-03-03 15:01:45 +01:00
Simon Martín
b6ff37d33f
changing param name
2018-03-02 19:57:33 +01:00
Simon Martín
967cdf8a06
refactoring user_limits
2018-03-02 19:54:22 +01:00
Simon Martín
5cd7c24776
refactoring middleware and updating tests
2018-03-02 16:42:45 +01:00
Simon Martín
a11d9870a3
exporting RATE_LIMIT_REDIS_DB
2018-03-02 13:17:50 +01:00
Simon Martín
25d1e5442c
new redis keys
2018-03-01 15:51:19 +01:00
Simon Martín
7a6ef637ae
make jshint happy
2018-03-01 14:47:34 +01:00
Simon Martín
f3adad57aa
fix assignment
2018-03-01 13:15:32 +01:00
Simon Martín
779ab3b83f
adding rate limit middleware to controllers
2018-03-01 12:19:57 +01:00
Simon Martín
a3dfd2767b
rate limit middleware
2018-03-01 12:19:28 +01:00
Simon Martín
2834975d14
controllers using user limits service
2018-03-01 11:31:35 +01:00
Simon Martín
3fdb7abeaf
user limits service
2018-03-01 11:30:34 +01:00
Daniel García Aubert
5802b2182e
Improve profiling
2018-02-27 13:56:00 +01:00
Daniel García Aubert
78f8c430ea
Merge branch 'master' into auth-api
2018-02-27 12:24:40 +01:00
Daniel García Aubert
6f4c79eaef
Fix common interface for auth-backends
2018-02-26 19:02:05 +01:00
Daniel García Aubert
418ec1304f
Fix issue with auth fallback, it wasn't authenticated when apikey master was provided
2018-02-26 18:13:49 +01:00
Daniel García Aubert
cf0214f5c3
Respond with 403 intead of 401 when the request has no permissions to the specific resource
2018-02-23 15:50:23 +01:00
Daniel García Aubert
f85bdf53a5
Missing middleware
2018-02-23 13:19:26 +01:00
Daniel García Aubert
897d38c3f4
Do not set undefined step
2018-02-23 13:05:41 +01:00
Daniel García Aubert
47b54612c7
Place send-response-middleware to the very end of middleware stack
2018-02-23 13:03:56 +01:00
Rafa de la Torre
90b0ec61c2
Add RESIZE=yes param to shapefiles export
...
Add the parameter `RESIZE=yes` to gdal shapefile driver. This resizes
fields to their optimal size. See
http://www.gdal.org/drv_shapefile.html for details.
This is needed because some customers are having trouble exporting
shapefiles, then reimporting into ArcGIS.
2018-02-23 10:34:44 +01:00
Daniel García Aubert
3807d9f94d
Template for User not found error
2018-02-22 17:49:02 +01:00
