Simon Martín
1cf7032c9a
adding user and date in copy logs
2018-06-04 18:08:34 +02:00
Rafa de la Torre
994e8a702b
Add callbacks to pg.connect
...
And call them to return connections to the pool.
2018-05-31 18:59:19 +02:00
Rafa de la Torre
b59ae1d057
Make sure the COPY TO query is cancelled
...
Issue a CancelRequest upon client disconnection, to make sure the COPY
TO query is cancelled and the connection/session is put back to the pg
pool.
2018-05-31 17:27:35 +02:00
Rafa de la Torre
332f7096d3
Listen to response events (on behalf of @oleurud)
2018-05-31 17:06:19 +02:00
Rafa de la Torre
0eab03a7e7
Add a more informative message
2018-05-31 16:41:22 +02:00
Eneko Lakasta
46e3a87f41
add some auth TODOs
2018-05-30 18:15:35 +02:00
Eneko Lakasta
cf8bf6e5e6
get DB configuration from job instead from metadata in Job Canceller
2018-05-30 17:36:55 +02:00
Simon Martín
310f652ae4
send CopyFail when user close connection
2018-05-30 12:59:49 +02:00
Eneko Lakasta
7764975c09
please jshint: remove unnecessary require (underscore)
2018-05-30 12:08:04 +02:00
Simon Martín
a083eb909c
fix req events
2018-05-29 16:19:53 +02:00
Eneko Lakasta
fa5a99211c
check user exists in user middleware
...
This way, we keep sending a 404 error if the user does not exist.
2018-05-29 13:23:50 +02:00
Simon Martín
008fad3d13
undo 'ensuring data ingestion log'
2018-05-29 12:45:57 +02:00
Eneko Lakasta
ebe04d04ad
refactor user database service. Get proper DB config based on auth type: oauth vs apikey
...
Oauth uses only master api key configuration
Api key uses master and the provided api key configurations
Also move default api key fallback to this service
2018-05-28 15:53:51 +02:00
Simon Martín
e092647246
ensuring data ingestion log
2018-05-28 11:25:09 +02:00
Simon Martín
7d32ae293f
rename buyan_logger file name
2018-05-28 11:18:30 +02:00
Simon Martín
3cf28bb7ff
jshint happy
2018-05-25 18:50:56 +02:00
Simon Martín
22caa049ee
StreamCopyMetrics
2018-05-25 18:47:41 +02:00
Simon Martín
b31984cbc6
simplify response
2018-05-25 17:57:40 +02:00
Simon Martín
1fa5afd759
unify error handler
2018-05-25 17:50:59 +02:00
Simon Martín
6c3f9cf1d3
simplify controllers
2018-05-25 17:42:30 +02:00
Eneko Lakasta
d3b6ebd260
Fallback to default api key if no api key is provided
...
What happens with oauth authorization? This way we will always have an api_key and oauth will never trigger....
2018-05-25 17:28:56 +02:00
Eneko Lakasta
ec75227aa2
use apikeys instead of metadata for the database connections parameters
2018-05-25 17:23:24 +02:00
Simon Martín
fd70673d88
more details
2018-05-25 16:46:25 +02:00
Simon Martín
8a2c777577
details
2018-05-25 16:00:27 +02:00
Simon Martín
9b0b92fb6b
removing try catch
2018-05-25 15:46:12 +02:00
Simon Martín
5bed04a38c
fix gzip problem with req close event
2018-05-25 15:34:30 +02:00
Simon Martín
fe3bd4fd37
handling errors from request in COPYfrom
2018-05-25 14:33:54 +02:00
Simon Martín
e2bba967f9
handling copyto errors
2018-05-24 19:48:24 +02:00
Eneko Lakasta
6bc90eb9b3
fix typo: Backed -> Backend
2018-05-24 11:04:14 +02:00
Simon Martín
fca6ee8232
changing name of dataIngestionLogPath
2018-05-23 17:32:44 +02:00
Simon Martín
b2a36eb556
copy metrics with BunyanLogger
2018-05-23 17:25:46 +02:00
Simon Martín
fdc542f7b5
BunyanLogger
2018-05-23 17:24:48 +02:00
Simon Martín
5ba7dca79c
copyfrom metrics to kibana
2018-05-23 10:30:37 +02:00
Paul Ramsey
f77707f6da
Fix reponse copy from
2018-05-22 10:48:11 -04:00
Paul Ramsey
a4654f82b2
Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q
2018-05-22 10:37:50 -04:00
Paul Ramsey
ed696a96ec
Merge branch 'pgcopy-stream' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream-q
2018-05-22 10:23:04 -04:00
Simon Martín
6d73d97ae2
copyFrom metrics to statsd
2018-05-22 16:02:14 +02:00
Simon Martín
e347985465
changing query parameter name from sql to q
...
unify query validatrion
2018-05-22 15:42:57 +02:00
Simon Martín
b9474e7fc3
total_rows in copyto metrics
2018-05-22 15:29:08 +02:00
Simon Martín
0bac7a484c
improving query info
2018-05-22 14:16:22 +02:00
Simon Martín
185f708d5b
small style refactor
2018-05-22 14:15:55 +02:00
Simon Martín
86e0ebc148
small refactor in copy format regex
2018-05-22 12:02:41 +02:00
Simon Martín
f01191472b
refactoring copy controller middlewares
2018-05-22 11:56:50 +02:00
Simon Martín
433bd01c27
copyto metrics
2018-05-22 11:54:10 +02:00
Simon Martín
81be15fbc3
adding format to copy metrics
2018-05-21 19:13:44 +02:00
Paul Ramsey
64fc0c32e3
Change from 'sql' as query parameter to 'q', aping existing sql api
2018-05-21 07:47:16 -07:00
Simon Martín
816779fefd
erratum
2018-05-18 17:32:59 +02:00
Simon Martín
82c9fec18a
copy from metrics
2018-05-18 17:31:29 +02:00
Eneko Lakasta
fc7e246704
please jshint
2018-05-18 11:35:54 +02:00
Simon Martín
bf4d12868b
Merge branch 'master' into pgcopy-stream
2018-05-18 10:57:32 +02:00
Eneko Lakasta
1e8c6e198c
remove api key fallback
2018-05-17 17:13:59 +02:00
Simon Martín
9016647a6f
reverting use of busboy instead of multer
2018-05-14 18:31:44 +02:00
Simon Martín
60b48df4d6
default rate limits params to server.js
2018-05-11 15:41:35 +02:00
Simon Martín
d70013b0f4
object assing for default rate limits params
2018-05-11 15:36:59 +02:00
Simon Martín
bc8a867334
jshinty happy
2018-05-11 14:51:43 +02:00
Simon Martín
f2acc8f653
forgotten log
2018-05-11 14:16:26 +02:00
Simon Martín
eee8bc2674
moving copyto headers to correct position
2018-05-11 14:12:23 +02:00
Simon Martín
791967877c
addining gzip support to copyfrom
2018-05-11 13:33:54 +02:00
Simon Martín
f749798ca5
validating inputs
2018-05-11 10:38:33 +02:00
Paul Ramsey
8f211f905c
Remove wrong examples
2018-05-10 12:21:33 -07:00
Paul Ramsey
f0021264d2
Change from form-multipart to POST with chunked upload
2018-05-10 12:11:55 -07:00
Simon Martín
0bb7cec9f4
params existence responsability for UserLimits
2018-05-10 11:51:58 +02:00
Simon Martín
d400926387
using busboy instead of multer
2018-05-08 18:39:16 +02:00
Simon Martín
a997881048
ensuring existence of rate limits params in code
2018-05-08 16:00:16 +02:00
Simon Martín
82d14ab98f
busboy configuration
2018-05-08 13:15:19 +02:00
Simon Martín
7c4409bcf5
from var to const/let
2018-05-08 13:10:25 +02:00
Simon Martín
c40ab801b1
refactoring handleCopyTo
2018-05-08 13:08:29 +02:00
Simon Martín
b036b876ff
using Busboy instead of multer
2018-05-08 12:52:33 +02:00
Simon Martín
6b96032fcf
ensuring return error
2018-05-04 16:50:17 +02:00
Simon Martín
383b86cf9b
simplify sql paramter handler and removing underscore
2018-05-04 16:47:50 +02:00
Simon Martín
2b066eadfa
removing unused dependencies
2018-05-04 15:35:23 +02:00
Simon Martín
02238fefe1
moving body-parser from global to routes of query and job
2018-05-04 15:15:37 +02:00
Simon Martín
0346cf11d8
enabling multipart/form-data again
2018-05-04 15:11:45 +02:00
Simon Martín
4322ccdd09
adding new endpoints to rate limits
2018-05-03 18:50:13 +02:00
Simon Martín
af4c5906c8
adding /sql tro the endpoints
2018-05-03 18:31:49 +02:00
Paul Ramsey
36bdf84f71
Merge branch 'master' of github.com:CartoDB/CartoDB-SQL-API into pgcopy-stream
2018-04-27 15:56:13 +02:00
Paul Ramsey
e2d8896307
Typo in curl example
2018-04-27 12:32:47 +02:00
Paul Ramsey
5f9cc37dba
Jshint change, DRY the content-disposition handler
2018-04-27 12:19:53 +02:00
Paul Ramsey
fe52af71ac
Add documentation SQL API copyfrom/copyto end points,
...
and make small modifications arising from that task
2018-04-27 11:55:06 +02:00
Paul Ramsey
c78cac9bd6
Use the correct database credentials, so we obey security
2018-04-26 10:45:52 +02:00
Paul Ramsey
f13028b497
More jshint changes
2018-04-26 10:20:21 +02:00
Simon Martín
5f8e1d8f53
fix validatePGEntitiesAccess access
2018-04-26 10:13:03 +02:00
Paul Ramsey
cc5fa5c6ce
Changes suggested by jshint
2018-04-26 10:04:38 +02:00
Paul Ramsey
ea66076255
Remove 'console.debug', try and get travis tests clean
2018-04-26 09:40:13 +02:00
Paul Ramsey
a98781d335
Working code for /copyfrom (streaming) and /copyto (streaming)
2018-04-25 18:37:04 +02:00
Paul Ramsey
b59bda5780
Get working copyto implementation in place. Many test cases
...
needed still and thoughts on how to communicate errors back
that are maybe nicer than stack messages?
2018-04-24 15:55:20 +02:00
Simon Martín
e12e16507a
create object and export
2018-04-24 13:21:43 +02:00
Paul Ramsey
4914100205
CopyFrom works, but still needs a decent return payload and
...
a lot of work on returning useful information for error cases (post empty input,
db errors returned more nicely? etc)
2018-04-24 13:07:57 +02:00
Paul Ramsey
72bce5732b
WIP on /copyfrom
2018-04-24 11:26:15 +02:00
Simon Martín
3b228ac17b
solving jshint crazy error
2018-04-24 10:10:37 +02:00
Simon Martín
985f9ef7f5
pgEntitiesAccessValidator
2018-04-23 18:17:44 +02:00
Rafa de la Torre
2229d0ee57
Fix jshint issues
2018-04-13 16:36:43 +02:00
Paul Ramsey
0161696627
Update copy_controller.js
2018-04-13 05:43:23 -07:00
Paul Ramsey
d54e2f5a07
Implementation including multer, custom storage engine, and pg-copy,
...
but without turning over pg-copy, and demonstrating the missing 'sql'
parameter in the custom storage engine.
2018-04-12 12:25:28 -07:00
Simon Martín
2a729b7da8
destructuring error_handler parameters
2018-04-03 13:43:17 +02:00
Simon Martín
3f6b6d3cff
middleware naming
2018-04-03 13:41:18 +02:00
Simon Martín
db6ab4a591
let & const vars
2018-03-28 13:52:45 +02:00
Simon Martín
adab7f21d8
setErrorHeader refactor
2018-03-28 13:50:45 +02:00
Simon Martín
720c3eb8cb
changing msg var name for errorResponse
2018-03-28 13:35:19 +02:00
Simon Martín
4301b63189
fix error
2018-03-28 13:31:31 +02:00
Simon Martín
616f3f3c07
errorHandlerFactory
2018-03-28 13:06:39 +02:00
Simon Martín
16ea727fc3
move ErrorHandler logic from error middleware
2018-03-28 12:58:10 +02:00
Simon Martín
5b6f876cd6
change pgErrorHandler for more generic errorHandler
2018-03-28 12:50:07 +02:00
Simon Martín
b45f5fe31e
move ErrorHandler to services folder
2018-03-28 12:49:33 +02:00
Simon Martín
287d4634c6
rename createPgError to createGenericError
2018-03-28 12:43:34 +02:00
Simon Martín
e611d21029
getName refactor
2018-03-28 12:38:45 +02:00
Simon Martín
2bfa00ab57
getStatus refactor
2018-03-28 12:35:01 +02:00
Simon Martín
d727a50407
jshint
2018-03-28 12:17:42 +02:00
Simon Martín
96435d28e0
removing underscore dependency
2018-03-28 12:15:55 +02:00
Simon Martín
e2641200f7
extracting error type check
2018-03-28 12:13:31 +02:00
Simon Martín
575f1b3a05
simplify error response on models
2018-03-28 11:55:52 +02:00
Simon Martín
050c598e90
using new error handler as a Error
2018-03-28 11:30:11 +02:00
Simon Martín
2722302a89
using error handler as a Error
2018-03-28 11:29:38 +02:00
Simon Martín
60021a70dc
extends Error
2018-03-28 10:56:06 +02:00
Simon Martín
522363c419
removing semicolons
2018-03-28 10:49:50 +02:00
Simon Martín
eefea03b80
error handler to class
2018-03-28 10:48:25 +02:00
Simon Martín
67e1c04efd
indentation details
2018-03-28 10:48:08 +02:00
Simon Martín
c6eb0ad4c7
error middleware name
2018-03-28 09:58:05 +02:00
Simon Martín
0998e50014
refactor getFields method
2018-03-27 15:56:53 +02:00
Simon Martín
b75334f1e7
Merge branch 'master' into limits-error-refactor
2018-03-27 15:32:24 +02:00
Simon Martín
483c263f61
adding and testing limits error response with params
2018-03-26 19:10:23 +02:00
Simon Martín
54876fa203
addind properties to limits errors
2018-03-26 15:56:50 +02:00
Simon Martín
50b8a9ed17
remove unused parameter
2018-03-23 18:02:44 +01:00
Simon Martín
0b708165b3
simplify timeout error catch
2018-03-23 17:48:21 +01:00
Simon Martín
8c86c2b3f8
correct error message in rate limit
2018-03-23 15:35:06 +01:00
Simon Martín
26baaf6081
removing retry after when no necessary
2018-03-23 15:09:16 +01:00
Simon Martín
80818c3cbb
fix userLimits reference
2018-03-20 16:20:56 +01:00
Simon Martín
581bd0dd89
Merge branch 'master' into rateLimits
2018-03-14 12:47:45 +01:00
Simon Martín
1ffeb456e8
Headers following rfc6648
2018-03-14 12:17:26 +01:00
Daniel García Aubert
8dea7615c4
Use user's port instead of the generic one
2018-03-13 19:08:51 +01:00
Daniel García Aubert
328d6f253e
Honor batch api port
2018-03-13 18:49:01 +01:00
Eneko Lakasta
2ec767c1fc
Set Vary Header to honor Authorization header when caching (fastly)
2018-03-13 11:59:07 +01:00
Simon Martín
9cd4e85768
using cartodb-redis in rate limits
2018-03-03 15:01:45 +01:00
Simon Martín
b6ff37d33f
changing param name
2018-03-02 19:57:33 +01:00
Simon Martín
967cdf8a06
refactoring user_limits
2018-03-02 19:54:22 +01:00
Simon Martín
5cd7c24776
refactoring middleware and updating tests
2018-03-02 16:42:45 +01:00
Simon Martín
a11d9870a3
exporting RATE_LIMIT_REDIS_DB
2018-03-02 13:17:50 +01:00
Simon Martín
25d1e5442c
new redis keys
2018-03-01 15:51:19 +01:00
Simon Martín
7a6ef637ae
make jshint happy
2018-03-01 14:47:34 +01:00
Simon Martín
f3adad57aa
fix assignment
2018-03-01 13:15:32 +01:00
Simon Martín
779ab3b83f
adding rate limit middleware to controllers
2018-03-01 12:19:57 +01:00
Simon Martín
a3dfd2767b
rate limit middleware
2018-03-01 12:19:28 +01:00
Simon Martín
2834975d14
controllers using user limits service
2018-03-01 11:31:35 +01:00
Simon Martín
3fdb7abeaf
user limits service
2018-03-01 11:30:34 +01:00
Daniel García Aubert
5802b2182e
Improve profiling
2018-02-27 13:56:00 +01:00
Daniel García Aubert
78f8c430ea
Merge branch 'master' into auth-api
2018-02-27 12:24:40 +01:00
Daniel García Aubert
6f4c79eaef
Fix common interface for auth-backends
2018-02-26 19:02:05 +01:00
Daniel García Aubert
418ec1304f
Fix issue with auth fallback, it wasn't authenticated when apikey master was provided
2018-02-26 18:13:49 +01:00
Daniel García Aubert
cf0214f5c3
Respond with 403 intead of 401 when the request has no permissions to the specific resource
2018-02-23 15:50:23 +01:00
Daniel García Aubert
f85bdf53a5
Missing middleware
2018-02-23 13:19:26 +01:00
Daniel García Aubert
897d38c3f4
Do not set undefined step
2018-02-23 13:05:41 +01:00
Daniel García Aubert
47b54612c7
Place send-response-middleware to the very end of middleware stack
2018-02-23 13:03:56 +01:00
Rafa de la Torre
90b0ec61c2
Add RESIZE=yes param to shapefiles export
...
Add the parameter `RESIZE=yes` to gdal shapefile driver. This resizes
fields to their optimal size. See
http://www.gdal.org/drv_shapefile.html for details.
This is needed because some customers are having trouble exporting
shapefiles, then reimporting into ArcGIS.
2018-02-23 10:34:44 +01:00
Daniel García Aubert
3807d9f94d
Template for User not found error
2018-02-22 17:49:02 +01:00
Daniel García Aubert
ec24019cc0
Apply some ES6 goodies
2018-02-22 17:35:03 +01:00
Daniel García Aubert
868dc798b2
Cosmetic line break
2018-02-22 15:59:13 +01:00
Daniel García Aubert
095c34358e
Remove step
2018-02-22 15:58:52 +01:00
Daniel García Aubert
8c3b9fa05b
Remove bad assignment
2018-02-22 13:27:50 +01:00
Daniel García Aubert
e4067132b0
Add profile to user limit middleware
2018-02-22 12:48:39 +01:00
Daniel García Aubert
700c64bba3
Extract to a middleware user timeout limit from user-database-services
2018-02-22 12:45:55 +01:00
Daniel García Aubert
afceac7369
Improve profile step name
2018-02-22 12:23:37 +01:00
Daniel García Aubert
94c5bd11df
Split authorization middleware, it was actually doing two things: authorize and get database connection params
2018-02-22 12:22:39 +01:00
Daniel García Aubert
1252f32579
Get api_key token from res.locals
2018-02-22 12:07:27 +01:00
Daniel García Aubert
8730b5c517
Read if the request is authenticated through res.locals
2018-02-22 12:04:05 +01:00
Daniel García Aubert
b399abee18
Move authorization to auth-api and extract it from user-database-service
2018-02-22 11:46:34 +01:00
Daniel García Aubert
6c92781600
Remove empty line
2018-02-21 16:19:12 +01:00
Daniel García Aubert
9088cbf150
Improve naming
2018-02-21 13:51:28 +01:00
Daniel García Aubert
28a7cf9259
Don't authenticate when default_public api-key is provided
2018-02-21 11:14:31 +01:00
Daniel García Aubert
ba113d8628
Improve naming
2018-02-20 16:43:43 +01:00
Daniel García Aubert
33baa399ae
Create function to compose middlewares to perfom job operations
2018-02-20 16:25:16 +01:00
Daniel García Aubert
864ec30058
Remove duplication
2018-02-20 13:22:33 +01:00
Daniel García Aubert
d4d54648d4
Improve naming
2018-02-20 13:14:28 +01:00
Daniel García Aubert
03e9b8006b
Actually pass just apikey instead of the entire "res" object
2018-02-19 19:16:33 +01:00
Daniel García Aubert
d6776189bd
Remove unused param
2018-02-19 16:04:57 +01:00
Daniel García Aubert
4af751ff62
Rename param
2018-02-19 15:58:48 +01:00
Daniel García Aubert
b6a666a8b4
Miss rename middleware
2018-02-19 15:54:05 +01:00
Daniel García Aubert
8c13add7a8
Rename middleware
2018-02-19 15:49:17 +01:00
Daniel García Aubert
939443ef81
Unify credentials and authenticated request midllewares
2018-02-19 15:44:28 +01:00
Daniel García Aubert
7898b49e59
Extract profiler middleware to used in query and job controllers
2018-02-19 15:13:36 +01:00
Daniel García Aubert
85fbd7c6b2
Create middlewares to initialize and finish profiling
2018-02-19 14:42:52 +01:00
Daniel García Aubert
633d06bf2d
Do not use underscore
2018-02-19 14:41:06 +01:00
Daniel García Aubert
e04529c501
Rename middleware
2018-02-19 14:20:09 +01:00
Daniel García Aubert
75c2d85dbb
Use authenticated middleware in query controller
2018-02-19 13:24:44 +01:00
Daniel García Aubert
114070ef96
Rename middleware
2018-02-19 12:37:19 +01:00
Daniel García Aubert
adfe3dd52b
Remove comment
2018-02-19 11:10:39 +01:00
Daniel García Aubert
ea310db62c
Cosmetic chenges
2018-02-19 11:04:28 +01:00
Daniel García Aubert
edd382fb8f
Check user is the same user that sends the request when basic-auth is provided
2018-02-16 18:21:06 +01:00
Daniel García Aubert
bcf88093e9
Support basic-auth for batch api
2018-02-16 11:52:57 +01:00
Daniel García Aubert
2cadfe6f18
Add support for Basic Auth
2018-02-16 10:46:58 +01:00
Daniel García Aubert
756fbe42c9
Use apikey midlleware
2018-02-15 17:24:04 +01:00
Daniel García Aubert
ad772246d0
Get api_key from specific middleware and save it into res.locals
2018-02-15 17:23:35 +01:00
Daniel García Aubert
67f2a85abd
Add database credentials to jobs
2018-02-15 12:30:04 +01:00
Daniel García Aubert
ea6e8b5315
Implement fallback mechanism to be able to authenticate as usual in case of apikey is not found
2018-02-14 16:22:36 +01:00
Daniel García Aubert
7c859f0d7b
Use actual apikey properties
2018-02-12 17:49:50 +01:00
Daniel García Aubert
c003a491c0
Use current implementation of cartodb-redis
2018-02-12 16:41:35 +01:00
Daniel García Aubert
5c7f5fbaa5
Merge branch 'master' into auth-api
2018-02-12 16:30:09 +01:00
Simon Martín
bd410fb5f6
using same way to search a string
2018-01-09 17:16:38 +01:00
Simon Martín
df86cc16e4
method to detect timeout error
2018-01-09 17:14:11 +01:00
Simon Martín
6e45c39b1f
Merge branch 'master' into pythonTimeout
2018-01-08 11:29:17 +01:00
Simon Martín
ee511d3762
stringifyForLogs more usual case first
2017-12-18 14:54:23 +01:00
