Simon Martín
80818c3cbb
fix userLimits reference
2018-03-20 16:20:56 +01:00
Simon Martín
581bd0dd89
Merge branch 'master' into rateLimits
2018-03-14 12:47:45 +01:00
Simon Martín
1ffeb456e8
Headers following rfc6648
2018-03-14 12:17:26 +01:00
Daniel García Aubert
8dea7615c4
Use user's port instead of the generic one
2018-03-13 19:08:51 +01:00
Daniel García Aubert
328d6f253e
Honor batch api port
2018-03-13 18:49:01 +01:00
Eneko Lakasta
2ec767c1fc
Set Vary Header to honor Authorization header when caching (fastly)
2018-03-13 11:59:07 +01:00
Simon Martín
9cd4e85768
using cartodb-redis in rate limits
2018-03-03 15:01:45 +01:00
Simon Martín
b6ff37d33f
changing param name
2018-03-02 19:57:33 +01:00
Simon Martín
967cdf8a06
refactoring user_limits
2018-03-02 19:54:22 +01:00
Simon Martín
5cd7c24776
refactoring middleware and updating tests
2018-03-02 16:42:45 +01:00
Simon Martín
a11d9870a3
exporting RATE_LIMIT_REDIS_DB
2018-03-02 13:17:50 +01:00
Simon Martín
25d1e5442c
new redis keys
2018-03-01 15:51:19 +01:00
Simon Martín
7a6ef637ae
make jshint happy
2018-03-01 14:47:34 +01:00
Simon Martín
f3adad57aa
fix assignment
2018-03-01 13:15:32 +01:00
Simon Martín
779ab3b83f
adding rate limit middleware to controllers
2018-03-01 12:19:57 +01:00
Simon Martín
a3dfd2767b
rate limit middleware
2018-03-01 12:19:28 +01:00
Simon Martín
2834975d14
controllers using user limits service
2018-03-01 11:31:35 +01:00
Simon Martín
3fdb7abeaf
user limits service
2018-03-01 11:30:34 +01:00
Daniel García Aubert
5802b2182e
Improve profiling
2018-02-27 13:56:00 +01:00
Daniel García Aubert
78f8c430ea
Merge branch 'master' into auth-api
2018-02-27 12:24:40 +01:00
Daniel García Aubert
6f4c79eaef
Fix common interface for auth-backends
2018-02-26 19:02:05 +01:00
Daniel García Aubert
418ec1304f
Fix issue with auth fallback, it wasn't authenticated when apikey master was provided
2018-02-26 18:13:49 +01:00
Daniel García Aubert
cf0214f5c3
Respond with 403 intead of 401 when the request has no permissions to the specific resource
2018-02-23 15:50:23 +01:00
Daniel García Aubert
f85bdf53a5
Missing middleware
2018-02-23 13:19:26 +01:00
Daniel García Aubert
897d38c3f4
Do not set undefined step
2018-02-23 13:05:41 +01:00
Daniel García Aubert
47b54612c7
Place send-response-middleware to the very end of middleware stack
2018-02-23 13:03:56 +01:00
Rafa de la Torre
90b0ec61c2
Add RESIZE=yes param to shapefiles export
...
Add the parameter `RESIZE=yes` to gdal shapefile driver. This resizes
fields to their optimal size. See
http://www.gdal.org/drv_shapefile.html for details.
This is needed because some customers are having trouble exporting
shapefiles, then reimporting into ArcGIS.
2018-02-23 10:34:44 +01:00
Daniel García Aubert
3807d9f94d
Template for User not found error
2018-02-22 17:49:02 +01:00
Daniel García Aubert
ec24019cc0
Apply some ES6 goodies
2018-02-22 17:35:03 +01:00
Daniel García Aubert
868dc798b2
Cosmetic line break
2018-02-22 15:59:13 +01:00
Daniel García Aubert
095c34358e
Remove step
2018-02-22 15:58:52 +01:00
Daniel García Aubert
8c3b9fa05b
Remove bad assignment
2018-02-22 13:27:50 +01:00
Daniel García Aubert
e4067132b0
Add profile to user limit middleware
2018-02-22 12:48:39 +01:00
Daniel García Aubert
700c64bba3
Extract to a middleware user timeout limit from user-database-services
2018-02-22 12:45:55 +01:00
Daniel García Aubert
afceac7369
Improve profile step name
2018-02-22 12:23:37 +01:00
Daniel García Aubert
94c5bd11df
Split authorization middleware, it was actually doing two things: authorize and get database connection params
2018-02-22 12:22:39 +01:00
Daniel García Aubert
1252f32579
Get api_key token from res.locals
2018-02-22 12:07:27 +01:00
Daniel García Aubert
8730b5c517
Read if the request is authenticated through res.locals
2018-02-22 12:04:05 +01:00
Daniel García Aubert
b399abee18
Move authorization to auth-api and extract it from user-database-service
2018-02-22 11:46:34 +01:00
Daniel García Aubert
6c92781600
Remove empty line
2018-02-21 16:19:12 +01:00
Daniel García Aubert
9088cbf150
Improve naming
2018-02-21 13:51:28 +01:00
Daniel García Aubert
28a7cf9259
Don't authenticate when default_public api-key is provided
2018-02-21 11:14:31 +01:00
Daniel García Aubert
ba113d8628
Improve naming
2018-02-20 16:43:43 +01:00
Daniel García Aubert
33baa399ae
Create function to compose middlewares to perfom job operations
2018-02-20 16:25:16 +01:00
Daniel García Aubert
864ec30058
Remove duplication
2018-02-20 13:22:33 +01:00
Daniel García Aubert
d4d54648d4
Improve naming
2018-02-20 13:14:28 +01:00
Daniel García Aubert
03e9b8006b
Actually pass just apikey instead of the entire "res" object
2018-02-19 19:16:33 +01:00
Daniel García Aubert
d6776189bd
Remove unused param
2018-02-19 16:04:57 +01:00
Daniel García Aubert
4af751ff62
Rename param
2018-02-19 15:58:48 +01:00
Daniel García Aubert
b6a666a8b4
Miss rename middleware
2018-02-19 15:54:05 +01:00
Daniel García Aubert
8c13add7a8
Rename middleware
2018-02-19 15:49:17 +01:00
Daniel García Aubert
939443ef81
Unify credentials and authenticated request midllewares
2018-02-19 15:44:28 +01:00
Daniel García Aubert
7898b49e59
Extract profiler middleware to used in query and job controllers
2018-02-19 15:13:36 +01:00
Daniel García Aubert
85fbd7c6b2
Create middlewares to initialize and finish profiling
2018-02-19 14:42:52 +01:00
Daniel García Aubert
633d06bf2d
Do not use underscore
2018-02-19 14:41:06 +01:00
Daniel García Aubert
e04529c501
Rename middleware
2018-02-19 14:20:09 +01:00
Daniel García Aubert
75c2d85dbb
Use authenticated middleware in query controller
2018-02-19 13:24:44 +01:00
Daniel García Aubert
114070ef96
Rename middleware
2018-02-19 12:37:19 +01:00
Daniel García Aubert
adfe3dd52b
Remove comment
2018-02-19 11:10:39 +01:00
Daniel García Aubert
ea310db62c
Cosmetic chenges
2018-02-19 11:04:28 +01:00
Daniel García Aubert
edd382fb8f
Check user is the same user that sends the request when basic-auth is provided
2018-02-16 18:21:06 +01:00
Daniel García Aubert
bcf88093e9
Support basic-auth for batch api
2018-02-16 11:52:57 +01:00
Daniel García Aubert
2cadfe6f18
Add support for Basic Auth
2018-02-16 10:46:58 +01:00
Daniel García Aubert
756fbe42c9
Use apikey midlleware
2018-02-15 17:24:04 +01:00
Daniel García Aubert
ad772246d0
Get api_key from specific middleware and save it into res.locals
2018-02-15 17:23:35 +01:00
Daniel García Aubert
67f2a85abd
Add database credentials to jobs
2018-02-15 12:30:04 +01:00
Daniel García Aubert
ea6e8b5315
Implement fallback mechanism to be able to authenticate as usual in case of apikey is not found
2018-02-14 16:22:36 +01:00
Daniel García Aubert
7c859f0d7b
Use actual apikey properties
2018-02-12 17:49:50 +01:00
Daniel García Aubert
c003a491c0
Use current implementation of cartodb-redis
2018-02-12 16:41:35 +01:00
Daniel García Aubert
5c7f5fbaa5
Merge branch 'master' into auth-api
2018-02-12 16:30:09 +01:00
Simon Martín
bd410fb5f6
using same way to search a string
2018-01-09 17:16:38 +01:00
Simon Martín
df86cc16e4
method to detect timeout error
2018-01-09 17:14:11 +01:00
Simon Martín
6e45c39b1f
Merge branch 'master' into pythonTimeout
2018-01-08 11:29:17 +01:00
Simon Martín
ee511d3762
stringifyForLogs more usual case first
2017-12-18 14:54:23 +01:00
Simon Martín
b531922459
add line at EOF
2017-12-18 12:59:20 +01:00
Simon Martín
485d9adddd
escape chars function
2017-12-18 12:45:44 +01:00
Simon Martín
3fa09a007e
line at EOF and better comment
2017-12-12 17:16:40 +01:00
Simon Martín
6481d14192
escape quotes for logs
2017-12-12 12:58:31 +01:00
Simon Martín
c478e8e62e
removing default error log value
2017-12-12 10:55:28 +01:00
Simon Martín
61b351535c
change funcion name
2017-11-30 15:06:25 +01:00
Simon Martín
0ab87df644
extract error message from array
2017-11-28 17:19:10 +01:00
Simon Martín
4d598eacd7
dont modify err, works with errorsLog
2017-11-28 16:11:04 +01:00
Simon Martín
a17e1fc5ec
logErrors function
2017-11-28 15:58:14 +01:00
Simon Martín
3e23be2087
add errors header with default value
2017-11-28 15:57:58 +01:00
Daniel García Aubert
88015aaf0f
Implement new auth system based on api-key tokens with scoped permissions
2017-11-24 17:57:54 +01:00
Simon Martín
670cb6eb8b
Merge branch 'master' into pythonTimeout
2017-11-24 16:02:13 +01:00
Daniel García Aubert
15949d2fc2
Fix wrong function name
2017-11-24 15:55:16 +01:00
Daniel García Aubert
b10937a8fb
Use res.locals instead of req.context to share object among middlewares
2017-11-24 15:49:25 +01:00
Daniel García Aubert
c2d1a4653e
Improve naming for connection database params holder
2017-11-24 15:45:49 +01:00
Simon Martín
11e0ac89d2
undo test of postgres9.5 error
2017-11-24 12:22:30 +01:00
Rafa de la Torre
296d1878fd
Trying regexp instead of exact match
2017-11-24 10:55:02 +01:00
Daniel García Aubert
833b7b962c
Order geo-columns to export 'the_geom' by default while exporting SHP format
2017-11-17 14:45:58 +01:00
Simon Martín
27e43c8d91
handling python timeout error
2017-11-17 13:30:58 +01:00
Daniel García Aubert
b1112e2bb1
Remove unneeded header
2017-09-20 16:48:22 +02:00
Daniel García Aubert
980e762ab7
Skip user auth middlewares
2017-09-20 16:35:33 +02:00
Daniel García Aubert
0597a3b8a6
Change job work in progress endpoint
2017-09-20 15:46:29 +02:00
Raul Ochoa
32154b67c6
Merge branch 'master' into upgrade-cartodb-psql
2017-08-11 11:57:16 +02:00
Daniel García Aubert
f13c16e523
Typo
2017-08-10 11:22:08 +02:00
Daniel García Aubert
01a71ee60e
Apply user timeout to ogr2ogr command
2017-08-09 12:50:16 +02:00
Raul Ochoa
3e536b822e
Remove unnecessary destroyOnError option
2017-08-08 18:21:10 +02:00
Daniel García Aubert
fb477260ec
Respond with 429 and proper message when database fires a timeout error
2017-08-03 17:19:08 +02:00
Rafa de la Torre
07890ec608
Change tableCache.get for tableCache.peek #244
...
This will make it consider age of the key when it was initially set,
instead of renewing it with the time of the last access.
2017-07-03 11:13:21 +02:00
Rafa de la Torre
1db935a24b
Merge pull request #423 from CartoDB/422-disable-query-lru-cache
...
422 disable query lru cache
2017-06-27 17:40:33 +02:00
Rafa de la Torre
2d9a5e9655
Move cyclomatic complexity to the factory #422
2017-06-27 16:05:05 +02:00
Rafa de la Torre
c8e42f561d
Reduce cyclomatic complexity (jshint 2/2) #422
2017-06-27 15:35:51 +02:00
Rafa de la Torre
34ccfc10df
Please jshint (1/2) #422
2017-06-27 13:26:14 +02:00
Rafa de la Torre
bf478cabb2
Fix initialization of NoCache #422
2017-06-27 12:44:11 +02:00
Rafa de la Torre
bc71889f93
First attempt at removing tableCache #422
2017-06-27 12:10:24 +02:00
Daniel García Aubert
aa66000848
Going green: use replacer parameter in JSON.stringify method to not cast special number values to null
2017-06-08 12:42:37 +02:00
Mario de Frutos
3a99020632
Added regexp for the test to matches both 0.10 and 6.X node version
2017-04-18 12:15:47 +02:00
Mario de Frutos
da42daa536
Improved error message for zip spawn command
2017-04-18 12:15:47 +02:00
Daniel García Aubert
e4eeb9e1f6
Clean commented code
2017-04-18 12:15:47 +02:00
Daniel García Aubert
e0f9209017
Attach error handler while spawning zip command
2017-04-18 12:15:47 +02:00
Daniel García Aubert
884d3aea0a
Dont call 'next' callback twice, include error message if zip don't exit normally
2017-04-18 12:15:47 +02:00
Daniel García Aubert
4b98b7056a
Make zip command path configurable
2017-04-18 12:15:47 +02:00
Daniel García Aubert
6dba06130b
Retur a proper error message when ogr2ogr command fails
2017-04-18 11:40:37 +02:00
Daniel García Aubert
d60e2107af
Add error callback to ogr command while spawning
2017-04-11 09:43:25 +02:00
Raul Ochoa
db9bfacf07
Extract stats client creation
2017-03-30 16:13:17 +02:00
Raul Ochoa
d48a901533
Merge branch 'master' into node-v6
2017-01-12 19:49:56 +01:00
Javier Torres
62a67f148c
Correctly accept custom options for formats in ogr.js
2017-01-11 19:07:50 +01:00
Javier Torres
c67a2e7d09
Use cartodb_id as FID column in gpkg
2017-01-11 19:07:50 +01:00
Daniel García Aubert
ef124d689e
Handle query end in error listener since pg no longer emits end if and error occurs
2016-12-07 16:22:21 +01:00
Daniel García Aubert
99c7a6e4f9
Raise job query size to 16kb
2016-11-07 16:50:44 +01:00
Daniel García Aubert
4a64d37c6c
Add work in progrees list endpoint
2016-10-28 15:08:42 +02:00
Raul Ochoa
cdde1be29e
Re-use redis pool as much as possible
2016-10-17 15:02:34 +02:00
Raul Ochoa
e401c01d78
Only log on non-test environments
2016-10-12 01:40:14 +02:00
Raul Ochoa
e4b1711e8e
pub/sub package
2016-10-11 18:28:46 +02:00
Raul Ochoa
66820a67bb
Make possible to specify a name for batch
2016-10-10 19:46:07 +02:00
Raul Ochoa
42a1f3ad4c
Accept multipart requests
2016-10-05 17:29:16 +02:00
Raul Ochoa
6309318534
Use body-parser from old connect module
2016-10-05 14:22:44 +02:00
Raul Ochoa
025b3f3cc7
Increase body limit to 20mb
2016-10-05 11:14:06 +02:00
Raul Ochoa
2a2a54a073
DRY in job response handler
2016-10-04 16:07:13 +02:00
Raul Ochoa
19c9bec633
Callback with job
2016-10-04 15:57:13 +02:00
Raul Ochoa
05cbd55b95
Use db host from request's context
2016-10-04 15:50:39 +02:00
Raul Ochoa
c32a2199fa
Use request bootstrapper to add host header
2016-10-04 15:43:19 +02:00
Raul Ochoa
7b7d651d8f
DRY while authenticating requests
2016-10-04 15:40:56 +02:00
Raul Ochoa
20f50d988e
Use user middleware in job controller
2016-10-04 15:19:31 +02:00
Raul Ochoa
b139b9ab21
Add context object in all requests
2016-10-04 15:12:46 +02:00
Raul Ochoa
2edc7505e7
Do not condition req.profiler
2016-10-04 15:08:31 +02:00
Raul Ochoa
cb9db0c4cb
Make request.profiler always available
2016-10-04 14:40:04 +02:00
Raul Ochoa
3b6bc14d17
Increment errors on err
2016-10-04 13:19:29 +02:00
Raul Ochoa
bf1a67780d
Merge branch 'master' into express-4.x
2016-09-30 17:37:30 +02:00
Daniel García Aubert
59e5c10f85
Set batch queries log path
2016-09-29 15:37:14 +02:00
Raul Ochoa
2aa1f045ff
Use .status() API
2016-09-26 18:10:20 +02:00
Raul Ochoa
abc2f130c9
Migrate to express 4.x series
...
- Remove express logger
- Error handler responds with application/[json|javascript]
- Fix all tests relying on res.headers
- assert.response based on request module
2016-09-26 18:09:27 +02:00
Raul Ochoa
f92d50cccf
Rename from app to server
...
Removes app_root dependency in requires
2016-09-14 20:54:53 +02:00
Raul Ochoa
0626d80e24
Removes support for optional rollbar logging
2016-09-14 19:22:31 +02:00
Raul Ochoa
6f4fb931f7
Remove user indexer from app
2016-08-30 19:04:36 +02:00
Raul Ochoa
ba0f2f1066
Remove endpoint to retrieve jobs list
2016-08-30 18:49:01 +02:00
Raul Ochoa
704ba110ba
Log job creation
2016-08-30 18:43:09 +02:00
Raul Ochoa
ad0d101bfd
Remove patch/put endpoints for jobs
2016-08-30 17:43:34 +02:00
Daniel García Aubert
395635d246
Merge branch 'master' into fix-publisher-connection
2016-07-11 17:49:11 +02:00
Raul Ochoa
3ebbd9f7c4
Skip tables with no updated_at registered in cdb_tablemetadata
2016-07-11 16:39:12 +02:00
Daniel García Aubert
ccff602bbf
Merge branch 'master' into fix-publisher-connection
2016-07-07 16:07:41 +02:00
Raul Ochoa
368fe2403e
Allow to setup more than one domain to validate oauth against
2016-07-07 14:20:36 +02:00
Daniel García Aubert
5eaad4d5d9
Uses redis-mpool for pubsub in Batch API
2016-07-07 14:14:46 +02:00
Daniel García Aubert
a1f31df92e
Now Batch API broadcast to other APIs everytime that re-enqueues a multiple-query job
2016-06-29 18:29:53 +02:00
Daniel García Aubert
282da58ffe
Set default value for statsd-client in job-controller to avoid check it every time it's going to be used
2016-05-30 12:27:19 +02:00
Daniel García Aubert
976bf5b039
Implemented profiling for job-runner and job-controller
2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577
Merge branch 'master' into batch-add-profile
2016-05-26 17:46:52 +02:00
Raul Ochoa
a91c7a6a88
Increase job payload size to 8kb
2016-05-24 15:41:18 +02:00
Raul Ochoa
23228b2d73
Payload size validates multiple queries and fallback queries
...
It uses a middleware to check the body size
2016-05-24 14:28:00 +02:00
Daniel García Aubert
64ad284c9c
WIP: adding metrics to Batch API
2016-05-24 11:19:00 +02:00
Daniel
8c8da8e39c
Merge pull request #301 from CartoDB/job-model-refactor
...
Job model refactor
2016-05-18 11:38:41 +02:00
Daniel García Aubert
d2d3ba8159
Passed tests
2016-05-16 01:22:47 +02:00
Daniel García Aubert
cc7dd7a0d2
Job model refactor
2016-05-13 18:50:55 +02:00
Raul Ochoa
4e3238c2e8
Fix control flow when using OGR with _needSRS=true and empty queries
...
Continue on empty results as srid and geom type are not critical
when there are no results
Fixes #299
2016-05-12 17:21:07 +02:00
Juan Ignacio Sánchez Lara
e979b79f49
gpkg format #291
2016-05-10 18:42:23 +02:00
Juan Ignacio Sánchez Lara
c82cafb1ed
Geopackage format support #291
2016-04-27 16:56:06 +02:00
Daniel García Aubert
5de931daa5
Improver error message in job max payload limit.
2016-04-25 11:18:30 +02:00
Daniel García Aubert
1dcbb1afdf
Improved error message
2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124
Fixed typos
2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944
Implemented limit to query size in Batch API.
2016-04-18 15:30:16 +02:00
Daniel García Aubert
056f22b156
Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series
2016-03-18 14:57:18 +01:00
Raul Ochoa
66cd1f400f
Skip query tables cache for authenticated requests
2016-03-10 19:20:56 +01:00
Raul Ochoa
5ebc8e43d5
Start hits=0 as first time is not a hit
2016-03-08 14:50:08 +01:00
Raul Ochoa
b8460d033e
Add a cache decorator over QueryTables
2016-03-08 14:48:56 +01:00
Raul Ochoa
dbe033112d
Rethrow error but do not fail when getting affected tables
2016-03-08 14:00:31 +01:00
Alejandro Martínez
cd593e12ce
Join all Surrogate-Keys in one line
2016-02-24 11:43:43 +01:00
Alejandro Martínez
7d024dbf3f
Fix include in query_controller
2016-02-22 19:10:59 +01:00
Alejandro Martínez
5411aa3a01
Use Date.now() fallback value for getLastUpdatedTime
2016-02-22 19:08:04 +01:00
Alejandro Martínez
2c4b659100
Properly discard errors fetching updated tables
2016-02-22 19:07:41 +01:00
Alejandro Martínez
5e06711b4b
Use node-cartodb-querytables library
2016-02-22 12:24:05 +01:00
Raul Ochoa
15f90c1a78
QueryTablesApi only caches affected tables and always retrieve last modification
2016-02-02 01:16:24 +01:00
Raul Ochoa
9deab814c9
Skip query-tables-api for authenticated requests
2016-02-01 16:53:28 +01:00
Raul Ochoa
24ecf130a1
Namespace queries so two users don't share results
...
For instance 'select * from table' should result in different affected
tables (schema+table) depending on the database.
2016-01-28 15:38:08 +01:00
Raul Ochoa
65bcdad657
Fix cyclomatic complexity
2016-01-28 14:48:12 +01:00
Raul Ochoa
86ecc3ad0a
Implements authDbParams interface
...
Uses user and pass from redis and relies on rest of params prepared
for normal dbParams
2016-01-28 14:38:02 +01:00
Raul Ochoa
e70a21a2b4
Adds documentation about values passed over callback in getConnectionParams
2016-01-28 14:25:03 +01:00
Raul Ochoa
a654c90b6d
X-Cache-Channel generation only requires affected tables
...
mayWrite is redundant, and it's not its responsibility
2016-01-28 14:18:38 +01:00
Raul Ochoa
131d798653
Change interface to return also authenticated connection params
2016-01-28 14:18:10 +01:00
Raul Ochoa
235e7f9bc1
Rename method to match returning connection params behaviour
2016-01-28 14:14:31 +01:00
Raul Ochoa
023d06fd19
Affected tables and last modified behind QueryTablesApi
...
It handles the internal cache and exposes an unified result
2016-01-28 14:08:18 +01:00
Raul Ochoa
7dad90a108
Remove unused dependencies from job controller
2016-01-28 13:10:21 +01:00
Raul Ochoa
3be769a67e
Fix style
2016-01-28 12:33:32 +01:00
Raul Ochoa
9881a6f7d4
Use readonly flag to run affected tables query
2016-01-28 12:33:00 +01:00
Daniel García Aubert
e9bd93cf3f
Now batch service is linked to express app in order to make it accesible from main app module to expose drain mechanism
2016-01-25 16:28:19 +01:00
Daniel García Aubert
f2ea01f627
Removed unnecessary check in job controller
2016-01-25 14:50:43 +01:00
Daniel García Aubert
e6f714a51b
Implemented some new unit test
2016-01-22 19:31:25 +01:00
Daniel García Aubert
968caae9b7
Removed void comment
2016-01-21 18:26:48 +01:00
