Daniel García Aubert
114070ef96
Rename middleware
2018-02-19 12:37:19 +01:00
Daniel García Aubert
adfe3dd52b
Remove comment
2018-02-19 11:10:39 +01:00
Daniel García Aubert
ea310db62c
Cosmetic chenges
2018-02-19 11:04:28 +01:00
Daniel García Aubert
edd382fb8f
Check user is the same user that sends the request when basic-auth is provided
2018-02-16 18:21:06 +01:00
Daniel García Aubert
2cadfe6f18
Add support for Basic Auth
2018-02-16 10:46:58 +01:00
Daniel García Aubert
756fbe42c9
Use apikey midlleware
2018-02-15 17:24:04 +01:00
Daniel García Aubert
67f2a85abd
Add database credentials to jobs
2018-02-15 12:30:04 +01:00
Daniel García Aubert
b10937a8fb
Use res.locals instead of req.context to share object among middlewares
2017-11-24 15:49:25 +01:00
Daniel García Aubert
c2d1a4653e
Improve naming for connection database params holder
2017-11-24 15:45:49 +01:00
Daniel García Aubert
b1112e2bb1
Remove unneeded header
2017-09-20 16:48:22 +02:00
Daniel García Aubert
980e762ab7
Skip user auth middlewares
2017-09-20 16:35:33 +02:00
Daniel García Aubert
0597a3b8a6
Change job work in progress endpoint
2017-09-20 15:46:29 +02:00
Raul Ochoa
32154b67c6
Merge branch 'master' into upgrade-cartodb-psql
2017-08-11 11:57:16 +02:00
Daniel García Aubert
01a71ee60e
Apply user timeout to ogr2ogr command
2017-08-09 12:50:16 +02:00
Raul Ochoa
3e536b822e
Remove unnecessary destroyOnError option
2017-08-08 18:21:10 +02:00
Daniel García Aubert
99c7a6e4f9
Raise job query size to 16kb
2016-11-07 16:50:44 +01:00
Daniel García Aubert
4a64d37c6c
Add work in progrees list endpoint
2016-10-28 15:08:42 +02:00
Raul Ochoa
e401c01d78
Only log on non-test environments
2016-10-12 01:40:14 +02:00
Raul Ochoa
2a2a54a073
DRY in job response handler
2016-10-04 16:07:13 +02:00
Raul Ochoa
19c9bec633
Callback with job
2016-10-04 15:57:13 +02:00
Raul Ochoa
05cbd55b95
Use db host from request's context
2016-10-04 15:50:39 +02:00
Raul Ochoa
c32a2199fa
Use request bootstrapper to add host header
2016-10-04 15:43:19 +02:00
Raul Ochoa
7b7d651d8f
DRY while authenticating requests
2016-10-04 15:40:56 +02:00
Raul Ochoa
20f50d988e
Use user middleware in job controller
2016-10-04 15:19:31 +02:00
Raul Ochoa
2edc7505e7
Do not condition req.profiler
2016-10-04 15:08:31 +02:00
Raul Ochoa
3b6bc14d17
Increment errors on err
2016-10-04 13:19:29 +02:00
Raul Ochoa
2aa1f045ff
Use .status() API
2016-09-26 18:10:20 +02:00
Raul Ochoa
ba0f2f1066
Remove endpoint to retrieve jobs list
2016-08-30 18:49:01 +02:00
Raul Ochoa
704ba110ba
Log job creation
2016-08-30 18:43:09 +02:00
Raul Ochoa
ad0d101bfd
Remove patch/put endpoints for jobs
2016-08-30 17:43:34 +02:00
Raul Ochoa
3ebbd9f7c4
Skip tables with no updated_at registered in cdb_tablemetadata
2016-07-11 16:39:12 +02:00
Daniel García Aubert
282da58ffe
Set default value for statsd-client in job-controller to avoid check it every time it's going to be used
2016-05-30 12:27:19 +02:00
Daniel García Aubert
976bf5b039
Implemented profiling for job-runner and job-controller
2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577
Merge branch 'master' into batch-add-profile
2016-05-26 17:46:52 +02:00
Raul Ochoa
a91c7a6a88
Increase job payload size to 8kb
2016-05-24 15:41:18 +02:00
Raul Ochoa
23228b2d73
Payload size validates multiple queries and fallback queries
...
It uses a middleware to check the body size
2016-05-24 14:28:00 +02:00
Daniel García Aubert
64ad284c9c
WIP: adding metrics to Batch API
2016-05-24 11:19:00 +02:00
Daniel García Aubert
cc7dd7a0d2
Job model refactor
2016-05-13 18:50:55 +02:00
Daniel García Aubert
5de931daa5
Improver error message in job max payload limit.
2016-04-25 11:18:30 +02:00
Daniel García Aubert
1dcbb1afdf
Improved error message
2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124
Fixed typos
2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944
Implemented limit to query size in Batch API.
2016-04-18 15:30:16 +02:00
Daniel García Aubert
056f22b156
Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series
2016-03-18 14:57:18 +01:00
Raul Ochoa
66cd1f400f
Skip query tables cache for authenticated requests
2016-03-10 19:20:56 +01:00
Raul Ochoa
b8460d033e
Add a cache decorator over QueryTables
2016-03-08 14:48:56 +01:00
Raul Ochoa
dbe033112d
Rethrow error but do not fail when getting affected tables
2016-03-08 14:00:31 +01:00
Alejandro Martínez
cd593e12ce
Join all Surrogate-Keys in one line
2016-02-24 11:43:43 +01:00
Alejandro Martínez
7d024dbf3f
Fix include in query_controller
2016-02-22 19:10:59 +01:00
Alejandro Martínez
5411aa3a01
Use Date.now() fallback value for getLastUpdatedTime
2016-02-22 19:08:04 +01:00
Alejandro Martínez
2c4b659100
Properly discard errors fetching updated tables
2016-02-22 19:07:41 +01:00
Alejandro Martínez
5e06711b4b
Use node-cartodb-querytables library
2016-02-22 12:24:05 +01:00
Raul Ochoa
15f90c1a78
QueryTablesApi only caches affected tables and always retrieve last modification
2016-02-02 01:16:24 +01:00
Raul Ochoa
9deab814c9
Skip query-tables-api for authenticated requests
2016-02-01 16:53:28 +01:00
Raul Ochoa
a654c90b6d
X-Cache-Channel generation only requires affected tables
...
mayWrite is redundant, and it's not its responsibility
2016-01-28 14:18:38 +01:00
Raul Ochoa
131d798653
Change interface to return also authenticated connection params
2016-01-28 14:18:10 +01:00
Raul Ochoa
235e7f9bc1
Rename method to match returning connection params behaviour
2016-01-28 14:14:31 +01:00
Raul Ochoa
023d06fd19
Affected tables and last modified behind QueryTablesApi
...
It handles the internal cache and exposes an unified result
2016-01-28 14:08:18 +01:00
Raul Ochoa
7dad90a108
Remove unused dependencies from job controller
2016-01-28 13:10:21 +01:00
Raul Ochoa
9881a6f7d4
Use readonly flag to run affected tables query
2016-01-28 12:33:00 +01:00
Daniel García Aubert
f2ea01f627
Removed unnecessary check in job controller
2016-01-25 14:50:43 +01:00
Daniel García Aubert
79e626a71e
Added cors middleware, decoupled query and job controllers from its dependencies
2016-01-21 17:24:48 +01:00
Daniel García Aubert
048d2cd0ea
Refactored user database service ans updated controllers to use it.
2016-01-21 16:17:17 +01:00
Daniel García Aubert
f42727212b
Removed unnecessary checkAborted function in job controller and changed signature for getUserDatabase.
2016-01-21 15:05:46 +01:00
Daniel García Aubert
2913aed697
Placed job controller endpoit under /api/vX/sql
2016-01-18 20:12:44 +01:00
Daniel García Aubert
2d13bde301
Implemented uses cases test for Batch API
2016-01-11 20:14:15 +01:00
Daniel García Aubert
20f00d58d9
Refactored batch service to avoid event noise, doing in callback way
2016-01-08 15:47:59 +01:00
Daniel García Aubert
e6a4e0f7eb
Returned 201 HTTP code in job creation
...
Fixed bug listing jobs, now user job index is cleaned before respond to client
2016-01-07 12:07:30 +01:00
Daniel García Aubert
40d0161218
Fixed bug in authentication for PUT methods in Job Controller
2016-01-05 13:54:18 +01:00
Daniel García Aubert
246655de94
Changed redis data structure for users jobs
2016-01-04 19:08:13 +01:00
Daniel García Aubert
35650985db
Implemented job modification whether job is pending
2015-12-31 15:42:31 +01:00
Daniel García Aubert
ada39d84b8
Implemented job cancelation in Batch API
2015-12-30 20:16:18 +01:00
Daniel García Aubert
851c393f75
Refactored batch manager avoiding concurrence issues
2015-12-29 10:19:10 +01:00
Daniel García Aubert
4dcd7c0a50
Fixed typo and prepared for listing user's job
2015-12-24 17:47:41 +01:00
Daniel García Aubert
c4f96b4a59
Implemented support for listing jobs for one user
2015-12-24 17:42:49 +01:00
Daniel García Aubert
ecb76ca1de
Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs.
2015-12-23 17:29:11 +01:00
Daniel García Aubert
62db809476
Return 401 when no API KEY is provided
2015-12-23 14:55:49 +01:00
Daniel García Aubert
30d0b2906b
Implemented GET for jobs API
2015-12-22 11:45:25 +01:00
Daniel García Aubert
ef59a1d942
Fixed typo in batch test
2015-12-22 11:06:46 +01:00
Daniel García Aubert
328d4c6bfe
Fixed bad assertion in batch acceptance test
2015-12-22 11:02:16 +01:00
Daniel García Aubert
3762ad7a39
Now jobs are stored in redis instead of user's database
2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964
Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout
2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a
Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it.
2015-12-16 15:57:58 +01:00
Daniel García Aubert
6cc48bf9dd
Implemented batch service
2015-12-09 20:17:45 +01:00
Daniel García Aubert
c7680722ca
Passed tests
2015-12-09 12:35:20 +01:00
Daniel García Aubert
00721bcd02
Implementing batch service
2015-12-09 00:02:08 +01:00
Daniel García Aubert
6f741827cd
Fixed jshint error
2015-12-07 12:19:16 +01:00
vagrant
35d6600f36
Implemented test for job manegement
2015-12-07 10:29:55 +00:00
Daniel García Aubert
ee41db99ad
Returned job_id to client/user after job enqueue
2015-12-07 09:59:05 +01:00
Daniel García Aubert
a1243ad64d
Implemented job controller to enqueue jobs
2015-12-07 09:40:51 +01:00
Daniel García Aubert
f7a3c6ac4e
Renamed controllers routing method
2015-12-03 18:43:13 +01:00
Daniel García Aubert
c6bae6d951
app module moved to right place
2015-12-03 18:33:17 +01:00
Daniel García Aubert
d31d496a2e
Add useful comment
2015-12-03 18:25:35 +01:00
Daniel García Aubert
a697fe6faa
Making test pass
2015-12-03 18:19:39 +01:00
Daniel García Aubert
e5dedb6315
Split query controller and separated from app
2015-12-03 17:28:18 +01:00
Daniel García Aubert
df59405dca
Split some functionallity from app to dedicated controllers
2015-12-03 15:00:35 +01:00
Raul Ochoa
b160f4051a
Merge pull request #250 from CartoDB/issue-238
...
Stop adding X-Cache-Channel header when no tables involved
2015-09-08 12:30:26 +02:00
Raul Ochoa
5ec6d7c77b
Remove no longer needed health check params
2015-09-07 18:34:50 +02:00
Raul Ochoa
875eccba62
Stop adding X-Cache-Channel header when no tables were identified in SQL query
...
Fixes #238 and fixes #157
2015-09-07 18:22:24 +02:00
Raul Ochoa
335d71bc04
Adds a custom log4js format function to log sql query from POST requests
...
Tokens in log4js are not provided with request/response for context,
using a custom format function allow to access them
In the case of express logger token functions receive request/response
for context so it uses a custom token
Closes #206
2015-09-04 12:29:20 +02:00
Raul Ochoa
f816093036
Set Last-Modified header based on affected tables
...
Closes #101
2015-09-02 16:25:56 +02:00
Raul Ochoa
c6c614b5bf
Do not return results from health check
...
It also removed old dependencies and takes disabled file path in ctor.
2015-08-28 18:16:04 +02:00
Raul Ochoa
d42e5a6b75
Do not expose remove from pool after query error as a config option
2015-08-04 16:23:36 +02:00
Raul Ochoa
7c696317d0
New option to remove client from pool after error happens
...
This help to avoid issues with transactions.
Closes #241
2015-08-04 15:53:50 +02:00
Raul Ochoa
a297a14300
Revert "Uses tables ogr2ogr param to avoid heavy query on pg_class and friends"
...
This reverts commit b6e53f7326
2015-06-16 16:58:22 +02:00
Francisco Dans
3a8f94641b
Merge pull request #225 from CartoDB/log-addr
...
Uses :remote-addr instead of :req[X-Real-IP]
2015-06-01 16:17:47 +02:00
Francisco Dans
996bd971cd
replaces default log format string to remote-addr
2015-05-26 19:18:54 +02:00
Raul Ochoa
debeb8ca3e
Update news
2015-05-25 16:38:07 +02:00
Raul Ochoa
b6e53f7326
Uses tables ogr2ogr param to avoid heavy query on pg_class and friends
...
Fixes #204
Newer versions of ogr2ogr don't use that heavy query so it should work
Ref 2e3eeefd0f
2015-05-25 16:29:18 +02:00
Raul Ochoa
9dd887dc4d
jshint fixes, not addressing complexity
2015-05-13 12:22:41 +02:00
Raul Ochoa
9dcb397737
Merge pull request #205 from CartoDB/url_rewrite
...
added /u/:user routing
2015-03-20 12:22:07 +01:00
Raul Ochoa
9b31df6793
Improve row size limit error message
2015-03-02 14:34:01 +01:00
Raul Ochoa
302a856d37
Move note about username extraction to cartodb_request
2015-02-02 12:15:53 +01:00
Raul Ochoa
e5ab4272eb
Use a multiple params route to be able to extract the username from
...
the path or default to host header.
2015-02-02 12:09:34 +01:00
javi
824b7c084e
added /u/:user routing
2015-01-30 11:37:29 +01:00
Seth Fitzsimmons
bcc1577ec4
Add 'refresh' to the list of keywords that write
2015-01-06 12:40:01 -08:00
Raul Ochoa
d50ddbb10f
Add more fields to error responses with hint, detail and context for SQL errors.
2014-11-21 12:59:48 +01:00
Raul Ochoa
74429f82e1
Improve topojson output by streaming json
2014-11-12 11:36:59 +01:00
Raul Ochoa
dc601a5feb
Improve statement timeout error messages
2014-11-11 13:57:15 +01:00
Raul Ochoa
25d2e64891
Removes query tables console.log
2014-09-17 15:54:46 +02:00
Raul Ochoa
2d232c20ef
Returns 401 Unauthorized for queries without permission
2014-09-02 15:00:04 +02:00
Raul Ochoa
7c62632ea8
Health check endpoint
2014-08-26 18:40:58 +02:00
Raul Ochoa
3a26692448
New header for database host serving the request
2014-08-22 12:29:27 +02:00
Raul Ochoa
0035f04edd
Metrics revamp: removes and adds some metrics
2014-08-18 19:45:17 +02:00
Raul Ochoa
77cb86154c
Starts using cartodb-psql node module in SQL API
2014-08-11 20:15:55 +02:00
Raul Ochoa
951636892c
Removes unused vars
2014-08-08 12:54:36 +02:00
Raul Ochoa
21b8e6947c
Non authenticated request cannot use pg_ catalogs/functions
2014-08-08 12:48:29 +02:00
Raul Ochoa
f6c364b3b9
CDB-3032 Removes sql statements restriction on pg_ queries
2014-08-07 16:22:48 +02:00
Raul Ochoa
bd60f8f748
Using new method from cartodb-redis to retrieve oauth values so we
...
can reuse the same redis-mpool
2014-08-06 12:51:55 +02:00
Raul Ochoa
a7027f4b05
Removes console.log
2014-08-05 16:48:11 +02:00
Raul Ochoa
480a9f27b4
New authentication mechanism: checks in advance if credentials are provided
...
in order to do a single request to redis to retrieve the required database
connection parameters.
2014-08-05 16:20:06 +02:00
Raul Ochoa
49406c99fa
Moves auth functionality to its own directory
2014-08-05 11:57:43 +02:00
Raul Ochoa
e16f278087
CDB-3629 Uses one request to redis to retrieve all connection params
2014-08-05 02:29:07 +02:00
Raul Ochoa
ce70e7252b
Callback requests send 200 status error even if the query failed
2014-08-04 15:56:43 +02:00
Raul Ochoa
73a195a7fa
Use streaming for json responses. Number of buffered rows can be specified by config.
2014-07-31 02:17:07 +02:00
Luis Bosque
6bb015bb14
[CDB-3678] Add X-Served-By-Host header
2014-07-30 12:39:03 +02:00
Raul Ochoa
2653314818
CDB-3657 Sends profiler data as json in X-SQLAPI-Profiler header
2014-07-29 19:29:16 +02:00
Raul Ochoa
0f14118235
CDB-3629 Pickup redis pool config values
2014-07-29 13:58:16 +02:00
Raul Ochoa
a7d444ca92
Changes authentication to start using public user if it is defined in redis.
2014-07-29 13:52:04 +02:00
Raul Ochoa
1a390a3d40
Some style changes
2014-07-29 13:52:04 +02:00
Raul Ochoa
21d6924088
Removes the call to set the search path as it should get the correct public user from redis
2014-07-29 13:52:04 +02:00
javi
ca7b87d81a
fixed problem when a query raised a database error
2014-07-29 13:52:04 +02:00
Raul Ochoa
9c255c504b
Updates CDB_QueryTables to match the one from the extension. Fixes some tests to show the schema presence.
...
Skip some tests that we need to review because with the current implementation of CDB_QueryTables dont make sense.
CDB_QueryTables should be used from the extension as a dependency.
2014-07-29 13:52:04 +02:00
javi
755ed362d4
set scheme to host user when public user is accessing
2014-07-29 13:52:04 +02:00
Raul Ochoa
b3609696a3
requests associated with formats based on postgres expose a cancel method that will be called on client request abortion/cancelling so postgres can cancel ongoing queries
2014-06-02 14:48:38 +02:00
Sandro Santilli
4b5a5921d5
Fix application_name to "cartodb_sqlapi"
2014-05-07 16:15:46 +02:00
Sandro Santilli
51b135c0ac
Set default PostgreSQL application name to "cartodb_sqlapi"
2014-05-07 16:14:17 +02:00
Raul Ochoa
2d2c056d6b
Removes spawn unused dependency
2014-04-22 11:18:52 +02:00
Raul Ochoa
537cb238c6
CDB-2081 Adds support for order_by through http query params.
2014-04-14 15:01:12 +02:00
Raul Ochoa
909530cfb3
documentation for sql parameter
2014-04-11 12:13:44 +02:00
Raul Ochoa
d10bd8e3b0
precompiled query may write regex
2014-04-11 12:03:43 +02:00
