Paul Ramsey
4914100205
CopyFrom works, but still needs a decent return payload and
...
a lot of work on returning useful information for error cases (post empty input,
db errors returned more nicely? etc)
2018-04-24 13:07:57 +02:00
Paul Ramsey
72bce5732b
WIP on /copyfrom
2018-04-24 11:26:15 +02:00
Simon Martín
985f9ef7f5
pgEntitiesAccessValidator
2018-04-23 18:17:44 +02:00
Rafa de la Torre
2229d0ee57
Fix jshint issues
2018-04-13 16:36:43 +02:00
Paul Ramsey
0161696627
Update copy_controller.js
2018-04-13 05:43:23 -07:00
Paul Ramsey
d54e2f5a07
Implementation including multer, custom storage engine, and pg-copy,
...
but without turning over pg-copy, and demonstrating the missing 'sql'
parameter in the custom storage engine.
2018-04-12 12:25:28 -07:00
Simon Martín
80818c3cbb
fix userLimits reference
2018-03-20 16:20:56 +01:00
Simon Martín
581bd0dd89
Merge branch 'master' into rateLimits
2018-03-14 12:47:45 +01:00
Daniel García Aubert
8dea7615c4
Use user's port instead of the generic one
2018-03-13 19:08:51 +01:00
Daniel García Aubert
328d6f253e
Honor batch api port
2018-03-13 18:49:01 +01:00
Simon Martín
b6ff37d33f
changing param name
2018-03-02 19:57:33 +01:00
Simon Martín
7a6ef637ae
make jshint happy
2018-03-01 14:47:34 +01:00
Simon Martín
f3adad57aa
fix assignment
2018-03-01 13:15:32 +01:00
Simon Martín
779ab3b83f
adding rate limit middleware to controllers
2018-03-01 12:19:57 +01:00
Simon Martín
2834975d14
controllers using user limits service
2018-03-01 11:31:35 +01:00
Daniel García Aubert
f85bdf53a5
Missing middleware
2018-02-23 13:19:26 +01:00
Daniel García Aubert
47b54612c7
Place send-response-middleware to the very end of middleware stack
2018-02-23 13:03:56 +01:00
Daniel García Aubert
700c64bba3
Extract to a middleware user timeout limit from user-database-services
2018-02-22 12:45:55 +01:00
Daniel García Aubert
94c5bd11df
Split authorization middleware, it was actually doing two things: authorize and get database connection params
2018-02-22 12:22:39 +01:00
Daniel García Aubert
8730b5c517
Read if the request is authenticated through res.locals
2018-02-22 12:04:05 +01:00
Daniel García Aubert
b399abee18
Move authorization to auth-api and extract it from user-database-service
2018-02-22 11:46:34 +01:00
Daniel García Aubert
9088cbf150
Improve naming
2018-02-21 13:51:28 +01:00
Daniel García Aubert
ba113d8628
Improve naming
2018-02-20 16:43:43 +01:00
Daniel García Aubert
33baa399ae
Create function to compose middlewares to perfom job operations
2018-02-20 16:25:16 +01:00
Daniel García Aubert
864ec30058
Remove duplication
2018-02-20 13:22:33 +01:00
Daniel García Aubert
d4d54648d4
Improve naming
2018-02-20 13:14:28 +01:00
Daniel García Aubert
8c13add7a8
Rename middleware
2018-02-19 15:49:17 +01:00
Daniel García Aubert
939443ef81
Unify credentials and authenticated request midllewares
2018-02-19 15:44:28 +01:00
Daniel García Aubert
7898b49e59
Extract profiler middleware to used in query and job controllers
2018-02-19 15:13:36 +01:00
Daniel García Aubert
85fbd7c6b2
Create middlewares to initialize and finish profiling
2018-02-19 14:42:52 +01:00
Daniel García Aubert
75c2d85dbb
Use authenticated middleware in query controller
2018-02-19 13:24:44 +01:00
Daniel García Aubert
114070ef96
Rename middleware
2018-02-19 12:37:19 +01:00
Daniel García Aubert
adfe3dd52b
Remove comment
2018-02-19 11:10:39 +01:00
Daniel García Aubert
ea310db62c
Cosmetic chenges
2018-02-19 11:04:28 +01:00
Daniel García Aubert
edd382fb8f
Check user is the same user that sends the request when basic-auth is provided
2018-02-16 18:21:06 +01:00
Daniel García Aubert
2cadfe6f18
Add support for Basic Auth
2018-02-16 10:46:58 +01:00
Daniel García Aubert
756fbe42c9
Use apikey midlleware
2018-02-15 17:24:04 +01:00
Daniel García Aubert
67f2a85abd
Add database credentials to jobs
2018-02-15 12:30:04 +01:00
Daniel García Aubert
b10937a8fb
Use res.locals instead of req.context to share object among middlewares
2017-11-24 15:49:25 +01:00
Daniel García Aubert
c2d1a4653e
Improve naming for connection database params holder
2017-11-24 15:45:49 +01:00
Daniel García Aubert
b1112e2bb1
Remove unneeded header
2017-09-20 16:48:22 +02:00
Daniel García Aubert
980e762ab7
Skip user auth middlewares
2017-09-20 16:35:33 +02:00
Daniel García Aubert
0597a3b8a6
Change job work in progress endpoint
2017-09-20 15:46:29 +02:00
Raul Ochoa
32154b67c6
Merge branch 'master' into upgrade-cartodb-psql
2017-08-11 11:57:16 +02:00
Daniel García Aubert
01a71ee60e
Apply user timeout to ogr2ogr command
2017-08-09 12:50:16 +02:00
Raul Ochoa
3e536b822e
Remove unnecessary destroyOnError option
2017-08-08 18:21:10 +02:00
Daniel García Aubert
99c7a6e4f9
Raise job query size to 16kb
2016-11-07 16:50:44 +01:00
Daniel García Aubert
4a64d37c6c
Add work in progrees list endpoint
2016-10-28 15:08:42 +02:00
Raul Ochoa
e401c01d78
Only log on non-test environments
2016-10-12 01:40:14 +02:00
Raul Ochoa
2a2a54a073
DRY in job response handler
2016-10-04 16:07:13 +02:00
Raul Ochoa
19c9bec633
Callback with job
2016-10-04 15:57:13 +02:00
Raul Ochoa
05cbd55b95
Use db host from request's context
2016-10-04 15:50:39 +02:00
Raul Ochoa
c32a2199fa
Use request bootstrapper to add host header
2016-10-04 15:43:19 +02:00
Raul Ochoa
7b7d651d8f
DRY while authenticating requests
2016-10-04 15:40:56 +02:00
Raul Ochoa
20f50d988e
Use user middleware in job controller
2016-10-04 15:19:31 +02:00
Raul Ochoa
2edc7505e7
Do not condition req.profiler
2016-10-04 15:08:31 +02:00
Raul Ochoa
3b6bc14d17
Increment errors on err
2016-10-04 13:19:29 +02:00
Raul Ochoa
2aa1f045ff
Use .status() API
2016-09-26 18:10:20 +02:00
Raul Ochoa
ba0f2f1066
Remove endpoint to retrieve jobs list
2016-08-30 18:49:01 +02:00
Raul Ochoa
704ba110ba
Log job creation
2016-08-30 18:43:09 +02:00
Raul Ochoa
ad0d101bfd
Remove patch/put endpoints for jobs
2016-08-30 17:43:34 +02:00
Raul Ochoa
3ebbd9f7c4
Skip tables with no updated_at registered in cdb_tablemetadata
2016-07-11 16:39:12 +02:00
Daniel García Aubert
282da58ffe
Set default value for statsd-client in job-controller to avoid check it every time it's going to be used
2016-05-30 12:27:19 +02:00
Daniel García Aubert
976bf5b039
Implemented profiling for job-runner and job-controller
2016-05-26 19:44:59 +02:00
Daniel García Aubert
ea00c22577
Merge branch 'master' into batch-add-profile
2016-05-26 17:46:52 +02:00
Raul Ochoa
a91c7a6a88
Increase job payload size to 8kb
2016-05-24 15:41:18 +02:00
Raul Ochoa
23228b2d73
Payload size validates multiple queries and fallback queries
...
It uses a middleware to check the body size
2016-05-24 14:28:00 +02:00
Daniel García Aubert
64ad284c9c
WIP: adding metrics to Batch API
2016-05-24 11:19:00 +02:00
Daniel García Aubert
cc7dd7a0d2
Job model refactor
2016-05-13 18:50:55 +02:00
Daniel García Aubert
5de931daa5
Improver error message in job max payload limit.
2016-04-25 11:18:30 +02:00
Daniel García Aubert
1dcbb1afdf
Improved error message
2016-04-18 16:24:52 +02:00
Daniel García Aubert
790b9c6124
Fixed typos
2016-04-18 15:44:48 +02:00
Daniel García Aubert
7cc7482944
Implemented limit to query size in Batch API.
2016-04-18 15:30:16 +02:00
Daniel García Aubert
056f22b156
Implemented multi-jobs, user is able to send an array of jobs and batch service will run them in series
2016-03-18 14:57:18 +01:00
Raul Ochoa
66cd1f400f
Skip query tables cache for authenticated requests
2016-03-10 19:20:56 +01:00
Raul Ochoa
b8460d033e
Add a cache decorator over QueryTables
2016-03-08 14:48:56 +01:00
Raul Ochoa
dbe033112d
Rethrow error but do not fail when getting affected tables
2016-03-08 14:00:31 +01:00
Alejandro Martínez
cd593e12ce
Join all Surrogate-Keys in one line
2016-02-24 11:43:43 +01:00
Alejandro Martínez
7d024dbf3f
Fix include in query_controller
2016-02-22 19:10:59 +01:00
Alejandro Martínez
5411aa3a01
Use Date.now() fallback value for getLastUpdatedTime
2016-02-22 19:08:04 +01:00
Alejandro Martínez
2c4b659100
Properly discard errors fetching updated tables
2016-02-22 19:07:41 +01:00
Alejandro Martínez
5e06711b4b
Use node-cartodb-querytables library
2016-02-22 12:24:05 +01:00
Raul Ochoa
15f90c1a78
QueryTablesApi only caches affected tables and always retrieve last modification
2016-02-02 01:16:24 +01:00
Raul Ochoa
9deab814c9
Skip query-tables-api for authenticated requests
2016-02-01 16:53:28 +01:00
Raul Ochoa
a654c90b6d
X-Cache-Channel generation only requires affected tables
...
mayWrite is redundant, and it's not its responsibility
2016-01-28 14:18:38 +01:00
Raul Ochoa
131d798653
Change interface to return also authenticated connection params
2016-01-28 14:18:10 +01:00
Raul Ochoa
235e7f9bc1
Rename method to match returning connection params behaviour
2016-01-28 14:14:31 +01:00
Raul Ochoa
023d06fd19
Affected tables and last modified behind QueryTablesApi
...
It handles the internal cache and exposes an unified result
2016-01-28 14:08:18 +01:00
Raul Ochoa
7dad90a108
Remove unused dependencies from job controller
2016-01-28 13:10:21 +01:00
Raul Ochoa
9881a6f7d4
Use readonly flag to run affected tables query
2016-01-28 12:33:00 +01:00
Daniel García Aubert
f2ea01f627
Removed unnecessary check in job controller
2016-01-25 14:50:43 +01:00
Daniel García Aubert
79e626a71e
Added cors middleware, decoupled query and job controllers from its dependencies
2016-01-21 17:24:48 +01:00
Daniel García Aubert
048d2cd0ea
Refactored user database service ans updated controllers to use it.
2016-01-21 16:17:17 +01:00
Daniel García Aubert
f42727212b
Removed unnecessary checkAborted function in job controller and changed signature for getUserDatabase.
2016-01-21 15:05:46 +01:00
Daniel García Aubert
2913aed697
Placed job controller endpoit under /api/vX/sql
2016-01-18 20:12:44 +01:00
Daniel García Aubert
2d13bde301
Implemented uses cases test for Batch API
2016-01-11 20:14:15 +01:00
Daniel García Aubert
20f00d58d9
Refactored batch service to avoid event noise, doing in callback way
2016-01-08 15:47:59 +01:00
Daniel García Aubert
e6a4e0f7eb
Returned 201 HTTP code in job creation
...
Fixed bug listing jobs, now user job index is cleaned before respond to client
2016-01-07 12:07:30 +01:00
Daniel García Aubert
40d0161218
Fixed bug in authentication for PUT methods in Job Controller
2016-01-05 13:54:18 +01:00
Daniel García Aubert
246655de94
Changed redis data structure for users jobs
2016-01-04 19:08:13 +01:00
Daniel García Aubert
35650985db
Implemented job modification whether job is pending
2015-12-31 15:42:31 +01:00
Daniel García Aubert
ada39d84b8
Implemented job cancelation in Batch API
2015-12-30 20:16:18 +01:00
Daniel García Aubert
851c393f75
Refactored batch manager avoiding concurrence issues
2015-12-29 10:19:10 +01:00
Daniel García Aubert
4dcd7c0a50
Fixed typo and prepared for listing user's job
2015-12-24 17:47:41 +01:00
Daniel García Aubert
c4f96b4a59
Implemented support for listing jobs for one user
2015-12-24 17:42:49 +01:00
Daniel García Aubert
ecb76ca1de
Improved batch API being consistent with naming, humam readability for job dates, hide redis key details for jobs.
2015-12-23 17:29:11 +01:00
Daniel García Aubert
62db809476
Return 401 when no API KEY is provided
2015-12-23 14:55:49 +01:00
Daniel García Aubert
30d0b2906b
Implemented GET for jobs API
2015-12-22 11:45:25 +01:00
Daniel García Aubert
ef59a1d942
Fixed typo in batch test
2015-12-22 11:06:46 +01:00
Daniel García Aubert
328d4c6bfe
Fixed bad assertion in batch acceptance test
2015-12-22 11:02:16 +01:00
Daniel García Aubert
3762ad7a39
Now jobs are stored in redis instead of user's database
2015-12-21 19:57:10 +01:00
Daniel García Aubert
fb8feeb964
Now Job is an event emitter instance and we are able to know what happen wwith any job. Implemented test without a silly timeout
2015-12-16 18:13:48 +01:00
Daniel García Aubert
43f759e96a
Refactored Batch API using streams instead a interval to consume the job queue. Limited one job running at the same time per queue instead of using a job's counter to limit it.
2015-12-16 15:57:58 +01:00
Daniel García Aubert
6cc48bf9dd
Implemented batch service
2015-12-09 20:17:45 +01:00
Daniel García Aubert
c7680722ca
Passed tests
2015-12-09 12:35:20 +01:00
Daniel García Aubert
00721bcd02
Implementing batch service
2015-12-09 00:02:08 +01:00
Daniel García Aubert
6f741827cd
Fixed jshint error
2015-12-07 12:19:16 +01:00
vagrant
35d6600f36
Implemented test for job manegement
2015-12-07 10:29:55 +00:00
Daniel García Aubert
ee41db99ad
Returned job_id to client/user after job enqueue
2015-12-07 09:59:05 +01:00
Daniel García Aubert
a1243ad64d
Implemented job controller to enqueue jobs
2015-12-07 09:40:51 +01:00
Daniel García Aubert
f7a3c6ac4e
Renamed controllers routing method
2015-12-03 18:43:13 +01:00
Daniel García Aubert
c6bae6d951
app module moved to right place
2015-12-03 18:33:17 +01:00
Daniel García Aubert
d31d496a2e
Add useful comment
2015-12-03 18:25:35 +01:00
Daniel García Aubert
a697fe6faa
Making test pass
2015-12-03 18:19:39 +01:00
Daniel García Aubert
e5dedb6315
Split query controller and separated from app
2015-12-03 17:28:18 +01:00
Daniel García Aubert
df59405dca
Split some functionallity from app to dedicated controllers
2015-12-03 15:00:35 +01:00
Raul Ochoa
b160f4051a
Merge pull request #250 from CartoDB/issue-238
...
Stop adding X-Cache-Channel header when no tables involved
2015-09-08 12:30:26 +02:00
Raul Ochoa
5ec6d7c77b
Remove no longer needed health check params
2015-09-07 18:34:50 +02:00
Raul Ochoa
875eccba62
Stop adding X-Cache-Channel header when no tables were identified in SQL query
...
Fixes #238 and fixes #157
2015-09-07 18:22:24 +02:00
Raul Ochoa
335d71bc04
Adds a custom log4js format function to log sql query from POST requests
...
Tokens in log4js are not provided with request/response for context,
using a custom format function allow to access them
In the case of express logger token functions receive request/response
for context so it uses a custom token
Closes #206
2015-09-04 12:29:20 +02:00
Raul Ochoa
f816093036
Set Last-Modified
header based on affected tables
...
Closes #101
2015-09-02 16:25:56 +02:00
Raul Ochoa
c6c614b5bf
Do not return results from health check
...
It also removed old dependencies and takes disabled file path in ctor.
2015-08-28 18:16:04 +02:00
Raul Ochoa
d42e5a6b75
Do not expose remove from pool after query error as a config option
2015-08-04 16:23:36 +02:00
Raul Ochoa
7c696317d0
New option to remove client from pool after error happens
...
This help to avoid issues with transactions.
Closes #241
2015-08-04 15:53:50 +02:00
Raul Ochoa
a297a14300
Revert "Uses tables ogr2ogr param to avoid heavy query on pg_class and friends"
...
This reverts commit b6e53f7326
.
2015-06-16 16:58:22 +02:00
Francisco Dans
3a8f94641b
Merge pull request #225 from CartoDB/log-addr
...
Uses :remote-addr instead of :req[X-Real-IP]
2015-06-01 16:17:47 +02:00
Francisco Dans
996bd971cd
replaces default log format string to remote-addr
2015-05-26 19:18:54 +02:00
Raul Ochoa
debeb8ca3e
Update news
2015-05-25 16:38:07 +02:00
Raul Ochoa
b6e53f7326
Uses tables ogr2ogr param to avoid heavy query on pg_class and friends
...
Fixes #204
Newer versions of ogr2ogr don't use that heavy query so it should work
Ref 2e3eeefd0f
2015-05-25 16:29:18 +02:00
Raul Ochoa
9dd887dc4d
jshint fixes, not addressing complexity
2015-05-13 12:22:41 +02:00
Raul Ochoa
9dcb397737
Merge pull request #205 from CartoDB/url_rewrite
...
added /u/:user routing
2015-03-20 12:22:07 +01:00
Raul Ochoa
9b31df6793
Improve row size limit error message
2015-03-02 14:34:01 +01:00
Raul Ochoa
302a856d37
Move note about username extraction to cartodb_request
2015-02-02 12:15:53 +01:00
Raul Ochoa
e5ab4272eb
Use a multiple params route to be able to extract the username from
...
the path or default to host header.
2015-02-02 12:09:34 +01:00
javi
824b7c084e
added /u/:user routing
2015-01-30 11:37:29 +01:00
Seth Fitzsimmons
bcc1577ec4
Add 'refresh' to the list of keywords that write
2015-01-06 12:40:01 -08:00
Raul Ochoa
d50ddbb10f
Add more fields to error responses with hint, detail and context for SQL errors.
2014-11-21 12:59:48 +01:00
Raul Ochoa
74429f82e1
Improve topojson output by streaming json
2014-11-12 11:36:59 +01:00
Raul Ochoa
dc601a5feb
Improve statement timeout error messages
2014-11-11 13:57:15 +01:00
Raul Ochoa
25d2e64891
Removes query tables console.log
2014-09-17 15:54:46 +02:00