cartodb4/dataservices-api
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,485 Commits 1 Branch 183 Tags 3.3 MiB
8674dabeb2
Commit Graph

331 Commits

Author SHA1 Message Date
Mario de Frutos
6177f0ac76 Remove info trace not needed in metrics 2016-11-07 11:02:09 +01:00
Mario de Frutos
88ed777785 Added plpy_mock and check multiple instances for metricsdatagatherer 2016-11-04 13:39:11 +01:00
Mario de Frutos
73aa26590b Make the data gather singleton support multiple instances 
Due to we use pgbouncer, we could share the same instance of the metrics
gatherer singleton in multiples requests. Added support to have
multiple instances identified by transaction id
 2016-11-04 13:39:11 +01:00
Mario de Frutos
c571edb21e Split response and function data, added uuid and type of element 2016-11-04 13:39:11 +01:00
Mario de Frutos
e878031744 Added url to the response data 2016-11-04 13:39:11 +01:00
Mario de Frutos
49fd51f6da Added logger to metrics context manager 2016-11-04 13:39:10 +01:00
Mario de Frutos
cf3c6f2ce5 New metrics logger system 
- It's not mandatory to have the log_path defined for all the services
- Now we add metrics to a MetricsGatherer using the add method in order
  to gather data from all the function scope, not just in one defined
  point
- There is a `metrics` context manager to wrap the function block. This
  context manager get the execution time for the block among other
  things
 2016-11-02 15:26:39 +01:00
Mario de Frutos
6058960ec5 Added timeout for all the third-party connections 2016-10-27 16:17:37 +02:00
Mario de Frutos
336d8be977 Hotfix: Typo in status code check 2016-10-26 17:49:07 +02:00
Mario de Frutos
80963e2589 504 errors return empty data instead of raise exception 
Due to some problems in Mapzen, we're receiving 504 errors from their
servers. To mitigate this problem, instead of raise an exception, we're
going to return empty data for that point
 2016-10-26 16:39:35 +02:00
Mario de Frutos
19d6cacdb3 Fix empty response from matrix 
https://github.com/CartoDB/dataservices-api/issues/293
 2016-10-26 16:00:50 +02:00
Rafa de la Torre
0d22942a72 Update version of python lib server package to 0.9.2 2016-10-21 17:38:32 +02:00
Rafa de la Torre
d4ac2eb5e6 Tests for ServicesRedisConfig (mapzen quota) #281 2016-10-21 15:40:22 +02:00
Rafa de la Torre
db80d389e0 Do not require org geocoder and isolines quota to be present 2016-10-21 15:38:58 +02:00
Rafa de la Torre
8e02c64aeb Tests for UserMetricsService, mapzen routing #281 2016-10-21 15:21:03 +02:00
Rafa de la Torre
cc2ab1bc0c Fix test: make it independent of current date #281 2016-10-21 15:08:17 +02:00
Rafa de la Torre
e1a7d1751c Fix corner case: used_quota == monthly_quota #281 2016-10-20 15:36:48 +02:00
Rafa de la Torre
0c49107f96 Some tests for the QuotaChecker #281 2016-10-20 15:31:21 +02:00
Rafa de la Torre
247034c21e Some tests for RoutingConfig #281 2016-10-19 18:40:17 +02:00
Rafa de la Torre
2b1b1c981f Add routing to UserMetricsService.used_quota 2016-10-19 17:41:43 +02:00
Rafa de la Torre
aaff5564ec Use soft_limit for __check_routing_quota #281 2016-10-18 17:44:56 +02:00
Rafa de la Torre
72998c324a Simplify the code for regularity's sake #281 2016-10-18 17:40:50 +02:00
Rafa de la Torre
bbd9b6b98e Add soft limit to config object #281 2016-10-18 17:35:36 +02:00
Rafa de la Torre
27be704bd6 Changes needed to read quota from redis #281 
Use per-user/org quota from redis, if present. Otherwise default to the
quota stored in server database (as it did to date).
 2016-10-18 17:26:45 +02:00
Rafa de la Torre
98d533b707 Bump version of python pip package to 0.9.1 2016-10-06 15:05:00 +02:00
Rafa de la Torre
6d0ad85d48 Fix for crash when sentinel_master_id is absent in config 2016-10-06 15:04:25 +02:00
Rafa de la Torre
00e6cace76 Update version of the python pip package 2016-10-06 11:06:28 +02:00
Rafa de la Torre
e9ad35ba1d Merge pull request #284 from CartoDB/redis-refactor-take2 
Redis refactor: Take 2
 2016-10-05 18:32:08 +02:00
Rafa de la Torre
cded6c2f08 Test for NullConfigStorage 2016-10-04 15:20:41 +02:00
Rafa de la Torre
e1b357137a Rename s/storage/backend for consistency 2016-10-04 14:59:57 +02:00
Rafa de la Torre
3844cfc226 Tests for environment module 2016-10-04 14:56:06 +02:00
Rafa de la Torre
2a1276f4f1 Add test for redis_config module 2016-10-04 12:53:52 +02:00
Rafa de la Torre
35da7e48fd Implement a ServerConfigBackendFactory 
mostly to keep layers separated.
 2016-10-03 17:42:19 +02:00
Rafa de la Torre
12aebb7eee Rename ConfigStorageInterface to ConfigBackendInterface 2016-10-03 17:35:33 +02:00
Rafa de la Torre
0d87a95270 Inject the environment into the LoggerConfigBuilder 2016-10-03 17:06:57 +02:00
Rafa de la Torre
18e1a5c7c9 Remove the class User and its tests 
as it is not used.
 2016-10-03 16:54:40 +02:00
Rafa de la Torre
fcca5da302 Add a RedisMetricsConnectionFactory class 2016-10-03 16:47:45 +02:00
Rafa de la Torre
1aec541906 Add ServerEnvironment and ServerEnvironmentBuilder 2016-10-03 16:21:28 +02:00
Rafa de la Torre
9e98e0794d Add backends for user and org configs 2016-10-03 16:05:08 +02:00
Rafa de la Torre
8fbb41742c Rename redis conf factories to builders 2016-10-03 15:25:01 +02:00
Rafa de la Torre
275a6dc27f Move redis_config related classes to separate file 2016-10-03 13:33:46 +02:00
Rafa de la Torre
d522083d5c Rename redis_connection_config test file as well 2016-10-03 13:22:47 +02:00
Rafa de la Torre
073163eb1a Rename redis_config.py to redis_connection_config.py 2016-10-03 13:20:30 +02:00
Rafa de la Torre
0c62c4bada Move the NullConfigStorage to a separate file 2016-10-03 13:17:04 +02:00
Rafa de la Torre
3361960cfc Move InMemoryConfigStorage to a separate file 2016-10-03 13:12:49 +02:00
Rafa de la Torre
86ab3abc53 Rename mapzen_geocoder module to mapzen_geocoder_config 2016-10-03 12:58:15 +02:00
Rafa de la Torre
fb812ee15e Add tests for redis connection configs 2016-10-03 12:33:38 +02:00
Rafa de la Torre
c1dd410201 Make RedisConnectionConfigBuilder abstract 
and also use custom ConfigException instead of generic one.
 2016-10-03 12:21:45 +02:00
Rafa de la Torre
34ddd28e6b Move ConfigException to a separate file 2016-10-03 12:20:17 +02:00
Carla
443fe88d5a Add newline to end fle 2016-09-30 13:08:27 +02:00
Carla Iriberri
74d2fba763 Add Redis Mock -- tested and working 2016-09-30 12:28:27 +02:00
Rafa de la Torre
1e6ee8d5c1 Add an Environment class 2016-09-30 11:21:35 +02:00
Rafa de la Torre
3a6cc4c364 Add mapzen config and integrate into legacy code 2016-09-29 18:41:16 +02:00
Rafa de la Torre
0b7b44d8a5 Add a couple of factories to abstract user/org configs 2016-09-29 17:46:17 +02:00
Rafa de la Torre
4b4a02905c Implement the RedisConfigStorage 2016-09-29 17:18:45 +02:00
Rafa de la Torre
1f3a655ae5 Add all the code to instantiate a user_config_storage (WIP) 2016-09-29 16:57:31 +02:00
Rafa de la Torre
9d60fde0b8 A stubb for RedisConfigStorage (WIP) 2016-09-29 13:07:40 +02:00
Rafa de la Torre
efdc151282 Add things to get logger config from storage 2016-09-28 18:22:51 +02:00
Rafa de la Torre
fd2cc21942 Add server config storage classes 2016-09-28 17:32:26 +02:00
Rafa de la Torre
18f05fbd4f User class with tests 2016-09-28 16:30:37 +02:00
Carla Iriberri
96a93e3c56 Python package version 0.8.1 2016-09-28 15:09:29 +02:00
Rafa de la Torre
64fc18b9e0 Fix and improve test speed 2016-09-28 13:43:50 +02:00
Mario de Frutos
9381d5644b Fixed QPS retry decorator 2016-09-16 13:46:02 +02:00
Carla
9f55f2ee3b Update README.md 2016-09-09 11:44:57 +02:00
Mario de Frutos
18ae2525b6 Add search type filter for Mapzen geocoder 2016-08-29 14:32:49 +02:00
Mario de Frutos
71d5ce951a Use mapzen as first option for the namedplace geocoding 2016-08-26 16:39:25 +02:00
Mario de Frutos
d602c42559 Strip spaces in the mapzen geocode parameters 2016-08-19 13:32:11 +02:00
Mario de Frutos
3cc4e06420 Add more logs for mapzen services 2016-08-19 11:44:59 +02:00
Mario de Frutos
0fb3cdd8b1 JSON response is not always present, better show the reponse object complete 2016-08-05 11:35:16 +02:00
Mario de Frutos
7cca8e95c3 Remove testing exception 2016-08-03 14:45:57 +02:00
Mario de Frutos
a33999ed04 Dont re-raise the exception to avoid leak sensitive information 2016-08-03 12:08:23 +02:00
Mario de Frutos
9186eb5d5b Log part decoupled from the main config class 2016-08-02 18:40:01 +02:00
Mario de Frutos
ed5cf25e9c Added new logger to all the functions 2016-08-02 16:44:03 +02:00
Mario de Frutos
5d5e8d6f9a Added logger with rollbar integration 2016-08-01 18:03:42 +02:00
Mario de Frutos
993059eafb Added server config with environment data 2016-08-01 14:00:32 +02:00
Mario de Frutos
f9f39dcf9c Refactor metrics logger 2016-08-01 13:17:06 +02:00
Mario de Frutos
4d47b905e8 Revert include the gateway error in QPS 2016-08-01 11:23:06 +02:00
Mario de Frutos
c23a57c03f Add QPS retry for mapzen isolines and add the 504 error to retry 2016-07-29 16:22:34 +02:00
Mario de Frutos
efde2ee1b7 Fix typo in the default value for service providers 2016-07-25 14:00:29 +02:00
Mario de Frutos
fc399c12c1 Change notices for debug 2016-07-25 13:58:01 +02:00
Mario de Frutos
f03897257e Better than lower, change to empty string if not provider setting in redis 2016-07-25 13:40:06 +02:00
Mario de Frutos
90998d4819 Fix users without provider setting setted 2016-07-25 13:35:59 +02:00
Mario de Frutos
79d0b5ba7c Add service providers that come from the user configuration 2016-07-25 12:25:52 +02:00
Mario de Frutos
ff8eed9750 Fix empty component return in isolines 2016-07-22 14:54:53 +02:00
Carla
6fb9f67e64 Update README.md 2016-07-11 10:50:19 +02:00
Carla Iriberri
aff5dd13d7 Bump python library to 0.7.0 2016-07-07 10:04:54 +02:00
Carla Iriberri
1022097300 Bump server to 0.12.0 2016-07-07 09:58:51 +02:00
Carla
5c20866277 Merge pull request #216 from CartoDB/mapzen-isolines-reloaded 
Add mapzen isolines
 2016-07-07 09:31:04 +02:00
Rafa de la Torre
f5d51da673 Fix another typo (hello Carla!!) 
This feature is dedicated to you. Keep rocking.
 2016-07-06 20:29:08 +02:00
Rafa de la Torre
bcc6bc35d3 Fix None*unit_factor error 
Also make the code more explicit about what happens when getting cost ==
None.
 2016-07-06 19:48:23 +02:00
Rafa de la Torre
230112b7e5 Add calculate_isodistance function 2016-07-06 19:20:21 +02:00
Rafa de la Torre
523eda2cc7 Generalize calculate_isochrone to calculate_isoline 2016-07-06 18:43:09 +02:00
Rafa de la Torre
54221fa671 Add transport mode car 2016-07-06 18:05:51 +02:00
Rafa de la Torre
6d888a7a62 Fix for points getting None cost 
Sometimes there's no route information for the point in a particular
angle we're interested in. In this case it is better to use more
points/angles and discard the ones we're not interested in.
 2016-07-06 18:03:24 +02:00
Rafa de la Torre
075edf0e0d More precission for earth's radius 2016-07-06 18:01:33 +02:00
Rafa de la Torre
6c4829df01 Small refactor for sanity 2016-07-06 16:05:51 +02:00
Carla Iriberri
7ddb3da60d Remove useless cost_per_hit line 2016-07-06 15:43:26 +02:00
Carla Iriberri
ff4eb5b348 Mock mapzen matrix config 2016-07-06 14:21:32 +02:00
Rafa de la Torre
2147d190a1 Unit test for the mapzen isolines 2016-07-06 13:19:28 +02:00
Carla Iriberri
a046d3ce97 Add Mapzen Matrix to config and metrics services 2016-07-06 13:19:28 +02:00
Rafa de la Torre
cdcac2dc1f Fix typo in test case 2016-07-06 13:19:28 +02:00
Carla
e9346faf42 Fix bug on exception raise 2016-07-06 12:33:59 +02:00
Rafa de la Torre
b78bd05754 Be resilient to None cost estimation 2016-07-05 20:52:41 +02:00
Rafa de la Torre
77cdc3d8ff Only refine individual solutions when error > TOLERANCE 2016-07-05 18:48:21 +02:00
Rafa de la Torre
2d95601c5a Fix: max_abs_error should be a scalar 2016-07-05 18:48:21 +02:00
Rafa de la Torre
9a9f35d9c2 Fix silly typos spotted by jgoizueta (WIP) 2016-07-05 18:48:21 +02:00
Rafa de la Torre
9becf1adb4 Iterative part of the algorithm (WIP) 2016-07-05 18:48:21 +02:00
Rafa de la Torre
87413255af Major rewrite of MapzenIsolines (WIP) 2016-07-05 18:48:21 +02:00
Rafa de la Torre
46971fe96f Raise error when response not OK 2016-07-05 18:48:21 +02:00
Rafa de la Torre
96199b0d6d Add example to code doc 2016-07-05 18:48:21 +02:00
Rafa de la Torre
a70560e566 Minimal Mapzen Time-Distance Matrix client 2016-07-05 18:48:21 +02:00
Rafa de la Torre
53fe4ce21d An attempt to adapt paremetrs (WIP) 2016-07-05 18:48:21 +02:00
Mario de Frutos
893b8db374 First stage is calculating the matrix of points 2016-07-05 18:48:21 +02:00
Carla Iriberri
6d35cff9c7 Exposes geocoder providers in public geocoder functions 
Add config function in postgres explicitly to get MapzenGeocoderConfig.
Bump versions for client and server APIs. New MapzenGeocoderConfig
included to be able to use current QuotaServices with non-configured
users.
 2016-07-05 11:13:43 +02:00
Rafa de la Torre
9ed059f4cc Bump server python lib version to 0.6.4 #204 2016-06-14 18:42:29 +02:00
Rafa de la Torre
972970a12d Use DAY_OF_MONTH_ZERO_PADDED elsewhere #204 2016-06-14 15:25:38 +02:00
Rafa de la Torre
37bcaeada3 Fix test_orgs_should_write_zero_padded_dates (WIP) #204 2016-06-14 13:18:42 +02:00
Rafa de la Torre
25bf9b6372 Add test_orgs_should_write_zero_padded_dates #204 2016-06-14 13:18:34 +02:00
Rafa de la Torre
f76a5cdfcc Fix test_should_write_zero_padded_dates #204 2016-06-14 10:13:14 +02:00
Rafa de la Torre
e046ca2c4d Add test_should_write_zero_padded_dates #204 2016-06-14 10:04:17 +02:00
Rafa de la Torre
c9f57259be Instructions on how to run integration tests #204 2016-06-13 19:05:31 +02:00
Rafa de la Torre
41b9db4144 Update python lib to 0.6.3 #201 2016-06-13 17:21:15 +02:00
Rafa de la Torre
25d42a75e4 test_should_not_request_redis_twice_when_unneeded #201 2016-06-13 15:45:56 +02:00
Rafa de la Torre
f9e9617d6f test_should_sum_amounts_from_both_key_formats #201 2016-06-13 13:10:07 +02:00
Rafa de la Torre
b0bb60736c Implement test and fix for that case #201 
Add test_should_account_for_zero_paddded_keys plus its corresponding fix
for it.
 2016-06-13 13:08:35 +02:00
Rafa de la Torre
a2863d7135 Allow for the manipulation of datetime.date.today() #201 2016-06-13 12:57:00 +02:00
Rafa de la Torre
a114f857fd Add the first test that makes sense #201 2016-06-13 12:21:06 +02:00
Rafa de la Torre
47962c9ccb Stub tests to be implemented (WIP) #201 2016-06-13 11:33:14 +02:00
Rafa de la Torre
901f5d7b8b Add a README.md file for dummies like me #201 2016-06-13 11:19:14 +02:00
Rafa de la Torre
c7bcbddaa9 Delete copy/pasted comment from requirements.txt #201 2016-06-13 11:16:49 +02:00
Rafa de la Torre
9ad55a4d53 Update url of pip package 2016-06-10 13:03:09 +02:00
Carla Iriberri
21aac960a6 Add function cdb_routing_with_waypoints 
Add routing with waypoints functions to client and server. Includes
signature checks tests for the Postgresql functions and unit and
integration tests for the Python library.

Add client v0.6.0 and server v0.9.0
 2016-05-25 11:38:37 +02:00
Carla
f46305de6f Update test_helper.py 2016-05-24 13:18:43 +02:00
Mario de Frutos
bb17b3bff7 Hotifix: obs_general_quota logic missing in the redis config 2016-05-16 14:19:08 +02:00
Mario de Frutos
c7c9e6e284 Server side new data observatory functions (v0.8.0) 2016-05-16 10:25:26 +02:00
Mario de Frutos
3e09605757 On 4xx we have to return an empty response instead of an exception 2016-05-10 16:46:38 +02:00
Mario de Frutos
e81dadaf2e Green/Blue deploy system for the observatory functions 2016-05-03 13:49:14 +02:00
Mario de Frutos
a9c420ba7b Use redis or db data observatory quota value 2016-04-25 09:33:18 +02:00
Mario de Frutos
3c5325c1d7 Server data observatory functions 2016-04-20 15:58:38 +02:00
Mario de Frutos
1e30b21d35 Fixed tests with the isolines changes 2016-04-18 18:37:46 +02:00
Mario de Frutos
15a7ed5a09 Fixed isolines overwrite for organizations 2016-04-18 11:49:11 +02:00
Mario de Frutos
8cd94973ff Version 0.6.2 2016-04-15 14:43:45 +02:00
Mario de Frutos
c9ddc37051 Mapzen geocoder and routing quota check 2016-04-15 10:21:40 +02:00
Mario de Frutos
b4c1991c90 Mapzen geocoder checks for the iso3 for the country passed 2016-04-13 18:49:13 +02:00
Mario de Frutos
a8dd211acf Use retry on timeout flag for Redis connections 
In order to avoid some fails due some high load peaks in our Redis
databases, we are going to active this flag that retries 1 more time
in case it receives a socket.Timeout error while reading or connecting
 2016-04-04 10:17:59 +02:00
Carla Iriberri
752d8317a2 Merge master 2016-04-01 10:18:46 +02:00
Carla
8c5a4d32de Update setup.py 2016-04-01 10:15:27 +02:00
Mario de Frutos
d625b26e57 Fixed timeout and random problems 2016-04-01 08:45:40 +02:00
Carla
62da0eff21 Update test_helper.py 2016-03-31 16:45:27 +02:00
Carla Iriberri
1505e0e6ea Make isoline credentials independent 2016-03-31 15:52:52 +02:00
Mario de Frutos
89ec3dcfa6 Mapzen geocoder integrated 2016-03-23 13:37:30 +01:00
Mario de Frutos
357458f44d Check for some issues like empty input, recoverable exceptions, etc 2016-03-22 15:19:03 +01:00
Mario de Frutos
4061009df0 4xx generated empty responses 2016-03-17 22:54:19 +01:00
Mario de Frutos
b9f18dda69 Added exception info for config retrieving 2016-03-17 11:36:52 +01:00
Mario de Frutos
7acf70131b Change db and timeout attributes too 2016-03-17 11:01:56 +01:00
Mario de Frutos
0e44910165 Change to sentinel_id 2016-03-17 10:58:49 +01:00
Mario de Frutos
1c33a72fdc Change sentinel attribute to use the config one 2016-03-17 10:56:25 +01:00
Mario de Frutos
5d57624fd5 Merge pull request #107 from CartoDB/rename_and_metrics 
Rename and metrics
 2016-03-17 10:23:14 +01:00
Mario de Frutos
f0a49d5fbc Deleted the used_credits field 2016-03-17 10:15:12 +01:00
Mario de Frutos
60f17fdb09 QPS manager for Mapzen 2016-03-14 11:21:26 +01:00
Mario de Frutos
bd2e380225 Add logger for the internal geocoder too 2016-03-08 16:47:49 +01:00
Mario de Frutos
c293b55c3c Added log and cost_per_hit config to the db config 2016-03-08 16:47:49 +01:00
Mario de Frutos
b97e838416 Refactor to move logic from SQL functions to Python 
- Moved the logic the retrieve the redis connection params to
  RedisDBConfig class
- Moved the logic that retrieve the services configuration to
  ServicesDBConfig
 2016-03-08 16:47:49 +01:00
Mario de Frutos
ac1627b5c5 Logger for geocoder metrics 2016-03-08 16:47:48 +01:00
Mario de Frutos
7237e01601 Change metrics increment naming to reflect service instead of geocoder 2016-03-07 16:03:28 +01:00
Mario de Frutos
df08b27068 Removed dependency with polyline and use our implementation 2016-03-02 10:49:02 +01:00
Mario de Frutos
4d04882c18 Polyline decoder 2016-03-02 10:43:27 +01:00
Mario de Frutos
d2e73a69fa Python library supports Mapzen routing 2016-02-26 10:49:10 +01:00
Luis Bosque
6c5b1b4e99 Use strictredis if sentinel_master_id is not null 
Also, use only a redis_host and redis_port attributes either if the
connection is to redis or to sentinel
 2016-02-26 10:49:09 +01:00
Luis Bosque
93463b3032 Support using a common redis connection 
If sentinel_host or sentinel_cluster_id is not provided it will try to
connect with a redis_host parameter
 2016-02-26 10:49:08 +01:00
Mario de Frutos
d27b2a2b78 Block google services users to use isolines functions 
Due some service restrictions, google services users couldn't use
the HERE services which includes the isolines functions.
 2016-02-26 10:42:05 +01:00
Mario de Frutos
e764b9036d Quota check for the routing feature 
We have refactor all the quota logic and extracted to a new QuotaChecker
class in order to have it ready to create a factory when needed

Added the logic for the routing quota check
 2016-02-18 15:40:55 +01:00
Mario de Frutos
3a4130282e Limit the mode types accepted 2016-02-12 10:01:42 +01:00
Mario de Frutos
f36a345db2 Updated python library version with routing part 2016-02-10 22:26:53 +01:00
Carla Iriberri
d4a2184ce0 Multiple fixes 2016-02-10 19:59:00 +01:00
Mario de Frutos
54622d5c81 Added the routing config and metrics 2016-02-10 17:01:41 +01:00
Mario de Frutos
75a915019f Helper to transform to Postgis multipolygon 2016-02-10 16:01:03 +01:00
Mario de Frutos
13cfee2d4c Fixed tests for the heremaps geocoder 2016-02-10 15:24:25 +01:00
Mario de Frutos
3b0f92d5e7 Routing isolines python library 2016-02-10 15:11:45 +01:00
Mario de Frutos
17b8b5942e Refactor to make the service config extensible to other services 2016-02-05 10:18:21 +01:00
Mario de Frutos
3960c13484 Refactor python library to unify and rename as cdb_services 2016-02-03 10:21:57 +01:00