cartodb4/dataservices-api
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,095 Commits 1 Branch 183 Tags 3.3 MiB
2117afc336
Commit Graph

567 Commits

Author SHA1 Message Date
Rafa de la Torre
27fde5a910 Use "{id}:{secret}" as caching key 2017-10-06 15:12:05 +02:00
Rafa de la Torre
b8e1120169 Revert "Use a different client_id for the test" 
This reverts commit 854fbb803c.
 2017-10-06 15:03:08 +02:00
Rafa de la Torre
8e33cf9723 Fix google services key validity check #382 2017-10-06 14:40:53 +02:00
Rafa de la Torre
7b988e3f78 Add tests for #382 
The examples are not arbitrary: base64 by default ignores characters
that are not in its alphabet. When decoding, the dashes and underscores
are not b64-valid and therefore ignored, resulting in a sequence with
the wrong padding.
 2017-10-06 14:37:26 +02:00
Rafa de la Torre
f79ac9297d Add again the check for valid credentials 
As our check is more strict that the one provided in googlemaps library.
 2017-10-06 13:44:08 +02:00
Rafa de la Torre
854fbb803c Use a different client_id for the test 2017-10-06 13:41:50 +02:00
Rafa de la Torre
d029ad7041 GoogleMapsClientFactory to persist clients #401 
Add a GoogleMapsClientFactory and remove the check for valid key,
which is no longer needed, as it is done in the google library.
 2017-10-06 12:05:34 +02:00
Rafa de la Torre
445d4cf97d Upgrade googlemaps python library to v2.5.1 
This closes #383
 2017-10-04 16:22:33 +02:00
Rafa de la Torre
f0a3249b4e Keep metrics keys consistent 
See cartodb repo and our internal documentation:
https://github.com/CartoDB/cartodb/search?utf8=%E2%9C%93&q=failed_responses&type=
 2017-09-07 11:24:38 +02:00
Mario de Frutos
475df918c7 Fix invalid geometries due to generalize option 
They add a note saying that use their simplification option could
lead to a self-intersection (which is the problem we have) and it's
creating invalid geometries

See it here https://mapzen.com/documentation/mobility/isochrone/api-reference/#other-request-parameters
 2017-08-30 17:35:08 +02:00
Mario de Frutos
c13dba71a4 Deactivated DO credits usage for users that have quota 
See https://github.com/CartoDB/bigmetadata/issues/215
 2017-08-24 14:35:20 +02:00
Daniel García Aubert
13a876b854 Fix typo while logging caught error in geocoder 2017-07-21 14:43:41 +02:00
Daniel García Aubert
1503b5bb2b Release artifacts: 
- Server extension version 0.25.0
 - Client extension version 0.18.0
 - Python library version 0.15.2
 2017-07-13 15:34:44 +02:00
Rafa de la Torre
49a53e4cd7 Fix non-zero-padded days in tests 
See https://github.com/CartoDB/dataservices-api/issues/204 and related
tickets.
 2017-07-07 15:14:27 +02:00
Mario de Frutos
61efb66aba Add an extra check for google credentials 
If the user has a wrong base64 padded secret key the googlemaps
python library is returning "TypeError: Incorrect padding" which
is very hard to understand. So now we check if the secret key is
a valid base64 string
 2017-05-26 11:51:32 +02:00
Rafael Porres Molina
37eeab5b9e past participle of set is set 2017-05-24 15:41:12 +02:00
Javier Goizueta
524d15c1a9 Fix uses of cartodb cdb_conf functions 2017-03-30 18:10:53 +02:00
Javier Goizueta
aac89e0236 New versions 0.16.0 (client), 0.23.0 (server), 0.15.0 (python) 2017-03-28 17:53:40 +02:00
Javier Goizueta
f0b0a9e7f2 Merge pull request #355 from CartoDB/346-user-rate-limits 
Service rate limits
 2017-03-28 17:48:46 +02:00
Rafa de la Torre
77dfda11d9 Merge pull request #353 from CartoDB/udpate-docs 
Fix links and improve readability
 2017-03-28 11:53:07 +02:00
Javier Goizueta
2c15110255 Rename constructor arguments for consistency 2017-03-28 10:53:16 +02:00
Javier Goizueta
63c03894cc Fix bugs 2017-03-21 15:45:05 +01:00
Javier Goizueta
942c6ac63d Fix indentation 2017-03-21 15:44:50 +01:00
Javier Goizueta
208469f534 Add rate limits configuration writing tests 2017-03-21 15:43:33 +01:00
Javier Goizueta
73f97128ed Reorganization of configuration modules, fix circular dependencies 
The new module cartodb_services/config is intended for services configuration objects
Some legacy configuration objects remain under cartodb_services/metrics.
The refactored configuration backends are also not moved here
 2017-03-21 15:42:53 +01:00
Javier Goizueta
945c6cd685 WIP: support for storing rate limits configuration 2017-03-16 19:12:39 +01:00
Javier Goizueta
7f6c19b292 Minor changes for clarity 
* ServiceManager check method renamed as assert_within_limits
* Legacy classes moved to separate files
 2017-03-16 17:59:22 +01:00
Javier Goizueta
9a04ad06a0 Move rate limits out of Config 
The (legacy) Config object rate limit-related modifications are reverted.
For the legacy case, configuration is handled in a specific RateLimits builder class.
 2017-03-15 19:32:50 +01:00
Javier Goizueta
cebd7d2141 Apply ServiceManager to Google Geocoder 
Note that the Goolge geocoder uses QuotaService to account for used quota,
but it doesn't have a quota limit.
 2017-03-15 17:58:14 +01:00
Javier Goizueta
680206a437 Missing import 2017-03-15 12:51:56 +01:00
Javier Goizueta
77b5ff0b9e Fix problem with db config default arguments 2017-03-15 12:46:25 +01:00
Javier Goizueta
05b29967c7 Merge branch 'development' into 346-user-rate-limits 
# Conflicts:
#	README.md
#	server/extension/sql/20_geocode_street.sql
#	server/lib/python/cartodb_services/cartodb_services/metrics/config.py
 2017-03-14 19:00:53 +01:00
Javier Goizueta
6c5ca97468 Complete rate-limiting for Mapzen & Here gecoding 
ServiceManager class has been introduced to handle service configuration at
SQL level (with a LegacyServiceManager alternative for non-refactored services).
These new classes take the responsibility of rate limits and quota checking.

Tests have been added for ServiceManager and rate limits, but currently they
check only the limits configuration since Lua support would be needed
to use rratelimit with MockRedis.
 2017-03-14 18:51:18 +01:00
Rafa de la Torre
a64557b50e New version 0.14.1 of python library 2017-03-13 15:10:03 +01:00
Rafa de la Torre
4e14e9b0c0 Fix links and improve readability 2017-03-13 13:12:06 +01:00
Rafa de la Torre
26d025a5d1 Remove tests that no longer apply #206 2017-03-10 13:15:56 +01:00
Rafa de la Torre
80e7ed90a8 Make get_metrics read just zero-padded dates #206 2017-03-10 13:14:36 +01:00
Javier Goizueta
250d384d06 Geocoder, per user, cofigurable rate limits 
WIP, specially the GeocoderConfig part is flaky and ugly
 2017-03-09 18:42:48 +01:00
Rafa de la Torre
fc1e05658e Bump version of python library to 0.14.0 2017-03-08 13:13:19 +01:00
Rafa de la Torre
b8ec3dc306 Merge pull request #342 from CartoDB/remove-mapzen-forcing-layers-equal-address 
Remove the search_type if its address from the params sent to Mapzen
 2017-03-08 13:07:07 +01:00
Rafa de la Torre
ffdb6524da Fix bug: search_type can be None 
Found via unit test:

```sh
$ nosetests -x test/
....................................................................................E
======================================================================
ERROR: test_geocode_address_with_valid_params (test.test_mapzengeocoder.MapzenGeocoderTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/requests_mock/mocker.py", line 177, in inner
    return func(*args, **kwargs)
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/test/test_mapzengeocoder.py", line 100, in test_geocode_address_with_valid_params
    country='ESP')
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 28, in wrapped_function
    return QPSService(retry_timeout=timeout, queries_per_second=qps).call(original_function, *args, **kwargs)
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 53, in call
    raise e
AttributeError: 'NoneType' object has no attribute 'lower'

----------------------------------------------------------------------
Ran 85 tests in 0.140s

FAILED (errors=1)
```
 2017-03-08 10:19:32 +01:00
Rafa de la Torre
82a7477d37 Beautify code 2017-03-08 10:14:37 +01:00
Rafa de la Torre
25ba9866ae Small PoC with rratelimit #346 2017-03-07 13:31:06 +01:00
Javier Goizueta
6dca3f4e58 Change default arguent value for clarity 
It is valie to pass None in service_params arguments, so make it the default.
 2017-03-06 19:01:05 +01:00
Javier Goizueta
d01170471e Missing @property 2017-03-06 18:58:00 +01:00
Javier Goizueta
89762c1a7f Use service params configuration in Mapzen services 2017-03-06 15:25:11 +01:00
Javier Goizueta
1081e81047 Add Mapzen service params to configuration 2017-03-06 14:34:23 +01:00
Javier Goizueta
29cdd8099a Add service configuration (URL, etc) for Mapzen Geocoding 2017-03-06 10:20:20 +01:00
Javier Goizueta
c0374f5800 Add service configuration (URL, etc) for Here Routing 2017-03-02 17:55:08 +01:00
Javier Goizueta
677d2acead Add service configuration (URL, etc) for Here Geocoding 
See #343
 2017-03-02 17:15:14 +01:00
nygeog
a9e942a48b remove the search_type if its address from the params sent to mapzen 
remove the search_type if its address from the params sent to mapzen
 2017-02-02 10:23:09 -05:00
Mario de Frutos
c364c7f5ed Merge pull request #339 from CartoDB/obs-getdata-getmeta-wrappers 
Obs_GetData OBS_GetMeta wrappers
 2017-02-01 10:05:52 +01:00
Mario de Frutos
c550a5d4e9 Merge pull request #321 from CartoDB/fix-default-isolines-provider 
Change default isolines provider
 2017-02-01 09:41:21 +01:00
John Krauss
8204bb3451 fix error-causing raise (which obscures real error) 2017-01-26 12:27:29 +01:00
Rafa de la Torre
a0dd7c7497 Fix tests: zero-pad months in redis key #335 2017-01-04 11:11:27 +01:00
Rafa de la Torre
9700ae966c Remove uneeded math for simplicity's sake 2016-12-28 14:44:48 +01:00
Javier Goizueta
bdf9627586 Fix typo 
I needed to fix this; I felt some kitten was being killed somewhere
because of this. And it made baby Jesus cry.
 2016-12-27 17:11:39 +01:00
Mario de Frutos
80b23c62c3 QPS timeout was badly calculated 
timedelta microseconds is just the microseconds part of the timedelta
object not the elapsed time in microseconds.

I've change to use the total_seconds method to get all the elapsed time
in seconds and transform to miliseconds.
 2016-12-23 11:40:30 +01:00
Mario de Frutos
23a2de0321 Add mapzen isolines to the quota checker 2016-12-22 17:38:10 +01:00
Javier Goizueta
9ee1d045c8 Fix tests 2016-12-21 09:54:08 +01:00
Rafa de la Torre
360b5bd57f Merge remote-tracking branch 'origin/development' into mapzen_services_use_quotas 
Solve merge conflict in setup.py (version of python lib).
 2016-12-20 18:14:01 +01:00
Javier Goizueta
32010669e8 New version of the python lib 
To fix bug with period end date; see #322
 2016-12-20 16:39:19 +01:00
Mario de Frutos
38754fec26 Cover refactor mapzen geocoder config with tests 2016-12-20 16:17:42 +01:00
Mario de Frutos
356135672b Sanitize quota get in the refactor part 2016-12-20 16:17:42 +01:00
Mario de Frutos
4d0abf9026 Bump python library version 2016-12-20 16:17:42 +01:00
Mario de Frutos
0672f2752b Improve tests and add more unit tests for the quota functionality 2016-12-20 16:17:41 +01:00
Javier Goizueta
b5514aea60 Avoid invalid end of period dates 
Fixes #322
 2016-12-20 12:48:36 +01:00
Javier Goizueta
53acd4d30e Add tests that reveal end-of-period date problem 
See #322
 2016-12-20 12:41:54 +01:00
Mario de Frutos
6c71d73498 Use the current quota for mapzen services 2016-12-20 11:52:06 +01:00
Rafa de la Torre
a00fca6d13 Update README.md 
Add a bit about versioning stuff
 2016-12-16 18:00:22 +01:00
Rafa de la Torre
29caaf9297 Update README.md 
Remove misleading paragraph about `requirements.txt` and `setup.py` dependencies. Refer to https://packaging.python.org/requirements/ for an authoritative discussion.
 2016-12-16 18:00:22 +01:00
Javier Goizueta
4fc90626ab New version of the python lib 
New release to change number of http connection retries to 1
 2016-12-13 18:03:37 +01:00
Javier Goizueta
1f53af65b9 Set number of http connection retries to 1 
Change the maximum number of retries for connection to external services to 1.
 2016-12-13 11:32:22 +01:00
Javier Goizueta
96a147a242 The default isolines provider should be Mapzen 2016-12-13 10:19:27 +01:00
Javier Goizueta
79fb796180 Fix bug in retries usage; mantain behaviour 
The number of retries was being passed as the pool size parameter.
This is corrected but number of retries changed to 1 to maintain previous behaviour.
 2016-12-13 10:07:10 +01:00
Javier Goizueta
2953fda75c Revert num of isochrone retries to 3 
See #317
 2016-12-12 12:14:10 +01:00
Rafa de la Torre
98fa248fff New version of the python lib #317 2016-12-07 16:39:28 +01:00
Rafa de la Torre
7c348dee0f Mapzen isochrones: tweak retry params #317 2016-12-07 16:38:47 +01:00
Rafa de la Torre
bdeaadf33f Fix QPS: Compare everything in microseconds #317 2016-12-07 16:37:25 +01:00
Mario de Frutos
51face5593 Make the HTTP requests retry on timeout 2016-12-02 17:01:05 +01:00
Mario de Frutos
77b2ac9155 Bump python version 2016-11-29 16:34:59 +01:00
Mario de Frutos
8edbf33ff1 Added denoise and generalize options to make the ischrones less heavy 2016-11-29 16:10:59 +01:00
Mario de Frutos
bd439d8985 Car is auto in mapzen isochrones 2016-11-29 13:18:52 +01:00
Mario de Frutos
77f4f3e7ff Mapzen isochtrones integration 2016-11-29 12:57:23 +01:00
Rafa de la Torre
2b2691d0b9 New minor version of the python library #302 2016-11-25 10:43:06 +01:00
Rafa de la Torre
4c75c5f261 Revert internal geocoder related stuff #302 
This reverts commits 579d11e, 4af5975 and a2da597
 2016-11-24 17:33:06 +01:00
Rafa de la Torre
efb10e43e4 Merge remote-tracking branch 'origin/development' into 302-expose-quotas 2016-11-24 12:36:23 +01:00
Rafa de la Torre
1dc6060570 Fix: deal with string values for soft limits #302 2016-11-23 18:51:13 +01:00
Rafa de la Torre
579d11ebd5 Add params to InternalGeocoderConfig #302 
Add a bunch of params to InternalGeocoderConfig for consistency with
other services:
- monthly_quota
- period_end_date
- soft_limit
 2016-11-23 17:04:50 +01:00
Rafa de la Torre
a1eae3934c Add OBS to UserMetricsService #302 2016-11-23 16:47:35 +01:00
Mario de Frutos
d1b17da70b Don't make mandatory to have a metrics file 2016-11-11 11:48:08 +01:00
Rafa de la Torre
3fa686ec65 Remove the class Service in favor of service_type ENUM #302 2016-11-10 19:21:29 +01:00
Rafa de la Torre
7a56dc698f List of services bound to quota #302 2016-11-10 16:59:58 +01:00
Rafa de la Torre
b00edf218a Fix: quotas should be integers #302 2016-11-08 18:46:58 +01:00
Rafa de la Torre
2c76fa4501 Fix: quotas should be integers #302 2016-11-08 18:28:28 +01:00
Rafa de la Torre
4169bc0f88 Fix doc: how to run python unit tests #302 2016-11-08 17:52:18 +01:00
Mario de Frutos
6177f0ac76 Remove info trace not needed in metrics 2016-11-07 11:02:09 +01:00
Mario de Frutos
88ed777785 Added plpy_mock and check multiple instances for metricsdatagatherer 2016-11-04 13:39:11 +01:00
Mario de Frutos
73aa26590b Make the data gather singleton support multiple instances 
Due to we use pgbouncer, we could share the same instance of the metrics
gatherer singleton in multiples requests. Added support to have
multiple instances identified by transaction id
 2016-11-04 13:39:11 +01:00
Mario de Frutos
c571edb21e Split response and function data, added uuid and type of element 2016-11-04 13:39:11 +01:00
Mario de Frutos
e878031744 Added url to the response data 2016-11-04 13:39:11 +01:00
Mario de Frutos
49fd51f6da Added logger to metrics context manager 2016-11-04 13:39:10 +01:00
Mario de Frutos
cf3c6f2ce5 New metrics logger system 
- It's not mandatory to have the log_path defined for all the services
- Now we add metrics to a MetricsGatherer using the add method in order
  to gather data from all the function scope, not just in one defined
  point
- There is a `metrics` context manager to wrap the function block. This
  context manager get the execution time for the block among other
  things
 2016-11-02 15:26:39 +01:00
Mario de Frutos
6058960ec5 Added timeout for all the third-party connections 2016-10-27 16:17:37 +02:00
Mario de Frutos
336d8be977 Hotfix: Typo in status code check 2016-10-26 17:49:07 +02:00
Mario de Frutos
80963e2589 504 errors return empty data instead of raise exception 
Due to some problems in Mapzen, we're receiving 504 errors from their
servers. To mitigate this problem, instead of raise an exception, we're
going to return empty data for that point
 2016-10-26 16:39:35 +02:00
Mario de Frutos
19d6cacdb3 Fix empty response from matrix 
https://github.com/CartoDB/dataservices-api/issues/293
 2016-10-26 16:00:50 +02:00
Rafa de la Torre
0d22942a72 Update version of python lib server package to 0.9.2 2016-10-21 17:38:32 +02:00
Rafa de la Torre
d4ac2eb5e6 Tests for ServicesRedisConfig (mapzen quota) #281 2016-10-21 15:40:22 +02:00
Rafa de la Torre
db80d389e0 Do not require org geocoder and isolines quota to be present 2016-10-21 15:38:58 +02:00
Rafa de la Torre
8e02c64aeb Tests for UserMetricsService, mapzen routing #281 2016-10-21 15:21:03 +02:00
Rafa de la Torre
cc2ab1bc0c Fix test: make it independent of current date #281 2016-10-21 15:08:17 +02:00
Rafa de la Torre
e1a7d1751c Fix corner case: used_quota == monthly_quota #281 2016-10-20 15:36:48 +02:00
Rafa de la Torre
0c49107f96 Some tests for the QuotaChecker #281 2016-10-20 15:31:21 +02:00
Rafa de la Torre
247034c21e Some tests for RoutingConfig #281 2016-10-19 18:40:17 +02:00
Rafa de la Torre
2b1b1c981f Add routing to UserMetricsService.used_quota 2016-10-19 17:41:43 +02:00
Rafa de la Torre
aaff5564ec Use soft_limit for __check_routing_quota #281 2016-10-18 17:44:56 +02:00
Rafa de la Torre
72998c324a Simplify the code for regularity's sake #281 2016-10-18 17:40:50 +02:00
Rafa de la Torre
bbd9b6b98e Add soft limit to config object #281 2016-10-18 17:35:36 +02:00
Rafa de la Torre
27be704bd6 Changes needed to read quota from redis #281 
Use per-user/org quota from redis, if present. Otherwise default to the
quota stored in server database (as it did to date).
 2016-10-18 17:26:45 +02:00
Rafa de la Torre
98d533b707 Bump version of python pip package to 0.9.1 2016-10-06 15:05:00 +02:00
Rafa de la Torre
6d0ad85d48 Fix for crash when sentinel_master_id is absent in config 2016-10-06 15:04:25 +02:00
Rafa de la Torre
00e6cace76 Update version of the python pip package 2016-10-06 11:06:28 +02:00
Rafa de la Torre
e9ad35ba1d Merge pull request #284 from CartoDB/redis-refactor-take2 
Redis refactor: Take 2
 2016-10-05 18:32:08 +02:00
Rafa de la Torre
cded6c2f08 Test for NullConfigStorage 2016-10-04 15:20:41 +02:00
Rafa de la Torre
e1b357137a Rename s/storage/backend for consistency 2016-10-04 14:59:57 +02:00
Rafa de la Torre
3844cfc226 Tests for environment module 2016-10-04 14:56:06 +02:00
Rafa de la Torre
2a1276f4f1 Add test for redis_config module 2016-10-04 12:53:52 +02:00
Rafa de la Torre
35da7e48fd Implement a ServerConfigBackendFactory 
mostly to keep layers separated.
 2016-10-03 17:42:19 +02:00
Rafa de la Torre
12aebb7eee Rename ConfigStorageInterface to ConfigBackendInterface 2016-10-03 17:35:33 +02:00
Rafa de la Torre
0d87a95270 Inject the environment into the LoggerConfigBuilder 2016-10-03 17:06:57 +02:00
Rafa de la Torre
18e1a5c7c9 Remove the class User and its tests 
as it is not used.
 2016-10-03 16:54:40 +02:00
Rafa de la Torre
fcca5da302 Add a RedisMetricsConnectionFactory class 2016-10-03 16:47:45 +02:00
Rafa de la Torre
1aec541906 Add ServerEnvironment and ServerEnvironmentBuilder 2016-10-03 16:21:28 +02:00
Rafa de la Torre
9e98e0794d Add backends for user and org configs 2016-10-03 16:05:08 +02:00
Rafa de la Torre
8fbb41742c Rename redis conf factories to builders 2016-10-03 15:25:01 +02:00
Rafa de la Torre
275a6dc27f Move redis_config related classes to separate file 2016-10-03 13:33:46 +02:00
Rafa de la Torre
d522083d5c Rename redis_connection_config test file as well 2016-10-03 13:22:47 +02:00
Rafa de la Torre
073163eb1a Rename redis_config.py to redis_connection_config.py 2016-10-03 13:20:30 +02:00
Rafa de la Torre
0c62c4bada Move the NullConfigStorage to a separate file 2016-10-03 13:17:04 +02:00
Rafa de la Torre
3361960cfc Move InMemoryConfigStorage to a separate file 2016-10-03 13:12:49 +02:00
Rafa de la Torre
86ab3abc53 Rename mapzen_geocoder module to mapzen_geocoder_config 2016-10-03 12:58:15 +02:00
Rafa de la Torre
fb812ee15e Add tests for redis connection configs 2016-10-03 12:33:38 +02:00
Rafa de la Torre
c1dd410201 Make RedisConnectionConfigBuilder abstract 
and also use custom ConfigException instead of generic one.
 2016-10-03 12:21:45 +02:00
Rafa de la Torre
34ddd28e6b Move ConfigException to a separate file 2016-10-03 12:20:17 +02:00
Carla
443fe88d5a Add newline to end fle 2016-09-30 13:08:27 +02:00
Carla Iriberri
74d2fba763 Add Redis Mock -- tested and working 2016-09-30 12:28:27 +02:00
Rafa de la Torre
1e6ee8d5c1 Add an Environment class 2016-09-30 11:21:35 +02:00
Rafa de la Torre
3a6cc4c364 Add mapzen config and integrate into legacy code 2016-09-29 18:41:16 +02:00
Rafa de la Torre
0b7b44d8a5 Add a couple of factories to abstract user/org configs 2016-09-29 17:46:17 +02:00
Rafa de la Torre
4b4a02905c Implement the RedisConfigStorage 2016-09-29 17:18:45 +02:00
Rafa de la Torre
1f3a655ae5 Add all the code to instantiate a user_config_storage (WIP) 2016-09-29 16:57:31 +02:00
Rafa de la Torre
9d60fde0b8 A stubb for RedisConfigStorage (WIP) 2016-09-29 13:07:40 +02:00
Rafa de la Torre
efdc151282 Add things to get logger config from storage 2016-09-28 18:22:51 +02:00
Rafa de la Torre
fd2cc21942 Add server config storage classes 2016-09-28 17:32:26 +02:00
Rafa de la Torre
18f05fbd4f User class with tests 2016-09-28 16:30:37 +02:00
Carla Iriberri
96a93e3c56 Python package version 0.8.1 2016-09-28 15:09:29 +02:00
Rafa de la Torre
64fc18b9e0 Fix and improve test speed 2016-09-28 13:43:50 +02:00
Mario de Frutos
9381d5644b Fixed QPS retry decorator 2016-09-16 13:46:02 +02:00
Carla
9f55f2ee3b Update README.md 2016-09-09 11:44:57 +02:00
Mario de Frutos
18ae2525b6 Add search type filter for Mapzen geocoder 2016-08-29 14:32:49 +02:00
Mario de Frutos
71d5ce951a Use mapzen as first option for the namedplace geocoding 2016-08-26 16:39:25 +02:00
Mario de Frutos
d602c42559 Strip spaces in the mapzen geocode parameters 2016-08-19 13:32:11 +02:00
Mario de Frutos
3cc4e06420 Add more logs for mapzen services 2016-08-19 11:44:59 +02:00
Mario de Frutos
0fb3cdd8b1 JSON response is not always present, better show the reponse object complete 2016-08-05 11:35:16 +02:00
Mario de Frutos
7cca8e95c3 Remove testing exception 2016-08-03 14:45:57 +02:00
Mario de Frutos
a33999ed04 Dont re-raise the exception to avoid leak sensitive information 2016-08-03 12:08:23 +02:00
Mario de Frutos
9186eb5d5b Log part decoupled from the main config class 2016-08-02 18:40:01 +02:00
Mario de Frutos
ed5cf25e9c Added new logger to all the functions 2016-08-02 16:44:03 +02:00
Mario de Frutos
5d5e8d6f9a Added logger with rollbar integration 2016-08-01 18:03:42 +02:00
Mario de Frutos
993059eafb Added server config with environment data 2016-08-01 14:00:32 +02:00
Mario de Frutos
f9f39dcf9c Refactor metrics logger 2016-08-01 13:17:06 +02:00
Mario de Frutos
4d47b905e8 Revert include the gateway error in QPS 2016-08-01 11:23:06 +02:00
Mario de Frutos
c23a57c03f Add QPS retry for mapzen isolines and add the 504 error to retry 2016-07-29 16:22:34 +02:00
Mario de Frutos
efde2ee1b7 Fix typo in the default value for service providers 2016-07-25 14:00:29 +02:00
Mario de Frutos
fc399c12c1 Change notices for debug 2016-07-25 13:58:01 +02:00
Mario de Frutos
f03897257e Better than lower, change to empty string if not provider setting in redis 2016-07-25 13:40:06 +02:00
Mario de Frutos
90998d4819 Fix users without provider setting setted 2016-07-25 13:35:59 +02:00
Mario de Frutos
79d0b5ba7c Add service providers that come from the user configuration 2016-07-25 12:25:52 +02:00
Mario de Frutos
ff8eed9750 Fix empty component return in isolines 2016-07-22 14:54:53 +02:00
Carla
6fb9f67e64 Update README.md 2016-07-11 10:50:19 +02:00
Carla Iriberri
aff5dd13d7 Bump python library to 0.7.0 2016-07-07 10:04:54 +02:00
Carla Iriberri
1022097300 Bump server to 0.12.0 2016-07-07 09:58:51 +02:00
Carla
5c20866277 Merge pull request #216 from CartoDB/mapzen-isolines-reloaded 
Add mapzen isolines
 2016-07-07 09:31:04 +02:00
Rafa de la Torre
f5d51da673 Fix another typo (hello Carla!!) 
This feature is dedicated to you. Keep rocking.
 2016-07-06 20:29:08 +02:00
Rafa de la Torre
bcc6bc35d3 Fix None*unit_factor error 
Also make the code more explicit about what happens when getting cost ==
None.
 2016-07-06 19:48:23 +02:00
Rafa de la Torre
230112b7e5 Add calculate_isodistance function 2016-07-06 19:20:21 +02:00
Rafa de la Torre
523eda2cc7 Generalize calculate_isochrone to calculate_isoline 2016-07-06 18:43:09 +02:00
Rafa de la Torre
54221fa671 Add transport mode car 2016-07-06 18:05:51 +02:00
Rafa de la Torre
6d888a7a62 Fix for points getting None cost 
Sometimes there's no route information for the point in a particular
angle we're interested in. In this case it is better to use more
points/angles and discard the ones we're not interested in.
 2016-07-06 18:03:24 +02:00
Rafa de la Torre
075edf0e0d More precission for earth's radius 2016-07-06 18:01:33 +02:00
Rafa de la Torre
6c4829df01 Small refactor for sanity 2016-07-06 16:05:51 +02:00
Carla Iriberri
7ddb3da60d Remove useless cost_per_hit line 2016-07-06 15:43:26 +02:00
Carla Iriberri
ff4eb5b348 Mock mapzen matrix config 2016-07-06 14:21:32 +02:00
Rafa de la Torre
2147d190a1 Unit test for the mapzen isolines 2016-07-06 13:19:28 +02:00
Carla Iriberri
a046d3ce97 Add Mapzen Matrix to config and metrics services 2016-07-06 13:19:28 +02:00
Rafa de la Torre
cdcac2dc1f Fix typo in test case 2016-07-06 13:19:28 +02:00
Carla
e9346faf42 Fix bug on exception raise 2016-07-06 12:33:59 +02:00
Rafa de la Torre
b78bd05754 Be resilient to None cost estimation 2016-07-05 20:52:41 +02:00
Rafa de la Torre
77cdc3d8ff Only refine individual solutions when error > TOLERANCE 2016-07-05 18:48:21 +02:00
Rafa de la Torre
2d95601c5a Fix: max_abs_error should be a scalar 2016-07-05 18:48:21 +02:00
Rafa de la Torre
9a9f35d9c2 Fix silly typos spotted by jgoizueta (WIP) 2016-07-05 18:48:21 +02:00
Rafa de la Torre
9becf1adb4 Iterative part of the algorithm (WIP) 2016-07-05 18:48:21 +02:00
Rafa de la Torre
87413255af Major rewrite of MapzenIsolines (WIP) 2016-07-05 18:48:21 +02:00
Rafa de la Torre
46971fe96f Raise error when response not OK 2016-07-05 18:48:21 +02:00
Rafa de la Torre
96199b0d6d Add example to code doc 2016-07-05 18:48:21 +02:00
Rafa de la Torre
a70560e566 Minimal Mapzen Time-Distance Matrix client 2016-07-05 18:48:21 +02:00
Rafa de la Torre
53fe4ce21d An attempt to adapt paremetrs (WIP) 2016-07-05 18:48:21 +02:00
Mario de Frutos
893b8db374 First stage is calculating the matrix of points 2016-07-05 18:48:21 +02:00
Carla Iriberri
6d35cff9c7 Exposes geocoder providers in public geocoder functions 
Add config function in postgres explicitly to get MapzenGeocoderConfig.
Bump versions for client and server APIs. New MapzenGeocoderConfig
included to be able to use current QuotaServices with non-configured
users.
 2016-07-05 11:13:43 +02:00
Rafa de la Torre
9ed059f4cc Bump server python lib version to 0.6.4 #204 2016-06-14 18:42:29 +02:00
Rafa de la Torre
972970a12d Use DAY_OF_MONTH_ZERO_PADDED elsewhere #204 2016-06-14 15:25:38 +02:00
Rafa de la Torre
37bcaeada3 Fix test_orgs_should_write_zero_padded_dates (WIP) #204 2016-06-14 13:18:42 +02:00
Rafa de la Torre
25bf9b6372 Add test_orgs_should_write_zero_padded_dates #204 2016-06-14 13:18:34 +02:00
Rafa de la Torre
f76a5cdfcc Fix test_should_write_zero_padded_dates #204 2016-06-14 10:13:14 +02:00
Rafa de la Torre
e046ca2c4d Add test_should_write_zero_padded_dates #204 2016-06-14 10:04:17 +02:00
Rafa de la Torre
c9f57259be Instructions on how to run integration tests #204 2016-06-13 19:05:31 +02:00
Rafa de la Torre
41b9db4144 Update python lib to 0.6.3 #201 2016-06-13 17:21:15 +02:00
Rafa de la Torre
25d42a75e4 test_should_not_request_redis_twice_when_unneeded #201 2016-06-13 15:45:56 +02:00
Rafa de la Torre
f9e9617d6f test_should_sum_amounts_from_both_key_formats #201 2016-06-13 13:10:07 +02:00
Rafa de la Torre
b0bb60736c Implement test and fix for that case #201 
Add test_should_account_for_zero_paddded_keys plus its corresponding fix
for it.
 2016-06-13 13:08:35 +02:00
Rafa de la Torre
a2863d7135 Allow for the manipulation of datetime.date.today() #201 2016-06-13 12:57:00 +02:00
Rafa de la Torre
a114f857fd Add the first test that makes sense #201 2016-06-13 12:21:06 +02:00
Rafa de la Torre
47962c9ccb Stub tests to be implemented (WIP) #201 2016-06-13 11:33:14 +02:00
Rafa de la Torre
901f5d7b8b Add a README.md file for dummies like me #201 2016-06-13 11:19:14 +02:00
Rafa de la Torre
c7bcbddaa9 Delete copy/pasted comment from requirements.txt #201 2016-06-13 11:16:49 +02:00
Rafa de la Torre
9ad55a4d53 Update url of pip package 2016-06-10 13:03:09 +02:00
Carla Iriberri
21aac960a6 Add function cdb_routing_with_waypoints 
Add routing with waypoints functions to client and server. Includes
signature checks tests for the Postgresql functions and unit and
integration tests for the Python library.

Add client v0.6.0 and server v0.9.0
 2016-05-25 11:38:37 +02:00
Carla
f46305de6f Update test_helper.py 2016-05-24 13:18:43 +02:00
Mario de Frutos
bb17b3bff7 Hotifix: obs_general_quota logic missing in the redis config 2016-05-16 14:19:08 +02:00
Mario de Frutos
c7c9e6e284 Server side new data observatory functions (v0.8.0) 2016-05-16 10:25:26 +02:00
Mario de Frutos
3e09605757 On 4xx we have to return an empty response instead of an exception 2016-05-10 16:46:38 +02:00
Mario de Frutos
e81dadaf2e Green/Blue deploy system for the observatory functions 2016-05-03 13:49:14 +02:00
Mario de Frutos
a9c420ba7b Use redis or db data observatory quota value 2016-04-25 09:33:18 +02:00
Mario de Frutos
3c5325c1d7 Server data observatory functions 2016-04-20 15:58:38 +02:00
Mario de Frutos
1e30b21d35 Fixed tests with the isolines changes 2016-04-18 18:37:46 +02:00
Mario de Frutos
15a7ed5a09 Fixed isolines overwrite for organizations 2016-04-18 11:49:11 +02:00
Mario de Frutos
8cd94973ff Version 0.6.2 2016-04-15 14:43:45 +02:00
Mario de Frutos
c9ddc37051 Mapzen geocoder and routing quota check 2016-04-15 10:21:40 +02:00
Mario de Frutos
b4c1991c90 Mapzen geocoder checks for the iso3 for the country passed 2016-04-13 18:49:13 +02:00
Mario de Frutos
a8dd211acf Use retry on timeout flag for Redis connections 
In order to avoid some fails due some high load peaks in our Redis
databases, we are going to active this flag that retries 1 more time
in case it receives a socket.Timeout error while reading or connecting
 2016-04-04 10:17:59 +02:00
Carla Iriberri
752d8317a2 Merge master 2016-04-01 10:18:46 +02:00
Carla
8c5a4d32de Update setup.py 2016-04-01 10:15:27 +02:00
Mario de Frutos
d625b26e57 Fixed timeout and random problems 2016-04-01 08:45:40 +02:00
Carla
62da0eff21 Update test_helper.py 2016-03-31 16:45:27 +02:00
Carla Iriberri
1505e0e6ea Make isoline credentials independent 2016-03-31 15:52:52 +02:00
Mario de Frutos
89ec3dcfa6 Mapzen geocoder integrated 2016-03-23 13:37:30 +01:00
Mario de Frutos
357458f44d Check for some issues like empty input, recoverable exceptions, etc 2016-03-22 15:19:03 +01:00
Mario de Frutos
4061009df0 4xx generated empty responses 2016-03-17 22:54:19 +01:00
Mario de Frutos
b9f18dda69 Added exception info for config retrieving 2016-03-17 11:36:52 +01:00
Mario de Frutos
7acf70131b Change db and timeout attributes too 2016-03-17 11:01:56 +01:00
Mario de Frutos
0e44910165 Change to sentinel_id 2016-03-17 10:58:49 +01:00
Mario de Frutos
1c33a72fdc Change sentinel attribute to use the config one 2016-03-17 10:56:25 +01:00
Mario de Frutos
5d57624fd5 Merge pull request #107 from CartoDB/rename_and_metrics 
Rename and metrics
 2016-03-17 10:23:14 +01:00
Mario de Frutos
f0a49d5fbc Deleted the used_credits field 2016-03-17 10:15:12 +01:00
Mario de Frutos
60f17fdb09 QPS manager for Mapzen 2016-03-14 11:21:26 +01:00
Mario de Frutos
bd2e380225 Add logger for the internal geocoder too 2016-03-08 16:47:49 +01:00
Mario de Frutos
c293b55c3c Added log and cost_per_hit config to the db config 2016-03-08 16:47:49 +01:00
Mario de Frutos
b97e838416 Refactor to move logic from SQL functions to Python 
- Moved the logic the retrieve the redis connection params to
  RedisDBConfig class
- Moved the logic that retrieve the services configuration to
  ServicesDBConfig
 2016-03-08 16:47:49 +01:00
Mario de Frutos
ac1627b5c5 Logger for geocoder metrics 2016-03-08 16:47:48 +01:00
Mario de Frutos
7237e01601 Change metrics increment naming to reflect service instead of geocoder 2016-03-07 16:03:28 +01:00
Mario de Frutos
df08b27068 Removed dependency with polyline and use our implementation 2016-03-02 10:49:02 +01:00
Mario de Frutos
4d04882c18 Polyline decoder 2016-03-02 10:43:27 +01:00
Mario de Frutos
d2e73a69fa Python library supports Mapzen routing 2016-02-26 10:49:10 +01:00
Luis Bosque
6c5b1b4e99 Use strictredis if sentinel_master_id is not null 
Also, use only a redis_host and redis_port attributes either if the
connection is to redis or to sentinel
 2016-02-26 10:49:09 +01:00
Luis Bosque
93463b3032 Support using a common redis connection 
If sentinel_host or sentinel_cluster_id is not provided it will try to
connect with a redis_host parameter
 2016-02-26 10:49:08 +01:00
Mario de Frutos
d27b2a2b78 Block google services users to use isolines functions 
Due some service restrictions, google services users couldn't use
the HERE services which includes the isolines functions.
 2016-02-26 10:42:05 +01:00
Mario de Frutos
e764b9036d Quota check for the routing feature 
We have refactor all the quota logic and extracted to a new QuotaChecker
class in order to have it ready to create a factory when needed

Added the logic for the routing quota check
 2016-02-18 15:40:55 +01:00
Mario de Frutos
3a4130282e Limit the mode types accepted 2016-02-12 10:01:42 +01:00
Mario de Frutos
f36a345db2 Updated python library version with routing part 2016-02-10 22:26:53 +01:00
Carla Iriberri
d4a2184ce0 Multiple fixes 2016-02-10 19:59:00 +01:00
Mario de Frutos
54622d5c81 Added the routing config and metrics 2016-02-10 17:01:41 +01:00
Mario de Frutos
75a915019f Helper to transform to Postgis multipolygon 2016-02-10 16:01:03 +01:00
Mario de Frutos
13cfee2d4c Fixed tests for the heremaps geocoder 2016-02-10 15:24:25 +01:00
Mario de Frutos
3b0f92d5e7 Routing isolines python library 2016-02-10 15:11:45 +01:00
Mario de Frutos
17b8b5942e Refactor to make the service config extensible to other services 2016-02-05 10:18:21 +01:00
Mario de Frutos
3960c13484 Refactor python library to unify and rename as cdb_services 2016-02-03 10:21:57 +01:00
Mario de Frutos
e7c58b9a51 Google geocoder working 2016-02-03 10:21:34 +01:00
Mario de Frutos
e73736cbec Restrict the python-dateutil library to avoid conflict with other projects that need that specific version 2016-01-26 19:57:18 +01:00
Mario de Frutos
4e53397f1c Change heremaps module to support host as parameter 2016-01-25 17:36:18 +01:00
Mario de Frutos
fffcef90a5 Check for the mandatories configurations in Redis before proceed 2016-01-25 08:17:03 +01:00
Mario de Frutos
e46600325b Use the real heremaps url 2016-01-22 16:39:55 +01:00
Mario de Frutos
984519e7f1 Change to have functions for all the geocoder types but only one function to call them 2016-01-22 10:05:46 +01:00
Mario de Frutos
c74947b0a5 Cartodb geocoder python module tests fixed 2016-01-22 10:05:40 +01:00
Mario de Frutos
aced68096b Heremaps tests fixed 2016-01-21 18:07:07 +01:00
Mario de Frutos
8a51558068 We need to have one date at least to check the quota 2016-01-21 18:06:04 +01:00
Mario de Frutos
3488d3f990 Pass the amount always 2016-01-21 18:05:34 +01:00
Mario de Frutos
111804dafc Refactor of the google geocoder keys in the config object 2016-01-21 18:05:08 +01:00
Mario de Frutos
af051e2ed5 Set the soft geocoding limit always 2016-01-21 18:04:08 +01:00
Mario de Frutos
4d0cabb429 Add username and orgname to the config object instead of pass them to every object 2016-01-21 18:03:23 +01:00
Mario de Frutos
7c3ab87b78 When geocoder API calls heremaps increments the usage metrics 2016-01-21 13:45:29 +01:00
Mario de Frutos
ad9c16b4df Nokia geocoder config moved to the config helper 2016-01-21 08:40:25 +01:00
Mario de Frutos
fc35911b91 Geocoder street function with quota checking 2016-01-20 20:57:24 +01:00
Mario de Frutos
b2fa8739e2 User config now is retrieved from redis 2016-01-20 20:56:19 +01:00
Mario de Frutos
d2a096907e Removed unused examples for python geocoder library 2015-11-23 09:50:10 +01:00
Mario de Frutos
9e30bf2223 New Redis structure for services 2015-11-23 09:40:10 +01:00
Mario de Frutos
928e33b489 Added config helper for user and geocoder config 2015-11-23 09:39:47 +01:00
Mario de Frutos
e8983283da Redis connection get from sentinel master 2015-11-23 09:39:47 +01:00
Mario de Frutos
b38a9b2fd3 Added redis config logic 2015-11-23 09:39:47 +01:00
Mario de Frutos
da12d6628d Extracted redis connection and quota check to be reusable 2015-11-23 09:39:47 +01:00