Commit Graph

623 Commits

Author SHA1 Message Date
Javier Goizueta
6c5ca97468 Complete rate-limiting for Mapzen & Here gecoding
ServiceManager class has been introduced to handle service configuration at
SQL level (with a LegacyServiceManager alternative for non-refactored services).
These new classes take the responsibility of rate limits and quota checking.

Tests have been added for ServiceManager and rate limits, but currently they
check only the limits configuration since Lua support would be needed
to use rratelimit with MockRedis.
2017-03-14 18:51:18 +01:00
Rafa de la Torre
a64557b50e New version 0.14.1 of python library 2017-03-13 15:10:03 +01:00
Rafa de la Torre
4e14e9b0c0 Fix links and improve readability 2017-03-13 13:12:06 +01:00
Rafa de la Torre
26d025a5d1 Remove tests that no longer apply #206 2017-03-10 13:15:56 +01:00
Rafa de la Torre
80e7ed90a8 Make get_metrics read just zero-padded dates #206 2017-03-10 13:14:36 +01:00
Javier Goizueta
250d384d06 Geocoder, per user, cofigurable rate limits
WIP, specially the GeocoderConfig part is flaky and ugly
2017-03-09 18:42:48 +01:00
Rafa de la Torre
c6c4255242 Server 0.22.0 version release artifacts 2017-03-08 15:22:58 +01:00
Rafa de la Torre
87a2e02721 Make more apparent what happens with make release
Instead of getting (deleted and untracked files):

  ```
  $ git status
  # On branch master
  # Your branch is ahead of 'origin/master' by 20 commits.
  #
  # Changes not staged for commit:
  #   (use "git add/rm <file>..." to update what will be committed)
  #   (use "git checkout -- <file>..." to discard changes in working directory)
  #
  #       deleted:    cdb_dataservices_server--0.20.0--0.21.0.sql
  #       deleted:    cdb_dataservices_server--0.21.0--0.20.0.sql
  #       deleted:    cdb_dataservices_server--0.21.0.sql
  #       modified:   cdb_dataservices_server.control
  #
  # Untracked files:
  #   (use "git add <file>..." to include in what will be committed)
  #
  #       cdb_dataservices_server--0.21.0--0.22.0.sql
  #       cdb_dataservices_server--0.22.0--0.21.0.sql
  #       cdb_dataservices_server--0.22.0.sql
  #       old_versions/cdb_dataservices_server--0.20.0--0.21.0.sql
  #       old_versions/cdb_dataservices_server--0.21.0--0.20.0.sql
  #       old_versions/cdb_dataservices_server--0.21.0.sql
  no changes added to commit (use "git add" and/or "git commit -a")
  ```

you'd get something like:

  ```
  $ git status
  # On branch master
  # Your branch is ahead of 'origin/master' by 20 commits.
  #
  # Changes to be committed:
  #   (use "git reset HEAD <file>..." to unstage)
  #
  #       renamed:    cdb_dataservices_server--0.20.0--0.21.0.sql -> old_versions/cdb_dataservices_server--0.20.0--0.21.0.sql
  #       renamed:    cdb_dataservices_server--0.21.0--0.20.0.sql -> old_versions/cdb_dataservices_server--0.21.0--0.20.0.sql
  #       renamed:    cdb_dataservices_server--0.21.0.sql -> old_versions/cdb_dataservices_server--0.21.0.sql
  #
  # Changes not staged for commit:
  #   (use "git add <file>..." to update what will be committed)
  #   (use "git checkout -- <file>..." to discard changes in working directory)
  #
  #       modified:   cdb_dataservices_server.control
  #
  # Untracked files:
  #   (use "git add <file>..." to include in what will be committed)
  #
  #       cdb_dataservices_server--0.21.0--0.22.0.sql
  #       cdb_dataservices_server--0.22.0--0.21.0.sql
  #       cdb_dataservices_server--0.22.0.sql
  ```

which is nicer IMHO.
2017-03-08 13:22:28 +01:00
Rafa de la Torre
fc1e05658e Bump version of python library to 0.14.0 2017-03-08 13:13:19 +01:00
Rafa de la Torre
b8ec3dc306 Merge pull request #342 from CartoDB/remove-mapzen-forcing-layers-equal-address
Remove the search_type if its address from the params sent to Mapzen
2017-03-08 13:07:07 +01:00
Rafa de la Torre
ffdb6524da Fix bug: search_type can be None
Found via unit test:

```sh
$ nosetests -x test/
....................................................................................E
======================================================================
ERROR: test_geocode_address_with_valid_params (test.test_mapzengeocoder.MapzenGeocoderTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/requests_mock/mocker.py", line 177, in inner
    return func(*args, **kwargs)
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/test/test_mapzengeocoder.py", line 100, in test_geocode_address_with_valid_params
    country='ESP')
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 28, in wrapped_function
    return QPSService(retry_timeout=timeout, queries_per_second=qps).call(original_function, *args, **kwargs)
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 53, in call
    raise e
AttributeError: 'NoneType' object has no attribute 'lower'

----------------------------------------------------------------------
Ran 85 tests in 0.140s

FAILED (errors=1)
```
2017-03-08 10:19:32 +01:00
Rafa de la Torre
82a7477d37 Beautify code 2017-03-08 10:14:37 +01:00
Rafa de la Torre
049c8e8657 Merge branch 'development' into remove-mapzen-forcing-layers-equal-address 2017-03-07 14:56:55 +01:00
Rafa de la Torre
25ba9866ae Small PoC with rratelimit #346 2017-03-07 13:31:06 +01:00
Javier Goizueta
6dca3f4e58 Change default arguent value for clarity
It is valie to pass None in service_params arguments, so make it the default.
2017-03-06 19:01:05 +01:00
Javier Goizueta
d01170471e Missing @property 2017-03-06 18:58:00 +01:00
Javier Goizueta
89762c1a7f Use service params configuration in Mapzen services 2017-03-06 15:25:11 +01:00
Javier Goizueta
1081e81047 Add Mapzen service params to configuration 2017-03-06 14:34:23 +01:00
Javier Goizueta
29cdd8099a Add service configuration (URL, etc) for Mapzen Geocoding 2017-03-06 10:20:20 +01:00
Javier Goizueta
c0374f5800 Add service configuration (URL, etc) for Here Routing 2017-03-02 17:55:08 +01:00
Javier Goizueta
677d2acead Add service configuration (URL, etc) for Here Geocoding
See #343
2017-03-02 17:15:14 +01:00
Mario de Frutos
435a84f294 OBS_GetData functions shall increase the DO quota usage 2017-02-06 17:47:15 +01:00
nygeog
a9e942a48b remove the search_type if its address from the params sent to mapzen
remove the search_type if its address from the params sent to mapzen
2017-02-02 10:23:09 -05:00
Mario de Frutos
061956f250 Remove quota usage for obs_meta functions 2017-02-02 10:59:21 +01:00
Mario de Frutos
3d94d1dfaf Server 0.21.0 version release artifacts 2017-02-01 10:19:32 +01:00
Mario de Frutos
c364c7f5ed Merge pull request #339 from CartoDB/obs-getdata-getmeta-wrappers
Obs_GetData OBS_GetMeta wrappers
2017-02-01 10:05:52 +01:00
Mario de Frutos
c550a5d4e9 Merge pull request #321 from CartoDB/fix-default-isolines-provider
Change default isolines provider
2017-02-01 09:41:21 +01:00
John Krauss
1f225e0978 fix extraneous newline in expectation 2017-01-26 12:27:29 +01:00
John Krauss
e81f005ce9 adding server tests, interface to generate client sql code 2017-01-26 12:27:29 +01:00
John Krauss
8204bb3451 fix error-causing raise (which obscures real error) 2017-01-26 12:27:29 +01:00
John Krauss
56de523498 add wrappers for getmeta, getdata(geoms) and getdata(ids) 2017-01-26 12:27:29 +01:00
Rafa de la Torre
a0dd7c7497 Fix tests: zero-pad months in redis key #335 2017-01-04 11:11:27 +01:00
Rafa de la Torre
9700ae966c Remove uneeded math for simplicity's sake 2016-12-28 14:44:48 +01:00
Javier Goizueta
bdf9627586 Fix typo
I needed to fix this; I felt some kitten was being killed somewhere
because of this. And it made baby Jesus cry.
2016-12-27 17:11:39 +01:00
Mario de Frutos
80b23c62c3 QPS timeout was badly calculated
timedelta microseconds is just the microseconds part of the timedelta
object not the elapsed time in microseconds.

I've change to use the total_seconds method to get all the elapsed time
in seconds and transform to miliseconds.
2016-12-23 11:40:30 +01:00
Mario de Frutos
23a2de0321 Add mapzen isolines to the quota checker 2016-12-22 17:38:10 +01:00
Javier Goizueta
9ee1d045c8 Fix tests 2016-12-21 09:54:08 +01:00
Rafa de la Torre
360b5bd57f Merge remote-tracking branch 'origin/development' into mapzen_services_use_quotas
Solve merge conflict in setup.py (version of python lib).
2016-12-20 18:14:01 +01:00
Javier Goizueta
32010669e8 New version of the python lib
To fix bug with period end date; see #322
2016-12-20 16:39:19 +01:00
Mario de Frutos
38754fec26 Cover refactor mapzen geocoder config with tests 2016-12-20 16:17:42 +01:00
Mario de Frutos
356135672b Sanitize quota get in the refactor part 2016-12-20 16:17:42 +01:00
Mario de Frutos
4d0abf9026 Bump python library version 2016-12-20 16:17:42 +01:00
Mario de Frutos
0672f2752b Improve tests and add more unit tests for the quota functionality 2016-12-20 16:17:41 +01:00
Javier Goizueta
b5514aea60 Avoid invalid end of period dates
Fixes #322
2016-12-20 12:48:36 +01:00
Javier Goizueta
53acd4d30e Add tests that reveal end-of-period date problem
See #322
2016-12-20 12:41:54 +01:00
Mario de Frutos
6c71d73498 Use the current quota for mapzen services 2016-12-20 11:52:06 +01:00
Rafa de la Torre
a00fca6d13 Update README.md
Add a bit about versioning stuff
2016-12-16 18:00:22 +01:00
Rafa de la Torre
29caaf9297 Update README.md
Remove misleading paragraph about `requirements.txt` and `setup.py` dependencies. Refer to https://packaging.python.org/requirements/ for an authoritative discussion.
2016-12-16 18:00:22 +01:00
Javier Goizueta
4fc90626ab New version of the python lib
New release to change number of http connection retries to 1
2016-12-13 18:03:37 +01:00
Javier Goizueta
1f53af65b9 Set number of http connection retries to 1
Change the maximum number of retries for connection to external services to 1.
2016-12-13 11:32:22 +01:00
Javier Goizueta
96a147a242 The default isolines provider should be Mapzen 2016-12-13 10:19:27 +01:00
Javier Goizueta
79fb796180 Fix bug in retries usage; mantain behaviour
The number of retries was being passed as the pool size parameter.
This is corrected but number of retries changed to 1 to maintain previous behaviour.
2016-12-13 10:07:10 +01:00
Javier Goizueta
2953fda75c Revert num of isochrone retries to 3
See #317
2016-12-12 12:14:10 +01:00
Rafa de la Torre
98fa248fff New version of the python lib #317 2016-12-07 16:39:28 +01:00
Rafa de la Torre
7c348dee0f Mapzen isochrones: tweak retry params #317 2016-12-07 16:38:47 +01:00
Rafa de la Torre
bdeaadf33f Fix QPS: Compare everything in microseconds #317 2016-12-07 16:37:25 +01:00
Mario de Frutos
51face5593 Make the HTTP requests retry on timeout 2016-12-02 17:01:05 +01:00
Mario de Frutos
77b2ac9155 Bump python version 2016-11-29 16:34:59 +01:00
Mario de Frutos
8edbf33ff1 Added denoise and generalize options to make the ischrones less heavy 2016-11-29 16:10:59 +01:00
Mario de Frutos
bd439d8985 Car is auto in mapzen isochrones 2016-11-29 13:18:52 +01:00
Mario de Frutos
16259862dd Version 0.20.0 artifact 2016-11-29 13:08:43 +01:00
Mario de Frutos
77f4f3e7ff Mapzen isochtrones integration 2016-11-29 12:57:23 +01:00
Mario de Frutos
4b714b3845 New version 0.20.0 2016-11-29 12:25:35 +01:00
Rafa de la Torre
ab4584bed3 New server version 0.19.0 #302 2016-11-25 10:56:41 +01:00
Rafa de la Torre
2b2691d0b9 New minor version of the python library #302 2016-11-25 10:43:06 +01:00
Rafa de la Torre
9c6eabc59e Do not enforce types for params #302
As types are tied to a schema, they don't get along very well with
pl/proxy. Do not use them for the service_type.
2016-11-24 18:38:33 +01:00
Rafa de la Torre
769b740ba4 Rename service_params to service_quota_info #302 2016-11-24 17:46:07 +01:00
Rafa de la Torre
c2ede37d75 Revert internal geocoder related stuff #302 2016-11-24 17:45:19 +01:00
Rafa de la Torre
4c75c5f261 Revert internal geocoder related stuff #302
This reverts commits 579d11e, 4af5975 and a2da597
2016-11-24 17:33:06 +01:00
Rafa de la Torre
efb10e43e4 Merge remote-tracking branch 'origin/development' into 302-expose-quotas 2016-11-24 12:36:23 +01:00
Rafa de la Torre
1dc6060570 Fix: deal with string values for soft limits #302 2016-11-23 18:51:13 +01:00
Rafa de la Torre
a2da597e00 Support null quota (internal_geocoder) #302 2016-11-23 18:10:44 +01:00
Rafa de la Torre
4af5975ac1 Add the internal geocoder to cdb_service_params #302 2016-11-23 17:07:23 +01:00
Rafa de la Torre
579d11ebd5 Add params to InternalGeocoderConfig #302
Add a bunch of params to InternalGeocoderConfig for consistency with
other services:
- monthly_quota
- period_end_date
- soft_limit
2016-11-23 17:04:50 +01:00
Rafa de la Torre
a1eae3934c Add OBS to UserMetricsService #302 2016-11-23 16:47:35 +01:00
Rafa de la Torre
d70d149eed Fix bug: the service_type to used_quota #302 2016-11-23 16:40:00 +01:00
Rafa de la Torre
decc7626a8 Create types only if they don't exist #302 2016-11-23 16:38:47 +01:00
Rafa de la Torre
abbb227ad5 Fix typo: use obs config for obs service #302 2016-11-23 11:40:02 +01:00
Mario de Frutos
f0474852f2 Client 0.12.1 and server 0.18.1 artifacts 2016-11-11 18:27:37 +01:00
Mario de Frutos
8396eae733 Server update to type obs_meta_geometry with new types 2016-11-11 18:13:01 +01:00
Mario de Frutos
dfaa037475 Server init new version 0.18.1 2016-11-11 17:53:20 +01:00
Mario de Frutos
d1b17da70b Don't make mandatory to have a metrics file 2016-11-11 11:48:08 +01:00
Rafa de la Torre
2e6f37af56 Add observatory #302 2016-11-11 11:27:57 +01:00
Mario de Frutos
b5f4c507b3 Fixed some typos in SQL artifact v0.18 and v0.17 2016-11-11 11:13:08 +01:00
Rafa de la Torre
95dda082e7 Add routing #302 2016-11-11 10:24:03 +01:00
Rafa de la Torre
9f9034f4f1 Make cdb_enough_quota check the table got from cdb_service_params #302 2016-11-10 19:23:39 +01:00
Rafa de la Torre
603fbbbc3f Make cdb_service_params return a table for all services #302 2016-11-10 19:23:01 +01:00
Rafa de la Torre
3fa686ec65 Remove the class Service in favor of service_type ENUM #302 2016-11-10 19:21:29 +01:00
Rafa de la Torre
429a71ef37 Add a function to check for enough quota #302 2016-11-10 18:56:50 +01:00
Rafa de la Torre
b2cdb1c74b Move all into one func returning a tuple #302 2016-11-10 18:40:18 +01:00
Rafa de la Torre
84061dec4f Move everything to cdb_service_params (WIP) #302 2016-11-10 18:12:24 +01:00
Mario de Frutos
08debb83ed Server side change JSONB to JSON in the legacybuildermedatada function 2016-11-10 18:08:35 +01:00
Rafa de la Torre
7a32ace1ac Add the hires_geocoder to cdb_monthly_quota #302 2016-11-10 17:00:25 +01:00
Rafa de la Torre
7a56dc698f List of services bound to quota #302 2016-11-10 16:59:58 +01:00
Mario de Frutos
4945b4d0bf Server version 0.18.0 artifact 2016-11-10 16:03:08 +01:00
Mario de Frutos
040b094b01 New observatory meta functions for server side 2016-11-10 11:29:19 +01:00
Mario de Frutos
a006f76285 New server version 0.18.0 2016-11-10 11:29:18 +01:00
Rafa de la Torre
eb7188235d Provider function #302 2016-11-10 11:14:15 +01:00
Rafa de la Torre
3e059003b6 Add soft limit function #302 2016-11-10 11:09:40 +01:00
Rafa de la Torre
0e38af6ecd Add cdb_remaining_quota to server #302 2016-11-08 18:47:22 +01:00
Rafa de la Torre
b00edf218a Fix: quotas should be integers #302 2016-11-08 18:46:58 +01:00
Rafa de la Torre
6f9feb07a0 Add cdb_monthly_quota (WIP) #302 2016-11-08 18:29:06 +01:00
Rafa de la Torre
2c76fa4501 Fix: quotas should be integers #302 2016-11-08 18:28:28 +01:00
Rafa de la Torre
4169bc0f88 Fix doc: how to run python unit tests #302 2016-11-08 17:52:18 +01:00
Mario de Frutos
6177f0ac76 Remove info trace not needed in metrics 2016-11-07 11:02:09 +01:00
Mario de Frutos
352d4217bc Fixed typo in the config cache for observatory snapshot functions 2016-11-07 10:31:53 +01:00
Mario de Frutos
2fd560f3cf Release 0.17.0 artifact 2016-11-04 13:41:12 +01:00
Mario de Frutos
88ed777785 Added plpy_mock and check multiple instances for metricsdatagatherer 2016-11-04 13:39:11 +01:00
Mario de Frutos
73aa26590b Make the data gather singleton support multiple instances
Due to we use pgbouncer, we could share the same instance of the metrics
gatherer singleton in multiples requests. Added support to have
multiple instances identified by transaction id
2016-11-04 13:39:11 +01:00
Mario de Frutos
c571edb21e Split response and function data, added uuid and type of element 2016-11-04 13:39:11 +01:00
Mario de Frutos
e878031744 Added url to the response data 2016-11-04 13:39:11 +01:00
Mario de Frutos
49fd51f6da Added logger to metrics context manager 2016-11-04 13:39:10 +01:00
Mario de Frutos
708bee2912 Added metrics context manager to public functions 2016-11-04 13:39:10 +01:00
Mario de Frutos
cf3c6f2ce5 New metrics logger system
- It's not mandatory to have the log_path defined for all the services
- Now we add metrics to a MetricsGatherer using the add method in order
  to gather data from all the function scope, not just in one defined
  point
- There is a `metrics` context manager to wrap the function block. This
  context manager get the execution time for the block among other
  things
2016-11-02 15:26:39 +01:00
Mario de Frutos
93579532e3 Move old versions 2016-10-28 10:42:25 +02:00
Mario de Frutos
6058960ec5 Added timeout for all the third-party connections 2016-10-27 16:17:37 +02:00
Mario de Frutos
336d8be977 Hotfix: Typo in status code check 2016-10-26 17:49:07 +02:00
Mario de Frutos
80963e2589 504 errors return empty data instead of raise exception
Due to some problems in Mapzen, we're receiving 504 errors from their
servers. To mitigate this problem, instead of raise an exception, we're
going to return empty data for that point
2016-10-26 16:39:35 +02:00
Mario de Frutos
19d6cacdb3 Fix empty response from matrix
https://github.com/CartoDB/dataservices-api/issues/293
2016-10-26 16:00:50 +02:00
Rafa de la Torre
0d22942a72 Update version of python lib server package to 0.9.2 2016-10-21 17:38:32 +02:00
Rafa de la Torre
d4ac2eb5e6 Tests for ServicesRedisConfig (mapzen quota) #281 2016-10-21 15:40:22 +02:00
Rafa de la Torre
db80d389e0 Do not require org geocoder and isolines quota to be present 2016-10-21 15:38:58 +02:00
Rafa de la Torre
8e02c64aeb Tests for UserMetricsService, mapzen routing #281 2016-10-21 15:21:03 +02:00
Rafa de la Torre
cc2ab1bc0c Fix test: make it independent of current date #281 2016-10-21 15:08:17 +02:00
Rafa de la Torre
e1a7d1751c Fix corner case: used_quota == monthly_quota #281 2016-10-20 15:36:48 +02:00
Rafa de la Torre
0c49107f96 Some tests for the QuotaChecker #281 2016-10-20 15:31:21 +02:00
Rafa de la Torre
247034c21e Some tests for RoutingConfig #281 2016-10-19 18:40:17 +02:00
Rafa de la Torre
2b1b1c981f Add routing to UserMetricsService.used_quota 2016-10-19 17:41:43 +02:00
Rafa de la Torre
aaff5564ec Use soft_limit for __check_routing_quota #281 2016-10-18 17:44:56 +02:00
Rafa de la Torre
72998c324a Simplify the code for regularity's sake #281 2016-10-18 17:40:50 +02:00
Rafa de la Torre
bbd9b6b98e Add soft limit to config object #281 2016-10-18 17:35:36 +02:00
Rafa de la Torre
27be704bd6 Changes needed to read quota from redis #281
Use per-user/org quota from redis, if present. Otherwise default to the
quota stored in server database (as it did to date).
2016-10-18 17:26:45 +02:00
Rafa de la Torre
98d533b707 Bump version of python pip package to 0.9.1 2016-10-06 15:05:00 +02:00
Rafa de la Torre
6d0ad85d48 Fix for crash when sentinel_master_id is absent in config 2016-10-06 15:04:25 +02:00
Rafa de la Torre
00e6cace76 Update version of the python pip package 2016-10-06 11:06:28 +02:00
Rafa de la Torre
e9ad35ba1d Merge pull request #284 from CartoDB/redis-refactor-take2
Redis refactor: Take 2
2016-10-05 18:32:08 +02:00
Rafa de la Torre
dcb3935021 Upgrade/downgrade files for server extension 2016-10-05 18:28:24 +02:00
Rafa de la Torre
cded6c2f08 Test for NullConfigStorage 2016-10-04 15:20:41 +02:00
Rafa de la Torre
e1b357137a Rename s/storage/backend for consistency 2016-10-04 14:59:57 +02:00
Rafa de la Torre
3844cfc226 Tests for environment module 2016-10-04 14:56:06 +02:00
Rafa de la Torre
2a1276f4f1 Add test for redis_config module 2016-10-04 12:53:52 +02:00
Rafa de la Torre
35da7e48fd Implement a ServerConfigBackendFactory
mostly to keep layers separated.
2016-10-03 17:42:19 +02:00
Rafa de la Torre
12aebb7eee Rename ConfigStorageInterface to ConfigBackendInterface 2016-10-03 17:35:33 +02:00
Rafa de la Torre
0d87a95270 Inject the environment into the LoggerConfigBuilder 2016-10-03 17:06:57 +02:00
Rafa de la Torre
18e1a5c7c9 Remove the class User and its tests
as it is not used.
2016-10-03 16:54:40 +02:00
Rafa de la Torre
fcca5da302 Add a RedisMetricsConnectionFactory class 2016-10-03 16:47:45 +02:00
Rafa de la Torre
1aec541906 Add ServerEnvironment and ServerEnvironmentBuilder 2016-10-03 16:21:28 +02:00
Rafa de la Torre
9e98e0794d Add backends for user and org configs 2016-10-03 16:05:08 +02:00
Rafa de la Torre
8fbb41742c Rename redis conf factories to builders 2016-10-03 15:25:01 +02:00
Rafa de la Torre
275a6dc27f Move redis_config related classes to separate file 2016-10-03 13:33:46 +02:00