Javier Goizueta
6c5ca97468
Complete rate-limiting for Mapzen & Here gecoding
...
ServiceManager class has been introduced to handle service configuration at
SQL level (with a LegacyServiceManager alternative for non-refactored services).
These new classes take the responsibility of rate limits and quota checking.
Tests have been added for ServiceManager and rate limits, but currently they
check only the limits configuration since Lua support would be needed
to use rratelimit with MockRedis.
2017-03-14 18:51:18 +01:00
Rafa de la Torre
a64557b50e
New version 0.14.1 of python library
2017-03-13 15:10:03 +01:00
Rafa de la Torre
4e14e9b0c0
Fix links and improve readability
2017-03-13 13:12:06 +01:00
Rafa de la Torre
26d025a5d1
Remove tests that no longer apply #206
2017-03-10 13:15:56 +01:00
Rafa de la Torre
80e7ed90a8
Make get_metrics read just zero-padded dates #206
2017-03-10 13:14:36 +01:00
Javier Goizueta
250d384d06
Geocoder, per user, cofigurable rate limits
...
WIP, specially the GeocoderConfig part is flaky and ugly
2017-03-09 18:42:48 +01:00
Rafa de la Torre
c6c4255242
Server 0.22.0 version release artifacts
2017-03-08 15:22:58 +01:00
Rafa de la Torre
87a2e02721
Make more apparent what happens with make release
...
Instead of getting (deleted and untracked files):
```
$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 20 commits.
#
# Changes not staged for commit:
# (use "git add/rm <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# deleted: cdb_dataservices_server--0.20.0--0.21.0.sql
# deleted: cdb_dataservices_server--0.21.0--0.20.0.sql
# deleted: cdb_dataservices_server--0.21.0.sql
# modified: cdb_dataservices_server.control
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# cdb_dataservices_server--0.21.0--0.22.0.sql
# cdb_dataservices_server--0.22.0--0.21.0.sql
# cdb_dataservices_server--0.22.0.sql
# old_versions/cdb_dataservices_server--0.20.0--0.21.0.sql
# old_versions/cdb_dataservices_server--0.21.0--0.20.0.sql
# old_versions/cdb_dataservices_server--0.21.0.sql
no changes added to commit (use "git add" and/or "git commit -a")
```
you'd get something like:
```
$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 20 commits.
#
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# renamed: cdb_dataservices_server--0.20.0--0.21.0.sql -> old_versions/cdb_dataservices_server--0.20.0--0.21.0.sql
# renamed: cdb_dataservices_server--0.21.0--0.20.0.sql -> old_versions/cdb_dataservices_server--0.21.0--0.20.0.sql
# renamed: cdb_dataservices_server--0.21.0.sql -> old_versions/cdb_dataservices_server--0.21.0.sql
#
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: cdb_dataservices_server.control
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# cdb_dataservices_server--0.21.0--0.22.0.sql
# cdb_dataservices_server--0.22.0--0.21.0.sql
# cdb_dataservices_server--0.22.0.sql
```
which is nicer IMHO.
2017-03-08 13:22:28 +01:00
Rafa de la Torre
fc1e05658e
Bump version of python library to 0.14.0
2017-03-08 13:13:19 +01:00
Rafa de la Torre
b8ec3dc306
Merge pull request #342 from CartoDB/remove-mapzen-forcing-layers-equal-address
...
Remove the search_type if its address from the params sent to Mapzen
2017-03-08 13:07:07 +01:00
Rafa de la Torre
ffdb6524da
Fix bug: search_type can be None
...
Found via unit test:
```sh
$ nosetests -x test/
....................................................................................E
======================================================================
ERROR: test_geocode_address_with_valid_params (test.test_mapzengeocoder.MapzenGeocoderTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/requests_mock/mocker.py", line 177, in inner
return func(*args, **kwargs)
File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/test/test_mapzengeocoder.py", line 100, in test_geocode_address_with_valid_params
country='ESP')
File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 28, in wrapped_function
return QPSService(retry_timeout=timeout, queries_per_second=qps).call(original_function, *args, **kwargs)
File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 53, in call
raise e
AttributeError: 'NoneType' object has no attribute 'lower'
----------------------------------------------------------------------
Ran 85 tests in 0.140s
FAILED (errors=1)
```
2017-03-08 10:19:32 +01:00
Rafa de la Torre
82a7477d37
Beautify code
2017-03-08 10:14:37 +01:00
Rafa de la Torre
049c8e8657
Merge branch 'development' into remove-mapzen-forcing-layers-equal-address
2017-03-07 14:56:55 +01:00
Rafa de la Torre
25ba9866ae
Small PoC with rratelimit #346
2017-03-07 13:31:06 +01:00
Javier Goizueta
6dca3f4e58
Change default arguent value for clarity
...
It is valie to pass None in service_params arguments, so make it the default.
2017-03-06 19:01:05 +01:00
Javier Goizueta
d01170471e
Missing @property
2017-03-06 18:58:00 +01:00
Javier Goizueta
89762c1a7f
Use service params configuration in Mapzen services
2017-03-06 15:25:11 +01:00
Javier Goizueta
1081e81047
Add Mapzen service params to configuration
2017-03-06 14:34:23 +01:00
Javier Goizueta
29cdd8099a
Add service configuration (URL, etc) for Mapzen Geocoding
2017-03-06 10:20:20 +01:00
Javier Goizueta
c0374f5800
Add service configuration (URL, etc) for Here Routing
2017-03-02 17:55:08 +01:00
Javier Goizueta
677d2acead
Add service configuration (URL, etc) for Here Geocoding
...
See #343
2017-03-02 17:15:14 +01:00
Mario de Frutos
435a84f294
OBS_GetData functions shall increase the DO quota usage
2017-02-06 17:47:15 +01:00
nygeog
a9e942a48b
remove the search_type if its address from the params sent to mapzen
...
remove the search_type if its address from the params sent to mapzen
2017-02-02 10:23:09 -05:00
Mario de Frutos
061956f250
Remove quota usage for obs_meta functions
2017-02-02 10:59:21 +01:00
Mario de Frutos
3d94d1dfaf
Server 0.21.0 version release artifacts
2017-02-01 10:19:32 +01:00
Mario de Frutos
c364c7f5ed
Merge pull request #339 from CartoDB/obs-getdata-getmeta-wrappers
...
Obs_GetData OBS_GetMeta wrappers
2017-02-01 10:05:52 +01:00
Mario de Frutos
c550a5d4e9
Merge pull request #321 from CartoDB/fix-default-isolines-provider
...
Change default isolines provider
2017-02-01 09:41:21 +01:00
John Krauss
1f225e0978
fix extraneous newline in expectation
2017-01-26 12:27:29 +01:00
John Krauss
e81f005ce9
adding server tests, interface to generate client sql code
2017-01-26 12:27:29 +01:00
John Krauss
8204bb3451
fix error-causing raise (which obscures real error)
2017-01-26 12:27:29 +01:00
John Krauss
56de523498
add wrappers for getmeta, getdata(geoms) and getdata(ids)
2017-01-26 12:27:29 +01:00
Rafa de la Torre
a0dd7c7497
Fix tests: zero-pad months in redis key #335
2017-01-04 11:11:27 +01:00
Rafa de la Torre
9700ae966c
Remove uneeded math for simplicity's sake
2016-12-28 14:44:48 +01:00
Javier Goizueta
bdf9627586
Fix typo
...
I needed to fix this; I felt some kitten was being killed somewhere
because of this. And it made baby Jesus cry.
2016-12-27 17:11:39 +01:00
Mario de Frutos
80b23c62c3
QPS timeout was badly calculated
...
timedelta microseconds is just the microseconds part of the timedelta
object not the elapsed time in microseconds.
I've change to use the total_seconds method to get all the elapsed time
in seconds and transform to miliseconds.
2016-12-23 11:40:30 +01:00
Mario de Frutos
23a2de0321
Add mapzen isolines to the quota checker
2016-12-22 17:38:10 +01:00
Javier Goizueta
9ee1d045c8
Fix tests
2016-12-21 09:54:08 +01:00
Rafa de la Torre
360b5bd57f
Merge remote-tracking branch 'origin/development' into mapzen_services_use_quotas
...
Solve merge conflict in setup.py (version of python lib).
2016-12-20 18:14:01 +01:00
Javier Goizueta
32010669e8
New version of the python lib
...
To fix bug with period end date; see #322
2016-12-20 16:39:19 +01:00
Mario de Frutos
38754fec26
Cover refactor mapzen geocoder config with tests
2016-12-20 16:17:42 +01:00
Mario de Frutos
356135672b
Sanitize quota get in the refactor part
2016-12-20 16:17:42 +01:00
Mario de Frutos
4d0abf9026
Bump python library version
2016-12-20 16:17:42 +01:00
Mario de Frutos
0672f2752b
Improve tests and add more unit tests for the quota functionality
2016-12-20 16:17:41 +01:00
Javier Goizueta
b5514aea60
Avoid invalid end of period dates
...
Fixes #322
2016-12-20 12:48:36 +01:00
Javier Goizueta
53acd4d30e
Add tests that reveal end-of-period date problem
...
See #322
2016-12-20 12:41:54 +01:00
Mario de Frutos
6c71d73498
Use the current quota for mapzen services
2016-12-20 11:52:06 +01:00
Rafa de la Torre
a00fca6d13
Update README.md
...
Add a bit about versioning stuff
2016-12-16 18:00:22 +01:00
Rafa de la Torre
29caaf9297
Update README.md
...
Remove misleading paragraph about `requirements.txt` and `setup.py` dependencies. Refer to https://packaging.python.org/requirements/ for an authoritative discussion.
2016-12-16 18:00:22 +01:00
Javier Goizueta
4fc90626ab
New version of the python lib
...
New release to change number of http connection retries to 1
2016-12-13 18:03:37 +01:00
Javier Goizueta
1f53af65b9
Set number of http connection retries to 1
...
Change the maximum number of retries for connection to external services to 1.
2016-12-13 11:32:22 +01:00
Javier Goizueta
96a147a242
The default isolines provider should be Mapzen
2016-12-13 10:19:27 +01:00
Javier Goizueta
79fb796180
Fix bug in retries usage; mantain behaviour
...
The number of retries was being passed as the pool size parameter.
This is corrected but number of retries changed to 1 to maintain previous behaviour.
2016-12-13 10:07:10 +01:00
Javier Goizueta
2953fda75c
Revert num of isochrone retries to 3
...
See #317
2016-12-12 12:14:10 +01:00
Rafa de la Torre
98fa248fff
New version of the python lib #317
2016-12-07 16:39:28 +01:00
Rafa de la Torre
7c348dee0f
Mapzen isochrones: tweak retry params #317
2016-12-07 16:38:47 +01:00
Rafa de la Torre
bdeaadf33f
Fix QPS: Compare everything in microseconds #317
2016-12-07 16:37:25 +01:00
Mario de Frutos
51face5593
Make the HTTP requests retry on timeout
2016-12-02 17:01:05 +01:00
Mario de Frutos
77b2ac9155
Bump python version
2016-11-29 16:34:59 +01:00
Mario de Frutos
8edbf33ff1
Added denoise and generalize options to make the ischrones less heavy
2016-11-29 16:10:59 +01:00
Mario de Frutos
bd439d8985
Car is auto in mapzen isochrones
2016-11-29 13:18:52 +01:00
Mario de Frutos
16259862dd
Version 0.20.0 artifact
2016-11-29 13:08:43 +01:00
Mario de Frutos
77f4f3e7ff
Mapzen isochtrones integration
2016-11-29 12:57:23 +01:00
Mario de Frutos
4b714b3845
New version 0.20.0
2016-11-29 12:25:35 +01:00
Rafa de la Torre
ab4584bed3
New server version 0.19.0 #302
2016-11-25 10:56:41 +01:00
Rafa de la Torre
2b2691d0b9
New minor version of the python library #302
2016-11-25 10:43:06 +01:00
Rafa de la Torre
9c6eabc59e
Do not enforce types for params #302
...
As types are tied to a schema, they don't get along very well with
pl/proxy. Do not use them for the service_type.
2016-11-24 18:38:33 +01:00
Rafa de la Torre
769b740ba4
Rename service_params to service_quota_info #302
2016-11-24 17:46:07 +01:00
Rafa de la Torre
c2ede37d75
Revert internal geocoder related stuff #302
2016-11-24 17:45:19 +01:00
Rafa de la Torre
4c75c5f261
Revert internal geocoder related stuff #302
...
This reverts commits 579d11e
, 4af5975
and a2da597
2016-11-24 17:33:06 +01:00
Rafa de la Torre
efb10e43e4
Merge remote-tracking branch 'origin/development' into 302-expose-quotas
2016-11-24 12:36:23 +01:00
Rafa de la Torre
1dc6060570
Fix: deal with string values for soft limits #302
2016-11-23 18:51:13 +01:00
Rafa de la Torre
a2da597e00
Support null quota (internal_geocoder) #302
2016-11-23 18:10:44 +01:00
Rafa de la Torre
4af5975ac1
Add the internal geocoder to cdb_service_params #302
2016-11-23 17:07:23 +01:00
Rafa de la Torre
579d11ebd5
Add params to InternalGeocoderConfig #302
...
Add a bunch of params to InternalGeocoderConfig for consistency with
other services:
- monthly_quota
- period_end_date
- soft_limit
2016-11-23 17:04:50 +01:00
Rafa de la Torre
a1eae3934c
Add OBS to UserMetricsService #302
2016-11-23 16:47:35 +01:00
Rafa de la Torre
d70d149eed
Fix bug: the service_type to used_quota #302
2016-11-23 16:40:00 +01:00
Rafa de la Torre
decc7626a8
Create types only if they don't exist #302
2016-11-23 16:38:47 +01:00
Rafa de la Torre
abbb227ad5
Fix typo: use obs config for obs service #302
2016-11-23 11:40:02 +01:00
Mario de Frutos
f0474852f2
Client 0.12.1 and server 0.18.1 artifacts
2016-11-11 18:27:37 +01:00
Mario de Frutos
8396eae733
Server update to type obs_meta_geometry with new types
2016-11-11 18:13:01 +01:00
Mario de Frutos
dfaa037475
Server init new version 0.18.1
2016-11-11 17:53:20 +01:00
Mario de Frutos
d1b17da70b
Don't make mandatory to have a metrics file
2016-11-11 11:48:08 +01:00
Rafa de la Torre
2e6f37af56
Add observatory #302
2016-11-11 11:27:57 +01:00
Mario de Frutos
b5f4c507b3
Fixed some typos in SQL artifact v0.18 and v0.17
2016-11-11 11:13:08 +01:00
Rafa de la Torre
95dda082e7
Add routing #302
2016-11-11 10:24:03 +01:00
Rafa de la Torre
9f9034f4f1
Make cdb_enough_quota check the table got from cdb_service_params #302
2016-11-10 19:23:39 +01:00
Rafa de la Torre
603fbbbc3f
Make cdb_service_params return a table for all services #302
2016-11-10 19:23:01 +01:00
Rafa de la Torre
3fa686ec65
Remove the class Service in favor of service_type ENUM #302
2016-11-10 19:21:29 +01:00
Rafa de la Torre
429a71ef37
Add a function to check for enough quota #302
2016-11-10 18:56:50 +01:00
Rafa de la Torre
b2cdb1c74b
Move all into one func returning a tuple #302
2016-11-10 18:40:18 +01:00
Rafa de la Torre
84061dec4f
Move everything to cdb_service_params (WIP) #302
2016-11-10 18:12:24 +01:00
Mario de Frutos
08debb83ed
Server side change JSONB to JSON in the legacybuildermedatada function
2016-11-10 18:08:35 +01:00
Rafa de la Torre
7a32ace1ac
Add the hires_geocoder to cdb_monthly_quota #302
2016-11-10 17:00:25 +01:00
Rafa de la Torre
7a56dc698f
List of services bound to quota #302
2016-11-10 16:59:58 +01:00
Mario de Frutos
4945b4d0bf
Server version 0.18.0 artifact
2016-11-10 16:03:08 +01:00
Mario de Frutos
040b094b01
New observatory meta functions for server side
2016-11-10 11:29:19 +01:00
Mario de Frutos
a006f76285
New server version 0.18.0
2016-11-10 11:29:18 +01:00
Rafa de la Torre
eb7188235d
Provider function #302
2016-11-10 11:14:15 +01:00
Rafa de la Torre
3e059003b6
Add soft limit function #302
2016-11-10 11:09:40 +01:00
Rafa de la Torre
0e38af6ecd
Add cdb_remaining_quota to server #302
2016-11-08 18:47:22 +01:00
Rafa de la Torre
b00edf218a
Fix: quotas should be integers #302
2016-11-08 18:46:58 +01:00
Rafa de la Torre
6f9feb07a0
Add cdb_monthly_quota (WIP) #302
2016-11-08 18:29:06 +01:00
Rafa de la Torre
2c76fa4501
Fix: quotas should be integers #302
2016-11-08 18:28:28 +01:00
Rafa de la Torre
4169bc0f88
Fix doc: how to run python unit tests #302
2016-11-08 17:52:18 +01:00
Mario de Frutos
6177f0ac76
Remove info trace not needed in metrics
2016-11-07 11:02:09 +01:00
Mario de Frutos
352d4217bc
Fixed typo in the config cache for observatory snapshot functions
2016-11-07 10:31:53 +01:00
Mario de Frutos
2fd560f3cf
Release 0.17.0 artifact
2016-11-04 13:41:12 +01:00
Mario de Frutos
88ed777785
Added plpy_mock and check multiple instances for metricsdatagatherer
2016-11-04 13:39:11 +01:00
Mario de Frutos
73aa26590b
Make the data gather singleton support multiple instances
...
Due to we use pgbouncer, we could share the same instance of the metrics
gatherer singleton in multiples requests. Added support to have
multiple instances identified by transaction id
2016-11-04 13:39:11 +01:00
Mario de Frutos
c571edb21e
Split response and function data, added uuid and type of element
2016-11-04 13:39:11 +01:00
Mario de Frutos
e878031744
Added url to the response data
2016-11-04 13:39:11 +01:00
Mario de Frutos
49fd51f6da
Added logger to metrics context manager
2016-11-04 13:39:10 +01:00
Mario de Frutos
708bee2912
Added metrics context manager to public functions
2016-11-04 13:39:10 +01:00
Mario de Frutos
cf3c6f2ce5
New metrics logger system
...
- It's not mandatory to have the log_path defined for all the services
- Now we add metrics to a MetricsGatherer using the add method in order
to gather data from all the function scope, not just in one defined
point
- There is a `metrics` context manager to wrap the function block. This
context manager get the execution time for the block among other
things
2016-11-02 15:26:39 +01:00
Mario de Frutos
93579532e3
Move old versions
2016-10-28 10:42:25 +02:00
Mario de Frutos
6058960ec5
Added timeout for all the third-party connections
2016-10-27 16:17:37 +02:00
Mario de Frutos
336d8be977
Hotfix: Typo in status code check
2016-10-26 17:49:07 +02:00
Mario de Frutos
80963e2589
504 errors return empty data instead of raise exception
...
Due to some problems in Mapzen, we're receiving 504 errors from their
servers. To mitigate this problem, instead of raise an exception, we're
going to return empty data for that point
2016-10-26 16:39:35 +02:00
Mario de Frutos
19d6cacdb3
Fix empty response from matrix
...
https://github.com/CartoDB/dataservices-api/issues/293
2016-10-26 16:00:50 +02:00
Rafa de la Torre
0d22942a72
Update version of python lib server package to 0.9.2
2016-10-21 17:38:32 +02:00
Rafa de la Torre
d4ac2eb5e6
Tests for ServicesRedisConfig (mapzen quota) #281
2016-10-21 15:40:22 +02:00
Rafa de la Torre
db80d389e0
Do not require org geocoder and isolines quota to be present
2016-10-21 15:38:58 +02:00
Rafa de la Torre
8e02c64aeb
Tests for UserMetricsService, mapzen routing #281
2016-10-21 15:21:03 +02:00
Rafa de la Torre
cc2ab1bc0c
Fix test: make it independent of current date #281
2016-10-21 15:08:17 +02:00
Rafa de la Torre
e1a7d1751c
Fix corner case: used_quota == monthly_quota #281
2016-10-20 15:36:48 +02:00
Rafa de la Torre
0c49107f96
Some tests for the QuotaChecker #281
2016-10-20 15:31:21 +02:00
Rafa de la Torre
247034c21e
Some tests for RoutingConfig #281
2016-10-19 18:40:17 +02:00
Rafa de la Torre
2b1b1c981f
Add routing to UserMetricsService.used_quota
2016-10-19 17:41:43 +02:00
Rafa de la Torre
aaff5564ec
Use soft_limit for __check_routing_quota
#281
2016-10-18 17:44:56 +02:00
Rafa de la Torre
72998c324a
Simplify the code for regularity's sake #281
2016-10-18 17:40:50 +02:00
Rafa de la Torre
bbd9b6b98e
Add soft limit to config object #281
2016-10-18 17:35:36 +02:00
Rafa de la Torre
27be704bd6
Changes needed to read quota from redis #281
...
Use per-user/org quota from redis, if present. Otherwise default to the
quota stored in server database (as it did to date).
2016-10-18 17:26:45 +02:00
Rafa de la Torre
98d533b707
Bump version of python pip package to 0.9.1
2016-10-06 15:05:00 +02:00
Rafa de la Torre
6d0ad85d48
Fix for crash when sentinel_master_id is absent in config
2016-10-06 15:04:25 +02:00
Rafa de la Torre
00e6cace76
Update version of the python pip package
2016-10-06 11:06:28 +02:00
Rafa de la Torre
e9ad35ba1d
Merge pull request #284 from CartoDB/redis-refactor-take2
...
Redis refactor: Take 2
2016-10-05 18:32:08 +02:00
Rafa de la Torre
dcb3935021
Upgrade/downgrade files for server extension
2016-10-05 18:28:24 +02:00
Rafa de la Torre
cded6c2f08
Test for NullConfigStorage
2016-10-04 15:20:41 +02:00
Rafa de la Torre
e1b357137a
Rename s/storage/backend for consistency
2016-10-04 14:59:57 +02:00
Rafa de la Torre
3844cfc226
Tests for environment module
2016-10-04 14:56:06 +02:00
Rafa de la Torre
2a1276f4f1
Add test for redis_config module
2016-10-04 12:53:52 +02:00
Rafa de la Torre
35da7e48fd
Implement a ServerConfigBackendFactory
...
mostly to keep layers separated.
2016-10-03 17:42:19 +02:00
Rafa de la Torre
12aebb7eee
Rename ConfigStorageInterface to ConfigBackendInterface
2016-10-03 17:35:33 +02:00
Rafa de la Torre
0d87a95270
Inject the environment into the LoggerConfigBuilder
2016-10-03 17:06:57 +02:00
Rafa de la Torre
18e1a5c7c9
Remove the class User and its tests
...
as it is not used.
2016-10-03 16:54:40 +02:00
Rafa de la Torre
fcca5da302
Add a RedisMetricsConnectionFactory class
2016-10-03 16:47:45 +02:00
Rafa de la Torre
1aec541906
Add ServerEnvironment and ServerEnvironmentBuilder
2016-10-03 16:21:28 +02:00
Rafa de la Torre
9e98e0794d
Add backends for user and org configs
2016-10-03 16:05:08 +02:00
Rafa de la Torre
8fbb41742c
Rename redis conf factories to builders
2016-10-03 15:25:01 +02:00
Rafa de la Torre
275a6dc27f
Move redis_config related classes to separate file
2016-10-03 13:33:46 +02:00