cartodb4/dataservices-api
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,484 Commits 1 Branch 183 Tags 3.3 MiB
972aba6cfb
Commit Graph

369 Commits

Author SHA1 Message Date
Rafa de la Torre
6c3260b6ee Move credentials tests to GoogleMapsClientFactoryTestCase 2017-10-06 15:23:44 +02:00
Rafa de la Torre
5005572f89 Add tests for GoogleMapsClientFactory 2017-10-06 15:18:51 +02:00
Rafa de la Torre
27fde5a910 Use "{id}:{secret}" as caching key 2017-10-06 15:12:05 +02:00
Rafa de la Torre
b8e1120169 Revert "Use a different client_id for the test" 
This reverts commit 854fbb803c.
 2017-10-06 15:03:08 +02:00
Rafa de la Torre
8e33cf9723 Fix google services key validity check #382 2017-10-06 14:40:53 +02:00
Rafa de la Torre
7b988e3f78 Add tests for #382 
The examples are not arbitrary: base64 by default ignores characters
that are not in its alphabet. When decoding, the dashes and underscores
are not b64-valid and therefore ignored, resulting in a sequence with
the wrong padding.
 2017-10-06 14:37:26 +02:00
Rafa de la Torre
f79ac9297d Add again the check for valid credentials 
As our check is more strict that the one provided in googlemaps library.
 2017-10-06 13:44:08 +02:00
Rafa de la Torre
854fbb803c Use a different client_id for the test 2017-10-06 13:41:50 +02:00
Rafa de la Torre
d029ad7041 GoogleMapsClientFactory to persist clients #401 
Add a GoogleMapsClientFactory and remove the check for valid key,
which is no longer needed, as it is done in the google library.
 2017-10-06 12:05:34 +02:00
Rafa de la Torre
445d4cf97d Upgrade googlemaps python library to v2.5.1 
This closes #383
 2017-10-04 16:22:33 +02:00
Rafa de la Torre
f0a3249b4e Keep metrics keys consistent 
See cartodb repo and our internal documentation:
https://github.com/CartoDB/cartodb/search?utf8=%E2%9C%93&q=failed_responses&type=
 2017-09-07 11:24:38 +02:00
Mario de Frutos
475df918c7 Fix invalid geometries due to generalize option 
They add a note saying that use their simplification option could
lead to a self-intersection (which is the problem we have) and it's
creating invalid geometries

See it here https://mapzen.com/documentation/mobility/isochrone/api-reference/#other-request-parameters
 2017-08-30 17:35:08 +02:00
Mario de Frutos
c13dba71a4 Deactivated DO credits usage for users that have quota 
See https://github.com/CartoDB/bigmetadata/issues/215
 2017-08-24 14:35:20 +02:00
Daniel García Aubert
13a876b854 Fix typo while logging caught error in geocoder 2017-07-21 14:43:41 +02:00
Daniel García Aubert
1503b5bb2b Release artifacts: 
- Server extension version 0.25.0
 - Client extension version 0.18.0
 - Python library version 0.15.2
 2017-07-13 15:34:44 +02:00
Rafa de la Torre
49a53e4cd7 Fix non-zero-padded days in tests 
See https://github.com/CartoDB/dataservices-api/issues/204 and related
tickets.
 2017-07-07 15:14:27 +02:00
Mario de Frutos
61efb66aba Add an extra check for google credentials 
If the user has a wrong base64 padded secret key the googlemaps
python library is returning "TypeError: Incorrect padding" which
is very hard to understand. So now we check if the secret key is
a valid base64 string
 2017-05-26 11:51:32 +02:00
Rafael Porres Molina
37eeab5b9e past participle of set is set 2017-05-24 15:41:12 +02:00
Javier Goizueta
524d15c1a9 Fix uses of cartodb cdb_conf functions 2017-03-30 18:10:53 +02:00
Javier Goizueta
aac89e0236 New versions 0.16.0 (client), 0.23.0 (server), 0.15.0 (python) 2017-03-28 17:53:40 +02:00
Javier Goizueta
f0b0a9e7f2 Merge pull request #355 from CartoDB/346-user-rate-limits 
Service rate limits
 2017-03-28 17:48:46 +02:00
Rafa de la Torre
77dfda11d9 Merge pull request #353 from CartoDB/udpate-docs 
Fix links and improve readability
 2017-03-28 11:53:07 +02:00
Javier Goizueta
2c15110255 Rename constructor arguments for consistency 2017-03-28 10:53:16 +02:00
Javier Goizueta
63c03894cc Fix bugs 2017-03-21 15:45:05 +01:00
Javier Goizueta
942c6ac63d Fix indentation 2017-03-21 15:44:50 +01:00
Javier Goizueta
208469f534 Add rate limits configuration writing tests 2017-03-21 15:43:33 +01:00
Javier Goizueta
73f97128ed Reorganization of configuration modules, fix circular dependencies 
The new module cartodb_services/config is intended for services configuration objects
Some legacy configuration objects remain under cartodb_services/metrics.
The refactored configuration backends are also not moved here
 2017-03-21 15:42:53 +01:00
Javier Goizueta
945c6cd685 WIP: support for storing rate limits configuration 2017-03-16 19:12:39 +01:00
Javier Goizueta
7f6c19b292 Minor changes for clarity 
* ServiceManager check method renamed as assert_within_limits
* Legacy classes moved to separate files
 2017-03-16 17:59:22 +01:00
Javier Goizueta
9a04ad06a0 Move rate limits out of Config 
The (legacy) Config object rate limit-related modifications are reverted.
For the legacy case, configuration is handled in a specific RateLimits builder class.
 2017-03-15 19:32:50 +01:00
Javier Goizueta
cebd7d2141 Apply ServiceManager to Google Geocoder 
Note that the Goolge geocoder uses QuotaService to account for used quota,
but it doesn't have a quota limit.
 2017-03-15 17:58:14 +01:00
Javier Goizueta
680206a437 Missing import 2017-03-15 12:51:56 +01:00
Javier Goizueta
77b5ff0b9e Fix problem with db config default arguments 2017-03-15 12:46:25 +01:00
Javier Goizueta
05b29967c7 Merge branch 'development' into 346-user-rate-limits 
# Conflicts:
#	README.md
#	server/extension/sql/20_geocode_street.sql
#	server/lib/python/cartodb_services/cartodb_services/metrics/config.py
 2017-03-14 19:00:53 +01:00
Javier Goizueta
6c5ca97468 Complete rate-limiting for Mapzen & Here gecoding 
ServiceManager class has been introduced to handle service configuration at
SQL level (with a LegacyServiceManager alternative for non-refactored services).
These new classes take the responsibility of rate limits and quota checking.

Tests have been added for ServiceManager and rate limits, but currently they
check only the limits configuration since Lua support would be needed
to use rratelimit with MockRedis.
 2017-03-14 18:51:18 +01:00
Rafa de la Torre
a64557b50e New version 0.14.1 of python library 2017-03-13 15:10:03 +01:00
Rafa de la Torre
4e14e9b0c0 Fix links and improve readability 2017-03-13 13:12:06 +01:00
Rafa de la Torre
26d025a5d1 Remove tests that no longer apply #206 2017-03-10 13:15:56 +01:00
Rafa de la Torre
80e7ed90a8 Make get_metrics read just zero-padded dates #206 2017-03-10 13:14:36 +01:00
Javier Goizueta
250d384d06 Geocoder, per user, cofigurable rate limits 
WIP, specially the GeocoderConfig part is flaky and ugly
 2017-03-09 18:42:48 +01:00
Rafa de la Torre
fc1e05658e Bump version of python library to 0.14.0 2017-03-08 13:13:19 +01:00
Rafa de la Torre
b8ec3dc306 Merge pull request #342 from CartoDB/remove-mapzen-forcing-layers-equal-address 
Remove the search_type if its address from the params sent to Mapzen
 2017-03-08 13:07:07 +01:00
Rafa de la Torre
ffdb6524da Fix bug: search_type can be None 
Found via unit test:

```sh
$ nosetests -x test/
....................................................................................E
======================================================================
ERROR: test_geocode_address_with_valid_params (test.test_mapzengeocoder.MapzenGeocoderTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/requests_mock/mocker.py", line 177, in inner
    return func(*args, **kwargs)
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/test/test_mapzengeocoder.py", line 100, in test_geocode_address_with_valid_params
    country='ESP')
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 28, in wrapped_function
    return QPSService(retry_timeout=timeout, queries_per_second=qps).call(original_function, *args, **kwargs)
  File "/home/ubuntu/src/dataservices-api/server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py", line 53, in call
    raise e
AttributeError: 'NoneType' object has no attribute 'lower'

----------------------------------------------------------------------
Ran 85 tests in 0.140s

FAILED (errors=1)
```
 2017-03-08 10:19:32 +01:00
Rafa de la Torre
82a7477d37 Beautify code 2017-03-08 10:14:37 +01:00
Rafa de la Torre
25ba9866ae Small PoC with rratelimit #346 2017-03-07 13:31:06 +01:00
Javier Goizueta
6dca3f4e58 Change default arguent value for clarity 
It is valie to pass None in service_params arguments, so make it the default.
 2017-03-06 19:01:05 +01:00
Javier Goizueta
d01170471e Missing @property 2017-03-06 18:58:00 +01:00
Javier Goizueta
89762c1a7f Use service params configuration in Mapzen services 2017-03-06 15:25:11 +01:00
Javier Goizueta
1081e81047 Add Mapzen service params to configuration 2017-03-06 14:34:23 +01:00
Javier Goizueta
29cdd8099a Add service configuration (URL, etc) for Mapzen Geocoding 2017-03-06 10:20:20 +01:00
Javier Goizueta
c0374f5800 Add service configuration (URL, etc) for Here Routing 2017-03-02 17:55:08 +01:00
Javier Goizueta
677d2acead Add service configuration (URL, etc) for Here Geocoding 
See #343
 2017-03-02 17:15:14 +01:00
nygeog
a9e942a48b remove the search_type if its address from the params sent to mapzen 
remove the search_type if its address from the params sent to mapzen
 2017-02-02 10:23:09 -05:00
Mario de Frutos
c364c7f5ed Merge pull request #339 from CartoDB/obs-getdata-getmeta-wrappers 
Obs_GetData OBS_GetMeta wrappers
 2017-02-01 10:05:52 +01:00
Mario de Frutos
c550a5d4e9 Merge pull request #321 from CartoDB/fix-default-isolines-provider 
Change default isolines provider
 2017-02-01 09:41:21 +01:00
John Krauss
8204bb3451 fix error-causing raise (which obscures real error) 2017-01-26 12:27:29 +01:00
Rafa de la Torre
a0dd7c7497 Fix tests: zero-pad months in redis key #335 2017-01-04 11:11:27 +01:00
Rafa de la Torre
9700ae966c Remove uneeded math for simplicity's sake 2016-12-28 14:44:48 +01:00
Javier Goizueta
bdf9627586 Fix typo 
I needed to fix this; I felt some kitten was being killed somewhere
because of this. And it made baby Jesus cry.
 2016-12-27 17:11:39 +01:00
Mario de Frutos
80b23c62c3 QPS timeout was badly calculated 
timedelta microseconds is just the microseconds part of the timedelta
object not the elapsed time in microseconds.

I've change to use the total_seconds method to get all the elapsed time
in seconds and transform to miliseconds.
 2016-12-23 11:40:30 +01:00
Mario de Frutos
23a2de0321 Add mapzen isolines to the quota checker 2016-12-22 17:38:10 +01:00
Javier Goizueta
9ee1d045c8 Fix tests 2016-12-21 09:54:08 +01:00
Rafa de la Torre
360b5bd57f Merge remote-tracking branch 'origin/development' into mapzen_services_use_quotas 
Solve merge conflict in setup.py (version of python lib).
 2016-12-20 18:14:01 +01:00
Javier Goizueta
32010669e8 New version of the python lib 
To fix bug with period end date; see #322
 2016-12-20 16:39:19 +01:00
Mario de Frutos
38754fec26 Cover refactor mapzen geocoder config with tests 2016-12-20 16:17:42 +01:00
Mario de Frutos
356135672b Sanitize quota get in the refactor part 2016-12-20 16:17:42 +01:00
Mario de Frutos
4d0abf9026 Bump python library version 2016-12-20 16:17:42 +01:00
Mario de Frutos
0672f2752b Improve tests and add more unit tests for the quota functionality 2016-12-20 16:17:41 +01:00
Javier Goizueta
b5514aea60 Avoid invalid end of period dates 
Fixes #322
 2016-12-20 12:48:36 +01:00
Javier Goizueta
53acd4d30e Add tests that reveal end-of-period date problem 
See #322
 2016-12-20 12:41:54 +01:00
Mario de Frutos
6c71d73498 Use the current quota for mapzen services 2016-12-20 11:52:06 +01:00
Rafa de la Torre
a00fca6d13 Update README.md 
Add a bit about versioning stuff
 2016-12-16 18:00:22 +01:00
Rafa de la Torre
29caaf9297 Update README.md 
Remove misleading paragraph about `requirements.txt` and `setup.py` dependencies. Refer to https://packaging.python.org/requirements/ for an authoritative discussion.
 2016-12-16 18:00:22 +01:00
Javier Goizueta
4fc90626ab New version of the python lib 
New release to change number of http connection retries to 1
 2016-12-13 18:03:37 +01:00
Javier Goizueta
1f53af65b9 Set number of http connection retries to 1 
Change the maximum number of retries for connection to external services to 1.
 2016-12-13 11:32:22 +01:00
Javier Goizueta
96a147a242 The default isolines provider should be Mapzen 2016-12-13 10:19:27 +01:00
Javier Goizueta
79fb796180 Fix bug in retries usage; mantain behaviour 
The number of retries was being passed as the pool size parameter.
This is corrected but number of retries changed to 1 to maintain previous behaviour.
 2016-12-13 10:07:10 +01:00
Javier Goizueta
2953fda75c Revert num of isochrone retries to 3 
See #317
 2016-12-12 12:14:10 +01:00
Rafa de la Torre
98fa248fff New version of the python lib #317 2016-12-07 16:39:28 +01:00
Rafa de la Torre
7c348dee0f Mapzen isochrones: tweak retry params #317 2016-12-07 16:38:47 +01:00
Rafa de la Torre
bdeaadf33f Fix QPS: Compare everything in microseconds #317 2016-12-07 16:37:25 +01:00
Mario de Frutos
51face5593 Make the HTTP requests retry on timeout 2016-12-02 17:01:05 +01:00
Mario de Frutos
77b2ac9155 Bump python version 2016-11-29 16:34:59 +01:00
Mario de Frutos
8edbf33ff1 Added denoise and generalize options to make the ischrones less heavy 2016-11-29 16:10:59 +01:00
Mario de Frutos
bd439d8985 Car is auto in mapzen isochrones 2016-11-29 13:18:52 +01:00
Mario de Frutos
77f4f3e7ff Mapzen isochtrones integration 2016-11-29 12:57:23 +01:00
Rafa de la Torre
2b2691d0b9 New minor version of the python library #302 2016-11-25 10:43:06 +01:00
Rafa de la Torre
4c75c5f261 Revert internal geocoder related stuff #302 
This reverts commits 579d11e, 4af5975 and a2da597
 2016-11-24 17:33:06 +01:00
Rafa de la Torre
efb10e43e4 Merge remote-tracking branch 'origin/development' into 302-expose-quotas 2016-11-24 12:36:23 +01:00
Rafa de la Torre
1dc6060570 Fix: deal with string values for soft limits #302 2016-11-23 18:51:13 +01:00
Rafa de la Torre
579d11ebd5 Add params to InternalGeocoderConfig #302 
Add a bunch of params to InternalGeocoderConfig for consistency with
other services:
- monthly_quota
- period_end_date
- soft_limit
 2016-11-23 17:04:50 +01:00
Rafa de la Torre
a1eae3934c Add OBS to UserMetricsService #302 2016-11-23 16:47:35 +01:00
Mario de Frutos
d1b17da70b Don't make mandatory to have a metrics file 2016-11-11 11:48:08 +01:00
Rafa de la Torre
3fa686ec65 Remove the class Service in favor of service_type ENUM #302 2016-11-10 19:21:29 +01:00
Rafa de la Torre
7a56dc698f List of services bound to quota #302 2016-11-10 16:59:58 +01:00
Rafa de la Torre
b00edf218a Fix: quotas should be integers #302 2016-11-08 18:46:58 +01:00
Rafa de la Torre
2c76fa4501 Fix: quotas should be integers #302 2016-11-08 18:28:28 +01:00
Rafa de la Torre
4169bc0f88 Fix doc: how to run python unit tests #302 2016-11-08 17:52:18 +01:00
Mario de Frutos
6177f0ac76 Remove info trace not needed in metrics 2016-11-07 11:02:09 +01:00
Mario de Frutos
88ed777785 Added plpy_mock and check multiple instances for metricsdatagatherer 2016-11-04 13:39:11 +01:00