Merge branch 'develop' into docs-781
This commit is contained in:
Carlos Matallín
commit
e938ee0c7b
63
NEWS.md
63
NEWS.md
@ -1,3 +1,62 @@
|
||||
0.0.1 (open date)
|
||||
1.0.1 (2016-07-01)
|
||||
---
|
||||
|
||||
__Bugfixes__
|
||||
|
||||
* Fix for ERROR: Operation on mixed SRID geometries #130
|
||||
|
||||
|
||||
1.0.0 (6/27/2016)
|
||||
-----
|
||||
|
||||
* Incremented to 1.0.0 to be in compliance with [SemVer](http://semver.org/),
|
||||
which disallows use of 0.x.x versions. This also reflects that we are
|
||||
already in production.
|
||||
|
||||
__API Changes__
|
||||
|
||||
* Added `OBS_DumpVersion` to look up version data ([#118](https://github.com/CartoDB/observatory-extension/pull/118))
|
||||
|
||||
__Improvements__
|
||||
|
||||
* Whether data exists for a geom now determined by polygon intersection instead of
|
||||
BBOX overlap ([#119](https://github.com/CartoDB/observatory-extension/pull/119))
|
||||
* Automated tests cover Spanish and UK data
|
||||
([#115](https://github.com/CartoDB/observatory-extension/pull/115))
|
||||
* Automated tests cover `OBS_GetUSCensusMeasure`
|
||||
([#105](https://github.com/CartoDB/observatory-extension/pull/105))
|
||||
|
||||
__Bugfixes__
|
||||
|
||||
* Geom table can have different `geomref_colname` than the data table
|
||||
([#123](https://github.com/CartoDB/observatory-extension/pull/123))
|
||||
|
||||
|
||||
0.0.5 (5/27/2016)
|
||||
-----
|
||||
* Adds new function `OBS_GetMeasureById` ([#96](https://github.com/CartoDB/observatory-extension/pull/96))
|
||||
|
||||
0.0.4 (5/25/2016)
|
||||
-----
|
||||
* Updates queries involving US Census measure tags to be more generic ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
|
||||
* Fixes tests which relied on an erroneous subset of block groups ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
|
||||
|
||||
0.0.3 (5/24/2016)
|
||||
-----
|
||||
* Generalizes internal queries to properly pull from multiple named geometry references
|
||||
* Adds tests for Who's on First boundaries
|
||||
* Improves automatic fixtures testing script
|
||||
|
||||
0.0.2 (5/19/2016)
|
||||
-----
|
||||
* Adds Data Observatory exploration functions
|
||||
* Adds Data Observatory boundary functions
|
||||
* Adds Data Observatory measure functions
|
||||
* Adds script to generate fixtures for tests
|
||||
* Adds script for the automatic testing of metadata
|
||||
* Adds full documentation for all included functions
|
||||
* removes `cartodb` extension dependency
|
||||
|
||||
0.0.1 (5/19/2016)
|
||||
------------------
|
||||
* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`;
|
||||
* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`
|
||||
|
||||
@ -20,12 +20,6 @@ script for the new release, `release/observatory--X.Y.Z.sql`:
|
||||
make release
|
||||
```
|
||||
|
||||
Then, the release manager shall produce upgrade and downgrade scripts
|
||||
to migrate to/from the previous release. In the case of minor/patch
|
||||
releases this simply consist in extracting the functions that have changed
|
||||
and placing them in the proper `release/observatory--X.Y.Z--A.B.C.sql`
|
||||
file.
|
||||
|
||||
The new release can be deployed for staging/smoke tests with this command:
|
||||
|
||||
```
|
||||
|
||||
@ -4,6 +4,8 @@ This file is for reference purposes only. It is intended for tracking the Data O
|
||||
|
||||
## Documentation
|
||||
|
||||
* Overview (local file in the Docs repo)
|
||||
* Accessing the Data Observatory (local file in the Docs repo)
|
||||
* [Measures Functions](measures_functions.md)
|
||||
* [Boundary Functions](boundary_functions.md)
|
||||
* [Discovery Functions](discovery_functions.md)
|
||||
|
||||
@ -29,7 +29,7 @@ Disputed Areas | whosonfirst.wof_disputed_geom | none
|
||||
|
||||
## OBS_GetUSCensusMeasure Names Table
|
||||
|
||||
This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
|
||||
This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
|
||||
|
||||
Measure name | Measure description
|
||||
------------------------ | --------------------
|
||||
|
||||
@ -1,6 +1,8 @@
|
||||
# License
|
||||
|
||||
The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. For competency, please still review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
|
||||
The Data Observatory is a collection of data sources with varying licenses and terms of use. We have endeavored to find you data that will work for the broadest set of use-cases. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
|
||||
|
||||
_**Legal Note**: The Data Observatory makes use of a variety of third party data and databases (collectively, the “Data”). You acknowledge that the included Data, and the licenses and terms of use, may be amended from time to time. Whenever you use the Data, you agree to the current relevant terms or license. Some Data will require that you provide attribution to the data source. Other Data may be protected by US or international copyright laws, treaties, or conventions. The Data and associated metadata are provided 'as-is', without express or implied warranty of any kind, including, but not limited to, infringement, merchantability and fitness for a particular purpose. CartoDB is not responsible for the accuracy, completeness, timeliness or quality of the Data._
|
||||
|
||||
Name | Terms link
|
||||
-------|---------
|
||||
@ -14,5 +16,5 @@ Natural Earth | [http://www.naturalearthdata.com/about/terms-of-use/](http://w
|
||||
Quattroshapes | [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
|
||||
Zetashapes | [http://zetashapes.com/license](http://zetashapes.com/license)
|
||||
Spielman & Singleton | [https://www.openicpsr.org/repoEntity/show/41329](https://www.openicpsr.org/repoEntity/show/41329)
|
||||
Instituto Nacional de Estadistica | [http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
|
||||
El Instituto Nacional de Estadística (INE) | The National Statistics Institute (INE) of Spain includes data from multiple sources. If you are re-using their data, they explicitly require that you reference them accordingly<br /><br />[http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
|
||||
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
[Data Observatory Measures](/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
|
||||
|
||||
There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
|
||||
There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
|
||||
|
||||
You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
|
||||
|
||||
@ -15,8 +15,8 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
|
||||
Name |Description
|
||||
--- | ---
|
||||
point | a WGS84 point geometry (the_geom)
|
||||
measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
|
||||
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://cartodb.github.io/bigmetadata/index.html) (optional)
|
||||
measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
|
||||
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
|
||||
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
|
||||
time_span | time span of interest (e.g., 2010 - 2014)
|
||||
|
||||
@ -46,8 +46,8 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
|
||||
Name |Description
|
||||
--- | ---
|
||||
polygon | a WGS84 polygon geometry (the_geom)
|
||||
measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
|
||||
normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
|
||||
measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
|
||||
normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
|
||||
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
|
||||
time_span | time span of interest (e.g., 2010 - 2014)
|
||||
|
||||
@ -70,7 +70,7 @@ SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
|
||||
|
||||
## OBS_GetMeasure(point geometry, measure_id text)
|
||||
|
||||
The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf)).
|
||||
The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
|
||||
|
||||
#### Arguments
|
||||
|
||||
@ -78,7 +78,7 @@ Name |Description
|
||||
--- | ---
|
||||
point | a WGS84 point geometry (the_geom)
|
||||
measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)). It is important to note that these are different than 'measure_name' used in the Census based functions above.
|
||||
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf). (optional)
|
||||
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf). (optional)
|
||||
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
|
||||
time_span | time span of interest (e.g., 2010 - 2014)
|
||||
|
||||
@ -109,7 +109,7 @@ Name |Description
|
||||
--- | ---
|
||||
polygon_geometry | a WGS84 polygon geometry (the_geom)
|
||||
measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))
|
||||
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
|
||||
normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
|
||||
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
|
||||
time_span | time span of interest (e.g., 2010 - 2014)
|
||||
|
||||
@ -134,9 +134,43 @@ SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
|
||||
|
||||
* If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
|
||||
|
||||
## OBS_GetMeasureById(geom_ref text, measure_id text, boundary_id text)
|
||||
|
||||
The ```OBS_GetMeasureById(geom_ref, measure_id, boundary_id)``` function returns any Data Observatory measure that corresponds to the boundary in ```boundary_id``` that has a geometry reference of ```geom_ref```.
|
||||
|
||||
#### Arguments
|
||||
|
||||
Name |Description
|
||||
--- | ---
|
||||
geom_ref | a geometry reference (e.g., a US Census geoid)
|
||||
measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))
|
||||
boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
|
||||
time_span (optional) | time span of interest (e.g., 2010 - 2014). If `NULL` is passed, the measure from the most recent data will be used.
|
||||
|
||||
#### Returns
|
||||
|
||||
A NUMERIC value
|
||||
|
||||
Key | Description
|
||||
--- | ---
|
||||
value | the raw measure associated with `geom_ref`
|
||||
|
||||
#### Example
|
||||
|
||||
Add a measure to an empty column based on county geoids in your table
|
||||
|
||||
```SQL
|
||||
UPDATE tablename
|
||||
SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001', 'us.census.tiger.county')
|
||||
```
|
||||
|
||||
#### Errors
|
||||
|
||||
* Returns `NULL` if there is a mismatch between the geometry reference and the boundary id such as using the geoid of a county with the boundary of block groups
|
||||
|
||||
## OBS_GetCategory(point geometry, category_id text)
|
||||
|
||||
The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) for more detail.
|
||||
The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) for more detail.
|
||||
|
||||
#### Arguments
|
||||
|
||||
|
||||
1799
release/observatory--0.0.5.sql
Normal file
1799
release/observatory--0.0.5.sql
Normal file
File diff suppressed because it is too large
Load Diff
1816
release/observatory--1.0.0.sql
Normal file
1816
release/observatory--1.0.0.sql
Normal file
File diff suppressed because it is too large
Load Diff
1816
release/observatory--1.0.1.sql
Normal file
1816
release/observatory--1.0.1.sql
Normal file
File diff suppressed because it is too large
Load Diff
@ -1,5 +1,5 @@
|
||||
comment = 'CartoDB Observatory backend extension'
|
||||
default_version = '0.0.4'
|
||||
default_version = '1.0.1'
|
||||
requires = 'postgis'
|
||||
superuser = true
|
||||
schema = cdb_observatory
|
||||
|
||||
@ -24,8 +24,9 @@ def query(q, is_meta=False, **options):
|
||||
params['api_key'] = META_API_KEY if is_meta else API_KEY
|
||||
return requests.get(url, params=params)
|
||||
|
||||
MEASURE_COLUMNS = [(r['id'], ) for r in query('''
|
||||
SELECT id FROM obs_column
|
||||
MEASURE_COLUMNS = [(r['id'], r['point_only'], ) for r in query('''
|
||||
SELECT id, aggregate NOT ILIKE 'sum' as point_only
|
||||
FROM obs_column
|
||||
WHERE type ILIKE 'numeric'
|
||||
AND weight > 0
|
||||
''', is_meta=True).json()['rows']]
|
||||
@ -42,7 +43,16 @@ WHERE type ILIKE 'geometry'
|
||||
AND weight > 0
|
||||
''', is_meta=True).json()['rows']]
|
||||
|
||||
def default_point(column_id):
|
||||
US_CENSUS_MEASURE_COLUMNS = [(r['name'], ) for r in query('''
|
||||
SELECT c.name FROM obs_column c, obs_column_tag ct
|
||||
WHERE type ILIKE 'numeric'
|
||||
AND c.id = ct.column_id
|
||||
AND ct.tag_id LIKE 'us.census%'
|
||||
AND weight > 0
|
||||
''', is_meta=True).json()['rows']]
|
||||
|
||||
|
||||
def default_geometry_id(column_id):
|
||||
'''
|
||||
Returns default test point for the column_id.
|
||||
'''
|
||||
@ -63,8 +73,71 @@ def default_point(column_id):
|
||||
return 'CDB_LatLng(40.7, -73.9)'
|
||||
|
||||
|
||||
def default_point(column_id):
|
||||
'''
|
||||
Returns default test point for the column_id.
|
||||
'''
|
||||
if column_id == 'whosonfirst.wof_disputed_geom':
|
||||
return 'CDB_LatLng(33.78, 76.57)'
|
||||
elif column_id == 'whosonfirst.wof_marinearea_geom':
|
||||
return 'CDB_LatLng(43.33, -68.47)'
|
||||
elif column_id in ('us.census.tiger.school_district_elementary',
|
||||
'us.census.tiger.school_district_secondary',
|
||||
'us.census.tiger.school_district_elementary_clipped',
|
||||
'us.census.tiger.school_district_secondary_clipped'):
|
||||
return 'CDB_LatLng(40.7025, -73.7067)'
|
||||
elif column_id.startswith('uk'):
|
||||
if 'WA' in column_id:
|
||||
return 'CDB_LatLng(51.46844551219723, -3.184833526611328)'
|
||||
else:
|
||||
return 'CDB_LatLng(51.51461834694225, -0.08883476257324219)'
|
||||
elif column_id.startswith('es'):
|
||||
return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
|
||||
elif column_id.startswith('us.zillow'):
|
||||
return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
|
||||
else:
|
||||
return 'CDB_LatLng(40.7, -73.9)'
|
||||
|
||||
|
||||
def default_area(column_id):
|
||||
'''
|
||||
Returns default test area for the column_id
|
||||
'''
|
||||
point = default_point(column_id)
|
||||
area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 1000), 4326)'.format(
|
||||
point=point)
|
||||
return area
|
||||
|
||||
@parameterized(US_CENSUS_MEASURE_COLUMNS)
|
||||
def test_get_us_census_measure_points(name):
|
||||
resp = query('''
|
||||
SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
|
||||
'''.format(name=name.replace("'", "''"),
|
||||
schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
point=default_point('')))
|
||||
assert_equal(resp.status_code, 200)
|
||||
rows = resp.json()['rows']
|
||||
assert_equal(1, len(rows))
|
||||
assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
|
||||
@parameterized(MEASURE_COLUMNS)
|
||||
def test_measure_points(column_id):
|
||||
def test_get_measure_areas(column_id, point_only):
|
||||
if point_only:
|
||||
return
|
||||
resp = query('''
|
||||
SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
|
||||
'''.format(column_id=column_id,
|
||||
schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
area=default_area(column_id)))
|
||||
assert_equal(resp.status_code, 200)
|
||||
rows = resp.json()['rows']
|
||||
assert_equal(1, len(rows))
|
||||
assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
|
||||
@parameterized(MEASURE_COLUMNS)
|
||||
def test_get_measure_points(column_id, point_only):
|
||||
resp = query('''
|
||||
SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
|
||||
'''.format(column_id=column_id,
|
||||
@ -75,8 +148,20 @@ SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
|
||||
assert_equal(1, len(rows))
|
||||
assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
#@parameterized(CATEGORY_COLUMNS)
|
||||
#def test_get_category_areas(column_id):
|
||||
# resp = query('''
|
||||
#SELECT * FROM {schema}OBS_GetCategory({area}, '{column_id}')
|
||||
# '''.format(column_id=column_id,
|
||||
# schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
# area=default_area(column_id)))
|
||||
# assert_equal(resp.status_code, 200)
|
||||
# rows = resp.json()['rows']
|
||||
# assert_equal(1, len(rows))
|
||||
# assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
@parameterized(CATEGORY_COLUMNS)
|
||||
def test_category_points(column_id):
|
||||
def test_get_category_points(column_id):
|
||||
resp = query('''
|
||||
SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
|
||||
'''.format(column_id=column_id,
|
||||
@ -87,14 +172,62 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
|
||||
assert_equal(1, len(rows))
|
||||
assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
@parameterized(BOUNDARY_COLUMNS)
|
||||
def test_boundary_points(column_id):
|
||||
resp = query('''
|
||||
SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
|
||||
'''.format(column_id=column_id,
|
||||
schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
point=default_point(column_id)))
|
||||
assert_equal(resp.status_code, 200)
|
||||
rows = resp.json()['rows']
|
||||
assert_equal(1, len(rows))
|
||||
assert_is_not_none(rows[0].values()[0])
|
||||
#@parameterized(BOUNDARY_COLUMNS)
|
||||
#def test_get_boundaries_by_geometry(column_id):
|
||||
# resp = query('''
|
||||
#SELECT * FROM {schema}OBS_GetBoundariesByGeometry({area}, '{column_id}')
|
||||
# '''.format(column_id=column_id,
|
||||
# schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
# area=default_area(column_id)))
|
||||
# assert_equal(resp.status_code, 200)
|
||||
# rows = resp.json()['rows']
|
||||
# assert_equal(1, len(rows))
|
||||
# assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
#@parameterized(BOUNDARY_COLUMNS)
|
||||
#def test_get_points_by_geometry(column_id):
|
||||
# resp = query('''
|
||||
#SELECT * FROM {schema}OBS_GetPointsByGeometry({area}, '{column_id}')
|
||||
# '''.format(column_id=column_id,
|
||||
# schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
# area=default_area(column_id)))
|
||||
# assert_equal(resp.status_code, 200)
|
||||
# rows = resp.json()['rows']
|
||||
# assert_equal(1, len(rows))
|
||||
# assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
#@parameterized(BOUNDARY_COLUMNS)
|
||||
#def test_get_boundary_points(column_id):
|
||||
# resp = query('''
|
||||
#SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
|
||||
# '''.format(column_id=column_id,
|
||||
# schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
# point=default_point(column_id)))
|
||||
# assert_equal(resp.status_code, 200)
|
||||
# rows = resp.json()['rows']
|
||||
# assert_equal(1, len(rows))
|
||||
# assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
#@parameterized(BOUNDARY_COLUMNS)
|
||||
#def test_get_boundary_id(column_id):
|
||||
# resp = query('''
|
||||
#SELECT * FROM {schema}OBS_GetBoundaryId({point}, '{column_id}')
|
||||
# '''.format(column_id=column_id,
|
||||
# schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
# point=default_point(column_id)))
|
||||
# assert_equal(resp.status_code, 200)
|
||||
# rows = resp.json()['rows']
|
||||
# assert_equal(1, len(rows))
|
||||
# assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
#@parameterized(BOUNDARY_COLUMNS)
|
||||
#def test_get_boundary_by_id(column_id):
|
||||
# resp = query('''
|
||||
#SELECT * FROM {schema}OBS_GetBoundaryById({geometry_id}, '{column_id}')
|
||||
# '''.format(column_id=column_id,
|
||||
# schema='cdb_observatory.' if USE_SCHEMA else '',
|
||||
# geometry_id=default_geometry_id(column_id)))
|
||||
# assert_equal(resp.status_code, 200)
|
||||
# rows = resp.json()['rows']
|
||||
# assert_equal(1, len(rows))
|
||||
# assert_is_not_none(rows[0].values()[0])
|
||||
|
||||
@ -33,7 +33,8 @@ def select_star(tablename):
|
||||
|
||||
cdb = Dumpr('observatory.cartodb.com','')
|
||||
|
||||
metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag', 'obs_tag', 'obs_column_to_column']
|
||||
metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
|
||||
'obs_tag', 'obs_column_to_column', 'obs_dump_version']
|
||||
|
||||
fixtures = [
|
||||
('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
comment = 'CartoDB Observatory backend extension'
|
||||
default_version = '0.0.4'
|
||||
default_version = '1.0.1'
|
||||
requires = 'postgis'
|
||||
superuser = true
|
||||
schema = cdb_observatory
|
||||
|
||||
@ -199,3 +199,20 @@ BEGIN
|
||||
RETURN result;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
-- Function that returns the currently deployed obs_dump_version from the
|
||||
-- remote table of the same name.
|
||||
|
||||
CREATE OR REPLACE FUNCTION cdb_observatory.OBS_DumpVersion(
|
||||
)
|
||||
RETURNS TEXT
|
||||
AS $$
|
||||
DECLARE
|
||||
result text;
|
||||
BEGIN
|
||||
EXECUTE '
|
||||
SELECT MAX(dump_id) FROM observatory.obs_dump_version
|
||||
' INTO result;
|
||||
RETURN result;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
@ -380,6 +380,53 @@ END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
|
||||
CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMeasureById(
|
||||
geom_ref TEXT,
|
||||
measure_id TEXT,
|
||||
boundary_id TEXT,
|
||||
time_span TEXT DEFAULT NULL
|
||||
)
|
||||
RETURNS NUMERIC
|
||||
AS $$
|
||||
DECLARE
|
||||
target_table TEXT;
|
||||
colname TEXT;
|
||||
measure_val NUMERIC;
|
||||
data_geoid_colname TEXT;
|
||||
test_query TEXT;
|
||||
BEGIN
|
||||
|
||||
SELECT x ->> 'colname', x ->> 'tablename' INTO colname, target_table
|
||||
FROM cdb_observatory._OBS_GetColumnData(boundary_id, Array[measure_id], time_span) As x;
|
||||
|
||||
EXECUTE
|
||||
format('SELECT ct.colname
|
||||
FROM observatory.obs_column_to_column c2c,
|
||||
observatory.obs_column_table ct,
|
||||
observatory.obs_table t
|
||||
WHERE c2c.reltype = ''geom_ref''
|
||||
AND ct.column_id = c2c.source_id
|
||||
AND ct.table_id = t.id
|
||||
AND t.tablename = %L'
|
||||
, target_table)
|
||||
INTO data_geoid_colname;
|
||||
|
||||
RAISE DEBUG 'target_table %, colname %', target_table, colname;
|
||||
|
||||
EXECUTE format(
|
||||
'SELECT %I
|
||||
FROM observatory.%I
|
||||
WHERE %I.%I = %L',
|
||||
colname,
|
||||
target_table,
|
||||
target_table, data_geoid_colname, geom_ref)
|
||||
INTO measure_val;
|
||||
|
||||
RETURN measure_val;
|
||||
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetCategory(
|
||||
geom geometry(Geometry, 4326),
|
||||
category_id TEXT,
|
||||
@ -586,7 +633,7 @@ BEGIN
|
||||
q := q || q_select || format('FROM observatory.%I ', ((data_table_info)[1]->>'tablename'));
|
||||
|
||||
q := format(q || ' ) ' || q_sum || ' ]::numeric[] FROM _overlaps, values
|
||||
WHERE values.%I = _overlaps.%I', geom_geoid_colname, geom_geoid_colname);
|
||||
WHERE values.%I = _overlaps.%I', data_geoid_colname, geom_geoid_colname);
|
||||
|
||||
EXECUTE
|
||||
q
|
||||
|
||||
@ -114,7 +114,7 @@ BEGIN
|
||||
AND
|
||||
observatory.OBS_column.type = 'Geometry'
|
||||
AND
|
||||
$1 && bounds::box2d
|
||||
ST_Intersects($1, st_setsrid(observatory.obs_table.the_geom, 4326))
|
||||
$string$ || timespan_query
|
||||
USING geom;
|
||||
RETURN;
|
||||
|
||||
@ -27,3 +27,6 @@ t
|
||||
_obs_standardizemeasurename_test
|
||||
t
|
||||
(1 row)
|
||||
obs_dumpversion_notnull
|
||||
t
|
||||
(1 row)
|
||||
|
||||
@ -76,3 +76,15 @@ t
|
||||
obs_getuscensuscategory_polygon
|
||||
t
|
||||
(1 row)
|
||||
obs_getmeasurebyid_cartodb_census_tract
|
||||
t
|
||||
(1 row)
|
||||
obs_getmeasurebyid_null_boundary_null_timespan
|
||||
t
|
||||
(1 row)
|
||||
obs_getmeasurebyid_cartodb_block_group
|
||||
t
|
||||
(1 row)
|
||||
obs_getmeasurebyid_nulls
|
||||
t
|
||||
(1 row)
|
||||
|
||||
1
src/pg/test/fixtures/drop_fixtures.sql
vendored
1
src/pg/test/fixtures/drop_fixtures.sql
vendored
@ -6,6 +6,7 @@ DROP TABLE IF EXISTS observatory.obs_column;
|
||||
DROP TABLE IF EXISTS observatory.obs_column_tag;
|
||||
DROP TABLE IF EXISTS observatory.obs_tag;
|
||||
DROP TABLE IF EXISTS observatory.obs_column_to_column;
|
||||
DROP TABLE IF EXISTS observatory.obs_dump_version;
|
||||
DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
|
||||
DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
|
||||
DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
|
||||
|
||||
18902
src/pg/test/fixtures/load_fixtures.sql
vendored
18902
src/pg/test/fixtures/load_fixtures.sql
vendored
File diff suppressed because one or more lines are too long
@ -75,4 +75,7 @@ SELECT cdb_observatory._OBS_GetRelatedColumn(
|
||||
-- should give back a standardized measure name
|
||||
SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') = 'test_343_2_qqq' As _OBS_StandardizeMeasureName_test;
|
||||
|
||||
SELECT cdb_observatory.OBS_DumpVersion()
|
||||
IS NOT NULL AS OBS_DumpVersion_notnull;
|
||||
|
||||
\i test/fixtures/drop_fixtures.sql
|
||||
|
||||
@ -204,4 +204,37 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
|
||||
cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Low income, mix of minorities' As OBS_GetUSCensusCategory_polygon;
|
||||
|
||||
|
||||
-- OBS_GetMeasureById tests
|
||||
-- typical query
|
||||
SELECT (cdb_observatory.OBS_GetMeasureById(
|
||||
'36047048500',
|
||||
'us.census.acs.B01003001',
|
||||
'us.census.tiger.census_tract',
|
||||
'2010 - 2014'
|
||||
) - 3241) / 3241 < 0.0001 As OBS_GetMeasureById_cartodb_census_tract;
|
||||
|
||||
-- no boundary_id should give null
|
||||
SELECT cdb_observatory.OBS_GetMeasureById(
|
||||
'36047048500',
|
||||
'us.census.acs.B01003001',
|
||||
NULL,
|
||||
NULL
|
||||
) IS NULL As OBS_GetMeasureById_null_boundary_null_timespan;
|
||||
|
||||
-- query at block_group level
|
||||
SELECT (cdb_observatory.OBS_GetMeasureById(
|
||||
'360470485002',
|
||||
'us.census.acs.B01003001',
|
||||
'us.census.tiger.block_group',
|
||||
'2010 - 2014'
|
||||
) - 1900) / 1900 < 0.0001 As OBS_GetMeasureById_cartodb_block_group;
|
||||
|
||||
-- geom ref / boundary mismatch
|
||||
SELECT cdb_observatory.OBS_GetMeasureById(
|
||||
'36047048500',
|
||||
'us.census.acs.B01003001',
|
||||
'us.census.tiger.block_group',
|
||||
'2010 - 2014'
|
||||
) IS NULL As OBS_GetMeasureById_nulls;
|
||||
|
||||
\i test/fixtures/drop_fixtures.sql
|
||||
|
||||
Loading…
Reference in New Issue
Block a user