cartodb4/observatory-extension
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #332 from CartoDB/develop

Browse Source 
Release 1.9.0
This commit is contained in:
Mario de Frutos 2018-04-20 10:57:20 +02:00 committed by GitHub
commit a0535a6d02
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
25 changed files with 189443 additions and 187211 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@ -2,6 +2,11 @@
I'd like to request a new data observatory extension deploy: dump + extension
**VERY IMPORTANT!!!**
PLEASE USE `python scripts/generate_fixtures.py` TO GENERATE NEW FIXTURES FOR
THE NEW DUMP AND OVERRIDE IT IN THIS PROJECT BEFORE PASS THE TESTS
## Performance comparison to last deployment
Please include link here to comparison perftests:
@ -20,4 +25,4 @@ Add down here the PR links to be added and deployed:
-
// @CartoDB/dataservices
// @CartoDB/datateam

40
.travis.yml
View File

@ -1,43 +1,23 @@
language: c
dist: precise
group: travis_stable
sudo: required
env:
global:
- PAGER=cat
matrix:
include:
- env: PGSQL_VERSION=9.5 POSTGIS_VERSION=2.2
- env: PGSQL_VERSION=9.6 POSTGIS_VERSION=2.3
- env: PGSQL_VERSION=10 POSTGIS_VERSION=2.4
before_install:
- sudo add-apt-repository -y ppa:cartodb/postgresql-9.5
- sudo add-apt-repository -y ppa:cartodb/gis
- sudo add-apt-repository -y ppa:cartodb/gis-testing
- sudo apt-get update
# Install postgres db and build deps
- sudo /etc/init.d/postgresql stop # stop travis default instance
- sudo apt-get -y remove --purge postgresql-9.1
- sudo apt-get -y remove --purge postgresql-9.2
- sudo apt-get -y remove --purge postgresql-9.3
- sudo apt-get -y remove --purge postgresql-9.4
- sudo apt-get -y remove --purge postgresql-9.5
- sudo rm -rf /var/lib/postgresql/
- sudo rm -rf /var/log/postgresql/
- sudo rm -rf /etc/postgresql/
- sudo apt-get -y remove --purge postgis-2.2
- sudo apt-get -y autoremove
- sudo apt-get -y install postgresql-9.5=9.5.2-3cdb3
- sudo apt-get -y install postgresql-server-dev-9.5=9.5.2-3cdb3
- sudo apt-get -y install postgresql-plpython-9.5=9.5.2-3cdb3
- sudo apt-get -y install postgresql-9.5-postgis-scripts=2.2.2.0-cdb2
- sudo apt-get -y install postgresql-9.5-postgis-2.2=2.2.2.0-cdb2
# configure it to accept local connections from postgres
- echo -e "# TYPE DATABASE USER ADDRESS METHOD \nlocal all postgres trust\nlocal all all trust\nhost all all 127.0.0.1/32 trust" \
| sudo tee /etc/postgresql/9.5/main/pg_hba.conf
- sudo /etc/init.d/postgresql restart 9.5
- sudo bash $TRAVIS_BUILD_DIR/scripts/ci/install_postgres.sh
install:
- sudo make install
script:
- cd src/pg
- make test || { cat src/pg/test/regression.diffs; false; }
- make test || { cat $TRAVIS_BUILD_DIR/src/pg/test/regression.diffs; false; }

4
CONTRIBUTING.md
View File

@ -28,8 +28,8 @@ Run the tests with `make test`.
Update extension in a working database with:
```
ALTER EXTENSION observatory VERSION TO 'current';
ALTER EXTENSION observatory VERSION TO 'dev';
ALTER EXTENSION observatory UPDATE TO 'current';
ALTER EXTENSION observatory UPDATE TO 'dev';
```
Note: we keep the current development version install as 'dev' always;

12
NEWS.md
View File

@ -1,3 +1,15 @@
1.9.0 (2018-04-20)
------------------
__Improvements__
* Improved `OBS_GetAvailableGeometries` for the DO Timespans project ([#325](https://github.com/CartoDB/observatory-extension/pull/325))
* Improved `OBS_GetAvailableTimespans` for the DO Timespans project ([#324](https://github.com/CartoDB/bigmetadata/issues/324))
* Modified the denominated suggested_name to mitigate collisions ([#327](https://github.com/CartoDB/observatory-extension/pull/327))
* Fixed some errors so now the extension supports PostgreSQL 10 ([#329](https://github.com/CartoDB/observatory-extension/pull/329))
* Fixed documentation
* Add support for multiple PostgreSQL and Postgis versions in our travis script for test purposes
1.8.0 (2017-10-18)
------------------

40
doc/discovery_functions.md
View File

@ -97,7 +97,7 @@ valid_timespan | Boolean | True if the `timespan` argument is a valid timespan f
Obtain all numerators that are available within a small rectangle.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
SELECT * FROM OBS_GetAvailableNumerators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326))
```
@ -105,7 +105,7 @@ Obtain all numerators that are available within a small rectangle and are for
the United States only.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
SELECT * FROM OBS_GetAvailableNumerators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
```
@ -113,7 +113,7 @@ Obtain all numerators that are available within a small rectangle and are
employment related for the United States only.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
SELECT * FROM OBS_GetAvailableNumerators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment}');
```
@ -121,7 +121,7 @@ Obtain all numerators that are available within a small rectangle and are
related to both employment and age & gender for the United States only.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
SELECT * FROM OBS_GetAvailableNumerators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment, subsection/tags.age_gender}');
```
@ -129,7 +129,7 @@ Obtain all numerators that work with US population (`us.census.acs.B01003001`)
as a denominator.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
SELECT * FROM OBS_GetAvailableNumerators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
WHERE valid_denom IS True;
```
@ -138,7 +138,7 @@ Obtain all numerators that work with US states (`us.census.tiger.state`)
as a geometry.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
SELECT * FROM OBS_GetAvailableNumerators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
WHERE valid_geom IS True;
```
@ -146,7 +146,7 @@ WHERE valid_geom IS True;
Obtain all numerators available in the timespan `2011 - 2015`.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
SELECT * FROM OBS_GetAvailableNumerators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
WHERE valid_timespan IS True;
```
@ -191,7 +191,7 @@ valid_timespan | Boolean | True if the `timespan` argument is a valid timespan f
Obtain all denominators that are available within a small rectangle.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
SELECT * FROM OBS_GetAvailableDenominators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326));
```
@ -199,14 +199,14 @@ Obtain all denominators that are available within a small rectangle and are for
the United States only.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
SELECT * FROM OBS_GetAvailableDenominators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
```
Obtain all denominators for male population (`us.census.acs.B01001002`).
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
SELECT * FROM OBS_GetAvailableDenominators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01001002')
WHERE valid_numer IS True;
```
@ -215,7 +215,7 @@ Obtain all denominators that work with US states (`us.census.tiger.state`)
as a geometry.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
SELECT * FROM OBS_GetAvailableDenominators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
WHERE valid_geom IS True;
```
@ -223,7 +223,7 @@ WHERE valid_geom IS True;
Obtain all denominators available in the timespan `2011 - 2015`.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
SELECT * FROM OBS_GetAvailableDenominators(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
WHERE valid_timespan IS True;
```
@ -242,7 +242,7 @@ filter_tags | Text[] | a list of filters. Only geometries for which all of thes
numer_id | Text | the ID of a numerator to check whether the geometry is valid against. Will not reduce length of returned table, but will change values for `valid_numer` (optional)
denom_id | Text | the ID of a denominator to check whether the geometry is valid against. Will not reduce length of returned table, but will change values for `valid_denom` (optional)
timespan | Text | the ID of a timespan to check whether the geometry is valid against. Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
number_geometries | Integer | Number of geometries of the source data in order to calculate more accurately the score value to know which geometry fits better with the provided extent. (optional)
number_geometries | Integer | an additional variable that is used to adjust the calculation of the [score](https://carto.com/docs/carto-engine/data/discovery-functions/#returns-4) (optional)
#### Returns
@ -275,7 +275,7 @@ meanmediansize | Numeric | Ignored
Obtain all geometries that are available within a small rectangle.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
SELECT * FROM OBS_GetAvailableGeometries(
ST_MakeEnvelope(-74, 41, -73, 40, 4326));
```
@ -283,14 +283,14 @@ Obtain all geometries that are available within a small rectangle and are for
the United States only.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
SELECT * FROM OBS_GetAvailableGeometries(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
```
Obtain all geometries that work with total population (`us.census.acs.B01003001`).
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
SELECT * FROM OBS_GetAvailableGeometries(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
WHERE valid_numer IS True;
```
@ -298,7 +298,7 @@ WHERE valid_numer IS True;
Obtain all geometries with timespan `2015`.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
SELECT * FROM OBS_GetAvailableGeometries(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2015')
WHERE valid_timespan IS True;
```
@ -343,14 +343,14 @@ valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for th
Obtain all timespans that are available within a small rectangle.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
SELECT * FROM OBS_GetAvailableTimespans(
ST_MakeEnvelope(-74, 41, -73, 40, 4326));
```
Obtain all timespans for total population (`us.census.acs.B01003001`).
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
SELECT * FROM OBS_GetAvailableTimespans(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
WHERE valid_numer IS True;
```
@ -359,7 +359,7 @@ Obtain all timespans that work with US states (`us.census.tiger.state`)
as a geometry.
```SQL
SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
SELECT * FROM OBS_GetAvailableTimespans(
ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, 'us.census.tiger.state')
WHERE valid_geom IS True;
```

2445
release/observatory--1.9.0.sql Normal file
View File

File diff suppressed because one or more lines are too long

2
release/observatory.control
View File

@ -1,5 +1,5 @@
comment = 'CartoDB Observatory backend extension'
default_version = '1.8.0'
default_version = '1.9.0'
requires = 'postgis'
superuser = true
schema = cdb_observatory

38
scripts/ci/install_postgres.sh Normal file
View File

@ -0,0 +1,38 @@
#!/bin/bash
# echo commands
set -x
# exit on error
set -e
dpkg -l | grep postgresql
# Add the PDGD repository
apt-key adv --keyserver keys.gnupg.net --recv-keys 7FCC7D46ACCC4CF8
add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
apt-get update
# Remove those all PgSQL versions except the one we're testing
PGSQL_VERSIONS=(9.2 9.3 9.4 9.5 9.6 10)
/etc/init.d/postgresql stop # stop travis default instance
for V in "${PGSQL_VERSIONS[@]}"; do
if [ "$V" != "$PGSQL_VERSION" ]; then
apt-get -y remove --purge postgresql-${V} postgresql-client-${V} postgresql-contrib-${V} postgresql-${V}-postgis-2.3-scripts
else
apt-get -y remove --purge postgresql-${V}-postgis-2.3-scripts
fi
done
apt-get -y autoremove
# Install PostgreSQL
apt-get -y install postgresql-${PGSQL_VERSION} postgresql-${PGSQL_VERSION}-postgis-${POSTGIS_VERSION} postgresql-server-dev-${PGSQL_VERSION}
# Configure it to accept local connections from postgres
echo -e "# TYPE DATABASE USER ADDRESS METHOD \nlocal all postgres trust\nlocal all all trust\nhost all all 127.0.0.1/32 trust" > /etc/postgresql/${PGSQL_VERSION}/main/pg_hba.conf
# Restart PostgreSQL
/etc/init.d/postgresql restart ${PGSQL_VERSION}
dpkg -l | grep postgresql

60
scripts/generate_fixtures.py
View File

@ -2,12 +2,18 @@ import os
import psycopg2
import subprocess
PGUSER = os.environ.get('PGUSER', 'postgres')
PGPASSWORD = os.environ.get('PGPASSWORD', '')
PGHOST=os.environ.get('PGHOST', 'localhost')
PGPORT=os.environ.get('PGPORT', '5432')
PGDATABASE=os.environ.get('PGDATABASE', 'postgres')
DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
user=os.environ.get('PGUSER', 'postgres'),
password=os.environ.get('PGPASSWORD', ''),
host=os.environ.get('PGHOST', 'localhost'),
port=os.environ.get('PGPORT', '5432'),
database=os.environ.get('PGDATABASE', 'postgres'),
user=PGUSER,
password=PGPASSWORD,
host=PGHOST,
port=PGPORT,
database=PGDATABASE
))
CURSOR = DB_CONN.cursor()
@ -51,8 +57,8 @@ def get_tablename_query(column_id, boundary_id, timespan):
METADATA_TABLES = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
'obs_tag', 'obs_column_to_column', 'obs_dump_version', 'obs_meta',
'obs_meta_numer', 'obs_meta_denom', 'obs_meta_geom',
'obs_meta_timespan', 'obs_meta_geom_numer_timespan',
'obs_table_to_table', 'obs_meta_numer', 'obs_meta_denom',
'obs_meta_geom', 'obs_meta_timespan', 'obs_meta_geom_numer_timespan',
'obs_column_table_tile', 'obs_column_table_tile_simple']
FIXTURES = [
@ -207,15 +213,13 @@ FIXTURES = [
('us.census.spielman_singleton_segments.X55', 'us.census.tiger.census_tract', '2010 - 2014'),
('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-06'),
('whosonfirst.wof_country_name', 'whosonfirst.wof_country_geom', '2016'),
('us.census.acs.B01003001', 'us.census.tiger.zcta5_clipped', '2010 - 2014'),
('us.census.acs.B01003001', 'us.census.tiger.block_group_clipped', '2010 - 2014'),
('us.census.acs.B01003001', 'us.census.tiger.census_tract_clipped', '2010 - 2014'),
('us.census.tiger.fullname', 'us.census.tiger.pointlm_geom', '2016'),
('us.census.tiger.fullname', 'us.census.tiger.prisecroads_geom', '2016'),
('us.census.tiger.name', 'us.census.tiger.county', '2015'),
('us.census.tiger.name', 'us.census.tiger.county_clipped', '2015'),
('us.census.tiger.name', 'us.census.tiger.block_group', '2015'),
('us.census.acs.B01003001', 'us.census.tiger.zcta5', '2010 - 2014'),
('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
('us.census.tiger.place_geoname', 'us.census.tiger.place_clipped', '2015'),
('us.census.tiger.county_geoname', 'us.census.tiger.county_clipped', '2015'),
('us.census.tiger.county_geoname', 'us.census.tiger.county', '2015'),
('us.census.tiger.block_group_geoname', 'us.census.tiger.block_group', '2015'),
]
OUTFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
@ -230,27 +234,35 @@ def dump(cols, tablename, where=''):
tablename=tablename,
))
subprocess.check_call('pg_dump -x --section=pre-data -t observatory.{tablename} '
subprocess.check_call('PGPASSWORD={pgpassword} PGUSER={pguser} PGHOST={pghost} PGDATABASE={pgdb} '
'pg_dump -x --section=pre-data -t observatory.{tablename} '
' | sed "s:SET search_path.*::" '
' | sed "s:CREATE TABLE :CREATE TABLE observatory.:" '
' | sed "s:ALTER TABLE.*OWNER.*::" '
' | sed "s:SET idle_in_transaction_session_timeout.*::" '
' >> {outfile}'.format(
tablename=tablename,
outfile=OUTFILE_PATH,
pgpassword=PGPASSWORD,
pghost=PGHOST,
pgdb=PGDATABASE,
pguser=PGUSER
), shell=True)
with open(OUTFILE_PATH, 'a') as outfile:
outfile.write('COPY observatory."{}" FROM stdin WITH CSV HEADER;\n'.format(tablename))
subprocess.check_call('''
psql -c "COPY (SELECT {cols} \
PGPASSWORD={pgpassword} psql -U {pguser} -d {pgdb} -h {pghost} -c "COPY (SELECT {cols} \
FROM observatory.{tablename} {where}) \
TO STDOUT WITH CSV HEADER" >> {outfile}'''.format(
cols=cols,
tablename=tablename,
where=where,
outfile=OUTFILE_PATH,
pgpassword=PGPASSWORD,
pghost=PGHOST,
pgdb=PGDATABASE,
pguser=PGUSER
), shell=True)
with open(OUTFILE_PATH, 'a') as outfile:
@ -347,6 +359,10 @@ def main():
timespans=','.join(["'{}'".format(x) for _, _, x in FIXTURES]),
table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
)
elif tablename in ('obs_table_to_table'):
where = '''WHERE source_id IN ({table_ids})'''.format(
table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
)
else:
where = ''
dump('*', tablename, where)
@ -355,12 +371,6 @@ def main():
if 'zcta5' in table_id or 'zillow_zip' in table_id:
where = '\'11%\''
compare = 'LIKE'
elif 'pri_sec_roads' in table_id or 'point_landmark' in table_id:
dump('*', tablename, 'WHERE geom && ST_MakeEnvelope(-74,40.69,-73.9,40.72, 4326)')
continue
elif 'whosonfirst' in table_id:
where = "('85632785','85633051','85633111','85633147','85633253','85633267')"
compare = 'IN'
elif 'county' in table_id and 'tiger' in table_id:
where = "('48061', '36047')"
compare = 'IN'

2
src/pg/observatory.control
View File

@ -1,5 +1,5 @@
comment = 'CartoDB Observatory backend extension'
default_version = '1.8.0'
default_version = '1.9.0'
requires = 'postgis'
superuser = true
schema = cdb_observatory

8
src/pg/sql/41_observatory_augmentation.sql
View File

@ -249,10 +249,10 @@ BEGIN
'suggested_name', cdb_observatory.FIRST(
LOWER(TRIM(BOTH '_' FROM regexp_replace(CASE WHEN numer_id IS NOT NULL
THEN CASE
WHEN normalization ILIKE 'area%%' THEN numer_colname || ' per sq km'
WHEN normalization ILIKE 'denom%%' THEN numer_colname || ' rate'
ELSE numer_colname
END || ' ' || numer_timespan
WHEN normalization ILIKE 'area%%' THEN numer_colname || ' per sq km' || ' ' || numer_timespan
WHEN normalization ILIKE 'denom%%' THEN numer_colname || ' ' || numer_timespan || ' by ' || denom_colname
ELSE numer_colname || ' ' || numer_timespan
END
ELSE geom_name || ' ' || geom_timespan
END, '[^a-zA-Z0-9]+', '_', 'g')))
),

10
src/pg/sql/42_observatory_exploration.sql
View File

@ -383,7 +383,7 @@ BEGIN
CASE WHEN $1 IS NOT NULL AND $1 != '' THEN
EXISTS (SELECT 1 FROM observatory.obs_meta_geom_numer_timespan onu WHERE o.geom_id = onu.geom_id AND onu.numer_id = $1 AND ($3 = ANY(onu.timespans) OR $3 IN (select(unnest(o.timespans)))))
ELSE
EXISTS (SELECT 1 FROM observatory.obs_meta_geom_numer_timespan onu WHERE o.geom_id = onu.geom_id AND ($3 = ANY(onu.timespans) OR $3 IN (select(unnest(o.timespans)))))
EXISTS (SELECT 1 FROM observatory.obs_meta_geom_numer_timespan onu WHERE o.geom_id = onu.geom_id AND ($3 = ANY(onu.geom_timespans) OR $3 IN (select(unnest(o.timespans)))))
END
ELSE
false
@ -448,12 +448,12 @@ BEGIN
NULL::TEXT timespan_aggregate,
NULL::TEXT timespan_license,
NULL::TEXT timespan_source,
NULL::TEXT timespan_type,
timespan_type::TEXT,
NULL::JSONB timespan_extra,
NULL::JSONB timespan_tags,
$1 = ANY(numers) valid_numer,
$2 = ANY(denoms) valid_denom,
$3 = ANY(geoms) valid_geom_id
COALESCE($1 = ANY(numers), false) valid_numer,
COALESCE($2 = ANY(denoms), false) valid_denom,
COALESCE($3 = ANY(geoms), false) valid_geom_id
FROM observatory.obs_meta_timespan
WHERE %s (timespan_tags ?& $4 OR CARDINALITY($4) = 0)
$string$, geom_clause)

168
src/pg/test/expected/01_install_test.out
View File

@ -2,3 +2,171 @@
CREATE EXTENSION postgis;
-- Install the extension
CREATE EXTENSION observatory VERSION 'dev';
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\set ECHO none
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)
set_config
------------
(1 row)

2
src/pg/test/expected/40_observatory_utility_test.out
View File

@ -1,6 +1,5 @@
\pset format unaligned
\set ECHO all
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\set ECHO none
_obs_geomtable_with_returned_table
@ -21,7 +20,6 @@ t
obs_dumpversion_notnull
t
(1 row)
ERROR: Error performing intersection: TopologyException: found non-noded intersection between LINESTRING (-97.1968 25.9574, -97.1971 25.9576) and LINESTRING (-97.197 25.9575, -97.1972 25.9576) at -97.19699802694231 25.957551976080605
complex_safe_intersection_works
t
(1 row)

8
src/pg/test/expected/41_observatory_augmentation_test.out
View File

@ -240,12 +240,6 @@ t|t|t
id|correct_num_geoms|correct_pop|correct_bg_names
t|t|t|t
(1 row)
id|correct_num_points
t|t
(1 row)
id|correct_num_points|pointgeom_names
t|t|t
(1 row)
id|obs_getdata_by_id_one_measure_null
t|t
(1 row)
@ -299,7 +293,7 @@ tract_sample|tract_max_error|tract_avg_error|tract_min_error
25|t|t|t
50|t|t|t
100|t|t|t
761|t|t|t
741|t|t|t
(9 rows)
no_bg_point_error
t

10
src/pg/test/expected/42_observatory_exploration_test.out
View File

@ -232,15 +232,15 @@ us.census.tiger.zcta5|9
us.census.tiger.county|0
(4 rows)
column_id|_obs_geometryscores_numgeoms_50km_buffer
us.census.tiger.block_group|10817
us.census.tiger.block_group|10818
us.census.tiger.census_tract|3396
us.census.tiger.zcta5|484
us.census.tiger.zcta5|483
us.census.tiger.county|11
(4 rows)
column_id|_obs_geometryscores_numgeoms_500km_buffer
us.census.tiger.block_group|48567
us.census.tiger.census_tract|15823
us.census.tiger.zcta5|6466
us.census.tiger.block_group|48569
us.census.tiger.census_tract|15825
us.census.tiger.zcta5|6465
us.census.tiger.county|295
(4 rows)
column_id|_obs_geometryscores_numgeoms_2500km_buffer

9
src/pg/test/expected/44_observatory_geometries_test.out
View File

@ -42,12 +42,6 @@ t
obs_getboundarybyid_boundary_id_mismatch_geom_id
t
(1 row)
_obs_getboundariesbygeometry_roads_around_cartodb
t
(1 row)
_obs_getboundariesbygeometry_points_around_cartodb
t
(1 row)
_obs_getboundariesbygeometry_tracts_around_cartodb
t
(1 row)
@ -60,9 +54,6 @@ t
obs_getboundariesbygeometry_tracts_around_null_island
t
(1 row)
obs_getboundariesbygeometry_wof
t
(1 row)
obs_getboundariesbypointandradius_around_cartodb
t
(1 row)

32
src/pg/test/fixtures/drop_fixtures.sql vendored
View File

@ -8,6 +8,7 @@ DROP TABLE IF EXISTS observatory.obs_tag;
DROP TABLE IF EXISTS observatory.obs_column_to_column;
DROP TABLE IF EXISTS observatory.obs_dump_version;
DROP TABLE IF EXISTS observatory.obs_meta;
DROP TABLE IF EXISTS observatory.obs_table_to_table;
DROP TABLE IF EXISTS observatory.obs_meta_numer;
DROP TABLE IF EXISTS observatory.obs_meta_denom;
DROP TABLE IF EXISTS observatory.obs_meta_geom;
@ -16,20 +17,19 @@ DROP TABLE IF EXISTS observatory.obs_meta_geom_numer_timespan;
DROP TABLE IF EXISTS observatory.obs_column_table_tile;
DROP TABLE IF EXISTS observatory.obs_column_table_tile_simple;
DROP TABLE IF EXISTS observatory.obs_78fb6c1d6ff6505225175922c2c389ce48d7632c;
DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
DROP TABLE IF EXISTS observatory.obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963;
DROP TABLE IF EXISTS observatory.obs_c4411eba732408d47d73281772dbf03d60645dec;
DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
DROP TABLE IF EXISTS observatory.obs_0310c639744a2014bb1af82709228f05b59e7d3d;
DROP TABLE IF EXISTS observatory.obs_87a814e485deabe3b12545a537f693d16ca702c2;
DROP TABLE IF EXISTS observatory.obs_e32f8e59c7c8861ee5ee4029b3ace2af9a5c9caf;
DROP TABLE IF EXISTS observatory.obs_23cb5063486bd7cf36f17e89e5e65cd31b331f6e;
DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
DROP TABLE IF EXISTS observatory.obs_8e30e6b3792430b410ba5b9e49cdc6a0d404d48f;
DROP TABLE IF EXISTS observatory.obs_08025e1287e3af2b5de571d06562ba8d3bdb48e9;
DROP TABLE IF EXISTS observatory.obs_fae094ddb7157380e2495b9867e1f067fdbdf288;
DROP TABLE IF EXISTS observatory.obs_d03c931c9b7f9df54c3fae95bb7f958fe3187c71;
DROP TABLE IF EXISTS observatory.obs_a6811c89ed79ab4339d89a86907b586439cc74df;
DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
DROP TABLE IF EXISTS observatory.obs_3b537fe9a1dcdd3be4a53f64429e30a836ecb6ee;
DROP TABLE IF EXISTS observatory.obs_c4411eba732408d47d73281772dbf03d60645dec;
DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
DROP TABLE IF EXISTS observatory.obs_8e30e6b3792430b410ba5b9e49cdc6a0d404d48f;
DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
DROP TABLE IF EXISTS observatory.obs_87a814e485deabe3b12545a537f693d16ca702c2;
DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
DROP TABLE IF EXISTS observatory.obs_9b319c207dfa600c2296a6d46055e54a4c00f646;
DROP TABLE IF EXISTS observatory.obs_9b319c207dfa600c2296a6d46055e54a4c00f646;
DROP TABLE IF EXISTS observatory.obs_0310c639744a2014bb1af82709228f05b59e7d3d;
DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;

372999
src/pg/test/fixtures/load_fixtures.sql vendored
View File

File diff suppressed because one or more lines are too long

2
src/pg/test/sql/01_install_test.sql
View File

@ -3,3 +3,5 @@ CREATE EXTENSION postgis;
-- Install the extension
CREATE EXTENSION observatory VERSION 'dev';
\i test/fixtures/load_fixtures.sql

7
src/pg/test/sql/40_observatory_utility_test.sql
View File

@ -1,6 +1,5 @@
\pset format unaligned
\set ECHO all
\i test/fixtures/load_fixtures.sql
SET client_min_messages TO WARNING;
\set ECHO none
@ -48,12 +47,6 @@ SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') =
SELECT cdb_observatory.OBS_DumpVersion()
IS NOT NULL AS OBS_DumpVersion_notnull;
-- Should fail to perform intersection
SELECT ST_IsValid(ST_Intersection(
cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county'),
cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county_clipped')
)) AS complex_intersection_fails;
-- Should succeed in intersecting
SELECT ST_IsValid(cdb_observatory.safe_intersection(
cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county'),

45
src/pg/test/sql/41_observatory_augmentation_test.sql
View File

@ -22,14 +22,14 @@ SELECT cdb_observatory.OBS_GetSegmentSnapshot(
)::text is null as null_island_segmentation;
-- Point-based OBS_GetMeasure with zillow
SELECT abs(OBS_GetMeasure_zhvi_point - 597900) / 597900 < 5.0 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
SELECT abs(OBS_GetMeasure_zhvi_point - 446000) / 446000 < 5.0 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
ST_SetSRID(ST_Point(-73.90820503234865, 40.69469600456701), 4326),
'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2014-01'
) As t(OBS_GetMeasure_zhvi_point);
-- Point-based OBS_GetMeasure with later measure
SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 995400) / 995400 < 5.0 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 701400) / 701400 < 5.0 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
ST_SetSRID(ST_Point(-73.90820503234865, 40.69469600456701), 4326),
'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2016-06'
) As t(OBS_GetMeasure_zhvi_point_default_latest);
@ -416,7 +416,7 @@ SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
-- OBS_GetMeta provides suggested name for simple meta request with denom
SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
'[{"numer_id": "us.census.acs.B01001002", "normalization": "denom"}]'
)->0->>'suggested_name' = 'male_pop_rate_2010_2014' obs_getmeta_suggested_name_denom;
)->0->>'suggested_name' = 'male_pop_2010_2014_by_total_pop' obs_getmeta_suggested_name_denom;
-- OBS_GetData/OBS_GetMeta by id with empty list/null
WITH data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::TEXT[], null))
@ -677,7 +677,7 @@ data AS (SELECT * FROM cdb_observatory.OBS_GetData(
(SELECT meta FROM meta)))
SELECT id = 1 id,
data->0->>'value' = 'Hispanic Black mix multilingual, high poverty, renters, uses public transport' data_poly_categorical,
abs((data->1->>'value')::Numeric - 15787) / 15787 < 0.0001 valcol
abs((data->1->>'value')::Numeric - 15790) / 15790 < 0.0001 valcol
FROM data;
-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon
@ -700,46 +700,22 @@ data AS (SELECT * FROM cdb_observatory.OBS_GetData(
(SELECT meta FROM meta), false))
SELECT every(id = 1) is TRUE id,
count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
abs(sum((data->1->>'value')::numeric) - 12327) / 12327 < 0.001 correct_pop
abs(sum((data->1->>'value')::numeric) - 12329) / 12329 < 0.001 correct_pop
FROM data;
-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon + one measure + one text
WITH
meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
'[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.tiger.name", "geom_id": "us.census.tiger.block_group"}]') meta),
'[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.tiger.block_group_geoname", "geom_id": "us.census.tiger.block_group"}]') meta),
data AS (SELECT * FROM cdb_observatory.OBS_GetData(
ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
(SELECT meta FROM meta), false))
SELECT every(id = 1) is TRUE id,
count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
abs(sum((data->1->>'value')::numeric) - 12327) / 12327 < 0.001 correct_pop,
abs(sum((data->1->>'value')::numeric) - 12329) / 12329 < 0.001 correct_pop,
array_agg(distinct data->2->>'value') = '{"Block Group 1","Block Group 2","Block Group 3","Block Group 4","Block Group 5"}' correct_bg_names
FROM data;
-- OBS_GetData/OBS_GetMeta by geom with points inside a polygon
WITH
meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
'[{"geom_id": "us.census.tiger.pointlm_geom"}]') meta),
data AS (SELECT * FROM cdb_observatory.OBS_GetData(
ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
(SELECT meta FROM meta), false))
SELECT every(id = 1) AS id,
count(distinct (data->0->>'value')::geometry(point, 4326)) = 3 correct_num_points
FROM data;
-- OBS_GetData/OBS_GetMeta by geom with points inside a polygon + one text
WITH
meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
'[{"geom_id": "us.census.tiger.pointlm_geom"}, {"geom_id": "us.census.tiger.pointlm_geom", "numer_id": "us.census.tiger.fullname"}]') meta),
data AS (SELECT * FROM cdb_observatory.OBS_GetData(
ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
(SELECT meta FROM meta), false))
SELECT every(id = 1) AS id,
count(distinct (data->0->>'value')::geometry(point, 4326)) = 3 correct_num_points,
array_agg(data->1->>'value') = '{"Bushwick Yards","Edward Block Square","Bushwick Houses"}' pointgeom_names
FROM data;
-- OBS_GetData by id with one standard measure
WITH
meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
@ -896,7 +872,8 @@ WITH _geoms AS (
FALSE
)
WHERE data->0->>'geomref' LIKE '36047%'
ORDER BY RANDOM()
and (data->1->>'value')::numeric > 1000
ORDER BY geom_ref
), geoms AS (
SELECT *, row_number() OVER () cartodb_id FROM _geoms
), samples AS (

10
src/pg/test/sql/42_observatory_exploration_test.sql
View File

@ -503,7 +503,7 @@ SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
SELECT ARRAY_AGG(column_id ORDER BY score DESC)
= ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
'us.census.tiger.county', 'us.census.tiger.zcta5']
'us.census.tiger.zcta5', 'us.census.tiger.county']
AS _obs_geometryscores_5km_buffer
FROM cdb_observatory._OBS_GetGeometryScores(
ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
@ -532,8 +532,8 @@ SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
WHERE table_id LIKE '%2015%';
SELECT ARRAY_AGG(column_id ORDER BY score DESC)
= ARRAY['us.census.tiger.county', 'us.census.tiger.census_tract',
'us.census.tiger.zcta5', 'us.census.tiger.block_group']
= ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
'us.census.tiger.census_tract', 'us.census.tiger.block_group']
AS _obs_geometryscores_2500km_buffer
FROM cdb_observatory._OBS_GetGeometryScores(
ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
@ -593,7 +593,7 @@ SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
SELECT ARRAY_AGG(column_id ORDER BY score DESC)
= ARRAY['us.census.tiger.zcta5', 'us.census.tiger.census_tract',
'us.census.tiger.county', 'us.census.tiger.block_group']
'us.census.tiger.block_group', 'us.census.tiger.county']
AS _obs_geometryscores_500km_buffer_500_geoms
FROM cdb_observatory._OBS_GetGeometryScores(
ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
@ -613,7 +613,7 @@ SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
SELECT ARRAY_AGG(column_id ORDER BY score DESC)
= ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
'us.census.tiger.county', 'us.census.tiger.zcta5']
'us.census.tiger.zcta5', 'us.census.tiger.county']
AS _obs_geometryscores_500km_buffer_25000_geoms
FROM cdb_observatory._OBS_GetGeometryScores(
ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),

64
src/pg/test/sql/44_observatory_geometries_test.sql
View File

File diff suppressed because one or more lines are too long

12
src/python/test/perftest.py
View File

@ -95,7 +95,7 @@ def record(params, results):
('complex', '_OBS_GetGeometryScores', 'NULL', 3000)
])
def test_getgeometryscores_performance(geom_complexity, api_method, filters, target_geoms):
print api_method, geom_complexity, filters, target_geoms
print(api_method, geom_complexity, filters, target_geoms)
rownums = (1, 5, 10, ) if 'complex' in geom_complexity else (5, 25, 50,)
results = []
@ -118,7 +118,7 @@ def test_getgeometryscores_performance(geom_complexity, api_method, filters, tar
'qps': qps,
'stmt': stmt
})
print rows, ': ', qps, ' QPS'
print(rows, ': ', qps, ' QPS')
if 'OBS_RECORD_TEST' in os.environ:
record({
@ -177,7 +177,7 @@ def test_getgeometryscores_performance(geom_complexity, api_method, filters, tar
('complex', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
])
def test_getmeasure_performance(geom_complexity, api_method, normalization, geom, boundary):
print api_method, geom_complexity, normalization, geom, boundary
print(api_method, geom_complexity, normalization, geom, boundary)
col = 'measure' if 'measure' in api_method.lower() else 'category'
results = []
@ -201,7 +201,7 @@ def test_getmeasure_performance(geom_complexity, api_method, normalization, geom
'qps': qps,
'stmt': stmt
})
print rows, ': ', qps, ' QPS'
print(rows, ': ', qps, ' QPS')
if 'OBS_RECORD_TEST' in os.environ:
record({
@ -249,7 +249,7 @@ def test_getmeasure_performance(geom_complexity, api_method, normalization, geom
('complex', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
])
def test_getdata_performance(geom_complexity, normalization, geom, boundary):
print geom_complexity, normalization, geom, boundary
print(geom_complexity, normalization, geom, boundary)
cols = ['us.census.acs.B01001002',
'us.census.acs.B01001003',
@ -305,7 +305,7 @@ def test_getdata_performance(geom_complexity, normalization, geom, boundary):
'qps': qps,
'stmt': stmt
})
print rows, ': ', qps, ' QPS'
print(rows, ': ', qps, ' QPS')
if 'OBS_RECORD_TEST' in os.environ:
record({