Merge remote-tracking branch 'origin/master' into remove_schema_triggers

This commit is contained in:
Alejandro Martínez 2015-12-23 17:09:41 +01:00
commit 2a2a7d534a
13 changed files with 215 additions and 41 deletions

View File

@ -1,7 +1,7 @@
# cartodb/Makefile
EXTENSION = cartodb
EXTVERSION = 0.11.3
EXTVERSION = 0.11.5
SED = sed
@ -54,6 +54,8 @@ UPGRADABLE = \
0.11.1 \
0.11.2 \
0.11.3 \
0.11.4 \
0.11.5 \
$(EXTVERSION)dev \
$(EXTVERSION)next \
$(END)

18
NEWS.md
View File

@ -1,12 +1,20 @@
next (2015-mm-dd)
-----------------
0.11.5 (2015-11-27)
-------------------
* Disable log invalidation time [#178](https://github.com/CartoDB/cartodb-postgresql/pull/178)
0.11.4 (2015-11-24)
-------------------
* Fix for existing PK cartodb_id problem [#174](https://github.com/CartoDB/cartodb-postgresql/issues/174)
* Add cartodbfication support for column names with embedded points to fix [#6114](https://github.com/CartoDB/cartodb/issues/6114)
* Add CDB_GreatCircle for creating great circle routes between two points [#171](https://github.com/CartoDB/cartodb-postgresql/pull/171)
* Fix to prevent cartodbfication problems [#155](https://github.com/CartoDB/cartodb-postgresql/issues/155)
0.11.3 (2015-10-27)
-------------------
* Added CDB_Helper.sql [#173](https://github.com/CartoDB/cartodb-postgresql/pull/173)
* Added _CDB_Unique_Identifier for creating UTF8 aware unique identifiers
* Added _CDB_Unique_Column_Identifier for creating UTF8 aware unique identifiers for columns
* Added _CDB_Octet_Truncate that truncates text to a certain amount of octets.
* Added `_CDB_Unique_Identifier` for creating UTF8 aware unique identifiers
* Added `_CDB_Unique_Column_Identifier` for creating UTF8 aware unique identifiers for columns
* Added `_CDB_Octet_Truncate` that truncates text to a certain amount of octets.
0.11.2 (2015-10-19)
-------------------

View File

@ -6,7 +6,7 @@ cartodb-postgresql
PostgreSQL extension for CartoDB
See https://github.com/CartoDB/cartodb/wiki/CartoDB-PostgreSQL-extension
See [the cartodb-postgresql wiki](https://github.com/CartoDB/cartodb-postgresql/wiki).
Dependencies
------------

16
doc/CDB_GreatCircle.md Normal file
View File

@ -0,0 +1,16 @@
Based on Paul Ramsey's [blog post](http://blog.cartodb.com/jets-and-datelines/).
#### Using the function
Creates a great circle line.
```sql
SELECT CDB_GreatCircle(start_point, end_point) FROM table_name
-- Results a line reprsenting the great circle between the two points
```
#### Arguments
CDB_GreatCircle(start_point, end_point)
* **start_point** ST_Point indicating the start of the line.
* **end_point** ST_point indicating the end of the line.

View File

@ -0,0 +1,59 @@
Introduction
============
This document aims at describing what cartodbfy is and what its formal requirements are, with the following goals in mind:
- clarify what are the expectations of the "cartodbfycation process".
- define an important part of what should be a stable, public API
- allow for better testing, which should in turn...
- ...ease modifications and increase quality of the code
What is the cartodbfycation
===========================
The cartodbfycation is the process of converting an arbitrary postgres table into a valid CartoDB table, and register it in the system so that it can be used in the CartoDB editor and platform to generate maps and analysis.
Valid CartoDB tables
====================
A valid CartoDB table shall meet the following conditions:
- Have a ``cartodb_id`` integer column as primary key with a sequence as default value
- Have a ``the_geom`` column of type ``Geometry`` with SRID 4326
- Have a ``the_geom_webmercator`` column of type ``Geometry`` with SRID 3857
- The columns ``the_geom`` and ``the_geom_webmercator`` shall be in sync
Additionally, a CartoDB table can contain other columns.
High level requirements
=======================
Here is a list of high level requirments for the public function ``CDB_CartodbfyTable()``:
- A call to ``CDB_CartodbfyTable()`` shall modify/rewrite the table and produce a valid CartoDB table with the same name.
- A call to ``CDB_CartodbfyTable()`` shall cause the registration of the table into the platform
- It shall be idempotent, meaning that successive calls to ``CDB_CartodbfyTable()`` shall not produce any visible effect in the system.
- If there's a column containing a geometry, it shall be used to generate ``the_geom`` and the ``the_geom_webmercator`` columns.
- Exporting and re-importing the same table in CartoDB shall produce equivalent tables, with the same features associated to the same ``cartodb_id``'s.
Note that there should be only one feature per row in the source table. If there's more than one, then which one is used for ``the_geom`` and ``the_geom_webmercator`` fields is not determined.
Low-level requirements
======================
- If the original table contains a valid (unique and not null) ``cartodb_id`` column, it shall be used
- If the original table contains a ``the_geom`` column or a ``the_geom_webmercator`` column in the expected projection (EPSG 4326 and EPSG 3857, respectively) they shall be used.
- A modification of a cartodbfy'ed table shall insert or update a row in ``CDB_TableMetadata``
- A cartodbfy'ed table shall have a ``btree`` index on ``cartodb_id``
- A cartodbfy'ed table shall have ``gist`` indices on ``the_geom`` and ``the_geom_webmercator``
- Cartodbfy shall deal with text columns for imports, regarding CartoDB columns

View File

@ -152,12 +152,12 @@ BEGIN
-- Copy existing values to new field
-- NOTE: using ALTER is a workaround to a PostgreSQL bug and is also known to be faster for tables with many rows
-- See http://www.postgresql.org/message-id/20140530143150.GA11051@localhost
sql := Format('ALTER TABLE %s ALTER cartodb_id TYPE int USING %I', reloid::text, new_name);
sql := Format('ALTER TABLE %s ALTER cartodb_id TYPE int USING %I::integer', reloid::text, new_name);
RAISE DEBUG 'Running %', sql;
EXECUTE sql;
-- Find max value
sql := Format('SELECT max(cartodb_id) FROM %s', reloid::text);
sql := Format('SELECT coalesce(max(cartodb_id), 0) as max FROM %s', reloid::text);
RAISE DEBUG 'Running %', sql;
EXECUTE sql INTO rec;
@ -166,7 +166,7 @@ BEGIN
AS seq INTO rec2;
-- Reset sequence name
sql := Format('ALTER SEQUENCE %s RESTART WITH %d', rec2.seq::text, rec.max + 1);
sql := Format('ALTER SEQUENCE %s RESTART WITH %s', rec2.seq::text, rec.max + 1);
RAISE DEBUG 'Running %', sql;
EXECUTE sql;
@ -504,7 +504,7 @@ BEGIN
IF rec.atttypid IN (20,21,23) THEN
-- And it's a unique primary key! Done!
IF rec.indisprimary AND rec.indisunique AND rec.attnotnull THEN
IF (rec.indisprimary OR rec.indisunique) AND rec.attnotnull THEN
RAISE DEBUG 'CDB(_CDB_Has_Usable_Primary_ID): %', Format('found good ''%s''', const.pkey);
RETURN true;
@ -1068,8 +1068,8 @@ BEGIN
-- Add now add all the rest of the columns
-- by selecting their names into an array and
-- joining the array with a comma
SELECT
',' || array_to_string(array_agg(a.attname),',') AS column_name_sql,
SELECT
',' || array_to_string(array_agg(Format('%I',a.attname)),',') AS column_name_sql,
Count(*) AS count
INTO rec
FROM pg_class c

View File

@ -0,0 +1,26 @@
-- Great circle point-to-point routes, based on:
-- http://blog.cartodb.com/jets-and-datelines/
--
CREATE OR REPLACE FUNCTION CDB_GreatCircle(start_point geometry, end_point geometry, max_segment_length NUMERIC DEFAULT 100000)
RETURNS geometry AS $$
DECLARE
line geometry;
BEGIN
line = ST_Segmentize(
ST_Makeline(
start_point,
end_point
)::geography,
max_segment_length
)::geometry;
IF ST_XMax(line) - ST_XMin(line) > 180 THEN
line = ST_Difference(
ST_Shift_Longitude(line),
ST_Buffer(ST_GeomFromText('LINESTRING(180 90, 180 -90)', 4326), 0.00001)
);
END IF;
RETURN line;
END;
$$
LANGUAGE 'plpgsql';

View File

@ -64,10 +64,6 @@ DECLARE
tabname TEXT;
rec RECORD;
found BOOL;
function_start timestamptz;
function_end timestamptz;
function_duration float;
log_error_verbosity_value text;
BEGIN
IF TG_OP = 'UPDATE' or TG_OP = 'INSERT' THEN
@ -76,8 +72,6 @@ BEGIN
tabname = OLD.tabname;
END IF;
function_start := clock_timestamp();
-- Notify table data update
-- This needs a little bit more of research regarding security issues
-- see https://github.com/CartoDB/cartodb/pull/241
@ -111,14 +105,7 @@ BEGIN
EXIT;
END LOOP;
IF NOT found THEN RAISE WARNING 'Missing cdb_invalidate_varnish()'; END IF;
function_end := clock_timestamp();
SELECT extract(epoch from (function_end - function_start)) INTO function_duration;
SELECT setting INTO log_error_verbosity_value FROM pg_settings WHERE name='log_error_verbosity';
SET log_error_verbosity=TERSE;
RAISE LOG 'invalidation_duration: %', function_duration::text;
PERFORM 'SET log_error_verbosity= ' || log_error_verbosity_value;
RETURN NULL;
END;
$$

View File

@ -0,0 +1 @@
../scripts-available/CDB_GreatCircle.sql

View File

@ -16,8 +16,8 @@ DECLARE
BEGIN
-- Save current constraints on geometry columns, if any
ogc_geom = ('','','','',0,0,'GEOMETRY');
ogc_merc = ogc_geom;
ogc_geom = ('','','','',0,0,'GEOMETRY');
ogc_merc = ogc_geom;
sql := 'SELECT gc.* FROM geometry_columns gc, pg_class c, pg_namespace n '
|| 'WHERE c.oid = ' || tabname::oid || ' AND n.oid = c.relnamespace'
|| ' AND gc.f_table_schema = n.nspname AND gc.f_table_name = c.relname'
@ -66,7 +66,7 @@ BEGIN
ELSE 3857 END as expsrid,
CASE WHEN gc.f_geometry_column = 'the_geom' THEN ogc_geom.type
ELSE ogc_merc.type END as exptype, gc.*
FROM geometry_columns gc, pg_class c, pg_namespace n
FROM geometry_columns gc, pg_class c, pg_namespace n
WHERE c.oid = tabname::oid AND n.oid = c.relnamespace
AND gc.f_table_schema = n.nspname AND gc.f_table_name = c.relname
AND gc.f_geometry_column IN ( 'the_geom', 'the_geom_webmercator')
@ -88,7 +88,7 @@ BEGIN
RAISE EXCEPTION '% entries found for table % in geometry_columns, expected 2', tmp, tabname;
END IF;
-- Check GiST index
-- Check GiST index
sql := 'SELECT a.attname, count(ri.relname) FROM'
|| ' pg_index i, pg_class c, pg_class ri, pg_attribute a, pg_opclass o'
|| ' WHERE i.indrelid = c.oid AND ri.oid = i.indexrelid'
@ -181,31 +181,31 @@ DROP TABLE t;
-- table with existing cartodb_id field of type int4 not sequenced
CREATE TABLE t AS SELECT 1::int4 as cartodb_id;
SELECT CDB_CartodbfyTableCheck('t', 'unsequenced cartodb_id');
SELECT cartodb_id FROM t;
SELECT cartodb_id FROM t;
DROP TABLE t;
-- table with text geometry column
CREATE TABLE t AS SELECT 'SRID=4326;POINT(1 1)'::text AS the_geom, 1::int4 as cartodb_id;
SELECT CDB_CartodbfyTableCheck('t', 'text the_geom column');
SELECT cartodb_id FROM t;
SELECT cartodb_id FROM t;
DROP TABLE t;
-- table with text geometry column, no SRS
CREATE TABLE t AS SELECT 'POINT(1 1)'::text AS the_geom, 1::int4 as cartodb_id;
SELECT CDB_CartodbfyTableCheck('t', 'text the_geom column, no srs');
SELECT cartodb_id FROM t;
SELECT cartodb_id FROM t;
DROP TABLE t;
-- table with text geometry column, unusual SRS
CREATE TABLE t AS SELECT 'SRID=26910;POINT(1 1)'::text AS the_geom, 1::int4 as cartodb_id;
SELECT CDB_CartodbfyTableCheck('t', 'text the_geom column, srs = 26819');
SELECT cartodb_id FROM t;
SELECT cartodb_id FROM t;
DROP TABLE t;
-- table with text unparseable geometry column
CREATE TABLE t AS SELECT 'SRID=26910;PONT(1 1)'::text AS the_geom, 1::int4 as cartodb_id;
SELECT CDB_CartodbfyTableCheck('t', 'text the_geom column, unparseable content');
SELECT cartodb_id FROM t;
SELECT cartodb_id FROM t;
DROP TABLE t;
-- table with existing cartodb_id serial primary key
@ -286,9 +286,9 @@ CREATE TABLE test (
cartodb_id integer
);
INSERT INTO test VALUES
(1),
(2),
(NULL),
(1),
(2),
(NULL),
(3);
SELECT CDB_CartodbfyTableCheck('test', 'Table with null cartodb_id #148');
SELECT cartodb_id, cartodb_id_0 from test;
@ -299,8 +299,8 @@ CREATE TABLE test (
cartodb_id integer
);
INSERT INTO test VALUES
(1),
(2),
(1),
(2),
(2);
SELECT CDB_CartodbfyTableCheck('test', 'Table with non unique cartodb_id #148');
SELECT cartodb_id, cartodb_id_0 from test;
@ -319,6 +319,47 @@ SELECT CDB_CartodbfyTableCheck('test', 'Table with non unique and null cartodb_i
SELECT cartodb_id, cartodb_id_0 from test;
DROP TABLE test;
CREATE TABLE test (
cartodb_id integer
);
CREATE UNIQUE INDEX "test_cartodb_id_key" ON test (cartodb_id);
CREATE UNIQUE INDEX "test_cartodb_id_pkey" ON test (cartodb_id);
ALTER TABLE test ADD CONSTRAINT "test_pkey" PRIMARY KEY USING INDEX test_cartodb_id_pkey;
INSERT INTO test VALUES
(1),
(2),
(3);
SELECT CDB_CartodbfyTableCheck('test', 'Table with primary key and unique index on it #174');
SELECT cartodb_id from test;
DROP TABLE test;
CREATE TABLE test (
name varchar,
"first.value" integer,
"second.value" integer
);
INSERT INTO test VALUES ('one', 1, 2), ('two', 3, 4);
SELECT CDB_CartodbfyTableCheck('test', 'Table with dots in name columns (cartodb #6114)');
SELECT name, "first.value" from test;
DROP TABLE test;
SET client_min_messages TO notice;
-- _CDB_create_cartodb_id_column with cartodb_id integer already present
CREATE TABLE test (cartodb_id integer);
SELECT _CDB_Create_Cartodb_ID_Column('test'::regclass);
SELECT column_name FROM information_schema.columns WHERE table_name = 'test' AND column_name = '_cartodb_id0';
DROP TABLE test;
-- _CDB_create_cartodb_id_column with cartodb_id text already present
CREATE TABLE test (cartodb_id text);
SELECT _CDB_Create_Cartodb_ID_Column('test'::regclass);
SELECT column_name FROM information_schema.columns WHERE table_name = 'test' AND column_name = '_cartodb_id0';
DROP TABLE test;
SET client_min_messages TO error;
-- TODO: table with existing custom-triggered the_geom

View File

@ -102,5 +102,35 @@ Table with non unique and null cartodb_id #148 cartodbfied fine
3|
4|2
DROP TABLE
CREATE TABLE
CREATE INDEX
CREATE INDEX
ALTER TABLE
INSERT 0 3
Table with primary key and unique index on it #174 cartodbfied fine
1
2
3
DROP TABLE
CREATE TABLE
INSERT 0 2
Table with dots in name columns (cartodb #6114) cartodbfied fine
one|1
two|3
DROP TABLE
SET
CREATE TABLE
NOTICE: Column cartodb_id already exists
NOTICE: Existing cartodb_id field does not have an associated sequence, renaming
NOTICE: Trying to recover data from _cartodb_id0 column
DROP TABLE
CREATE TABLE
NOTICE: Column cartodb_id already exists
NOTICE: Existing cartodb_id field is of invalid type text (need int2, int4 or int8), renaming
NOTICE: Trying to recover data from _cartodb_id0 column
DROP TABLE
SET
DROP FUNCTION
DROP FUNCTION

2
test/CDB_GreatCircle.sql Normal file
View File

@ -0,0 +1,2 @@
select ST_AsText(CDB_GreatCircle(CDB_LatLng(55.8580,4.2590), CDB_LatLng(40.7127,74.0059)));
select ST_AsText(CDB_GreatCircle(CDB_LatLng(55.8580,4.2590), CDB_LatLng(40.7127,74.0059), 50000));

View File

@ -0,0 +1,2 @@
LINESTRING(4.259 55.858,5.6692453115051 56.0150275120673,7.10720375678704 56.157400475677,8.5718366560563 56.2842986378254,10.0619272412891 56.3949153508462,11.5760785994189 56.4884642014437,13.1127142001617 56.564185934303,14.6700812655504 56.6213555706215,16.2462571744128 56.6592896061102,17.8391590143095 56.6773531596105,19.4465562981665 56.6749669334121,21.0660867567155 56.6516138405427,22.6952750058883 56.6068451534252,24.3315537765309 56.540286032869,25.9722872888145 56.4516403065472,27.6147962622065 56.3406943817481,29.2563839799455 56.207320197769,30.8943627796619 56.0514771479657,32.5260803224591 55.8732129290618,34.1489450028345 55.6726633044968,35.7604499005266 55.4500507979281,37.3581947399686 55.2056823610616,38.9399054089486 54.9399460854786,40.5034506895044 54.6533070499613,42.0468559644411 54.3463024122038,43.5683137754523 54.0195358662507,45.066191217402 53.673671594382,46.5390342525062 53.3094278446298,47.9855691138079 52.9275702630659,49.4047010366934 52.5289051040742,50.7955106088955 52.1142724327331,52.1572480633875 51.6845394219955,53.4893258557794 51.2405938343407,54.7913098701049 50.7833377637432,56.0629095865715 50.3136816997865,57.3039675245588 49.8325389621027,58.5144482465496 49.3408205404538,59.6944271762829 48.8394303639846,60.8440794494795 48.329261012675,61.9636689799149 47.811189874886,63.0535378889196 47.2860757471582,64.1140964137264 46.7547558660385,65.1458133802427 46.2180433565989,66.1492072992903 45.676725078361,67.1248381223566 45.131559846414,68.0732996734468 44.583277003498,68.9952127576034 44.0325753175597,69.8912189338183 43.4801221786776,70.7619749300985 42.9265530691612,71.6081476710291 42.3724712809595,72.4304098829428 41.8184478551838,73.2294362384225 41.2650217194684,74.0059 40.7127)
LINESTRING(4.259 55.858,4.96060044865294 55.9382939511593,5.6692453115051 56.0150275120673,6.38482117645567 56.0880973218335,7.10720375678705 56.157400475677,7.83625773770865 56.2228347173136,8.5718366560563 56.2842986378254,9.31378281572326 56.3416918804739,10.0619272412891 56.3949153508462,10.8160896721679 56.4438714316548,11.5760785994189 56.4884642014437,12.3416913471456 56.528599656387,13.1127142001617 56.5641859343031,13.8889225793161 56.5951335399513,14.6700812655504 56.6213555706215,15.4559446734179 56.6427679409819,16.2462571744128 56.6592896061102,17.0407534700619 56.6708427815999,17.8391590143095 56.6773531596105,18.6411904842936 56.6787501197174,19.4465562981665 56.6749669334121,20.2549571781681 56.6659409611101,21.0660867567155 56.6516138405428,21.8796322228404 56.6319316654367,22.6952750058883 56.6068451534252,23.5126914929996 56.5763098021872,24.3315537765309 56.5402860328691,25.1515304272452 56.4987393199198,25.9722872888145 56.4516403065472,26.7934882889404 56.3989649050969,27.6147962622065 56.3406943817482,28.4358737796488 56.2768154250305,29.2563839799456 56.207320197769,30.0759913971174 56.1322063721813,30.8943627796619 56.0514771479657,31.7111678961496 55.9651412533344,32.5260803224592 55.8732129290618,33.3387782060384 55.7757118957345,34.1489450028345 55.6726633044969,34.9562701828379 55.5640976716962,35.7604499005266 55.4500507979282,36.5611876268714 55.3305636720814,37.3581947399687 55.2056823610617,38.1511910717861 55.0754578859583,38.9399054089486 54.9399460854787,39.7240759459355 54.7992074675415,40.5034506895044 54.6533070499613,41.277787813601 54.5023141912026,42.0468559644411 54.3463024122039,42.8104345158644 54.1853492102971,43.5683137754524 54.0195358662508,44.3202951422663 53.8489472454711,45.066191217402 53.6736715943821,45.8058258688602 53.4938003329924,46.5390342525062 53.3094278446299,47.2656627911282 53.1206512637978,47.985569113808 52.9275702630661,48.6986219579803 52.7302868398744,49.4047010366934 52.5289051040743,50.1036968736777 52.3235310669909,50.7955106088955 52.1142724327332,51.4800537772815 51.9012383924278,52.1572480633875 51.6845394219956,52.8270250346284 51.4642870840378,53.4893258557795 51.2405938343408,54.1441009873167 51.0135728334539,54.791309870105 50.7833377637434,55.4309205988438 50.5500026522693,56.0629095865715 50.3136816997866,56.6872612224038 50.0744891161178,57.3039675245588 49.8325389621028,57.9130277905821 49.5879449982851,58.5144482465496 49.3408205404539,59.1082416968843 49.091278322122,59.6944271762829 48.8394303639847,60.2730296051101 48.5853878503721,60.8440794494795 48.3292610126751,61.40761238711 48.0711590197009,61.9636689799149 47.8111898748862,62.5122943541616 47.5494603202768,63.0535378889195 47.2860757471584,63.5874529134047 47.0211401132109,64.1140964137264 46.7547558660387,64.6335287494427 46.4870238729191,65.1458133802426 46.2180433565991,65.6510166029904 45.9479118369597,66.1492072992903 45.6767250783612,66.6404566936622 45.4045770424768,67.1248381223566 45.1315598464143,67.6024268127789 44.8577637259253,68.0732996734467 44.5832770034983,68.5375350943572 44.3081860611283,68.9952127576034 44.0325753175599,69.4464134580461 43.7565272097974,69.8912189338183 43.4801221786779,70.3297117064144 43.2034386583077,70.7619749300985 42.9265530691615,71.1880922503468 42.6495398146493,71.6081476710291 42.3724712809597,72.0222254300221 42.0954178399905,72.4304098829428 41.8184478551841,72.8327853946822 41.5416276900883,73.2294362384225 41.2650217194687,73.6204465018146 40.9886923428039,74.0059 40.7127)