Merge remote-tracking branch 'origin/master' into remove_schema_triggers

Makefile

@@ -1,7 +1,7 @@
 # cartodb/Makefile
 
 EXTENSION = cartodb
-EXTVERSION = 0.11.3
+EXTVERSION = 0.11.5
 
 SED = sed
 
@@ -54,6 +54,8 @@ UPGRADABLE = \
   0.11.1 \
   0.11.2 \
   0.11.3 \
+  0.11.4 \
+	0.11.5 \
   $(EXTVERSION)dev \
   $(EXTVERSION)next \
   $(END)

NEWS.md

@@ -1,12 +1,20 @@
-next (2015-mm-dd)
+0.11.5 (2015-11-27)
------------------
+-------------------
+* Disable log invalidation time [#178](https://github.com/CartoDB/cartodb-postgresql/pull/178)
+
+0.11.4 (2015-11-24)
+-------------------
+* Fix for existing PK cartodb_id problem [#174](https://github.com/CartoDB/cartodb-postgresql/issues/174)
+* Add cartodbfication support for column names with embedded points to fix [#6114](https://github.com/CartoDB/cartodb/issues/6114)
+* Add CDB_GreatCircle for creating great circle routes between two points [#171](https://github.com/CartoDB/cartodb-postgresql/pull/171)
+* Fix to prevent cartodbfication problems [#155](https://github.com/CartoDB/cartodb-postgresql/issues/155)
 
 0.11.3 (2015-10-27)
 -------------------
 * Added CDB_Helper.sql [#173](https://github.com/CartoDB/cartodb-postgresql/pull/173)
-* Added _CDB_Unique_Identifier for creating UTF8 aware unique identifiers
+* Added `_CDB_Unique_Identifier` for creating UTF8 aware unique identifiers
-* Added _CDB_Unique_Column_Identifier for creating UTF8 aware unique identifiers for columns
+* Added `_CDB_Unique_Column_Identifier` for creating UTF8 aware unique identifiers for columns
-* Added _CDB_Octet_Truncate that truncates text to a certain amount of octets.
+* Added `_CDB_Octet_Truncate` that truncates text to a certain amount of octets.
 
 0.11.2 (2015-10-19)
 -------------------

README.md

@@ -6,7 +6,7 @@ cartodb-postgresql
 
 PostgreSQL extension for CartoDB
 
-See https://github.com/CartoDB/cartodb/wiki/CartoDB-PostgreSQL-extension
+See [the cartodb-postgresql wiki](https://github.com/CartoDB/cartodb-postgresql/wiki).
 
 Dependencies
 ------------

doc/CDB_GreatCircle.md

@@ -0,0 +1,16 @@
+Based on Paul Ramsey's [blog post](http://blog.cartodb.com/jets-and-datelines/).
+#### Using the function
+
+Creates a great circle line.
+
+```sql
+SELECT CDB_GreatCircle(start_point, end_point) FROM table_name
+-- Results a line reprsenting the great circle between the two points
+```
+
+#### Arguments
+
+CDB_GreatCircle(start_point, end_point)
+
+* **start_point** ST_Point indicating the start of the line.
+* **end_point** ST_point indicating the end of the line.

doc/cartodbfy-requirements.rst

@@ -0,0 +1,59 @@
+Introduction
+============
+
+This document aims at describing what cartodbfy is and what its formal requirements are, with the following goals in mind:
+
+- clarify what are the expectations of the "cartodbfycation process".
+- define an important part of what should be a stable, public API
+- allow for better testing, which should in turn...
+- ...ease modifications and increase quality of the code
+
+
+
+What is the cartodbfycation
+===========================
+
+The cartodbfycation is the process of converting an arbitrary postgres table into a valid CartoDB table, and register it in the system so that it can be used in the CartoDB editor and platform to generate maps and analysis.
+
+
+
+Valid CartoDB tables
+====================
+
+A valid CartoDB table shall meet the following conditions:
+
+- Have a ``cartodb_id`` integer column as primary key with a sequence as default value
+- Have a ``the_geom`` column of type ``Geometry`` with SRID 4326
+- Have a ``the_geom_webmercator`` column of type ``Geometry`` with SRID 3857
+- The columns ``the_geom`` and ``the_geom_webmercator`` shall be in sync
+
+Additionally, a CartoDB table can contain other columns.
+
+
+
+High level requirements
+=======================
+
+Here is a list of high level requirments for the public function ``CDB_CartodbfyTable()``:
+
+- A call to ``CDB_CartodbfyTable()`` shall modify/rewrite the table and produce a valid CartoDB table with the same name.
+- A call to ``CDB_CartodbfyTable()`` shall cause the registration of the table into the platform
+- It shall be idempotent, meaning that successive calls to ``CDB_CartodbfyTable()`` shall not produce any visible effect in the system.
+- If there's a column containing a geometry, it shall be used to generate ``the_geom`` and the ``the_geom_webmercator`` columns.
+- Exporting and re-importing the same table in CartoDB shall produce equivalent tables, with the same features associated to the same ``cartodb_id``'s.
+
+Note that there should be only one feature per row in the source table. If there's more than one, then which one is used for ``the_geom`` and ``the_geom_webmercator`` fields is not determined.
+
+
+
+Low-level requirements
+======================
+
+- If the original table contains a valid (unique and not null) ``cartodb_id`` column, it shall be used
+- If the original table contains a ``the_geom`` column or a ``the_geom_webmercator`` column in the expected projection (EPSG 4326 and EPSG 3857, respectively) they shall be used.
+- A modification of a cartodbfy'ed table shall insert or update a row in ``CDB_TableMetadata``
+- A cartodbfy'ed table shall have a ``btree`` index on ``cartodb_id``
+- A cartodbfy'ed table shall have ``gist`` indices on ``the_geom`` and ``the_geom_webmercator``
+- Cartodbfy shall deal with text columns for imports, regarding CartoDB columns
+
+

scripts-available/CDB_CartodbfyTable.sql

@@ -152,12 +152,12 @@ BEGIN
       -- Copy existing values to new field
       -- NOTE: using ALTER is a workaround to a PostgreSQL bug and is also known to be faster for tables with many rows
       -- See http://www.postgresql.org/message-id/20140530143150.GA11051@localhost
-      sql := Format('ALTER TABLE %s ALTER cartodb_id TYPE int USING %I', reloid::text, new_name);
+      sql := Format('ALTER TABLE %s ALTER cartodb_id TYPE int USING %I::integer', reloid::text, new_name);
       RAISE DEBUG 'Running %', sql;
       EXECUTE sql;
 
       -- Find max value
-      sql := Format('SELECT max(cartodb_id) FROM %s', reloid::text);
+      sql := Format('SELECT coalesce(max(cartodb_id), 0) as max FROM %s', reloid::text);
       RAISE DEBUG 'Running %', sql;
       EXECUTE sql INTO rec;
 
@@ -166,7 +166,7 @@ BEGIN
         AS seq INTO rec2;
 
       -- Reset sequence name
-      sql := Format('ALTER SEQUENCE %s RESTART WITH %d', rec2.seq::text, rec.max + 1);
+      sql := Format('ALTER SEQUENCE %s RESTART WITH %s', rec2.seq::text, rec.max + 1);
       RAISE DEBUG 'Running %', sql;
       EXECUTE sql;
 
@@ -504,7 +504,7 @@ BEGIN
     IF rec.atttypid IN (20,21,23) THEN
           
       -- And it's a unique primary key! Done!
-      IF rec.indisprimary AND rec.indisunique AND rec.attnotnull THEN
+      IF (rec.indisprimary OR rec.indisunique) AND rec.attnotnull THEN
         RAISE DEBUG 'CDB(_CDB_Has_Usable_Primary_ID): %', Format('found good ''%s''', const.pkey);
         RETURN true;
 
@@ -1069,7 +1069,7 @@ BEGIN
   -- by selecting their names into an array and
   -- joining the array with a comma
   SELECT
-    ',' || array_to_string(array_agg(a.attname),',') AS column_name_sql, 
+    ',' || array_to_string(array_agg(Format('%I',a.attname)),',') AS column_name_sql,
     Count(*) AS count
   INTO rec
   FROM pg_class c 

scripts-available/CDB_GreatCircle.sql

@@ -0,0 +1,26 @@
+-- Great circle point-to-point routes, based on:
+--   http://blog.cartodb.com/jets-and-datelines/
+--
+CREATE OR REPLACE FUNCTION CDB_GreatCircle(start_point geometry, end_point geometry, max_segment_length NUMERIC DEFAULT 100000)
+RETURNS geometry AS $$
+DECLARE
+  line geometry;
+BEGIN
+  line = ST_Segmentize(
+    ST_Makeline(
+      start_point,
+      end_point
+    )::geography,
+    max_segment_length
+  )::geometry;
+
+  IF ST_XMax(line) - ST_XMin(line) > 180 THEN
+    line = ST_Difference(
+      ST_Shift_Longitude(line),
+			ST_Buffer(ST_GeomFromText('LINESTRING(180 90, 180 -90)', 4326), 0.00001)
+		);
+  END IF;
+RETURN line;
+END;
+$$
+LANGUAGE 'plpgsql';

scripts-available/CDB_TableMetadata.sql

@@ -64,10 +64,6 @@ DECLARE
   tabname TEXT;
   rec RECORD;
   found BOOL;
-  function_start timestamptz;
-  function_end timestamptz;
-  function_duration float;
-  log_error_verbosity_value text;
 BEGIN
 
   IF TG_OP = 'UPDATE' or TG_OP = 'INSERT' THEN
@@ -76,8 +72,6 @@ BEGIN
     tabname = OLD.tabname;
   END IF;
 
-  function_start := clock_timestamp();
- 
   -- Notify table data update
   -- This needs a little bit more of research regarding security issues
   -- see https://github.com/CartoDB/cartodb/pull/241
@@ -112,13 +106,6 @@ BEGIN
   END LOOP;
   IF NOT found THEN RAISE WARNING 'Missing cdb_invalidate_varnish()'; END IF;
 
-  function_end := clock_timestamp();
-  SELECT extract(epoch from (function_end - function_start)) INTO function_duration;
-  SELECT setting INTO log_error_verbosity_value FROM pg_settings WHERE name='log_error_verbosity';
-  SET log_error_verbosity=TERSE;
-  RAISE LOG 'invalidation_duration: %', function_duration::text;
-  PERFORM 'SET log_error_verbosity= ' || log_error_verbosity_value;
-  
   RETURN NULL;
 END;
 $$

scripts-enabled/241-CDB_GreatCircle.sql

@@ -0,0 +1 @@
+../scripts-available/CDB_GreatCircle.sql

test/CDB_CartodbfyTableTest.sql

@@ -319,6 +319,47 @@ SELECT CDB_CartodbfyTableCheck('test', 'Table with non unique and null cartodb_i
 SELECT cartodb_id, cartodb_id_0 from test;
 DROP TABLE test;
 
+CREATE TABLE test (
+  cartodb_id integer
+);
+CREATE UNIQUE INDEX "test_cartodb_id_key" ON test (cartodb_id);
+CREATE UNIQUE INDEX "test_cartodb_id_pkey" ON test (cartodb_id);
+ALTER TABLE test ADD CONSTRAINT "test_pkey" PRIMARY KEY USING INDEX test_cartodb_id_pkey;
+INSERT INTO test VALUES
+  (1),
+  (2),
+  (3);
+SELECT CDB_CartodbfyTableCheck('test', 'Table with primary key and unique index on it #174');
+SELECT cartodb_id from test;
+DROP TABLE test;
+
+CREATE TABLE test (
+  name varchar,
+  "first.value" integer,
+  "second.value" integer
+);
+INSERT INTO test VALUES ('one', 1, 2), ('two', 3, 4);
+SELECT CDB_CartodbfyTableCheck('test', 'Table with dots in name columns (cartodb #6114)');
+SELECT name, "first.value" from test;
+DROP TABLE test;
+
+SET client_min_messages TO notice;
+-- _CDB_create_cartodb_id_column with cartodb_id integer already present
+CREATE TABLE test (cartodb_id integer);
+
+SELECT _CDB_Create_Cartodb_ID_Column('test'::regclass);
+SELECT column_name FROM information_schema.columns WHERE table_name = 'test' AND column_name = '_cartodb_id0';
+
+DROP TABLE test;
+
+-- _CDB_create_cartodb_id_column with cartodb_id text already present
+CREATE TABLE test (cartodb_id text);
+
+SELECT _CDB_Create_Cartodb_ID_Column('test'::regclass);
+SELECT column_name FROM information_schema.columns WHERE table_name = 'test' AND column_name = '_cartodb_id0';
+
+DROP TABLE test;
+SET client_min_messages TO error;
 
 -- TODO: table with existing custom-triggered the_geom
 

test/CDB_CartodbfyTableTest_expect

@@ -102,5 +102,35 @@ Table with non unique and null cartodb_id #148 cartodbfied fine
 3|
 4|2
 DROP TABLE
+CREATE TABLE
+CREATE INDEX
+CREATE INDEX
+ALTER TABLE
+INSERT 0 3
+Table with primary key and unique index on it #174 cartodbfied fine
+1
+2
+3
+DROP TABLE
+CREATE TABLE
+INSERT 0 2
+Table with dots in name columns (cartodb #6114) cartodbfied fine
+one|1
+two|3
+DROP TABLE
+SET
+CREATE TABLE
+NOTICE:  Column cartodb_id already exists
+NOTICE:  Existing cartodb_id field does not have an associated sequence, renaming
+NOTICE:  Trying to recover data from _cartodb_id0 column
+
+DROP TABLE
+CREATE TABLE
+NOTICE:  Column cartodb_id already exists
+NOTICE:  Existing cartodb_id field is of invalid type text (need int2, int4 or int8), renaming
+NOTICE:  Trying to recover data from _cartodb_id0 column
+
+DROP TABLE
+SET
 DROP FUNCTION
 DROP FUNCTION

test/CDB_GreatCircle.sql

@@ -0,0 +1,2 @@
+select ST_AsText(CDB_GreatCircle(CDB_LatLng(55.8580,4.2590), CDB_LatLng(40.7127,74.0059)));
+select ST_AsText(CDB_GreatCircle(CDB_LatLng(55.8580,4.2590), CDB_LatLng(40.7127,74.0059), 50000));

test/CDB_GreatCircle_expect

@@ -0,0 +1,2 @@
+LINESTRING(4.259 55.858,5.6692453115051 56.0150275120673,7.10720375678704 56.157400475677,8.5718366560563 56.2842986378254,10.0619272412891 56.3949153508462,11.5760785994189 56.4884642014437,13.1127142001617 56.564185934303,14.6700812655504 56.6213555706215,16.2462571744128 56.6592896061102,17.8391590143095 56.6773531596105,19.4465562981665 56.6749669334121,21.0660867567155 56.6516138405427,22.6952750058883 56.6068451534252,24.3315537765309 56.540286032869,25.9722872888145 56.4516403065472,27.6147962622065 56.3406943817481,29.2563839799455 56.207320197769,30.8943627796619 56.0514771479657,32.5260803224591 55.8732129290618,34.1489450028345 55.6726633044968,35.7604499005266 55.4500507979281,37.3581947399686 55.2056823610616,38.9399054089486 54.9399460854786,40.5034506895044 54.6533070499613,42.0468559644411 54.3463024122038,43.5683137754523 54.0195358662507,45.066191217402 53.673671594382,46.5390342525062 53.3094278446298,47.9855691138079 52.9275702630659,49.4047010366934 52.5289051040742,50.7955106088955 52.1142724327331,52.1572480633875 51.6845394219955,53.4893258557794 51.2405938343407,54.7913098701049 50.7833377637432,56.0629095865715 50.3136816997865,57.3039675245588 49.8325389621027,58.5144482465496 49.3408205404538,59.6944271762829 48.8394303639846,60.8440794494795 48.329261012675,61.9636689799149 47.811189874886,63.0535378889196 47.2860757471582,64.1140964137264 46.7547558660385,65.1458133802427 46.2180433565989,66.1492072992903 45.676725078361,67.1248381223566 45.131559846414,68.0732996734468 44.583277003498,68.9952127576034 44.0325753175597,69.8912189338183 43.4801221786776,70.7619749300985 42.9265530691612,71.6081476710291 42.3724712809595,72.4304098829428 41.8184478551838,73.2294362384225 41.2650217194684,74.0059 40.7127)
+LINESTRING(4.259 55.858,4.96060044865294 55.9382939511593,5.6692453115051 56.0150275120673,6.38482117645567 56.0880973218335,7.10720375678705 56.157400475677,7.83625773770865 56.2228347173136,8.5718366560563 56.2842986378254,9.31378281572326 56.3416918804739,10.0619272412891 56.3949153508462,10.8160896721679 56.4438714316548,11.5760785994189 56.4884642014437,12.3416913471456 56.528599656387,13.1127142001617 56.5641859343031,13.8889225793161 56.5951335399513,14.6700812655504 56.6213555706215,15.4559446734179 56.6427679409819,16.2462571744128 56.6592896061102,17.0407534700619 56.6708427815999,17.8391590143095 56.6773531596105,18.6411904842936 56.6787501197174,19.4465562981665 56.6749669334121,20.2549571781681 56.6659409611101,21.0660867567155 56.6516138405428,21.8796322228404 56.6319316654367,22.6952750058883 56.6068451534252,23.5126914929996 56.5763098021872,24.3315537765309 56.5402860328691,25.1515304272452 56.4987393199198,25.9722872888145 56.4516403065472,26.7934882889404 56.3989649050969,27.6147962622065 56.3406943817482,28.4358737796488 56.2768154250305,29.2563839799456 56.207320197769,30.0759913971174 56.1322063721813,30.8943627796619 56.0514771479657,31.7111678961496 55.9651412533344,32.5260803224592 55.8732129290618,33.3387782060384 55.7757118957345,34.1489450028345 55.6726633044969,34.9562701828379 55.5640976716962,35.7604499005266 55.4500507979282,36.5611876268714 55.3305636720814,37.3581947399687 55.2056823610617,38.1511910717861 55.0754578859583,38.9399054089486 54.9399460854787,39.7240759459355 54.7992074675415,40.5034506895044 54.6533070499613,41.277787813601 54.5023141912026,42.0468559644411 54.3463024122039,42.8104345158644 54.1853492102971,43.5683137754524 54.0195358662508,44.3202951422663 53.8489472454711,45.066191217402 53.6736715943821,45.8058258688602 53.4938003329924,46.5390342525062 53.3094278446299,47.2656627911282 53.1206512637978,47.985569113808 52.9275702630661,48.6986219579803 52.7302868398744,49.4047010366934 52.5289051040743,50.1036968736777 52.3235310669909,50.7955106088955 52.1142724327332,51.4800537772815 51.9012383924278,52.1572480633875 51.6845394219956,52.8270250346284 51.4642870840378,53.4893258557795 51.2405938343408,54.1441009873167 51.0135728334539,54.791309870105 50.7833377637434,55.4309205988438 50.5500026522693,56.0629095865715 50.3136816997866,56.6872612224038 50.0744891161178,57.3039675245588 49.8325389621028,57.9130277905821 49.5879449982851,58.5144482465496 49.3408205404539,59.1082416968843 49.091278322122,59.6944271762829 48.8394303639847,60.2730296051101 48.5853878503721,60.8440794494795 48.3292610126751,61.40761238711 48.0711590197009,61.9636689799149 47.8111898748862,62.5122943541616 47.5494603202768,63.0535378889195 47.2860757471584,63.5874529134047 47.0211401132109,64.1140964137264 46.7547558660387,64.6335287494427 46.4870238729191,65.1458133802426 46.2180433565991,65.6510166029904 45.9479118369597,66.1492072992903 45.6767250783612,66.6404566936622 45.4045770424768,67.1248381223566 45.1315598464143,67.6024268127789 44.8577637259253,68.0732996734467 44.5832770034983,68.5375350943572 44.3081860611283,68.9952127576034 44.0325753175599,69.4464134580461 43.7565272097974,69.8912189338183 43.4801221786779,70.3297117064144 43.2034386583077,70.7619749300985 42.9265530691615,71.1880922503468 42.6495398146493,71.6081476710291 42.3724712809597,72.0222254300221 42.0954178399905,72.4304098829428 41.8184478551841,72.8327853946822 41.5416276900883,73.2294362384225 41.2650217194687,73.6204465018146 40.9886923428039,74.0059 40.7127)