Commit Graph

1172 Commits

Author SHA1 Message Date
Raul Marin
78077a6ec1 Bash tests improvements
- Stop loading sql files manually.
- Make tests independent.
- Single DB setup (not once per test)
2019-05-31 15:37:15 +02:00
Raul Marin
c45ef6c540 Fully qualify function calls 2019-05-31 15:37:15 +02:00
Rafa de la Torre
42dc03d77b Qualify functions with extension schema
This avoids some issues with search_path and scripting black magic.
2019-05-30 16:56:12 +02:00
Rafa de la Torre
9254723719 Make format expression more readable
Make an EXECUTE format('', param1, ..., paramN) more readable by
adding the index of the argument to print (`position`) in the format
specifier.
2019-05-29 13:13:30 +02:00
Rafa de la Torre
26ad966ab6 Add some timing info 2019-05-28 16:40:01 +02:00
Rafa de la Torre
a2723a3c90 Exclude certain columns from sync if instructed to do so
For the Geocoding (and in general for LDS use cases) it may come in
handy to exclude geometry columns from the list of stuff to
syncrhonize. Otherwise they may be lost, overwritten with NULL values.
2019-05-28 16:11:56 +02:00
Rafa de la Torre
2f8ea7e4ea Avoid tables name clashing when executing within same transaction
Generate more unique temp table names when the CDB_SyncTable function
is executed multiple times within the same transaction.

When executed in isolation, there will be always an implicit
surrounding transaction.

But when executed several times within the same transaction it can
give an `ERROR:  relation "src_sync_718794" already exists`.

E.g:

```
BEGIN;
SELECT cartodb.CDB_SyncTable('source1', 'public', 'dest1');
SELECT cartodb.CDB_SyncTable('source12, 'public', 'dest2');
COMMIT;
```
2019-05-28 15:39:02 +02:00
Rafa de la Torre
45c21d7f42 Make other tests resilient to CDB_SyncTable test failures
Use a transaction to avoid leaving stuff after our tests, that affect
Quota and User Tables tests.
2019-05-28 15:33:55 +02:00
Rafa de la Torre
f442c21fa4 Fix the test/CDB_AnalysisCheckTest.sql
These tests are failing in PG11:

```
*** /home/travis/build/CartoDB/cartodb-postgresql/expected/test/CDB_AnalysisCheckTest.out	2019-05-27 16:09:45.063543994 +0000
--- /home/travis/build/CartoDB/cartodb-postgresql/results/test/CDB_AnalysisCheckTest.out	2019-05-27 16:12:39.770847666 +0000
***************
*** 5,12 ****
  CREATE TABLE
  CREATE TABLE
  CREATE TABLE
- (analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5,public,analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5)
  (analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94,public,analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94)
  0

  1
--- 5,12 ----
  CREATE TABLE
  CREATE TABLE
  CREATE TABLE
  (analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94,public,analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94)
+ (analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5,public,analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5)
  0

  1
```

The reason for that is that they rely on row ordering that cannot be
guaranteed as per SQL Standard. Forcing that assumed ordering is
enough to get it working again.
2019-05-28 09:34:10 +02:00
Rafa de la Torre
ee9d08a2be Some functional E2E tests for the CDB_SyncTable() function 2019-05-27 17:56:36 +02:00
Rafa de la Torre
7606585672 Perf optimization: use EXCEPT instead of NOT IN
With javitonino's help, greatly reduce the processing time by using
EXCEPT instead of NOT IN, which causes it to use a `HashSetOp Except`
plan on the subqueries rather than a `Seq Scan` on `Materialize`'d
subtables.
2019-05-27 15:20:06 +02:00
Rafa de la Torre
81d0f338cf Create HASH indices on the temp tables 2019-05-27 15:18:13 +02:00
Rafa de la Torre
951f257654 Simplify code by using ON COMMIT DROP
Simplify code by relying on automatic removal of temp tables with ON
COMMIT DROP, so that we avoid the messy EXCEPTION management.
2019-05-27 12:02:12 +02:00
Rafa de la Torre
f461faf0b6 Simplify udpate: use * instead of column list 2019-05-27 11:56:41 +02:00
Rafa de la Torre
a8d57abda6 Update changed rows 2019-05-27 11:54:38 +02:00
Rafa de la Torre
982ddfdeff Helper function to generate UPDATE SET clause 2019-05-27 11:31:07 +02:00
Rafa de la Torre
da4331ac78 WIP: A function to sync tables
It assumes there's a cartodb_id column in both source and target. It
does not perform unnecessary actions. It respects augmented columns in
target table, if they exist. It is meant to be efficient.
2019-05-24 18:38:28 +02:00
Gonzalo Riestra
07d43275f8
Merge pull request #354 from CartoDB/update-readme
Add Redis dependency to Readme
2019-05-23 10:58:33 +02:00
Gonzalo Riestra
ec38444b7d better link 2019-05-23 08:12:57 +02:00
Gonzalo Riestra
ee9010dc23 add redis module to dependencies in readme 2019-05-23 08:03:37 +02:00
Gonzalo Riestra
74210c5b5c release 0.26.1 2019-03-19 17:43:36 +01:00
Gonzalo Riestra
d14d1d9994
Merge pull request #351 from CartoDB/remove_default_tis_config
Remove default TIS config in Ghost Table function
2019-03-14 12:23:06 +01:00
Gonzalo Riestra
5b19d0fc5e fix test correctly 2019-03-14 12:08:33 +01:00
Gonzalo Riestra
2baee24f30 fix test 2019-03-14 11:47:41 +01:00
Gonzalo Riestra
85d6164956 add log when there is no tig config 2019-03-14 11:39:10 +01:00
Gonzalo Riestra
a5b2b66bb6 fix test comment 2019-03-14 10:06:04 +01:00
Gonzalo Riestra
3faa389860 improve logs 2019-03-13 18:16:08 +01:00
Gonzalo Riestra
bc5d532735 remove default TIS config 2019-03-13 18:12:48 +01:00
Gonzalo Riestra
4f3d19ce7a
Merge pull request #350 from CartoDB/explicit-schema-for-ghost-tables
Add explicit schema for Ghost Tables functions
2019-03-13 10:42:13 +01:00
Gonzalo Riestra
45fed9cf1b more explicit schemas 2019-03-13 10:29:50 +01:00
Gonzalo Riestra
e19489144c simplify queries 2019-03-13 09:08:13 +01:00
Gonzalo Riestra
83707297de add explicit schema for all functions 2019-03-12 18:39:48 +01:00
Gonzalo Riestra
2f40261b8d
Merge pull request #349 from CartoDB/ghost_tables
Ghost tables functions
2019-03-11 15:29:17 +01:00
Gonzalo Riestra
b1202011f6 update news 2019-03-11 15:28:48 +01:00
Gonzalo Riestra
65d51fd8bd move cdb_ddl_execution table creation/drop to functions 2019-03-11 14:02:45 +01:00
Gonzalo Riestra
1a271d977b bump version 2019-03-11 12:24:14 +01:00
Gonzalo Riestra
9c6294d95b move trigger drop/creation inside functions 2019-03-11 11:28:45 +01:00
Gonzalo Riestra
c7bba14e9a simplify code 2019-03-11 09:20:05 +01:00
Gonzalo Riestra
667f896cfb add again views to the trigger 2019-03-08 13:21:11 +01:00
Gonzalo Riestra
cc1df0a708 use redis module version from carto 2019-03-08 13:01:56 +01:00
Gonzalo Riestra
d1ee383d9b add redis module for python to travis 2019-03-08 12:50:51 +01:00
Gonzalo Riestra
a794fb3d31 read tis config from cdb_conf 2019-03-08 12:03:55 +01:00
Gonzalo Riestra
50e41179fc fix test 2019-03-08 09:18:46 +01:00
Gonzalo Riestra
e3138cd56a make enable/disable trigger idempotent 2019-03-08 08:43:22 +01:00
Gonzalo Riestra
ab6720ad32 add create/drop/alter view to the trigger 2019-03-08 08:28:24 +01:00
Gonzalo Riestra
5f154a5859 use TIS instead of Redis 2019-03-08 08:28:06 +01:00
Gonzalo Riestra
99dd7cefc7 ghost tables functions using redis 2019-03-06 18:20:04 +01:00
Raúl Marín
85997e2445
Merge pull request #348 from Algunenano/shiftLongitude
Update ST_Shift_Longitude to ST_ShiftLongitude
2019-03-01 12:06:24 +01:00
Raul Marin
1bd4b9a6e3 Update NEWS 2019-03-01 11:47:30 +01:00
Raul Marin
5d4f1d98d7 Update ST_Shift_Longitude to ST_ShiftLongitude 2019-03-01 11:27:09 +01:00