Commit Graph

1359 Commits

Author SHA1 Message Date
Javier Goizueta
d8c840d126 Quote table and column names when necessary
Also use type NAME when appropriate. Note that quoted column names are not NAMES (may be longer).
2019-07-02 18:14:04 +02:00
Javier Goizueta
65483743b4 Remove unused vars 2019-07-02 17:44:06 +02:00
Javier Goizueta
4651883454 Fix order of test results 2019-07-02 17:43:13 +02:00
Javier Goizueta
470bae6268 Merge branch 'master' into table-syncer-notmps
# Conflicts:
#	scripts-available/CDB_SyncTable.sql
2019-07-02 12:21:49 +02:00
Javier Goizueta
75ba13b834
Merge pull request #364 from CartoDB/synctable-doc
Document CDB_SyncTable
2019-07-02 10:20:24 +02:00
Javier Goizueta
057695361d Fix typo 2019-07-01 17:49:42 +02:00
Javier Goizueta
0e1aeb0a76 Minor copy edit 2019-07-01 17:00:22 +02:00
Javier Goizueta
f2dae651b3 More complete sync table example 2019-07-01 16:58:24 +02:00
Javier Goizueta
9f414938ba Document CDB_SyncTable 2019-07-01 16:41:40 +02:00
Javier Goizueta
4f7b07f922
Merge pull request #363 from CartoDB/fix-sync-schema
Fix cartodb schema references
2019-07-01 16:15:25 +02:00
Javier Goizueta
69cc56b589 Remove schema from public function examples 2019-07-01 15:13:48 +02:00
Javier Goizueta
1028b24333 Fix cartodb schema references 2019-07-01 14:48:42 +02:00
Javier Goizueta
4d1c4f6a22 Release 0.28.0 2019-07-01 10:56:06 +02:00
Javier Goizueta
83ab128a01
Merge pull request #355 from CartoDB/table-syncer
[WIP] Table syncer
2019-07-01 10:49:06 +02:00
Rafa de la Torre
b0b4a92240 Add a test case for missing foreign CDB_Tablementadata 2019-06-28 16:19:57 +02:00
Rafa de la Torre
c06d24aa19 Make CDB_Get_Foreign_Updated_At robust to missing CDB_TableMetadata
This may happen with non-carto DB's, when checking the updated_at
times and not finding the corresponding remote.cdb_tablemetadata
imported from the foreign non-carto DB.

Instead of failing, return a NOW() timestampt, so that caching logic
just assumes there may have been changes.

This makes it work today, and leaves open the possibility of adding
the required carto metadata for homogeneous caching in the future.
2019-06-28 16:25:52 +02:00
Javier Goizueta
4f4df2de8d Merge branch 'table-syncer' into table-syncer-notmps 2019-06-28 15:28:06 +02:00
Javier Goizueta
b5f36902c5 Merge branch 'master' into table-syncer 2019-06-28 13:50:46 +02:00
Javier Goizueta
0bcbf6708a Remove usage of temporary tables 2019-06-28 13:49:15 +02:00
Gonzalo Riestra
2e665a56b4 release 0.27.2 2019-06-21 09:19:04 +02:00
Gonzalo Riestra
1e8ef2e1d9
Merge pull request #360 from CartoDB/fix-ghost-tables-trigger
Improve Ghost Tables functions
2019-06-21 09:15:20 +02:00
Gonzalo Riestra
e6ecde4346 add CREATE/ALTER/DROP FOREIGN TABLE to ddl commands 2019-06-20 11:20:48 +02:00
Gonzalo Riestra
2d42e6197a use bigint to store values from txid_current() 2019-06-20 11:20:03 +02:00
Gonzalo Riestra
5605fdd9b2 fix event trigger 2019-06-19 15:27:28 +02:00
Sergio Conde Gómez
ab7082d4c3
Merge pull request #359 from CartoDB/fix_carto-package
Fix carto-package requirements to be SemVer compliant
2019-06-17 13:40:35 +02:00
Sergio Conde Gomez
f9ac627e0e
Fix carto-package requirements to be SemVer compliant 2019-06-17 13:33:24 +02:00
Raul Marin
cc2066ea89 Release 0.27.1 2019-06-03 17:57:42 +02:00
Raúl Marín
aa380171b4
Merge pull request #358 from Algunenano/missing
Add missing qualifications
2019-06-03 16:26:10 +02:00
Raul Marin
a43abb37e0 Add missing qualifications 2019-06-03 15:22:52 +02:00
Raul Marin
787b513715 Release 0.27.0 2019-06-03 12:51:25 +02:00
Raúl Marín
362af5e6a0
Merge pull request #357 from Algunenano/full_schema_rebase
Qualify all function calls
2019-06-03 12:45:04 +02:00
Raul Marin
8dbd797429 Update CONTRIBUTING.md 2019-06-03 12:31:06 +02:00
Raul Marin
57ac26cbe7 Travis: Add PG 9.6 to the test matrix 2019-05-31 16:26:07 +02:00
Raul Marin
010cbe6e18 Remove unnecessary changes 2019-05-31 15:52:55 +02:00
Raul Marin
8e68f2a0a7 Add NEWS 2019-05-31 15:38:29 +02:00
Raul Marin
52aab9d564 Raise minimum requirement to Postgis 9.6 2019-05-31 15:37:15 +02:00
Raul Marin
06f563bb73 Tests: Only create publicuser if necessary
Also, don't try to drop it since it's used by default
CARTO installation
2019-05-31 15:37:15 +02:00
Raul Marin
78077a6ec1 Bash tests improvements
- Stop loading sql files manually.
- Make tests independent.
- Single DB setup (not once per test)
2019-05-31 15:37:15 +02:00
Raul Marin
c45ef6c540 Fully qualify function calls 2019-05-31 15:37:15 +02:00
Rafa de la Torre
42dc03d77b Qualify functions with extension schema
This avoids some issues with search_path and scripting black magic.
2019-05-30 16:56:12 +02:00
Rafa de la Torre
9254723719 Make format expression more readable
Make an EXECUTE format('', param1, ..., paramN) more readable by
adding the index of the argument to print (`position`) in the format
specifier.
2019-05-29 13:13:30 +02:00
Rafa de la Torre
26ad966ab6 Add some timing info 2019-05-28 16:40:01 +02:00
Rafa de la Torre
a2723a3c90 Exclude certain columns from sync if instructed to do so
For the Geocoding (and in general for LDS use cases) it may come in
handy to exclude geometry columns from the list of stuff to
syncrhonize. Otherwise they may be lost, overwritten with NULL values.
2019-05-28 16:11:56 +02:00
Rafa de la Torre
2f8ea7e4ea Avoid tables name clashing when executing within same transaction
Generate more unique temp table names when the CDB_SyncTable function
is executed multiple times within the same transaction.

When executed in isolation, there will be always an implicit
surrounding transaction.

But when executed several times within the same transaction it can
give an `ERROR:  relation "src_sync_718794" already exists`.

E.g:

```
BEGIN;
SELECT cartodb.CDB_SyncTable('source1', 'public', 'dest1');
SELECT cartodb.CDB_SyncTable('source12, 'public', 'dest2');
COMMIT;
```
2019-05-28 15:39:02 +02:00
Rafa de la Torre
45c21d7f42 Make other tests resilient to CDB_SyncTable test failures
Use a transaction to avoid leaving stuff after our tests, that affect
Quota and User Tables tests.
2019-05-28 15:33:55 +02:00
Rafa de la Torre
f442c21fa4 Fix the test/CDB_AnalysisCheckTest.sql
These tests are failing in PG11:

```
*** /home/travis/build/CartoDB/cartodb-postgresql/expected/test/CDB_AnalysisCheckTest.out	2019-05-27 16:09:45.063543994 +0000
--- /home/travis/build/CartoDB/cartodb-postgresql/results/test/CDB_AnalysisCheckTest.out	2019-05-27 16:12:39.770847666 +0000
***************
*** 5,12 ****
  CREATE TABLE
  CREATE TABLE
  CREATE TABLE
- (analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5,public,analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5)
  (analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94,public,analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94)
  0

  1
--- 5,12 ----
  CREATE TABLE
  CREATE TABLE
  CREATE TABLE
  (analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94,public,analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94)
+ (analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5,public,analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5)
  0

  1
```

The reason for that is that they rely on row ordering that cannot be
guaranteed as per SQL Standard. Forcing that assumed ordering is
enough to get it working again.
2019-05-28 09:34:10 +02:00
Rafa de la Torre
ee9d08a2be Some functional E2E tests for the CDB_SyncTable() function 2019-05-27 17:56:36 +02:00
Rafa de la Torre
7606585672 Perf optimization: use EXCEPT instead of NOT IN
With javitonino's help, greatly reduce the processing time by using
EXCEPT instead of NOT IN, which causes it to use a `HashSetOp Except`
plan on the subqueries rather than a `Seq Scan` on `Materialize`'d
subtables.
2019-05-27 15:20:06 +02:00
Rafa de la Torre
81d0f338cf Create HASH indices on the temp tables 2019-05-27 15:18:13 +02:00
Rafa de la Torre
951f257654 Simplify code by using ON COMMIT DROP
Simplify code by relying on automatic removal of temp tables with ON
COMMIT DROP, so that we avoid the messy EXCEPTION management.
2019-05-27 12:02:12 +02:00