Commit Graph

580 Commits

Author SHA1 Message Date
Rafa de la Torre
b7907ff82f Fix typo granting perms 2019-07-12 16:53:34 +02:00
Rafa de la Torre
12d955075a Fix bug iterating user_mapping options 2019-07-12 16:53:34 +02:00
Rafa de la Torre
10a4d85c01 Fix typo in example: missing closing } 2019-07-12 16:53:24 +02:00
Rafa de la Torre
524bb6ad42 Fix copy/paste typos 2019-07-12 12:49:59 +02:00
Rafa de la Torre
4da89d8abd First version of the function WIP 2019-07-12 12:40:02 +02:00
Rafa de la Torre
c7311ba48e A stub of a convenient FDW function 2019-07-12 12:00:30 +02:00
Javier Goizueta
bf140890d8
Merge pull request #365 from CartoDB/table-syncer-notmps
Sync tables without using temporary hash tables
2019-07-04 16:05:21 +02:00
Rafa de la Torre
29a31d4c40
Merge pull request #362 from CartoDB/fdw-affected-tables
Make CDB_Get_Foreign_Updated_At robust to missing CDB_TableMetadata
2019-07-04 16:02:46 +02:00
Javier Goizueta
dbd403a2f6 Fix cases of double-quoting identifiers 2019-07-04 12:47:58 +02:00
Javier Goizueta
7bdee5c13e Avoid double-quoting
Since dst_schema is a REGNAMESPACE, it is automatically quoted when casted to TEXT
2019-07-03 18:34:54 +02:00
Javier Goizueta
2e1fe2933c Fix whitespace 2019-07-03 16:56:29 +02:00
Rafa de la Torre
446f4113d9 Return NULL instead of NOW()
The absence of foreign CDB_TableMetadata actually means that we cannot
really tell when a remote table was modified.

Therefore we're using NULL with the meaning of "I don't know when it
was last modified".

To be taken in other caching layers, to adjust headers accordingly.
2019-07-03 16:19:46 +02:00
Javier Goizueta
5963c67376 Order the columns of a cartodbfied table consistently
The final order of the columns of a cartodbfied table wasn't uniquely specified, so could vary across PG versions.
This was a problem in particular for having deterministic test results.
2019-07-03 16:16:37 +02:00
Javier Goizueta
f5f18ca57c Fix the order of the columns 2019-07-02 18:14:18 +02:00
Javier Goizueta
d8c840d126 Quote table and column names when necessary
Also use type NAME when appropriate. Note that quoted column names are not NAMES (may be longer).
2019-07-02 18:14:04 +02:00
Javier Goizueta
65483743b4 Remove unused vars 2019-07-02 17:44:06 +02:00
Javier Goizueta
470bae6268 Merge branch 'master' into table-syncer-notmps
# Conflicts:
#	scripts-available/CDB_SyncTable.sql
2019-07-02 12:21:49 +02:00
Javier Goizueta
69cc56b589 Remove schema from public function examples 2019-07-01 15:13:48 +02:00
Javier Goizueta
1028b24333 Fix cartodb schema references 2019-07-01 14:48:42 +02:00
Rafa de la Torre
c06d24aa19 Make CDB_Get_Foreign_Updated_At robust to missing CDB_TableMetadata
This may happen with non-carto DB's, when checking the updated_at
times and not finding the corresponding remote.cdb_tablemetadata
imported from the foreign non-carto DB.

Instead of failing, return a NOW() timestampt, so that caching logic
just assumes there may have been changes.

This makes it work today, and leaves open the possibility of adding
the required carto metadata for homogeneous caching in the future.
2019-06-28 16:25:52 +02:00
Javier Goizueta
4f4df2de8d Merge branch 'table-syncer' into table-syncer-notmps 2019-06-28 15:28:06 +02:00
Javier Goizueta
b5f36902c5 Merge branch 'master' into table-syncer 2019-06-28 13:50:46 +02:00
Javier Goizueta
0bcbf6708a Remove usage of temporary tables 2019-06-28 13:49:15 +02:00
Gonzalo Riestra
e6ecde4346 add CREATE/ALTER/DROP FOREIGN TABLE to ddl commands 2019-06-20 11:20:48 +02:00
Gonzalo Riestra
2d42e6197a use bigint to store values from txid_current() 2019-06-20 11:20:03 +02:00
Gonzalo Riestra
5605fdd9b2 fix event trigger 2019-06-19 15:27:28 +02:00
Raul Marin
a43abb37e0 Add missing qualifications 2019-06-03 15:22:52 +02:00
Raul Marin
c45ef6c540 Fully qualify function calls 2019-05-31 15:37:15 +02:00
Rafa de la Torre
42dc03d77b Qualify functions with extension schema
This avoids some issues with search_path and scripting black magic.
2019-05-30 16:56:12 +02:00
Rafa de la Torre
9254723719 Make format expression more readable
Make an EXECUTE format('', param1, ..., paramN) more readable by
adding the index of the argument to print (`position`) in the format
specifier.
2019-05-29 13:13:30 +02:00
Rafa de la Torre
26ad966ab6 Add some timing info 2019-05-28 16:40:01 +02:00
Rafa de la Torre
a2723a3c90 Exclude certain columns from sync if instructed to do so
For the Geocoding (and in general for LDS use cases) it may come in
handy to exclude geometry columns from the list of stuff to
syncrhonize. Otherwise they may be lost, overwritten with NULL values.
2019-05-28 16:11:56 +02:00
Rafa de la Torre
2f8ea7e4ea Avoid tables name clashing when executing within same transaction
Generate more unique temp table names when the CDB_SyncTable function
is executed multiple times within the same transaction.

When executed in isolation, there will be always an implicit
surrounding transaction.

But when executed several times within the same transaction it can
give an `ERROR:  relation "src_sync_718794" already exists`.

E.g:

```
BEGIN;
SELECT cartodb.CDB_SyncTable('source1', 'public', 'dest1');
SELECT cartodb.CDB_SyncTable('source12, 'public', 'dest2');
COMMIT;
```
2019-05-28 15:39:02 +02:00
Rafa de la Torre
7606585672 Perf optimization: use EXCEPT instead of NOT IN
With javitonino's help, greatly reduce the processing time by using
EXCEPT instead of NOT IN, which causes it to use a `HashSetOp Except`
plan on the subqueries rather than a `Seq Scan` on `Materialize`'d
subtables.
2019-05-27 15:20:06 +02:00
Rafa de la Torre
81d0f338cf Create HASH indices on the temp tables 2019-05-27 15:18:13 +02:00
Rafa de la Torre
951f257654 Simplify code by using ON COMMIT DROP
Simplify code by relying on automatic removal of temp tables with ON
COMMIT DROP, so that we avoid the messy EXCEPTION management.
2019-05-27 12:02:12 +02:00
Rafa de la Torre
f461faf0b6 Simplify udpate: use * instead of column list 2019-05-27 11:56:41 +02:00
Rafa de la Torre
a8d57abda6 Update changed rows 2019-05-27 11:54:38 +02:00
Rafa de la Torre
982ddfdeff Helper function to generate UPDATE SET clause 2019-05-27 11:31:07 +02:00
Rafa de la Torre
da4331ac78 WIP: A function to sync tables
It assumes there's a cartodb_id column in both source and target. It
does not perform unnecessary actions. It respects augmented columns in
target table, if they exist. It is meant to be efficient.
2019-05-24 18:38:28 +02:00
Gonzalo Riestra
85d6164956 add log when there is no tig config 2019-03-14 11:39:10 +01:00
Gonzalo Riestra
3faa389860 improve logs 2019-03-13 18:16:08 +01:00
Gonzalo Riestra
bc5d532735 remove default TIS config 2019-03-13 18:12:48 +01:00
Gonzalo Riestra
45fed9cf1b more explicit schemas 2019-03-13 10:29:50 +01:00
Gonzalo Riestra
e19489144c simplify queries 2019-03-13 09:08:13 +01:00
Gonzalo Riestra
83707297de add explicit schema for all functions 2019-03-12 18:39:48 +01:00
Gonzalo Riestra
65d51fd8bd move cdb_ddl_execution table creation/drop to functions 2019-03-11 14:02:45 +01:00
Gonzalo Riestra
9c6294d95b move trigger drop/creation inside functions 2019-03-11 11:28:45 +01:00
Gonzalo Riestra
c7bba14e9a simplify code 2019-03-11 09:20:05 +01:00
Gonzalo Riestra
667f896cfb add again views to the trigger 2019-03-08 13:21:11 +01:00