Commit Graph

1144 Commits

Author SHA1 Message Date
Rafa de la Torre
c06d24aa19 Make CDB_Get_Foreign_Updated_At robust to missing CDB_TableMetadata
This may happen with non-carto DB's, when checking the updated_at
times and not finding the corresponding remote.cdb_tablemetadata
imported from the foreign non-carto DB.

Instead of failing, return a NOW() timestampt, so that caching logic
just assumes there may have been changes.

This makes it work today, and leaves open the possibility of adding
the required carto metadata for homogeneous caching in the future.
2019-06-28 16:25:52 +02:00
Javier Goizueta
4f4df2de8d Merge branch 'table-syncer' into table-syncer-notmps 2019-06-28 15:28:06 +02:00
Javier Goizueta
b5f36902c5 Merge branch 'master' into table-syncer 2019-06-28 13:50:46 +02:00
Javier Goizueta
0bcbf6708a Remove usage of temporary tables 2019-06-28 13:49:15 +02:00
Gonzalo Riestra
2e665a56b4 release 0.27.2 2019-06-21 09:19:04 +02:00
Gonzalo Riestra
1e8ef2e1d9
Merge pull request #360 from CartoDB/fix-ghost-tables-trigger
Improve Ghost Tables functions
2019-06-21 09:15:20 +02:00
Gonzalo Riestra
e6ecde4346 add CREATE/ALTER/DROP FOREIGN TABLE to ddl commands 2019-06-20 11:20:48 +02:00
Gonzalo Riestra
2d42e6197a use bigint to store values from txid_current() 2019-06-20 11:20:03 +02:00
Gonzalo Riestra
5605fdd9b2 fix event trigger 2019-06-19 15:27:28 +02:00
Sergio Conde Gómez
ab7082d4c3
Merge pull request #359 from CartoDB/fix_carto-package
Fix carto-package requirements to be SemVer compliant
2019-06-17 13:40:35 +02:00
Sergio Conde Gomez
f9ac627e0e
Fix carto-package requirements to be SemVer compliant 2019-06-17 13:33:24 +02:00
Raul Marin
cc2066ea89 Release 0.27.1 2019-06-03 17:57:42 +02:00
Raúl Marín
aa380171b4
Merge pull request #358 from Algunenano/missing
Add missing qualifications
2019-06-03 16:26:10 +02:00
Raul Marin
a43abb37e0 Add missing qualifications 2019-06-03 15:22:52 +02:00
Raul Marin
787b513715 Release 0.27.0 2019-06-03 12:51:25 +02:00
Raúl Marín
362af5e6a0
Merge pull request #357 from Algunenano/full_schema_rebase
Qualify all function calls
2019-06-03 12:45:04 +02:00
Raul Marin
8dbd797429 Update CONTRIBUTING.md 2019-06-03 12:31:06 +02:00
Raul Marin
57ac26cbe7 Travis: Add PG 9.6 to the test matrix 2019-05-31 16:26:07 +02:00
Raul Marin
010cbe6e18 Remove unnecessary changes 2019-05-31 15:52:55 +02:00
Raul Marin
8e68f2a0a7 Add NEWS 2019-05-31 15:38:29 +02:00
Raul Marin
52aab9d564 Raise minimum requirement to Postgis 9.6 2019-05-31 15:37:15 +02:00
Raul Marin
06f563bb73 Tests: Only create publicuser if necessary
Also, don't try to drop it since it's used by default
CARTO installation
2019-05-31 15:37:15 +02:00
Raul Marin
78077a6ec1 Bash tests improvements
- Stop loading sql files manually.
- Make tests independent.
- Single DB setup (not once per test)
2019-05-31 15:37:15 +02:00
Raul Marin
c45ef6c540 Fully qualify function calls 2019-05-31 15:37:15 +02:00
Rafa de la Torre
42dc03d77b Qualify functions with extension schema
This avoids some issues with search_path and scripting black magic.
2019-05-30 16:56:12 +02:00
Rafa de la Torre
9254723719 Make format expression more readable
Make an EXECUTE format('', param1, ..., paramN) more readable by
adding the index of the argument to print (`position`) in the format
specifier.
2019-05-29 13:13:30 +02:00
Rafa de la Torre
26ad966ab6 Add some timing info 2019-05-28 16:40:01 +02:00
Rafa de la Torre
a2723a3c90 Exclude certain columns from sync if instructed to do so
For the Geocoding (and in general for LDS use cases) it may come in
handy to exclude geometry columns from the list of stuff to
syncrhonize. Otherwise they may be lost, overwritten with NULL values.
2019-05-28 16:11:56 +02:00
Rafa de la Torre
2f8ea7e4ea Avoid tables name clashing when executing within same transaction
Generate more unique temp table names when the CDB_SyncTable function
is executed multiple times within the same transaction.

When executed in isolation, there will be always an implicit
surrounding transaction.

But when executed several times within the same transaction it can
give an `ERROR:  relation "src_sync_718794" already exists`.

E.g:

```
BEGIN;
SELECT cartodb.CDB_SyncTable('source1', 'public', 'dest1');
SELECT cartodb.CDB_SyncTable('source12, 'public', 'dest2');
COMMIT;
```
2019-05-28 15:39:02 +02:00
Rafa de la Torre
45c21d7f42 Make other tests resilient to CDB_SyncTable test failures
Use a transaction to avoid leaving stuff after our tests, that affect
Quota and User Tables tests.
2019-05-28 15:33:55 +02:00
Rafa de la Torre
f442c21fa4 Fix the test/CDB_AnalysisCheckTest.sql
These tests are failing in PG11:

```
*** /home/travis/build/CartoDB/cartodb-postgresql/expected/test/CDB_AnalysisCheckTest.out	2019-05-27 16:09:45.063543994 +0000
--- /home/travis/build/CartoDB/cartodb-postgresql/results/test/CDB_AnalysisCheckTest.out	2019-05-27 16:12:39.770847666 +0000
***************
*** 5,12 ****
  CREATE TABLE
  CREATE TABLE
  CREATE TABLE
- (analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5,public,analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5)
  (analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94,public,analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94)
  0

  1
--- 5,12 ----
  CREATE TABLE
  CREATE TABLE
  CREATE TABLE
  (analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94,public,analysis_2f13a3dbd7_f00cee44e9e6152b450bde3a92eb9ae0d099da94)
+ (analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5,public,analysis_2f13a3dbd7_41bd92976fc6dd97072afe4ee450054f4c0715d5)
  0

  1
```

The reason for that is that they rely on row ordering that cannot be
guaranteed as per SQL Standard. Forcing that assumed ordering is
enough to get it working again.
2019-05-28 09:34:10 +02:00
Rafa de la Torre
ee9d08a2be Some functional E2E tests for the CDB_SyncTable() function 2019-05-27 17:56:36 +02:00
Rafa de la Torre
7606585672 Perf optimization: use EXCEPT instead of NOT IN
With javitonino's help, greatly reduce the processing time by using
EXCEPT instead of NOT IN, which causes it to use a `HashSetOp Except`
plan on the subqueries rather than a `Seq Scan` on `Materialize`'d
subtables.
2019-05-27 15:20:06 +02:00
Rafa de la Torre
81d0f338cf Create HASH indices on the temp tables 2019-05-27 15:18:13 +02:00
Rafa de la Torre
951f257654 Simplify code by using ON COMMIT DROP
Simplify code by relying on automatic removal of temp tables with ON
COMMIT DROP, so that we avoid the messy EXCEPTION management.
2019-05-27 12:02:12 +02:00
Rafa de la Torre
f461faf0b6 Simplify udpate: use * instead of column list 2019-05-27 11:56:41 +02:00
Rafa de la Torre
a8d57abda6 Update changed rows 2019-05-27 11:54:38 +02:00
Rafa de la Torre
982ddfdeff Helper function to generate UPDATE SET clause 2019-05-27 11:31:07 +02:00
Rafa de la Torre
da4331ac78 WIP: A function to sync tables
It assumes there's a cartodb_id column in both source and target. It
does not perform unnecessary actions. It respects augmented columns in
target table, if they exist. It is meant to be efficient.
2019-05-24 18:38:28 +02:00
Gonzalo Riestra
07d43275f8
Merge pull request #354 from CartoDB/update-readme
Add Redis dependency to Readme
2019-05-23 10:58:33 +02:00
Gonzalo Riestra
ec38444b7d better link 2019-05-23 08:12:57 +02:00
Gonzalo Riestra
ee9010dc23 add redis module to dependencies in readme 2019-05-23 08:03:37 +02:00
Gonzalo Riestra
74210c5b5c release 0.26.1 2019-03-19 17:43:36 +01:00
Gonzalo Riestra
d14d1d9994
Merge pull request #351 from CartoDB/remove_default_tis_config
Remove default TIS config in Ghost Table function
2019-03-14 12:23:06 +01:00
Gonzalo Riestra
5b19d0fc5e fix test correctly 2019-03-14 12:08:33 +01:00
Gonzalo Riestra
2baee24f30 fix test 2019-03-14 11:47:41 +01:00
Gonzalo Riestra
85d6164956 add log when there is no tig config 2019-03-14 11:39:10 +01:00
Gonzalo Riestra
a5b2b66bb6 fix test comment 2019-03-14 10:06:04 +01:00
Gonzalo Riestra
3faa389860 improve logs 2019-03-13 18:16:08 +01:00
Gonzalo Riestra
bc5d532735 remove default TIS config 2019-03-13 18:12:48 +01:00