cartodb/cartodb-postgresql
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,391 Commits 35 Branches 94 Tags 1.8 MiB
c9ae99e4a7
Commit Graph

730 Commits

Author SHA1 Message Date
Raul Marin
b76b754811 CDB_FederatedServerTables skeleton 2019-10-29 18:53:05 +01:00
Raul Marin
664468b1e5 Rename CDB_FederatedServerListRemote files 2019-10-29 18:51:47 +01:00
Raul Marin
f7ab3d28a8 __CDB_FS_List_Foreign_Schemas_PG: Handle order and ignore pg_% schemas automatically 2019-10-29 18:35:02 +01:00
Raul Marin
144c4e722b Add federated server list remotes 
Includes:
- CDB_Federated_Server_List_Remote_Schemas
- CDB_Federated_Server_List_Remote_Tables
 2019-10-29 18:26:52 +01:00
Raul Marin
e22aaee5f5 Test re-registration of a server with different options 2019-10-29 16:47:35 +01:00
Raul Marin
da9ff0c750 Reimplement CDB_Federated_Server_Register_PG and CDB_Federated_Server_Unregister 2019-10-29 16:19:46 +01:00
Raul Marin
e6cb03300b Add auxiliar functions to handle internal naming 2019-10-29 15:38:58 +01:00
Raul Marin
bc7031d872 Move existence check to __CDB_FS_Generate_Object_Name 2019-10-29 13:19:11 +01:00
Raul Marin
1375894e99 CDB_Federated_Server_Unregister: Better error handling 2019-10-29 12:27:07 +01:00
Raul Marin
acf1f2dbd4 WIP: Add CDB_Federated_Server_Unregister 2019-10-28 18:34:46 +01:00
Raul Marin
b475eb4356 Federated Server List 2019-10-28 17:10:53 +01:00
Gonzalo Riestra
2ce4d8ee14 fix grant query 2019-10-25 12:02:08 +02:00
Gonzalo Riestra
11b30c8f59 fix oauth reassign ownership for functions 2019-10-25 11:34:59 +02:00
Raul Marin
9469871029 Remove unneeded change 2019-10-23 17:06:47 +02:00
Raul Marin
29397ff929 Make it work without postgis_raster 2019-10-23 14:21:45 +02:00
Raul Marin
34684507cb Cast sql_identifier to text 2019-10-23 13:46:04 +02:00
Raul Marin
e787b1d097 Missing qualification 2019-10-22 20:09:15 +02:00
Raul Marin
d1eb740a00 Missing qualifications 2019-10-22 19:45:42 +02:00
Raul Marin
8fac0c89eb python3: Try to adapt pathname2url 2019-10-22 18:06:29 +02:00
Raul Marin
9686ad5cfe Try to work with both python2 and python3 2019-10-22 16:49:08 +02:00
Raul Marin
048234cd80 CDB_Groups_API: Secure search_path 2019-10-03 18:58:27 +02:00
Raul Marin
a580bedefc Set safe schema on some functions 2019-10-03 18:58:27 +02:00
Raul Marin
0e891eff7f Remove cartodb from search path 
All calls to the extension functions should, and are,
be properly qualified, so there is no need to keep
the cartodb schema in the search_path
 2019-10-03 18:58:27 +02:00
Raul Marin
52b3290d26 CDB_TableMetadata: Use secure search_path 2019-10-03 18:58:27 +02:00
Raul Marin
186ee37a57 CDB_Username: Set secure search_path 2019-10-03 18:58:27 +02:00
Raul Marin
0898881470 Oauth: Set secure search_path 2019-10-03 18:58:27 +02:00
Raul Marin
4dfd71639a CDB_OAuthReassignTableOwnerOnCreation: Use CDB_Conf_GetConf 
Instead of accessing the cdb_conf table directly
 2019-10-03 18:58:27 +02:00
Raul Marin
c6b90aac8a GhostTables: Set secure search_path 2019-10-03 18:58:27 +02:00
Raul Marin
2f178bd89e _CDB_SetUp_User_PG_FDW_Server: Minor log changes 2019-10-01 11:02:34 +02:00
Raul Marin
32489c4eab Ghost tables: Add missing tags 
- Add materialized views commands.
- Add missing alter trigger.
- Add 'IMPORT FOREIGN SCHEMA' since it might create new tables.
- Add 'DROP EXTENSION' and 'DROP TYPE' since they alter table columns.
- Add 'DROP SCHEMA' and 'DROP SERVER' as they drop tables.
 2019-10-01 11:02:34 +02:00
Rafa de la Torre
402d97daa6 Merge remote-tracking branch 'origin/master' into user-defined-fdw 2019-07-17 09:18:35 +02:00
Rafa de la Torre
e41d2ec019 Add a flag to force drop of user FDW and related objects 
If force = true then it will add the subclause `CASCADE` to the SQL
DDL sentences that support it, otherwise it'll use `RESTRICT` which is
the default and exact opposite.
 2019-07-16 17:40:05 +02:00
Rafa de la Torre
3c460f1a85 Add a bunch of RAISE NOTICE's to inform user about progress 2019-07-16 17:03:23 +02:00
Rafa de la Torre
ce1e9ac41c Prefix all objects created with cdb_fdw_ 
Build the DB objects related to a user FDW with the following form:
`cdb_fdw_name`. This is aimed at easily inspect and filter them.

As requested in code review.
 2019-07-16 15:35:35 +02:00
Rafa de la Torre
0f33ee8b22 Prepend an underscore (_) to functions meant to be run by superuser 
_CDB_SetUp_User_PG_FDW_Server and _CDB_Drop_User_PG_FDW_Server are
meant to be executed by a superuser. Therefore they shouldn't be
considered part of the public API and hence the _CDB_Private_Function
naming convention.
 2019-07-16 14:32:32 +02:00
Rafa de la Torre
a32dea0282 Remove SECURITY DEFINER from user-defined FDW's 2019-07-16 13:26:03 +02:00
Rafa de la Torre
3a255df9d0 Rename PG-FDW's-specific functions to _PG_FDW_ 
As per review comment.
 2019-07-16 13:14:11 +02:00
Rafa de la Torre
99096d41e0 Drop the role when dropping a user-defined FDW 2019-07-15 17:25:48 +02:00
Rafa de la Torre
3a10ef7e76 Add ability to grant fdw role to org members 2019-07-15 16:54:23 +02:00
Rafa de la Torre
37004db047 Add new function to drop a user-defined foreign server 2019-07-15 16:14:07 +02:00
Rafa de la Torre
70220e04c1 Allow for imports of tables in different source schemas 2019-07-15 13:13:21 +02:00
Gonzalo Riestra
a5cb9f268d reassign ownership for sequences and functions as well 2019-07-15 12:31:44 +02:00
Rafa de la Torre
d2d909145d Add convenience function to import fdw tables 2019-07-12 16:53:34 +02:00
Rafa de la Torre
34dec227c4 Rename to CDB_SetUp_User_Foreign_Server 2019-07-12 16:53:34 +02:00
Rafa de la Torre
99e92e2505 Create a "PUBLIC" user mapping 2019-07-12 16:53:34 +02:00
Rafa de la Torre
c58a084102 Tweak ownership of db objects 2019-07-12 16:53:34 +02:00
Rafa de la Torre
b7907ff82f Fix typo granting perms 2019-07-12 16:53:34 +02:00
Rafa de la Torre
12d955075a Fix bug iterating user_mapping options 2019-07-12 16:53:34 +02:00
Rafa de la Torre
10a4d85c01 Fix typo in example: missing closing } 2019-07-12 16:53:24 +02:00
Rafa de la Torre
524bb6ad42 Fix copy/paste typos 2019-07-12 12:49:59 +02:00
Rafa de la Torre
4da89d8abd First version of the function WIP 2019-07-12 12:40:02 +02:00
Rafa de la Torre
c7311ba48e A stub of a convenient FDW function 2019-07-12 12:00:30 +02:00
Mario de Frutos Dieguez
0f1c98c743
Check for empty strings as well 2019-07-04 17:48:01 +02:00
Mario de Frutos Dieguez
8ecd2cd5e2
Reuser creator_role 
Co-Authored-By: Alberto Romeu <alrocar@users.noreply.github.com>
 2019-07-04 17:48:00 +02:00
Mario de Frutos Dieguez
f4be59cae0
Added tests 2019-07-04 17:48:00 +02:00
Mario de Frutos Dieguez
5744921065
OAuth functions 
- Create/drop reassign event trigger and the function with the logic
- Function that reassings owner to ownership role if defined
 2019-07-04 17:47:36 +02:00
Javier Goizueta
bf140890d8
Merge pull request #365 from CartoDB/table-syncer-notmps 
Sync tables without using temporary hash tables
 2019-07-04 16:05:21 +02:00
Rafa de la Torre
29a31d4c40
Merge pull request #362 from CartoDB/fdw-affected-tables 
Make CDB_Get_Foreign_Updated_At robust to missing CDB_TableMetadata
 2019-07-04 16:02:46 +02:00
Javier Goizueta
dbd403a2f6 Fix cases of double-quoting identifiers 2019-07-04 12:47:58 +02:00
Javier Goizueta
7bdee5c13e Avoid double-quoting 
Since dst_schema is a REGNAMESPACE, it is automatically quoted when casted to TEXT
 2019-07-03 18:34:54 +02:00
Javier Goizueta
2e1fe2933c Fix whitespace 2019-07-03 16:56:29 +02:00
Rafa de la Torre
446f4113d9 Return NULL instead of NOW() 
The absence of foreign CDB_TableMetadata actually means that we cannot
really tell when a remote table was modified.

Therefore we're using NULL with the meaning of "I don't know when it
was last modified".

To be taken in other caching layers, to adjust headers accordingly.
 2019-07-03 16:19:46 +02:00
Javier Goizueta
5963c67376 Order the columns of a cartodbfied table consistently 
The final order of the columns of a cartodbfied table wasn't uniquely specified, so could vary across PG versions.
This was a problem in particular for having deterministic test results.
 2019-07-03 16:16:37 +02:00
Javier Goizueta
f5f18ca57c Fix the order of the columns 2019-07-02 18:14:18 +02:00
Javier Goizueta
d8c840d126 Quote table and column names when necessary 
Also use type NAME when appropriate. Note that quoted column names are not NAMES (may be longer).
 2019-07-02 18:14:04 +02:00
Javier Goizueta
65483743b4 Remove unused vars 2019-07-02 17:44:06 +02:00
Javier Goizueta
470bae6268 Merge branch 'master' into table-syncer-notmps 
# Conflicts:
#	scripts-available/CDB_SyncTable.sql
 2019-07-02 12:21:49 +02:00
Javier Goizueta
69cc56b589 Remove schema from public function examples 2019-07-01 15:13:48 +02:00
Javier Goizueta
1028b24333 Fix cartodb schema references 2019-07-01 14:48:42 +02:00
Rafa de la Torre
c06d24aa19 Make CDB_Get_Foreign_Updated_At robust to missing CDB_TableMetadata 
This may happen with non-carto DB's, when checking the updated_at
times and not finding the corresponding remote.cdb_tablemetadata
imported from the foreign non-carto DB.

Instead of failing, return a NOW() timestampt, so that caching logic
just assumes there may have been changes.

This makes it work today, and leaves open the possibility of adding
the required carto metadata for homogeneous caching in the future.
 2019-06-28 16:25:52 +02:00
Javier Goizueta
4f4df2de8d Merge branch 'table-syncer' into table-syncer-notmps 2019-06-28 15:28:06 +02:00
Javier Goizueta
b5f36902c5 Merge branch 'master' into table-syncer 2019-06-28 13:50:46 +02:00
Javier Goizueta
0bcbf6708a Remove usage of temporary tables 2019-06-28 13:49:15 +02:00
Gonzalo Riestra
e6ecde4346 add CREATE/ALTER/DROP FOREIGN TABLE to ddl commands 2019-06-20 11:20:48 +02:00
Gonzalo Riestra
2d42e6197a use bigint to store values from txid_current() 2019-06-20 11:20:03 +02:00
Gonzalo Riestra
5605fdd9b2 fix event trigger 2019-06-19 15:27:28 +02:00
Raul Marin
a43abb37e0 Add missing qualifications 2019-06-03 15:22:52 +02:00
Raul Marin
c45ef6c540 Fully qualify function calls 2019-05-31 15:37:15 +02:00
Rafa de la Torre
42dc03d77b Qualify functions with extension schema 
This avoids some issues with search_path and scripting black magic.
 2019-05-30 16:56:12 +02:00
Rafa de la Torre
9254723719 Make format expression more readable 
Make an EXECUTE format('', param1, ..., paramN) more readable by
adding the index of the argument to print (`position`) in the format
specifier.
 2019-05-29 13:13:30 +02:00
Rafa de la Torre
26ad966ab6 Add some timing info 2019-05-28 16:40:01 +02:00
Rafa de la Torre
a2723a3c90 Exclude certain columns from sync if instructed to do so 
For the Geocoding (and in general for LDS use cases) it may come in
handy to exclude geometry columns from the list of stuff to
syncrhonize. Otherwise they may be lost, overwritten with NULL values.
 2019-05-28 16:11:56 +02:00
Rafa de la Torre
2f8ea7e4ea Avoid tables name clashing when executing within same transaction 
Generate more unique temp table names when the CDB_SyncTable function
is executed multiple times within the same transaction.

When executed in isolation, there will be always an implicit
surrounding transaction.

But when executed several times within the same transaction it can
give an `ERROR:  relation "src_sync_718794" already exists`.

E.g:

```
BEGIN;
SELECT cartodb.CDB_SyncTable('source1', 'public', 'dest1');
SELECT cartodb.CDB_SyncTable('source12, 'public', 'dest2');
COMMIT;
```
 2019-05-28 15:39:02 +02:00
Rafa de la Torre
7606585672 Perf optimization: use EXCEPT instead of NOT IN 
With javitonino's help, greatly reduce the processing time by using
EXCEPT instead of NOT IN, which causes it to use a `HashSetOp Except`
plan on the subqueries rather than a `Seq Scan` on `Materialize`'d
subtables.
 2019-05-27 15:20:06 +02:00
Rafa de la Torre
81d0f338cf Create HASH indices on the temp tables 2019-05-27 15:18:13 +02:00
Rafa de la Torre
951f257654 Simplify code by using ON COMMIT DROP 
Simplify code by relying on automatic removal of temp tables with ON
COMMIT DROP, so that we avoid the messy EXCEPTION management.
 2019-05-27 12:02:12 +02:00
Rafa de la Torre
f461faf0b6 Simplify udpate: use * instead of column list 2019-05-27 11:56:41 +02:00
Rafa de la Torre
a8d57abda6 Update changed rows 2019-05-27 11:54:38 +02:00
Rafa de la Torre
982ddfdeff Helper function to generate UPDATE SET clause 2019-05-27 11:31:07 +02:00
Rafa de la Torre
da4331ac78 WIP: A function to sync tables 
It assumes there's a cartodb_id column in both source and target. It
does not perform unnecessary actions. It respects augmented columns in
target table, if they exist. It is meant to be efficient.
 2019-05-24 18:38:28 +02:00
Gonzalo Riestra
85d6164956 add log when there is no tig config 2019-03-14 11:39:10 +01:00
Gonzalo Riestra
3faa389860 improve logs 2019-03-13 18:16:08 +01:00
Gonzalo Riestra
bc5d532735 remove default TIS config 2019-03-13 18:12:48 +01:00
Gonzalo Riestra
45fed9cf1b more explicit schemas 2019-03-13 10:29:50 +01:00
Gonzalo Riestra
e19489144c simplify queries 2019-03-13 09:08:13 +01:00
Gonzalo Riestra
83707297de add explicit schema for all functions 2019-03-12 18:39:48 +01:00
Gonzalo Riestra
65d51fd8bd move cdb_ddl_execution table creation/drop to functions 2019-03-11 14:02:45 +01:00
Gonzalo Riestra
9c6294d95b move trigger drop/creation inside functions 2019-03-11 11:28:45 +01:00
Gonzalo Riestra
c7bba14e9a simplify code 2019-03-11 09:20:05 +01:00
Gonzalo Riestra
667f896cfb add again views to the trigger 2019-03-08 13:21:11 +01:00
Gonzalo Riestra
a794fb3d31 read tis config from cdb_conf 2019-03-08 12:03:55 +01:00
Gonzalo Riestra
e3138cd56a make enable/disable trigger idempotent 2019-03-08 08:43:22 +01:00
Gonzalo Riestra
ab6720ad32 add create/drop/alter view to the trigger 2019-03-08 08:28:24 +01:00
Gonzalo Riestra
5f154a5859 use TIS instead of Redis 2019-03-08 08:28:06 +01:00
Gonzalo Riestra
99dd7cefc7 ghost tables functions using redis 2019-03-06 18:20:04 +01:00
Raul Marin
5d4f1d98d7 Update ST_Shift_Longitude to ST_ShiftLongitude 2019-03-01 11:27:09 +01:00
Gonzalo Riestra
0568b36a90 use session_user instead of current_user 2019-02-20 14:45:57 +01:00
Gonzalo Riestra
2a4ecd4850 Add CDB_Username() function 2019-02-20 10:38:00 +01:00
Raul Marin
6c57751579 CDB_TransformToWebmercator: Yet another hack over a function that shouldn't exist 2019-01-02 19:33:28 +01:00
Raul Marin
80fdd00541 Drop functions removed in 0.11.1 2019-01-02 18:56:24 +01:00
Raul Marin
d2a00852a8 Jenks: Iterate increasing the number of moved elements exponentially 2018-09-11 19:22:48 +02:00
Raul Marin
e605234d38 Jenks: Remove log messages 2018-09-10 15:53:02 +02:00
Raul Marin
0e7d797400 Jenks: Fix multiple bugs 2018-09-10 15:42:35 +02:00
Raul Marin
7e131143f9 _cdb_estimated_extent: Adapt to ST_EstimatedExtent change 
The behaviour was changed in Postgis 1.5.4 to return NULL instead of throwing
 2018-09-04 18:21:28 +02:00
Eneko Lakasta
b209726b1c
Merge pull request #320 from bloomberg/v0_18_5/cdb_tablemetadata_text_hotfix 
Fix incorrect timestamps in CDB_TableMetadata_Text
 2018-07-23 16:00:19 +02:00
Javier Torres
5659275c0c Don't rely on regexp to identify non explainable queries 2018-07-20 13:01:51 +02:00
Javier Goizueta
2766bbc83a Fix PARALLEL tags 
Fixes #333
 2018-07-18 12:13:31 +02:00
Rafa de la Torre
61d2024eb5 Make the code nicer by avoiding IF/THEN/ELSE 
As suggested by Algunenano.
 2018-07-03 15:35:05 +02:00
Rafa de la Torre
af142306aa Mark _CDB_Table_Exists() as PARALLEL UNSAFE 
As pointed out by Algunenano, PL/pgSQL function which establishes an
EXCEPTION block to catch errors must be qualified with it.
 2018-07-03 15:26:48 +02:00
Rafa de la Torre
82f90e618c Use CREATE OR REPLACE FUNCTION 2018-07-03 13:04:39 +02:00
Rafa de la Torre
55a77b0ef0 Add a new helper function _CDB_Table_Exists 2018-07-03 13:00:24 +02:00
Javier Torres
ccbabaa3b4 Missing quote idents 2018-05-31 17:06:39 +02:00
Javier Torres
7f5bef1203 Escape regclass call with quote_ident 2018-05-29 14:04:53 +02:00
Javier Torres
625d62c448 Remove unneeded variable 2018-05-28 20:29:36 +02:00
Javier Torres
f06418c99b Extract sequence-listing function 2018-05-28 20:14:57 +02:00
Javier Torres
1958f2de5b Working code to add permission to all table sequences 2018-05-28 20:07:16 +02:00
Mario de Frutos
c9f4685e5d Fix error with hyphens, moving logic to the Format part 2018-03-21 16:33:05 +01:00
Mario de Frutos
f96c334f48 Fixed a trucate problem with the seq rename. See #325 2018-03-21 12:05:56 +01:00
Andy Eschbacher
697a0a3a36
Merge branch 'master' into quantiles-bins-updates 2018-03-12 09:50:01 -04:00
Paul Ramsey
62c27ab140 Merge branch 'master' of github.com:CartoDB/cartodb-postgresql into seq-drop 2018-03-09 09:58:25 -08:00
Andy Eschbacher
b6b0ef704a Merge branch 'master' into quantiles-bins-updates 2018-03-08 10:33:01 -05:00
Andy Eschbacher
39998a9c88 fix syntax error oof oof 2018-03-08 10:21:43 -05:00
Andy Eschbacher
cf38d4cf25 oof include SELECT 2018-03-08 10:14:08 -05:00
Andy Eschbacher
0467c075f7 opts for array version of percentile_disc 2018-03-08 10:00:52 -05:00
Javier Goizueta
0a58c05049
Merge pull request #323 from CartoDB/321-grant-sequence 
grant usage on cartodb_id sequence when sharing read write
 2018-02-15 16:29:25 +01:00
Alberto Romeu
cd4653ecc0 grant usage on cartodb_id sequence when sharing read write 2018-02-15 08:40:17 +01:00
Javier Goizueta
7ea428913a Add limit to the grid-generating functions 
This adds an optinal maxcells parameter to the rectangle and hexagonal grids,
with a default value of 262144 (2^18).
An error occurs when the grids would need more cells.
 2018-02-14 17:50:06 +01:00
Tyler Parsons
d811a71da0 Fix incorrect timestamps in CDB_TableMetadata_Text 
Instead of performing a proper join on tabname, CDB_TableMetadata_Text joins
cdb_tablemetadata against pg_catalog.pg_class (i.e. All postgres tables,
views, indices, etc.) and gives a record for every possible tabname and
updated_at combination.  This results in the latest updated timestamp being
chosen for any table in CDB_Get_Foreign_Updated_At, which leads to unnecessary
and incorrect cache invalidation.
 2018-01-25 12:27:38 -05:00
Paul Ramsey
4050555801 Quiet errors where there's no existing sequence 2018-01-18 10:26:01 -08:00
Paul Ramsey
e4b0e7ea7a Make user final seq has the right desired name 2018-01-18 10:06:38 -08:00
Paul Ramsey
a79c2cb7a8 draft tweek on seq handling 2018-01-18 09:27:46 -08:00
Andy Eschbacher
691b9a8312 moves quantile bins to use postgres precentile functions 2017-12-05 16:16:39 -05:00
Raul Marin
9d980ab17c Mark CDB_JenksBins as VOLATILE as it uses random 2017-11-13 09:35:20 +01:00
Raul Marin
72c214a8e4 Fix regex for PARALLEL aggregations 
Worked in pg10, but it didn't in pg9.5
 2017-11-13 09:35:20 +01:00
Raul Marin
0abc2ba250 Add PARALLEL parameter to functions 2017-11-13 09:35:20 +01:00
Raul Marin
f24d15f6ca Fix issues when running under pg10 2017-10-25 14:38:06 +02:00
Rafa de la Torre
ffb779eb74 Increase search space of ids by 100x #305 2017-06-29 17:54:42 +02:00
Mario de Frutos
8a3d506a53 Add some debug logs to be used in the future 2017-06-02 15:41:18 +02:00
Mario de Frutos
4908bacc4b If existing overviews we regenerate them 
Instead of make a DROP/CREATE always, in case we have existing
overviews we have to make DELETE/INSERT because DDL operations require
AccessExclusiveLock which is not compatible with AccessShareLock that
is used by the queries used to gather data for the tiler.

This incompatibility above mentioned leads to have deadlocks
 2017-06-02 15:23:47 +02:00
Mario de Frutos
38fa3b485c Remove unused and old _CDB_GridCluster_Reduce_Strategy function for overviews 2017-06-02 13:05:05 +02:00