cartodb/cartodb-postgresql
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,078 Commits 35 Branches 94 Tags 1.8 MiB
446f4113d9
Commit Graph

547 Commits

Author SHA1 Message Date
Javier Goizueta
2a30eb2fd3 Fix zoom from scale condition for NULL result 
And rewrite in cleaner form.
 2016-05-27 15:10:20 +02:00
Javier Goizueta
0b3ad5e569 Limit the maximum zoom level 
Avoid returning zoom levels greater than the maximum
'safe' level. For a zero denominator (which would imply
and infinite zoom level) return the maximum level too.
 2016-05-27 12:29:52 +02:00
Paul Norman
aa302c237d Use standard formula for CDB_ZoomFromScale 
postgis-vt-util comes with a standard formula for zoom from
scaleDenominator, and this is well tested. This also fixes the
function throwing an error on NULL input.
 2016-05-27 12:29:43 +02:00
Javier Goizueta
9526f0448f Fix bug in feature density recursive query 
If the table had x and/or y columns they were picked by an inner
select instead of the recursive arguments.
Fixes #256
 2016-05-26 18:25:11 +02:00
Javier Goizueta
3399f2b9a5 Fix max overviews level usage 
Note that _CDB_Feature_Density_Ref_Z_Strategy returns the first
level for which overviews should not be used, and that in some
cases _CDB_Feature_Density should look beyond the max level to
compute a feature density estimate.
 2016-05-25 14:00:55 +02:00
Javier Goizueta
803b3671d0 Fixes for feature density computation 
* The initial iteration of the recursive feature density query shouldn't
  start beyond the maximum level
* Correct off-by-one limit the rest of iterations
 2016-05-25 11:21:12 +02:00
Javier Goizueta
c3fada29a8 Comment how the max overview level is defined 2016-05-25 11:07:06 +02:00
Javier Goizueta
86e5f6d317 Limit the maximum overview level to 23 
Fixes #257

23 = 32 bits per integer - 1 sign bit - 8 bits px/tile
 2016-05-24 18:18:58 +02:00
Javier Goizueta
8d7860dc7a Fixes #252 2016-05-09 11:54:56 +02:00
Javier Goizueta
b5427c65c8 Drop aggregate to be defined 
Otherwise future versions will fail to recreate the aggregate
 2016-04-29 08:46:01 +02:00
Paul Norman
956e56cd37 Use anyarray for equalintervalbins 2016-04-27 16:10:01 -07:00
Javier Goizueta
a067cc7da1 Generate stats used to identify category columns in overviews if needed 
This only generates the stats if no stats are available for a table.
This doesn't warrant that the stats are up to date or accurate.
 2016-04-27 15:06:09 +02:00
Javier Goizueta
2c43943df6 Fix syntax 2016-04-26 18:27:52 +02:00
Javier Goizueta
417cbe7902 Fix category columns aggregation in overviews 
Overviews are created in cascade, each one from the inmediate
lower level, but the stats to decide if a column is a category
should be taken always from the base table.
 2016-04-26 18:02:25 +02:00
Javier Goizueta
9a73703954 Use mode to aggregate categorical columns in overviews 
Fixes #245
 2016-04-26 15:15:24 +02:00
Paul Norman
b19a5fc3dc Convert CDB_EqualIntervalBins to a single SQL statement and add float version 2016-04-25 14:35:26 -07:00
Javier Goizueta
efe381ad94 Merge pull request #243 from CartoDB/241-webmercator 
Compute webmercator resolution with full accuracy
 2016-04-25 17:30:40 +02:00
Javier Goizueta
f7cce21eb7 Merge pull request #242 from CartoDB/240-overviews-pixels 
Adjust overview points to pixel centers
 2016-04-25 17:30:25 +02:00
Javier Goizueta
18267477da Merge pull request #238 from CartoDB/235-column-names 
Optimize column information functions
 2016-04-25 17:30:07 +02:00
Javier Goizueta
11ad45306f Remove unneeded pg_catalog schema name 2016-04-25 16:30:58 +02:00
Javier Goizueta
75c7ae98e4 Compute webmercator resolution with full accuracy 
Fixes #241
 2016-04-25 14:02:26 +02:00
Javier Goizueta
3c12cf629f Optimize overview pixel adjustment for integer-pixel cells 2016-04-25 13:53:59 +02:00
Javier Goizueta
7b2100b51e Adjust overview coordinates to pixel centers 
This makes the adjustment for all grid sizes, not only
for integral number of pixels.
 2016-04-25 13:33:43 +02:00
Javier Goizueta
580ec38ab8 Adjust overview clustered point to pixel centers 
Fixes #240
 2016-04-23 15:07:06 +02:00
Javier Goizueta
65415bb335 Optimize funcion CDB_COlumnType 2016-04-18 19:07:33 +02:00
Javier Goizueta
06ebb27160 Optimize internal funcion _cdb_unlimited_text_column 2016-04-18 18:50:37 +02:00
Javier Goizueta
bd5ae84e90 Optimize CDB_ColumnNames 
This implementation is about 1000 times faster
 2016-04-18 18:49:58 +02:00
Raul Ochoa
de5a702510 Adds table for storing camshaft analysis nodes 2016-04-18 17:41:39 +02:00
Javier Goizueta
ef43623f77 Remove unneeded variable 2016-04-15 17:58:03 +02:00
Javier Goizueta
1b0f77aa96 Always retain single-valued aggregated texts 
This makes columns which have the same value in a group to be aggregated
maintain that value (rather than replacing it by the multiple-value
indicator *) whatever the group value is. (Previously this happend
only for small groups)
 2016-04-15 17:49:00 +02:00
Javier Goizueta
45f063d469 Aggregate small number of text items in overviews 
Instead of nulling text fields for non-singleton aggregated records
concatenate distinct text values when they're few (5 or less).
Fixes #231
 2016-04-15 12:37:16 +02:00
Javier Goizueta
176d69d09e Fix AVG computation in overview tables 
Fixes #232
Averages of averages are not equal to overall averages.
 2016-04-15 10:48:08 +02:00
Javier Goizueta
9fdbfda60a Merge pull request #228 from CartoDB/225-no-centroid-master 
Use cell centers, not cluster centroids when grouping points
 2016-04-15 10:06:44 +02:00
Javier Goizueta
9a3d93976c Merge pull request #227 from CartoDB/226-add_count_aggregated_features 
Include and aggregate _vovw_count column to count aggregated features
 2016-04-15 10:06:05 +02:00
Javier Goizueta
46b45f6dd4 Merge pull request #224 from CartoDB/223-fix-dropoverviews 
Fix CDB_DropOverviews and CDB_Overviews problems
 2016-04-15 10:05:28 +02:00
Carla Iriberri
fd14750ce5 Rename _vovw_count to _feature_count 2016-04-14 18:23:09 +02:00
Javier Goizueta
c595e45c11 Add _vovw_count columnt to tables for which overviews are created 
Initially we planned to add this column to the queries sent to the
tiler only, but that makes the column hard to access from the editor.
 2016-04-14 17:32:18 +02:00
Javier Goizueta
f785e71d3b Fix: numeric is a valid numeric column type 
Actually this is the type of aggregated _vovw_count columns
 2016-04-14 15:46:03 +02:00
Carla
14b8cd7d99 Set default value to 1 and remove typo line 2016-04-14 12:12:17 +02:00
Carla
1a571c8a9c Set tolerance to 1 pixel 2016-04-14 11:13:59 +02:00
Carla Iriberri
8f44f5347a Fix indent for code clarity 2016-04-13 17:51:30 +02:00
Carla Iriberri
f96163265b Fix bug for tables without geom or with no potential overviews 
If the table doesn't have geometries but the createoverviews function is
called, the current geometry type checks won't work because "null" will
not give a boolean value in the type comparisons.

Also, if the createoverviews function is called over a simple table with
would not require overviews according to the strategies it is handled
correctly.
 2016-04-13 17:49:38 +02:00
Javier Goizueta
1c67214b09 Use cell centers, not cluster centroids when grouping points 
Fixes #225
 2016-04-13 11:14:09 +02:00
Carla Iriberri
16d08ef52b Include and aggregate _vovw_count column to count aggregated features 2016-04-12 11:10:55 +02:00
Javier Goizueta
15ac9a2cd9 Remove unneeded code 2016-04-07 10:30:10 +02:00
Javier Goizueta
ee61d46100 💄 rename variable for clarity 2016-04-07 10:24:02 +02:00
Javier Goizueta
49e7094c8a Make CDB_CreateOverview usable by superuser 
CDB_CreateOverview had to be executed with the user role
corresponding to the owner of the table; now it can be executed
by the postgres user.
 2016-04-07 07:52:58 +02:00
Javier Goizueta
fb910be12f Fix conversion of regclass to indentifier 2016-04-07 07:07:20 +02:00
Javier Goizueta
34c39662ec Replace use of CDB_UserTables in CDB_Overviews 
Use a function that returns reclasses and schema names properly instead.
 2016-04-07 00:07:45 +02:00
Javier Goizueta
84cac16d1c Temporary fix 2016-04-06 22:05:00 +02:00
Javier Goizueta
c1fc07d2ac Fix typo 
This function isn't beint actively used; should consider removing it
or testing it properly
 2016-04-06 18:58:37 +02:00
Javier Goizueta
5c3c0f5fc9 Fix bug in CDB_DropOverviews 
Fixes #223
 2016-04-06 18:57:52 +02:00
Javier Goizueta
06bb669d4c Add comment 
Clarify the reason why some functions are dropped at the
end of the file and not where their new definitions are.
 2016-04-05 13:22:07 +02:00
Javier Goizueta
00a3d6e650 Fix upgrades from prior versions 2016-04-05 13:10:00 +02:00
Javier Goizueta
f0ff197c56 Merge branch 'master' into 219-overviews-tolerance 
# Conflicts:
#	scripts-available/CDB_Overviews.sql
#	test/CDB_OverviewsTest_expect
 2016-04-05 13:08:22 +02:00
Javier Goizueta
57c32332e2 New function CDB_CreateOverviewsWithToleranceInPixels 
This function allows defining the previously fixed parameter grid_px.
The lim parameter used to define the reference Z level is also
correlated to this value.

Fixes #219
 2016-04-04 19:21:10 +02:00
Javier Goizueta
3c71eecbae Fix cases that require explicit schema name 
This allows using overview functions in situations where
the base tables require being qualified with the schema name.
 2016-04-04 17:45:20 +02:00
Javier Goizueta
c0262a05eb Fix CDB_Overviews for cases that require explicit schema name 
Fixes #218
 2016-04-04 17:02:59 +02:00
Javier Goizueta
4074173c05 Add support for boolean columns in overview creation 2016-03-21 12:58:29 +01:00
Carla
90fa45b59d Remove casting to bigint 2016-03-16 11:03:46 +01:00
Carla
16cf70bb4a Merge pull request #202 from CartoDB/cartodbfication_cartodb_id_text 
Add support to detect string cartodb_id columns
 2016-03-15 10:49:22 +01:00
Carla
0b8bada553 Remove unused variable 2016-03-09 11:54:59 +01:00
Carla Iriberri
7b48c2765e Fix error detection and fix tests 2016-03-08 14:06:58 +01:00
Carla Iriberri
d495bd45ba Merge branch 'cartodbfication_cartodb_id_text' of https://github.com/CartoDB/cartodb-postgresql into cartodbfication_cartodb_id_text 2016-03-08 11:59:00 +01:00
Carla Iriberri
f4b51807a1 Make cartodb_id inconsistencies fail and update tests 2016-03-08 11:58:30 +01:00
Carla
5bc725c8ab Add drop function if exists 2016-03-03 16:58:31 +01:00
Carla Iriberri
76a2cb9132 Improve exception error message 2016-03-02 17:36:45 +01:00
Alejandro Martínez
90c16fdb13 Fully qualify invalidations 2016-03-02 16:22:22 +01:00
Carla Iriberri
a18cbeb2cd Third iteration, expect a viable cartodb_id 2016-03-02 14:58:53 +01:00
Carla Iriberri
0d5f83b3c4 Rewriting function 2016-03-01 19:17:45 +01:00
Carla Iriberri
1198454046 Better exception handling 2016-03-01 17:43:21 +01:00
Carla Iriberri
32307ceef0 Add support to detect string cartodb_id columns 2016-03-01 15:24:40 +01:00
Rafa de la Torre
a0fe55bd5d Add a small bit of func doc about the quoting 2016-02-12 11:27:28 +01:00
Alejandro Martínez
56fed12392 Add CDB_TableMetadata_Text view as a proxy to access FDW tablemetadata 2016-02-11 19:16:00 +01:00
Rafa de la Torre
2f26b44142 tDo not quote dbname identifier 
Turns out that for caching it is our standard not to have the dbname
quoted.
 2016-02-11 16:10:27 +01:00
Rafa de la Torre
06036e2fe8 Quote identifiers returned by CDB_QueryTables_Updated_At 2016-02-10 18:58:01 +01:00
Rafa de la Torre
cd4ad29e39 Remove schema when selecting from CDB_TableMetadata 
to ease testing. When creating CDB_TableMetadata it is always put in
cartodb when done from the extension, but for tests it is done in
public.
 2016-02-09 18:49:20 +01:00
Rafa de la Torre
d59b826d37 Fix silly bug: ordering of functions 2016-02-09 17:11:16 +01:00
Rafa de la Torre
1c637f8689 Small fixes: qualify cartodb function calls 2016-02-09 16:38:11 +01:00
Rafa de la Torre
ecbdb4a430 Move fdw-aware functions to CDB_ForeignTable.sql 2016-02-09 13:40:18 +01:00
Rafa de la Torre
eb84dd04c9 Add func CDB_Last_Updated_Time 2016-02-09 13:30:00 +01:00
Rafa de la Torre
276b5cf9ea Rename func to CDB_QueryTables_Updated_At 
s/CDB_QueryTablesUpdatedAt/CDB_QueryTables_Updated_At and also add a bit
more inline doc.
 2016-02-09 13:29:10 +01:00
Rafa de la Torre
ec34b8ee28 Minor changes: use plain SQL func 2016-02-08 19:29:14 +01:00
Rafa de la Torre
0f21db51b6 Simplify the code and get the updated_at 2016-02-08 19:29:14 +01:00
Rafa de la Torre
a074f4df5d Adapt CDB_QueryTablesUpdatedAt to _cdb_fqtn_from_text 2016-02-08 19:29:14 +01:00
Rafa de la Torre
1a12fd3b69 Make _cdb_fqtn_from_text return a table 
with type instead of a record
 2016-02-08 19:29:14 +01:00
Rafa de la Torre
78a75cf22d Implementation of _cdb_fqtn_from_text (WIP) 2016-02-08 19:29:14 +01:00
Rafa de la Torre
1e3c7ace99 Implementation of _cdb_dbname_of_foreign_table (WIP) 2016-02-08 19:29:14 +01:00
Rafa de la Torre
c210008184 Skeleton of a possible solution (WIP) 2016-02-08 19:29:14 +01:00
Rafa de la Torre
0c43fe2731 Define API of CDB_QueryTablesUpdatedAt 2016-02-08 19:29:14 +01:00
Alejandro Martínez
edf79d9368 Add remote cdb_tablemetadata manually from public schema 2016-02-08 15:45:58 +01:00
Rafa de la Torre
8dedd2b3f4 Fix small typos: s/name/fdw_name/ 2016-02-05 13:15:20 +01:00
Rafa de la Torre
11834dfdab Fix typo: missing semicolon 2016-02-05 10:44:55 +01:00
Alejandro Martínez
b957635e78 Add CDB_Get_Foreign_Updated_At function 2016-02-04 18:26:43 +01:00
Alejandro Martínez
a9b9f1ff6c Rename CDB_FDW_Create to CDB_FDW_Setup, formatting fixes 2016-02-04 12:06:22 +01:00
Alejandro Martínez
d7b560324a Add _Create_FDW function to create a FDW defined on the config by name 2016-02-03 18:07:23 +01:00
Alejandro Martínez
7b52058265 Add CDB_ForeignTable functions 2016-02-03 17:50:13 +01:00
Javier Goizueta
5285943dbf Fix function declaration for create_from_unpackaged.sh use 
The script create_from_unpackaged.sh does not support
function declarations where the signature (name and parameter list
of the function, including parentheses) is not on a single line.
 2016-02-01 18:56:55 +01:00
Javier Goizueta
06dd31f4ad Fix: should be floating point divistion, not integer division 2016-01-29 16:51:52 +01:00
Javier Goizueta
6a11698a16 Merge branch 'overviews' into sql_lang 2016-01-29 16:29:23 +01:00
Javier Goizueta
00bd302f01 Avoid creating GridCluster overviews for non-point datasets 2016-01-29 16:19:35 +01:00
Javier Goizueta
93d4a6ead0 Restrict SECURITY DEFINER to overview registration 
And check it is applied only overview tables with valid names
 2016-01-28 17:04:06 +01:00
Javier Goizueta
66387c2d44 Make CDB_CreateOverviews a SECURITY DEFINER function 
So that it can change the permissions of overview tables
(by updating pg_class) even when executed by non-priviledeged users.
 2016-01-28 16:32:40 +01:00
Javier Goizueta
26c95347cd Merge branch 'master' into overviews 2016-01-27 16:56:15 +01:00
Javier Goizueta
b8d50204dd Avoid aggregation which causes out-of-memory crashes in PostgreSQL 
The use of multiple string_agg functions, even if applied to groups
of one single record causes out of memory crashes in PG 9.3.4 for
some (large) tables.
 2016-01-27 15:24:04 +01:00
Javier Goizueta
474de01757 Change the overview table naming scheme 
The scheme is changed from table_ovN to _vovw_N_table for
lower collision probability.
Also future naming changes will be easier by using the functions
_CDB_OverviewTableDiscriminator, _CDB_OverviewTableName,
_CDB_IsOverviewTableOf, etc.
 2016-01-26 13:20:28 +01:00
Javier Goizueta
1ebaeb76ac Overload CDB_Overviews to accept either a single table or an array 
The result of CDB_Overviews has been expandend with a base_table column
 2016-01-26 10:39:26 +01:00
Javier Goizueta
7033a8d9ac Fixes for table names that require quotes or which include a schema name 
Some cases left unsolved, with FIXME comments
 2016-01-13 18:49:27 +01:00
Javier Goizueta
46bc774d38 Fix CDB_Overviews for the case that the table name requires quoting 2016-01-13 18:24:06 +01:00
Javier Goizueta
a9e2d19918 Document CDB_DropOverviews 2016-01-12 16:20:11 +01:00
Javier Goizueta
5b47c51221 Add function to drop overviews 2016-01-12 15:35:50 +01:00
Javier Goizueta
ccdf8de59e Enhance documentation, update comments 2016-01-12 12:08:41 +01:00
Javier Goizueta
02f386be33 Regard the GridCluster reduction strategy as private 
The *strategy* parameteriation will be considered an
implementation detail for the time being.
 2016-01-12 11:59:42 +01:00
Javier Goizueta
a7c70fe497 Fix: CDB_ZoomFromScale deviation for lowest zoom levels 2016-01-12 11:46:47 +01:00
Javier Goizueta
0a066e0126 Change the default overview reduction strategy to GridCluster 2016-01-11 15:35:04 +01:00
Javier Goizueta
913640e2dc Preserve the column ordering of the base table in overviews 2016-01-11 15:34:10 +01:00
Javier Goizueta
ef7e613d41 Change CDB_ColumnNames to order columns names as in table 2016-01-11 15:28:27 +01:00
Javier Goizueta
fd7a8cff71 Change regular expression for consistency 2016-01-11 15:27:05 +01:00
Javier Goizueta
6ea63af974 Add function to obtain existing overviews of a table 2016-01-11 14:50:10 +01:00
Javier Goizueta
496f079b1c Copy dataset privileges to overview tables 2016-01-04 18:06:56 +01:00
Javier Goizueta
4580c9cd5c Change the cartodb_ids of aggregated overviews 
Instead of arbitrary ids, the id of one of the aggregated records
is used, so that if it is used in the UI to query point
the information of one of the records grouped at the point will be
retrieved.
 2016-01-04 10:33:39 +01:00
Javier Goizueta
08828b8b7d Set owner of overlays to the owner of the base table 2015-12-29 15:22:12 +01:00
Javier Goizueta
cecba655eb Preserve aggregated attributes of singleton groups 2015-12-29 14:56:47 +01:00
Javier Goizueta
b34a752172 Generate overviews for all Z levels 
Skipping levels produces visually inferior results
 2015-12-29 14:56:04 +01:00
Javier Goizueta
8ba9e74c4a Order columns of aggregated overlays as in the base table 2015-12-29 13:38:07 +01:00
Javier Goizueta
6a6a5bc96a Fix Sampling reduction to avoid RandomTids problems 
The fixed cases will not be common but do occur in tests.
This is an interim fix which should be reverted if CDB_randomTids changes.
 2015-12-28 19:41:14 +01:00
Javier Goizueta
2ff686de27 Clean up: remove spurious comment 2015-12-28 19:30:20 +01:00
Javier Goizueta
07326626b7 More efficient sampling strategy 
It is also renamed properly.
 2015-12-28 19:28:20 +01:00
Javier Goizueta
4afc427008 Rename Ref. Z strategy function 
It's not that *dummy* anymore, so choose a more descriptive name
 2015-12-28 17:47:57 +01:00
Alejandro Martínez
2a2a7d534a Merge remote-tracking branch 'origin/master' into remove_schema_triggers 2015-12-23 17:09:41 +01:00
Javier Goizueta
a8a2c04d71 Remove invalid comment 2015-12-23 14:20:57 +01:00
Javier Goizueta
a5bca7d715 Add function for post-processing new overview tables 2015-12-23 14:17:50 +01:00
Javier Goizueta
552206464e Make strategy function public 2015-12-23 12:50:04 +01:00
Javier Goizueta
2af0b9a57f Add function comments 2015-12-23 12:42:40 +01:00
Javier Goizueta
1b5de84c9a Add missing attribute-aggregationto the point reduction strategy 2015-12-23 12:17:37 +01:00
Javier Goizueta
5a78ee2896 Optimize the gridded clustering strategy 
The internal grid_px parameter is adjusted for best results with default symbol size
 2015-12-22 17:59:49 +01:00
Javier Goizueta
c8a1ef6f68 Slight optimization of gridded clustering 2015-12-22 15:25:21 +01:00
Paul Norman
ea7c16fbaf Convert some simple functions from plpgsql to sql 
SQL is a faster language to call, and these are very simple functions.
 2015-12-21 23:59:26 -08:00
Rafa de la Torre
564ab75d2d Use _cdb_estimated_extent instead of ST_Extent 
With a 3.8M points table, this was a ~30% gain in my local env.
 2015-12-21 18:41:50 +01:00
Rafa de la Torre
5010109c7d Add _cdb_estimated_extent to get the extent from stats 2015-12-21 18:41:45 +01:00
Rafa de la Torre
e60f73a31b A bit of internal documentation 2015-12-21 13:16:57 +01:00
Javier Goizueta
415a09392f Gridded clustering aggregation strategy for overviews 2015-12-16 17:45:36 +01:00
Javier Goizueta
e5cc9ef0bd Fix: typo 2015-12-16 17:26:17 +01:00
Javier Goizueta
f7857945c1 Change feature density computation algorithm 
Now parameterize by the number of levels to traverse and
start with the level that covers the extent of the table
with at least N*N tiles (N=4).
 2015-12-16 16:39:38 +01:00
Javier Goizueta
554464e43e Use tile-recursive computation of feature density 
This is a more adaptive way of estimating the feature density to
determine the base Z level.
Applying technique from http://javisantana.com/2014/10/22/traversing-quadtree.html
 2015-12-16 12:14:37 +01:00
Javier Goizueta
d7c8f3d7e8 Fix: overlay generation was not using the proper scale 2015-12-15 19:13:39 +01:00
Javier Goizueta
4c85d7f3ad Compute the reference Z level for a table 2015-12-15 19:12:48 +01:00
Javier Goizueta
d0e66910a0 Sketch for new Overview-creation functionality 2015-12-15 17:36:27 +01:00
Stuart Lynn
0ecbbd8e71 Make sure that empty arrays or arrays with all the same entry return 0 for Skewness and Kurtosis rather than throwing a division by zero error 2015-12-04 14:54:15 -05:00
Javier Goizueta
21d5ee5813 Merge pull request #178 from CartoDB/disable-log-invalidation-time 
Disable log invalidation time
 2015-11-27 15:03:46 +01:00