cartodb/Windshaft-cartodb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'separate-app-and-controllers-creation' into separate-routers

Browse Source
This commit is contained in:
Daniel García Aubert 2018-04-10 15:59:05 +02:00
commit bc45b50290
16 changed files with 1110 additions and 54 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
NEWS.md
View File

@ -1,8 +1,16 @@
# Changelog # Changelog
## 6.0.1 ## 6.1.0
Released 2018-mm-dd Released 2018-mm-dd
New features:
- Aggreation filters
Bug Fixes:
- Non-default aggregation selected the wrong columns (e.g. for vector tiles)
- Aggregation dimensions with alias where broken
- cartodb_id was not unique accross aggregated vector tiles
## 6.0.0 ## 6.0.0
Released 2018-03-19 Released 2018-03-19
Backward incompatible changes: Backward incompatible changes:

2
docs/MapConfig-Aggregation-extension.md
View File

@ -29,7 +29,7 @@ The value of this attribute can be `false` to explicitly disable aggregation for
// object, defines the columns of the aggregated datasets. Each property corresponds to a columns name and // object, defines the columns of the aggregated datasets. Each property corresponds to a columns name and
// should contain an object with two properties: "aggregate_function" (one of "sum", "max", "min", "avg", "mode" or "count"), // should contain an object with two properties: "aggregate_function" (one of "sum", "max", "min", "avg", "mode" or "count"),
// and "aggregated_column" (the name of a column of the original layer query or "*") // and "aggregated_column" (the name of a column of the original layer query or "*")
// A column defined as `"_cdb_features_count": {"aggregate_function": "count", aggregated_column: "*"}` // A column defined as `"_cdb_feature_count": {"aggregate_function": "count", aggregated_column: "*"}`
// is always generated in addition to the defined columns. // is always generated in addition to the defined columns.
// The column names `cartodb_id`, `the_geom`, `the_geom_webmercator` and `_cdb_feature_count` cannot be used // The column names `cartodb_id`, `the_geom`, `the_geom_webmercator` and `_cdb_feature_count` cannot be used
// for aggregated columns, as they correspond to columns always present in the result. // for aggregated columns, as they correspond to columns always present in the result.

81
docs/aggregation.md
View File

@ -10,7 +10,7 @@ Aggregation is available only for point geometries. During aggregation the point
When no placement or columns are specified a special default aggregation is performed. When no placement or columns are specified a special default aggregation is performed.
This special mode performs only spatial aggregation (using a grid defined by the requested tile and the resolution, parameter, as all the other cases), and returns a _random_ record from each group (grid cell) with all its columns and an additional `_cdb_features_count` with the number of features in the group. This special mode performs only spatial aggregation (using a grid defined by the requested tile and the resolution, parameter, as all the other cases), and returns a _random_ record from each group (grid cell) with all its columns and an additional `_cdb_feature_count` with the number of features in the group.
Regarding the randomness of the sample: currently we use the row with the minimum `cartodb_id` value in each group. Regarding the randomness of the sample: currently we use the row with the minimum `cartodb_id` value in each group.
@ -18,7 +18,7 @@ The rationale behind having this special aggregation with all the original colum
### User defined aggregations ### User defined aggregations
When either a explicit placement or columns are requested we no longer use the special, query; we use one determined by the placement (which will default to "centroid"), and it will have as columns only the aggregated columns specified, in addition to `_cdb_features_count`, which is always present. When either a explicit placement or columns are requested we no longer use the special, query; we use one determined by the placement (which will default to "centroid"), and it will have as columns only the aggregated columns specified, in addition to `_cdb_feature_count`, which is always present.
We might decide in the future to allow sampling column values for any of the different placement modes. We might decide in the future to allow sampling column values for any of the different placement modes.
@ -185,3 +185,80 @@ This is the minimum number of (estimated) rows in the dataset (query results) fo
] ]
} }
``` ```
### `filters`
Aggregated data can be filtered by imposing filtering conditions on the aggregated columns.
Each condition is represented by one or more parameters:
* `{ "equal": V }` selects an specific value of the aggregated column.
* `{ "not_equal": V }` selects values different from the one specified.
* `{ "in": [v1, v2, v3] }` selects any value from a list.
* `{ "not_in": [v1, v2, v3] }` selects any value not in a list.
* `{ "less_than": v }` selects values strictly less than the one given.
* `{ "less_than_or_equal_to": v }` selects values less than or equal to the one given.
* `{ "greater_than": v }` selects values strictly greater than the one given.
* `{ "greater_than_or_equal_to": v }` selects values greater than or equal to the one given.
One of the *less* conditions can be combined with one of the *greater* conditions to select a range of values, for example:
* `{ "greater_than": v1, "less_than": v2 }`
* `{ "greater_than_or_equal_to": v1, "less_than": v2 }`
* `{ "greater_than": v1, "less_than_or_equal_to": v2 }`
* `{ "greater_than_or_equal_to": v1, "less_than_or_equal_to": v2 }`
For a given column, multiple conditions can be passed in an array; the conditions will logically ORed (any of the conditions have to be verifid for the value to be selected):
* `"myvalue": [ { "equal": 10 }, { "less_than": 0 }]` will select values of the column `myvalue` which are equal to 10 **or** less than 0.
In addition, the filters applied to different columns are logically combined with AND (all the conditions have to be satisfied for an element to be selected); for example with the following `filters` parameter we'll select aggregated records which have a `total_value` > 100 **and** a category equal to "a".
```json
{
"total_value": { "greater_than": 100 },
"category": { "equal": "a" }
}
```
Note that the filtered columns have to be defined with the `columns` parameter, except for `_cdb_feature_count`, which is always implicitly defined and can be filtered too.
#### Example
```json
{
"version": "1.7.0",
"extent": [-20037508.5, -20037508.5, 20037508.5, 20037508.5],
"srid": 3857,
"maxzoom": 18,
"minzoom": 3,
"layers": [
{
"type": "mapnik",
"options": {
"sql": "select * from table",
"cartocss": "#table { marker-width: [total]; marker-fill: ramp(value, (red, green, blue), jenks); }",
"cartocss_version": "2.3.0",
"aggregation": {
"placement": "centroid",
"columns": {
"total_value": {
"aggregate_function": "sum",
"aggregated_column": "value"
},
"category": {
"aggregate_function": "mode",
"aggregated_column": "category"
}
},
"filters" : {
"total_value": { "greater_than": 100 },
"category": { "equal": "a" }
},
"resolution": 2,
"threshold": 500000
}
}
}
]
}
```

2
lib/cartodb/api/auth_api.js
View File

@ -109,7 +109,7 @@ AuthApi.prototype.authorizedByAPIKey = function(user, res, callback) {
return callback(error); return callback(error);
} }
return callback(null, true); return callback(null, true, apikey);
}); });
}; };

2
lib/cartodb/controllers/map/middlewares/last-updated-time-layergroup.js
View File

@ -3,7 +3,7 @@ module.exports = function setLastUpdatedTimeToLayergroup () {
const { mapConfigProvider, analysesResults } = res.locals; const { mapConfigProvider, analysesResults } = res.locals;
const layergroup = res.body; const layergroup = res.body;
mapConfigProvider.getAffectedTables((err, affectedTables) => { mapConfigProvider.createAffectedTables((err, affectedTables) => {
if (err) { if (err) {
return next(err); return next(err);
} }

11
lib/cartodb/controllers/template/admin.js
View File

@ -79,7 +79,7 @@ function authorizedByAPIKey ({ authApi, action, label }) {
return function authorizedByAPIKeyMiddleware (req, res, next) { return function authorizedByAPIKeyMiddleware (req, res, next) {
const { user } = res.locals; const { user } = res.locals;
authApi.authorizedByAPIKey(user, res, (err, authenticated) => { authApi.authorizedByAPIKey(user, res, (err, authenticated, apikey) => {
if (err) { if (err) {
return next(err); return next(err);
} }
@ -91,6 +91,15 @@ function authorizedByAPIKey ({ authApi, action, label }) {
return next(error); return next(error);
} }
if (apikey.type !== 'master') {
const error = new Error('Forbidden');
error.type = 'auth';
error.subtype = 'api-key-does-not-grant-access';
error.http_status = 403;
return next(error);
}
next(); next();
}); });
}; };

2
lib/cartodb/controllers/template/preview.js
View File

@ -153,7 +153,7 @@ function getStaticImageOptions ({ tablesExtentApi }) {
res.locals.imageOpts = DEFAULT_ZOOM_CENTER; res.locals.imageOpts = DEFAULT_ZOOM_CENTER;
mapConfigProvider.getAffectedTables((err, affectedTables) => { mapConfigProvider.createAffectedTables((err, affectedTables) => {
if (err) { if (err) {
return next(); return next();
} }

38
lib/cartodb/models/aggregation/aggregation-mapconfig.js
View File

@ -4,7 +4,8 @@ const aggregationValidator = require('./aggregation-validator');
const { const {
createPositiveNumberValidator, createPositiveNumberValidator,
createIncludesValueValidator, createIncludesValueValidator,
createAggregationColumnsValidator createAggregationColumnsValidator,
createAggregationFiltersValidator
} = aggregationValidator; } = aggregationValidator;
const SubstitutionTokens = require('../../utils/substitution-tokens'); const SubstitutionTokens = require('../../utils/substitution-tokens');
@ -43,6 +44,18 @@ module.exports = class AggregationMapConfig extends MapConfig {
]; ];
} }
static get FILTER_PARAMETERS () {
return [
// TODO: valid combinations of parameters:
// * Except for less/greater params, only one parameter allowed per filter.
// * Any less parameter can be combined with one of the greater paramters. (to define a range)
'less_than', 'less_than_or_equal_to',
'greater_than', 'greater_than_or_equal_to',
'equal', 'not_equal',
'between', 'in', 'not_in'
];
}
static supportsGeometryType(geometryType) { static supportsGeometryType(geometryType) {
return AggregationMapConfig.SUPPORTED_GEOMETRY_TYPES.includes(geometryType); return AggregationMapConfig.SUPPORTED_GEOMETRY_TYPES.includes(geometryType);
} }
@ -58,11 +71,15 @@ module.exports = class AggregationMapConfig extends MapConfig {
const positiveNumberValidator = createPositiveNumberValidator(this); const positiveNumberValidator = createPositiveNumberValidator(this);
const includesValidPlacementsValidator = createIncludesValueValidator(this, AggregationMapConfig.PLACEMENTS); const includesValidPlacementsValidator = createIncludesValueValidator(this, AggregationMapConfig.PLACEMENTS);
const aggregationColumnsValidator = createAggregationColumnsValidator(this, AggregationMapConfig.AGGREGATIONS); const aggregationColumnsValidator = createAggregationColumnsValidator(this, AggregationMapConfig.AGGREGATIONS);
const aggregationFiltersValidator = createAggregationFiltersValidator(
this, AggregationMapConfig.FILTER_PARAMETERS
);
validate('resolution', positiveNumberValidator); validate('resolution', positiveNumberValidator);
validate('placement', includesValidPlacementsValidator); validate('placement', includesValidPlacementsValidator);
validate('threshold', positiveNumberValidator); validate('threshold', positiveNumberValidator);
validate('columns', aggregationColumnsValidator); validate('columns', aggregationColumnsValidator);
validate('filters', aggregationFiltersValidator);
this.user = user; this.user = user;
this.pgConnection = connection; this.pgConnection = connection;
@ -77,7 +94,8 @@ module.exports = class AggregationMapConfig extends MapConfig {
threshold = AggregationMapConfig.THRESHOLD, threshold = AggregationMapConfig.THRESHOLD,
placement, placement,
columns = {}, columns = {},
dimensions = {} dimensions = {},
filters = {}
} = this.getAggregation(index); } = this.getAggregation(index);
return aggregationQuery({ return aggregationQuery({
@ -87,6 +105,7 @@ module.exports = class AggregationMapConfig extends MapConfig {
placement, placement,
columns, columns,
dimensions, dimensions,
filters,
isDefaultAggregation: this._isDefaultLayerAggregation(index) isDefaultAggregation: this._isDefaultLayerAggregation(index)
}); });
} }
@ -152,21 +171,19 @@ module.exports = class AggregationMapConfig extends MapConfig {
_getLayerAggregationRequiredColumns (index) { _getLayerAggregationRequiredColumns (index) {
const { columns, dimensions } = this.getAggregation(index); const { columns, dimensions } = this.getAggregation(index);
let finalColumns = ['cartodb_id', '_cdb_feature_count'];
let aggregatedColumns = []; let aggregatedColumns = [];
if (columns) { if (columns) {
aggregatedColumns = Object.keys(columns) aggregatedColumns = Object.keys(columns);
.map(key => columns[key].aggregated_column)
.filter(aggregatedColumn => typeof aggregatedColumn === 'string');
} }
let dimensionsColumns = []; let dimensionsColumns = [];
if (dimensions) { if (dimensions) {
dimensionsColumns = Object.keys(dimensions) dimensionsColumns = Object.keys(dimensions);
.map(key => dimensions[key])
.filter(dimension => typeof dimension === 'string');
} }
return removeDuplicates(aggregatedColumns.concat(dimensionsColumns)); return removeDuplicates(finalColumns.concat(aggregatedColumns).concat(dimensionsColumns));
} }
doesLayerReachThreshold(index, featureCount) { doesLayerReachThreshold(index, featureCount) {
@ -220,7 +237,8 @@ module.exports = class AggregationMapConfig extends MapConfig {
_isDefaultAggregation (aggregation) { _isDefaultAggregation (aggregation) {
return aggregation.placement === undefined && return aggregation.placement === undefined &&
aggregation.columns === undefined && aggregation.columns === undefined &&
this._isEmptyParameter(aggregation.dimensions); this._isEmptyParameter(aggregation.dimensions) &&
this._isEmptyParameter(aggregation.filters);
} }
_isEmptyParameter(parameter) { _isEmptyParameter(parameter) {

135
lib/cartodb/models/aggregation/aggregation-query.js
View File

@ -42,7 +42,8 @@ const queryForOptions = (options) => templateForOptions(options)({
sourceQuery: options.query, sourceQuery: options.query,
res: 256/options.resolution, res: 256/options.resolution,
columns: options.columns, columns: options.columns,
dimensions: options.dimensions dimensions: options.dimensions,
filters: options.filters
}); });
module.exports = queryForOptions; module.exports = queryForOptions;
@ -93,20 +94,23 @@ const aggregateColumnNames = (ctx, table) => {
return sep(Object.keys(columns)); return sep(Object.keys(columns));
}; };
const aggregateExpression = (column_name, column_parameters) => {
const aggregate_function = column_parameters.aggregate_function || 'count';
const aggregate_definition = SUPPORTED_AGGREGATE_FUNCTIONS[aggregate_function];
if (!aggregate_definition) {
throw new Error("Invalid Aggregate function: '" + aggregate_function + "'");
}
return aggregate_definition.sql(column_name, column_parameters);
};
const aggregateColumnDefs = ctx => { const aggregateColumnDefs = ctx => {
let columns = aggregateColumns(ctx); let columns = aggregateColumns(ctx);
return sep(Object.keys(columns).map(column_name => { return sep(Object.keys(columns).map(column_name => {
const aggregate_function = columns[column_name].aggregate_function || 'count'; const aggregate_expression = aggregateExpression(column_name, columns[column_name]);
const aggregate_definition = SUPPORTED_AGGREGATE_FUNCTIONS[aggregate_function];
if (!aggregate_definition) {
throw new Error("Invalid Aggregate function: '" + aggregate_function + "'");
}
const aggregate_expression = aggregate_definition.sql(column_name, columns[column_name]);
return `${aggregate_expression} AS ${column_name}`; return `${aggregate_expression} AS ${column_name}`;
})); }));
}; };
const aggregateDimensions = ctx => ctx.dimensions || {}; const aggregateDimensions = ctx => ctx.dimensions || {};
const dimensionNames = (ctx, table) => { const dimensionNames = (ctx, table) => {
@ -127,6 +131,111 @@ const dimensionDefs = ctx => {
})); }));
}; };
const aggregateFilters = ctx => ctx.filters || {};
const filterConditionSQL = (expr, filter) => {
// TODO: validate filter parameters (e.g. cannot have both greater_than and greater_than or equal to)
if (filter) {
if (!Array.isArray(filter)) {
filter = [filter];
}
if (filter.length > 0) {
return filter.map(f => filterSingleConditionSQL(expr, f)).join(' OR ');
}
}
};
const filterSingleConditionSQL = (expr, filter) => {
let cond;
Object.keys(FILTERS).some(f => {
cond = FILTERS[f](expr, filter);
return cond;
});
return cond;
};
const sqlQ = (value) => {
if (isFinite(value)) {
return String(value);
}
return `'${value}'`; // TODO: escape single quotes! (by doubling them)
};
/* jshint eqeqeq: false */
/* x != null is used to check for both null and undefined; triple !== wouldn't do the trick */
const FILTERS = {
between: (expr, filter) => {
const lo = filter.greater_than_or_equal_to, hi = filter.less_than_or_equal_to;
if (lo != null && hi != null) {
return `(${expr} BETWEEN ${sqlQ(lo)} AND ${sqlQ(hi)})`;
}
},
in: (expr, filter) => {
if (filter.in != null) {
return `(${expr} IN (${filter.in.map(v => sqlQ(v)).join(',')}))`;
}
},
notin: (expr, filter) => {
if (filter.not_in != null) {
return `(${expr} NOT IN (${filter.not_in.map(v => sqlQ(v)).join(',')}))`;
}
},
equal: (expr, filter) => {
if (filter.equal != null) {
return `(${expr} = ${sqlQ(filter.equal)})`;
}
},
not_equal: (expr, filter) => {
if (filter.not_equal != null) {
return `(${expr} <> ${sqlQ(filter.not_equal)})`;
}
},
range: (expr, filter) => {
let conds = [];
if (filter.greater_than_or_equal_to != null) {
conds.push(`(${expr} >= ${sqlQ(filter.greater_than_or_equal_to)})`);
}
if (filter.greater_than != null) {
conds.push(`(${expr} > ${sqlQ(filter.greater_than)})`);
}
if (filter.less_than_or_equal_to != null) {
conds.push(`(${expr} <= ${sqlQ(filter.less_than_or_equal_to)})`);
}
if (filter.less_than != null) {
conds.push(`(${expr} < ${sqlQ(filter.less_than)})`);
}
if (conds.length > 0) {
return conds.join(' AND ');
}
}
};
const filterConditions = ctx => {
let columns = aggregateColumns(ctx);
let dimensions = aggregateDimensions(ctx);
let filters = aggregateFilters(ctx);
return Object.keys(filters).map(filtered_column => {
let filtered_expr;
if (columns[filtered_column]) {
filtered_expr = aggregateExpression(filtered_column, columns[filtered_column]);
}
else if (dimensions[filtered_column]) {
filtered_expr = dimensions[filtered_column];
}
if (!filtered_expr) {
throw new Error("Invalid filtered column: '" + filtered_column + "'");
}
return filterConditionSQL(filtered_expr, filters[filtered_column]);
}).join(' AND ');
};
const havingClause = ctx => {
let cond = filterConditions(ctx);
return cond ? `HAVING ${cond}` : '';
};
// SQL expression to compute the aggregation resolution (grid cell size). // SQL expression to compute the aggregation resolution (grid cell size).
// This is equivalent to `${256/ctx.res}*CDB_XYZ_Resolution(CDB_ZoomFromScale(!scale_denominator!))` // This is equivalent to `${256/ctx.res}*CDB_XYZ_Resolution(CDB_ZoomFromScale(!scale_denominator!))`
// This is defined by the ctx.res parameter, which is the number of grid cells per tile linear dimension // This is defined by the ctx.res parameter, which is the number of grid cells per tile linear dimension
@ -181,7 +290,7 @@ const aggregationQueryTemplates = {
!bbox! AS bbox !bbox! AS bbox
) )
SELECT SELECT
row_number() over() AS cartodb_id, MIN(_cdb_query.cartodb_id) AS cartodb_id,
ST_SetSRID( ST_SetSRID(
ST_MakePoint( ST_MakePoint(
AVG(ST_X(_cdb_query.the_geom_webmercator)), AVG(ST_X(_cdb_query.the_geom_webmercator)),
@ -196,6 +305,7 @@ const aggregationQueryTemplates = {
Floor(ST_X(_cdb_query.the_geom_webmercator)/_cdb_params.res), Floor(ST_X(_cdb_query.the_geom_webmercator)/_cdb_params.res),
Floor(ST_Y(_cdb_query.the_geom_webmercator)/_cdb_params.res) Floor(ST_Y(_cdb_query.the_geom_webmercator)/_cdb_params.res)
${dimensionNames(ctx)} ${dimensionNames(ctx)}
${havingClause(ctx)}
`, `,
'point-grid': ctx => ` 'point-grid': ctx => `
@ -207,6 +317,7 @@ const aggregationQueryTemplates = {
), ),
_cdb_clusters AS ( _cdb_clusters AS (
SELECT SELECT
MIN(_cdb_query.cartodb_id) AS cartodb_id,
Floor(ST_X(_cdb_query.the_geom_webmercator)/_cdb_params.res)::int AS _cdb_gx, Floor(ST_X(_cdb_query.the_geom_webmercator)/_cdb_params.res)::int AS _cdb_gx,
Floor(ST_Y(_cdb_query.the_geom_webmercator)/_cdb_params.res)::int AS _cdb_gy Floor(ST_Y(_cdb_query.the_geom_webmercator)/_cdb_params.res)::int AS _cdb_gy
${dimensionDefs(ctx)} ${dimensionDefs(ctx)}
@ -214,9 +325,10 @@ const aggregationQueryTemplates = {
FROM (${ctx.sourceQuery}) _cdb_query, _cdb_params FROM (${ctx.sourceQuery}) _cdb_query, _cdb_params
WHERE the_geom_webmercator && _cdb_params.bbox WHERE the_geom_webmercator && _cdb_params.bbox
GROUP BY _cdb_gx, _cdb_gy ${dimensionNames(ctx)} GROUP BY _cdb_gx, _cdb_gy ${dimensionNames(ctx)}
${havingClause(ctx)}
) )
SELECT SELECT
row_number() over() AS cartodb_id, _cdb_clusters.cartodb_id AS cartodb_id,
ST_SetSRID(ST_MakePoint((_cdb_gx+0.5)*res, (_cdb_gy+0.5)*res), 3857) AS the_geom_webmercator ST_SetSRID(ST_MakePoint((_cdb_gx+0.5)*res, (_cdb_gy+0.5)*res), 3857) AS the_geom_webmercator
${dimensionNames(ctx)} ${dimensionNames(ctx)}
${aggregateColumnNames(ctx)} ${aggregateColumnNames(ctx)}
@ -241,11 +353,12 @@ const aggregationQueryTemplates = {
Floor(ST_X(_cdb_query.the_geom_webmercator)/_cdb_params.res), Floor(ST_X(_cdb_query.the_geom_webmercator)/_cdb_params.res),
Floor(ST_Y(_cdb_query.the_geom_webmercator)/_cdb_params.res) Floor(ST_Y(_cdb_query.the_geom_webmercator)/_cdb_params.res)
${dimensionNames(ctx)} ${dimensionNames(ctx)}
${havingClause(ctx)}
) )
SELECT SELECT
_cdb_clusters.cartodb_id, _cdb_clusters.cartodb_id,
the_geom, the_geom_webmercator the_geom, the_geom_webmercator
${dimensionNames(ctx, '_cdb_query')} ${dimensionNames(ctx, '_cdb_clusters')}
${aggregateColumnNames(ctx, '_cdb_clusters')} ${aggregateColumnNames(ctx, '_cdb_clusters')}
FROM FROM
_cdb_clusters INNER JOIN (${ctx.sourceQuery}) _cdb_query _cdb_clusters INNER JOIN (${ctx.sourceQuery}) _cdb_query

32
lib/cartodb/models/aggregation/aggregation-validator.js
View File

@ -42,6 +42,38 @@ module.exports.createAggregationColumnsValidator = function (mapconfig, validAgg
}; };
}; };
module.exports.createAggregationFiltersValidator = function (mapconfig, validParameters) {
return function validateAggregationFilters (value, key, index) {
const dims = mapconfig.getAggregation(index).dimensions || {};
const cols = mapconfig.getAggregation(index).columns || {};
const validKeys = Object.keys(dims).concat(Object.keys(cols));
Object.keys(value).forEach((filteredName) => {
// filteredName must be the name of either an aggregated column or a dimension in the same layer
if (!validKeys.includes(filteredName)) {
const message = `Invalid filtered column: ${filteredName}`;
throw createLayerError(message, mapconfig, index);
}
// The filter parameters must be valid
let filters = value[filteredName];
// a single filter or an array of filters (to be OR-combined) are accepted
if (!Array.isArray(filters)) {
filters = [filters];
}
filters.forEach(params => {
Object.keys(params).forEach(paramName => {
if (!validParameters.includes(paramName)) {
const message = `Invalid filter parameter name: ${paramName}`;
throw createLayerError(message, mapconfig, index);
}
});
// TODO: check parameter value (params[paramName]) to be of the correct type
});
// TODO: if multiple parameters within params check the combination is valid,
// i.e. one of the *less* parameters and one of the *greater* parameters.
});
};
};
function createAggregationColumnNamesValidator(mapconfig) { function createAggregationColumnNamesValidator(mapconfig) {
return function validateAggregationColumnNames (value, key, index) { return function validateAggregationColumnNames (value, key, index) {
Object.keys(value).forEach((columnName) => { Object.keys(value).forEach((columnName) => {

25
lib/cartodb/models/mapconfig/provider/create-layergroup-provider.js
View File

@ -58,7 +58,7 @@ CreateLayergroupMapConfigProvider.prototype.filter = MapStoreMapConfigProvider.p
CreateLayergroupMapConfigProvider.prototype.createKey = MapStoreMapConfigProvider.prototype.createKey; CreateLayergroupMapConfigProvider.prototype.createKey = MapStoreMapConfigProvider.prototype.createKey;
CreateLayergroupMapConfigProvider.prototype.getAffectedTables = function (callback) { CreateLayergroupMapConfigProvider.prototype.createAffectedTables = function (callback) {
this.getMapConfig((err, mapConfig) => { this.getMapConfig((err, mapConfig) => {
if (err) { if (err) {
return callback(err); return callback(err);
@ -67,11 +67,6 @@ CreateLayergroupMapConfigProvider.prototype.getAffectedTables = function (callba
const { dbname } = this.params; const { dbname } = this.params;
const token = mapConfig.id(); const token = mapConfig.id();
if (this.affectedTablesCache.hasAffectedTables(dbname, token)) {
const affectedTables = this.affectedTablesCache.get(dbname, token);
return callback(null, affectedTables);
}
const queries = []; const queries = [];
this.mapConfig.getLayers().forEach(layer => { this.mapConfig.getLayers().forEach(layer => {
@ -106,3 +101,21 @@ CreateLayergroupMapConfigProvider.prototype.getAffectedTables = function (callba
}); });
}); });
}; };
CreateLayergroupMapConfigProvider.prototype.getAffectedTables = function (callback) {
this.getMapConfig((err, mapConfig) => {
if (err) {
return callback(err);
}
const { dbname } = this.params;
const token = mapConfig.id();
if (this.affectedTablesCache.hasAffectedTables(dbname, token)) {
const affectedTables = this.affectedTablesCache.get(dbname, token);
return callback(null, affectedTables);
}
return this.createAffectedTables(callback);
});
};

26
lib/cartodb/models/mapconfig/provider/map-store-provider.js
View File

@ -89,7 +89,7 @@ MapStoreMapConfigProvider.prototype.createKey = function(base) {
return (base) ? baseKeyTpl(tplValues) : rendererKeyTpl(tplValues); return (base) ? baseKeyTpl(tplValues) : rendererKeyTpl(tplValues);
}; };
MapStoreMapConfigProvider.prototype.getAffectedTables = function(callback) { MapStoreMapConfigProvider.prototype.createAffectedTables = function(callback) {
this.getMapConfig((err, mapConfig) => { this.getMapConfig((err, mapConfig) => {
if (err) { if (err) {
return callback(err); return callback(err);
@ -98,12 +98,6 @@ MapStoreMapConfigProvider.prototype.getAffectedTables = function(callback) {
const { dbname } = this.params; const { dbname } = this.params;
const token = mapConfig.id(); const token = mapConfig.id();
if (this.affectedTablesCache.hasAffectedTables(dbname, token)) {
const affectedTables = this.affectedTablesCache.get(dbname, token);
return callback(null, affectedTables);
}
const queries = []; const queries = [];
mapConfig.getLayers().forEach(layer => { mapConfig.getLayers().forEach(layer => {
@ -138,3 +132,21 @@ MapStoreMapConfigProvider.prototype.getAffectedTables = function(callback) {
}); });
}); });
}; };
MapStoreMapConfigProvider.prototype.getAffectedTables = function (callback) {
this.getMapConfig((err, mapConfig) => {
if (err) {
return callback(err);
}
const { dbname } = this.params;
const token = mapConfig.id();
if (this.affectedTablesCache.hasAffectedTables(dbname, token)) {
const affectedTables = this.affectedTablesCache.get(dbname, token);
return callback(null, affectedTables);
}
return this.createAffectedTables(callback);
});
};

25
lib/cartodb/models/mapconfig/provider/named-map-provider.js
View File

@ -262,7 +262,7 @@ NamedMapMapConfigProvider.prototype.getTemplateName = function() {
return this.templateName; return this.templateName;
}; };
NamedMapMapConfigProvider.prototype.getAffectedTables = function(callback) { NamedMapMapConfigProvider.prototype.createAffectedTables = function(callback) {
this.getMapConfig((err, mapConfig) => { this.getMapConfig((err, mapConfig) => {
if (err) { if (err) {
return callback(err); return callback(err);
@ -271,11 +271,6 @@ NamedMapMapConfigProvider.prototype.getAffectedTables = function(callback) {
const { dbname } = this.rendererParams; const { dbname } = this.rendererParams;
const token = mapConfig.id(); const token = mapConfig.id();
if (this.affectedTablesCache.hasAffectedTables(dbname, token)) {
const affectedTables = this.affectedTablesCache.get(dbname, token);
return callback(null, affectedTables);
}
const queries = []; const queries = [];
mapConfig.getLayers().forEach(layer => { mapConfig.getLayers().forEach(layer => {
@ -310,3 +305,21 @@ NamedMapMapConfigProvider.prototype.getAffectedTables = function(callback) {
}); });
}); });
}; };
NamedMapMapConfigProvider.prototype.getAffectedTables = function (callback) {
this.getMapConfig((err, mapConfig) => {
if (err) {
return callback(err);
}
const { dbname } = this.params;
const token = mapConfig.id();
if (this.affectedTablesCache.hasAffectedTables(dbname, token)) {
const affectedTables = this.affectedTablesCache.get(dbname, token);
return callback(null, affectedTables);
}
return this.createAffectedTables(callback);
});
};

699
test/acceptance/aggregation.js
View File

@ -357,6 +357,103 @@ describe('aggregation', function () {
}); });
}); });
['centroid', 'point-sample', 'point-grid'].forEach(placement => {
it('should provide all the requested columns in non-default aggregation ',
function (done) {
const response = {
status: 200,
headers: {
'Content-Type': 'application/json; charset=utf-8'
}
};
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_2,
aggregation: {
placement: placement,
columns: {
'first_column': {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
dimensions: {
second_column: 'sqrt_value'
},
threshold: 1
},
cartocss: '#layer { marker-width: [first_column]; line-width: [second_column]; }',
cartocss_version: '2.3.0'
}
}
]);
this.testClient = new TestClient(this.mapConfig);
this.testClient.getLayergroup({ response }, (err, body) => {
if (err) {
return done(err);
}
assert.equal(typeof body.metadata, 'object');
assert.ok(Array.isArray(body.metadata.layers));
body.metadata.layers.forEach(layer => assert.ok(layer.meta.aggregation.mvt));
body.metadata.layers.forEach(layer => assert.ok(layer.meta.aggregation.png));
done();
});
});
it('should provide only the requested columns in non-default aggregation ',
function (done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_2,
aggregation: {
placement: placement,
columns: {
'first_column': {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
dimensions: {
second_column: 'sqrt_value'
},
threshold: 1
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
this.testClient.getTile(0, 0, 0, { format: 'mvt' }, function (err, res, mvt) {
if (err) {
return done(err);
}
const geojsonTile = JSON.parse(mvt.toGeoJSONSync(0));
let columns = new Set();
geojsonTile.features.forEach(f => {
Object.keys(f.properties).forEach(p => columns.add(p));
});
columns = Array.from(columns);
const expected_columns = [
'_cdb_feature_count', 'cartodb_id', 'first_column', 'second_column'
];
assert.deepEqual(columns.sort(), expected_columns.sort());
done();
});
});
});
it('should skip aggregation to create a layergroup with aggregation defined already', function (done) { it('should skip aggregation to create a layergroup with aggregation defined already', function (done) {
const mapConfig = createVectorMapConfig([ const mapConfig = createVectorMapConfig([
{ {
@ -689,6 +786,45 @@ describe('aggregation', function () {
}); });
}); });
['centroid', 'point-sample', 'point-grid'].forEach(placement => {
it(`dimensions with alias should work for ${placement} placement`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
placement: placement ,
threshold: 1,
dimensions: {
value2: "value"
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(
feature => assert.equal(typeof feature.properties.value2, 'number')
);
done();
});
});
});
it(`dimensions should trigger non-default aggregation`, function(done) { it(`dimensions should trigger non-default aggregation`, function(done) {
this.mapConfig = createVectorMapConfig([ this.mapConfig = createVectorMapConfig([
{ {
@ -1475,6 +1611,569 @@ describe('aggregation', function () {
done(); done();
}); });
}); });
['centroid', 'point-sample', 'point-grid'].forEach(placement => {
it(`filters should work for ${placement} placement`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
placement: placement ,
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: {
greater_than_or_equal_to: 0
}
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value >= 0);
});
done();
});
});
});
['centroid', 'point-sample', 'point-grid'].forEach(placement => {
it(`multiple ORed filters should work for ${placement} placement`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
placement: placement ,
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: [
{ greater_than: 0 },
{ less_than: -2 }
]
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value > 0 || row.properties.value < -2);
});
done();
});
});
});
['centroid', 'point-sample', 'point-grid'].forEach(placement => {
it(`multiple ANDed filters should work for ${placement} placement`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_2,
aggregation: {
placement: placement ,
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
},
value2: {
aggregate_function: 'sum',
aggregated_column: 'sqrt_value'
}
},
filters: {
value: { greater_than: 0 },
value2: { less_than: 9 }
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value > 0 && row.properties.value2 < 9);
});
done();
});
});
});
it(`supports IN filters`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: { in: [1, 3] }
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value === 1 || row.properties.value === 3);
});
done();
});
});
it(`supports NOT IN filters`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: { not_in: [1, 3] }
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value !== 1 && row.properties.value !== 3);
});
done();
});
});
it(`supports EQUAL filters`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: [{ equal: 1}, { equal: 3}]
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value === 1 || row.properties.value === 3);
});
done();
});
});
it(`supports NOT EQUAL filters`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: { not_equal: 1 }
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value !== 1);
});
done();
});
});
it(`supports BETWEEN filters`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: {
greater_than_or_equal_to: -1,
less_than_or_equal_to: 2
}
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value >= -1 || row.properties.value <= 2);
});
done();
});
});
it(`supports RANGE filters`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: {
greater_than: -1,
less_than_or_equal_to: 2
}
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
format: 'mvt'
};
this.testClient.getTile(0, 0, 0, options, (err, res, tile) => {
if (err) {
return done(err);
}
const tileJSON = tile.toJSON();
tileJSON[0].features.forEach(row => {
assert.ok(row.properties.value > -1 || row.properties.value <= 2);
});
done();
});
});
it(`invalid filters cause errors`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: {
not_a_valid_parameter: 0
}
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
response: {
status: 400
}
};
this.testClient.getLayergroup(options, (err, body) => {
if (err) {
return done(err);
}
assert.deepEqual(body, {
errors: [ 'Invalid filter parameter name: not_a_valid_parameter'],
errors_with_context:[{
type: 'layer',
message: 'Invalid filter parameter name: not_a_valid_parameter',
layer: {
id: "layer0",
index: 0,
type: "mapnik",
}
}]
});
done();
});
});
it(`filters on invalid columns cause errors`, function(done) {
this.mapConfig = createVectorMapConfig([
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
aggregation: {
threshold: 1,
columns: {
value_sum: {
aggregate_function: 'sum',
aggregated_column: 'value'
}
},
filters: {
value: {
not_a_valid_parameter: 0
}
}
}
}
}
]);
this.testClient = new TestClient(this.mapConfig);
const options = {
response: {
status: 400
}
};
this.testClient.getLayergroup(options, (err, body) => {
if (err) {
return done(err);
}
assert.deepEqual(body, {
errors: [ 'Invalid filtered column: value'],
errors_with_context:[{
type: 'layer',
message: 'Invalid filtered column: value',
layer: {
id: "layer0",
index: 0,
type: "mapnik",
}
}]
});
done();
});
});
['default', 'centroid', 'point-sample', 'point-grid'].forEach(placement => {
it(`aggregated ids are unique for ${placement} aggregation`, function (done) {
this.mapConfig = {
version: '1.6.0',
buffersize: { 'mvt': 0 },
layers: [
{
type: 'cartodb',
options: {
sql: POINTS_SQL_1,
resolution: 1,
aggregation: {
threshold: 1
}
}
}
]
};
if (placement !== 'default') {
this.mapConfig.layers[0].options.aggregation.placement = placement;
}
this.testClient = new TestClient(this.mapConfig);
this.testClient.getTile(1, 0, 1, { format: 'mvt' }, (err, res, mvt) => {
if (err) {
return done(err);
}
const tile1 = JSON.parse(mvt.toGeoJSONSync(0));
assert.ok(Array.isArray(tile1.features));
assert.ok(tile1.features.length > 0);
this.testClient.getTile(1, 1, 0, { format: 'mvt' }, (err, res, mvt) => {
if (err) {
return done(err);
}
const tile2 = JSON.parse(mvt.toGeoJSONSync(0));
assert.ok(Array.isArray(tile2.features));
assert.ok(tile2.features.length > 0);
const tile1Ids = tile1.features.map(f => f.properties.cartodb_id);
const tile2Ids = tile2.features.map(f => f.properties.cartodb_id);
const repeatedIds = tile1Ids.filter(id => tile2Ids.includes(id));
assert.equal(repeatedIds.length, 0);
done();
});
});
});
});
}); });
}); });
}); });

44
test/acceptance/auth/authorization.js
View File

@ -37,7 +37,7 @@ describe('authorization', function() {
}); });
}); });
it('should create and get a named map tile using a regular apikey token', function (done) { it.skip('should create and get a named map tile using a regular apikey token', function (done) {
const apikeyToken = 'regular1'; const apikeyToken = 'regular1';
const mapConfig = { const mapConfig = {
version: '1.7.0', version: '1.7.0',
@ -61,7 +61,7 @@ describe('authorization', function() {
testClient.drain(done); testClient.drain(done);
}); });
}); });
it('should fail getting a named map tile with default apikey token', function (done) { it('should fail getting a named map tile with default apikey token', function (done) {
const apikeyTokenCreate = 'regular1'; const apikeyTokenCreate = 'regular1';
@ -203,7 +203,7 @@ describe('authorization', function() {
assert.ok(layergroupResult.hasOwnProperty('errors')); assert.ok(layergroupResult.hasOwnProperty('errors'));
assert.equal(layergroupResult.errors.length, 1); assert.equal(layergroupResult.errors.length, 1);
assert.ok(layergroupResult.errors[0].match(/permission denied/), layergroupResult.errors[0]); assert.ok(layergroupResult.errors[0].match(/permission denied/), layergroupResult.errors[0]);
testClient.drain(done); testClient.drain(done);
}); });
}); });
@ -257,7 +257,7 @@ describe('authorization', function() {
type: 'source', type: 'source',
params: { params: {
query: 'select * from populated_places_simple_reduced' query: 'select * from populated_places_simple_reduced'
} }
} }
] ]
}; };
@ -354,7 +354,39 @@ describe('authorization', function() {
}); });
}); });
it('should create and get a named map tile using a regular apikey token', function (done) { it('should fail while listing named maps with a regular apikey token', function (done) {
const apikeyToken = 'regular1';
const testClient = new TestClient({}, apikeyToken);
testClient.getNamedMapList({ response: {status: 403 }}, function (err, res, body) {
assert.ifError(err);
assert.equal(res.statusCode, 403);
assert.equal(body.errors.length, 1);
assert.ok(body.errors[0].match(/Forbidden/), body.errors[0]);
testClient.drain(done);
});
});
it('should list named maps with master apikey token', function (done) {
const apikeyToken = 1234;
const testClient = new TestClient({}, apikeyToken);
testClient.getNamedMapList({}, function (err, res, body) {
assert.ifError(err);
assert.equal(res.statusCode, 200);
assert.ok(Array.isArray(body.template_ids));
testClient.drain(done);
});
});
it.skip('should create and get a named map tile using a regular apikey token', function (done) {
const apikeyToken = 'regular1'; const apikeyToken = 'regular1';
const template = { const template = {
@ -391,7 +423,7 @@ describe('authorization', function() {
}); });
}); });
it('should fail creating a named map using a regular apikey token and a private table', function (done) { it.skip('should fail creating a named map using a regular apikey token and a private table', function (done) {
const apikeyToken = 'regular1'; const apikeyToken = 'regular1';
const template = { const template = {

30
test/support/test-client.js
View File

@ -1254,6 +1254,36 @@ TestClient.prototype.getAnalysesCatalog = function (params, callback) {
); );
}; };
TestClient.prototype.getNamedMapList = function(params, callback) {
const request = {
url: `/api/v1/map/named?${qs.stringify({ api_key: this.apiKey })}`,
method: 'GET',
headers: {
host: 'localhost',
'Content-Type': 'application/json'
}
};
let expectedResponse = {
status: 200,
headers: {
'Content-Type': 'application/json; charset=utf-8'
}
};
if (params.response) {
expectedResponse = Object.assign(expectedResponse, params.response);
}
assert.response(this.server, request, expectedResponse, (res, err) => {
if (err) {
return callback(err);
}
const body = JSON.parse(res.body);
return callback(null, res, body);
});
};
TestClient.prototype.getNamedTile = function (name, z, x, y, format, options, callback) { TestClient.prototype.getNamedTile = function (name, z, x, y, format, options, callback) {
const { params } = options; const { params } = options;