cartodb4/Windshaft-cartodb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:CartoDB/Windshaft-cartodb into developer-center

Browse Source
This commit is contained in:
csubira 2019-03-01 15:06:41 +01:00
commit b7cf5ca174
304 changed files with 20401 additions and 8312 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
.travis.yml
View File

@ -1,14 +1,8 @@
sudo: required
dist: trusty
services:
- docker
before_install:
- docker pull cartoimages/windshaft-testing
script:
- docker run -e POSTGIS_VERSION=2.4 -v `pwd`:/srv cartoimages/windshaft-testing bash docker-test.sh
language: generic
jobs:
include:
- sudo: required
services:
- docker
language: generic
before_install: docker pull carto/nodejs-xenial-pg101:latest
script: npm run docker-test -- 10.15.1 # Node.js version

6
HOWTO_RELEASE
View File

@ -1,8 +1,8 @@
1. Test (make clean all check), fix if broken before proceeding
2. Ensure proper version in package.json
2. Ensure proper version in package.json and package-lock.json
3. Ensure NEWS section exists for the new version, review it, add release date
4. If there are modified dependencies in package.json, update them with `yarn upgrade {{package_name}}@{{version}}`
5. Commit package.json, yarn.lock, NEWS
4. If there are modified dependencies in package.json, update them with `npm upgrade {{package_name}}@{{version}}`
5. Commit package.json, package-lock.json, NEWS
6. git tag -a Major.Minor.Patch # use NEWS section as content
7. Stub NEWS/package for next version

50
INSTALL.md
View File

@ -1,35 +1,23 @@
# Installing Windshaft-CartoDB #
# Installing Windshaft-CartoDB
## Requirements ##
Make sure that you have the requirements needed. These are
## Requirements
- Core
- Node.js >=6.9.x
- yarn >=0.27.5 <1.0.0
- PostgreSQL >8.3.x, PostGIS >1.5.x
- Redis >2.4.0 (http://www.redis.io)
- Mapnik >3.x. See [Installing Mapnik](https://github.com/CartoDB/Windshaft#installing-mapnik).
- Windshaft: check [Windshaft dependencies and installation notes](https://github.com/CartoDB/Windshaft#dependencies)
- libcairo2-dev, libpango1.0-dev, libjpeg8-dev and libgif-dev for server side canvas support
Make sure that you have the requirements needed. These are:
- For cache control (optional)
- CartoDB 0.9.5+ (for `CDB_QueryTables`)
- Varnish (http://www.varnish-cache.org)
- Node 10.x
- npm 6.x
- PostgreSQL >= 10.0
- PostGIS >= 2.4
- CARTO Postgres Extension >= 0.24.1
- Redis >= 4
- libcairo2-dev, libpango1.0-dev, libjpeg8-dev and libgif-dev for server side canvas support
- C++11 (to build internal dependencies if needed)
On Ubuntu 14.04 the dependencies can be installed with
### Optional
```shell
sudo apt-get update
sudo apt-get install -y make g++ pkg-config git-core \
libgif-dev libjpeg-dev libcairo2-dev \
libhiredis-dev redis-server \
nodejs nodejs-legacy npm \
postgresql-9.3-postgis-2.1 postgresql-plpython-9.3 postgresql-server-dev-9.3
```
- Varnish (http://www.varnish-cache.org)
On Ubuntu 12.04 the [cartodb/cairo PPA](https://launchpad.net/~cartodb/+archive/ubuntu/cairo) may be useful.
## PostGIS setup ##
## PostGIS setup
A `template_postgis` database is expected. One can be set up with
@ -38,16 +26,16 @@ createdb --owner postgres --template template0 template_postgis
psql -d template_postgis -c 'CREATE EXTENSION postgis;'
```
## Build/install ##
## Build/install
To fetch and build all node-based dependencies, run:
```
yarn
```shell
npm install
```
Note that the ```yarn``` step will populate the node_modules/
Note that the ```npm``` step will populate the node_modules/
directory with modules, some of which being compiled on demand. If you
happen to have startup errors you may need to force rebuilding those
modules. At any time just wipe out the node_modules/ directory and run
```yarn``` again.
```npm``` again.

189
NEWS.md
View File

@ -1,8 +1,191 @@
# Changelog
## 5.3.2
Released yyyy-mm-dd
- Upgrades Windshaft to 4.5.3
## 7.0.1
Released 2019-mm-dd
## 7.0.0
Released 2019-02-22
Breaking changes:
- Drop support for Node.js 6
- Drop support for npm 3
- Stop supporting `yarn.lock`
- Drop support for Postgres 9.5
- Drop support for PosGIS 2.2
- Drop support for Redis 3
Announcements:
- In configuration, set `clipByBox2d` to true by default
- Update docs: compatible Node.js and npm versions
- Report fine-grained Garbage Collector stats
- Adding Authorization to Access-Control-Allow-Headers (https://github.com/CartoDB/CartoDB-SQL-API/issues/534)
- Update deps:
- windshaft@4.13.1: Upgrade tilelive-mapnik to version 0.6.18-cdb18
- camshaft@0.63.4: Improve error message for exceeded batch SQL API payload size: add suggestions about what the user can do about it.
- Update dev deps:
- jshint@2.9.7
- mocha@5.2.0
- Be able to customize max waiting workers parameter
- Handle 'max waitingClients count exceeded' error as "429, You are over platform's limits"
## 6.5.1
Released 2018-12-26
Bug Fixes:
- Update carto-package.json
## 6.5.0
Released 2018-12-26
New features
- Suport Node.js 10
- Configure travis to run docker tests against Node.js 6 & 10 versions
- Aggregation time dimensions
- Update sample configurations to use PostGIS to generate MVT's by default (as in production)
- Upgrades Windshaft to [4.12.1](https://github.com/CartoDB/Windshaft/blob/4.12.1/NEWS.md#version-4121)
- `pg-mvt`: Use `query-rewriter` to compose the query to render a MVT tile. If not defined, it will use a Default Query Rewriter.
- `pg-mvt`: Fix bug while building query and there is no columns defined for the layer.
- `pg-mvt`: Accept trailing semicolon in input queries.
- `Renderer Cache Entry`: Do not throw errors for integrity checks.
- Fix bug when releasing the renderer cache entry in some scenarios.
- Upgrade grainstore to [1.10.0](https://github.com/CartoDB/grainstore/releases/tag/1.10.0)
- Upgrade cartodb-redis to [2.1.0](https://github.com/CartoDB/node-cartodb-redis/releases/tag/2.1.0)
- Upgrade cartodb-query-tables to [0.4.0](https://github.com/CartoDB/node-cartodb-query-tables/releases/tag/0.4.0)
- Upgrade cartodb-psql to [0.13.1](https://github.com/CartoDB/node-cartodb-psql/releases/tag/0.13.1)
- Upgrade turbo-carto to [0.21.0](https://github.com/CartoDB/turbo-carto/releases/tag/0.21.0)
- Upgrade camshaft to [0.63.1](https://github.com/CartoDB/camshaft/releases/tag/0.63.1)
- Upgrade redis-mpool to [0.7.0](https://github.com/CartoDB/node-redis-mpool/releases/tag/0.7.0)
Bug Fixes:
- Prevent from uncaught exception: Range filter Error from camshaft when getting analysis query.
- Make all modules to use strict mode semantics.
## 6.4.0
Released 2018-09-24
- Upgrades Camshaft to [0.62.3](https://github.com/CartoDB/camshaft/releases/tag/0.61.11):
- Build query from node's cache to compute output columns when building analysis
- Adds metadata columns for street level geocoding
- Remove use of `step` module to handle asynchronous code, now it's defined as development dependency.
- Bug Fixes: (#1020)
- Fix bug in date-wrapper regarding columns with spaces
- Fix bug in aggregation-query regarding columns with spaces
- Upgrades Windshaft to [4.10.0](https://github.com/CartoDB/Windshaft/blob/4.10.0/NEWS.md#version-4100)
- `pg-mvt`:
- Now matches the behaviour of the `mapnik` renderer for MVTs.
- Removed undocummented filtering by `layer.options.columns`.
- Implement timeout in getTile.
- Several bugfixes.
- Dependency updates: Fixed a bug in Mapnik MVT renderer and cleanup in `tilelive-mapnik`.
- [MapConfig 1.8.0 released](https://github.com/CartoDB/Windshaft/blob/master/doc/MapConfig-1.8.0.md) with new options for MVTs:
- Add **`vector_extent`** option in MapConfig to setup the layer extent.
- Add **`vector_simplify_extent`** option in MapConfig to configure the simplification process.
- Remove use of `step` module to handle asynchronous code, now it's defined as development dependency.
## 6.3.0
Released 2018-07-26
- Upgrades Camshaft to [0.62.1](https://github.com/CartoDB/camshaft/releases/tag/0.62.1):
- Support for batch street-level geocoding. [0.62.1](https://github.com/CartoDB/camshaft/releases/tag/0.62.1)
## 6.2.0
Released 2018-07-20
Notice:
- This release changes the way that authentication works internally. You'll need to run `bundle exec rake carto:api_key:create_default` in your development environment to keep working.
New features:
- CI tests with Ubuntu Xenial + PostgreSQL 10.1 and Ubuntu Precise + PostgreSQL 9.5
- Upgrades Windshaft to [4.8.3](https://github.com/CartoDB/Windshaft/blob/4.8.3/NEWS.md#version-483) which includes:
- Update internal deps.
- A fix in mapnik-vector-tile to avoid grouping together properties with the same value but a different type.
- Performance improvements in the marker symbolizer (local cache, avoid building the collision matrix when possible).
- MVT: Disable simplify_distance to avoid multiple simplifications.
- Fix a bug with zero length lines not being rendered when using the marker symbolizer.
- Reduce size of npm package
- Omit attributes validation in layers with aggregation to avoid potentially long instantiation times
- Upgrades Camshaft to [0.61.11](https://github.com/CartoDB/camshaft/releases/tag/0.61.11):
- Use Dollar-Quoted String Constants to avoid Syntax Error while running moran analyses. [0.61.10](https://github.com/CartoDB/camshaft/releases/tag/0.61.10)
- Quote name columns when performing trade area analysis to avoid Syntax Errors. [0.61.11](https://github.com/CartoDB/camshaft/releases/tag/0.61.11)
- Update other deps:
- body-parser: 1.18.3
- cartodb-psql: 0.11.0
- cartodb-redis: 2.0.1
- dot: 1.1.2
- express: 4.16.3
- lru-cache: 4.1.3
- node-statsd: 0.1.1,
- queue-async: 1.1.0
- request: 2.87.0
- semver: 5.5.0
- step: 1.0.0
- turbo-carto: 0.20.4
- yargs: 11.1.0
- Update devel deps:
- istanbul: 0.4.5
- jshint: 2.9.5
- mocha: 3.5.3
- moment: 2.22.1
- nock: 9.2.6
- strftime: 0.10.0
- Optional instantiation metadata stats (https://github.com/CartoDB/Windshaft-cartodb/pull/952)
- Experimental dates_as_numbers support
- Tiles base urls with api key
Bug Fixes:
- Validates tile coordinates (z/x/y) from request params to be a valid integer value.
- Static maps fails for unsupported formats
- Handling errors extracting the column type on dataviews
- Fix `meta.stats.estimatedFeatureCount` for aggregations and queries with tokens
- Fix numeric histogram bounds when `start` and `end` are specified (#991)
- Static maps filters correctly if `layer` option is passed in the url.
- Aggregation doesn't return out-of-tile, partially aggregated clusters
- Aggregation was not accurate for high zoom, far away from the origin tiles
Announcements:
* Improve error message when the DB query is over the user's limits
## 6.1.0
Released 2018-04-16
New features:
- Aggreation filters
- Upgrades Windshaft to 4.7.0, which includes @carto/mapnik v3.6.2-carto.7 with improvements to metrics and markers caching. It also adds an option to disable the markers symbolizer caches in mapnik.
Bug Fixes:
- Non-default aggregation selected the wrong columns (e.g. for vector tiles)
- Aggregation dimensions with alias where broken
- cartodb_id was not unique accross aggregated vector tiles
## 6.0.0
Released 2018-03-19
Backward incompatible changes:
- Needs Redis v4
New features:
- Upgrades camshaft to 0.61.8
- Upgrades cartodb-redis to 1.0.0
- Rate limit feature (disabled by default)
- Fixes for tests with PG11
## 5.4.0
Released 2018-03-15
- Upgrades Windshaft to 4.5.7 ([Mapnik top metrics](https://github.com/CartoDB/Windshaft/pull/597), [AttributesBackend allows multiple features if all the attributes are the same](https://github.com/CartoDB/Windshaft/pull/602))
- Implemented middleware to authorize users via new Api Key system
- Keep the old authorization system as fallback
- Aggregation widget: Remove NULL categories in 'count' aggregations too
- Update request to 2.85.0
- Update camshaft to 0.61.4 (Fixes for AOI and Merge analyses)
- Update windshaft to 4.6.0, which in turn updates @carto/mapnik to 3.6.2-carto.4 and related dependencies. It brings in a cache for rasterized symbols. See https://github.com/CartoDB/node-mapnik/blob/v3.6.2-carto/CHANGELOG.carto.md#362-carto4
- PostGIS: Variables in postgis SQL queries must now additionally be wrapped in `!` (refs [#29](https://github.com/CartoDB/mapnik/issues/29), [mapnik/#3618](https://github.com/mapnik/mapnik/pull/3618)):
```sql
-- Before
SELECT ... WHERE trait = @variable
-- Now
SELECT ... WHERE trait = !@variable!
```
## 5.3.1
Released 2018-02-13

16
README.md
View File

@ -31,12 +31,10 @@ Upgrading
Checkout your commit/branch. If you need to reinstall dependencies (you can check [NEWS](NEWS.md)) do the following:
```sh
$ rm -rf node_modules
$ npm install
```
rm -rf node_modules; yarn
```
Run
---
```
node app.js <env>
@ -71,12 +69,12 @@ See [CONTRIBUTING.md](CONTRIBUTING.md).
### Developing with a custom windshaft version
If you plan or want to use a custom / not released yet version of windshaft (or any other dependency) the best option is
to use `yarn link`. You can read more about it at [yarn-link: Symlink a package folder](https://yarnpkg.com/en/docs/cli/link).
to use `npm link`. You can read more about it at [npm-link: Symlink a package folder](https://docs.npmjs.com/cli/link.html).
**Quick start**:
```shell
~/windshaft-directory $ yarn
~/windshaft-directory $ yarn link
~/windshaft-cartodb-directory $ yarn link windshaft
~/windshaft-directory $ npm install
~/windshaft-directory $ npm link
~/windshaft-cartodb-directory $ npm link windshaft
```

79
app.js
View File

@ -1,3 +1,5 @@
'use strict';
var http = require('http');
var https = require('https');
var path = require('path');
@ -100,8 +102,6 @@ if ( global.environment.log_filename ) {
global.log4js.configure(log4jsConfig);
global.logger = global.log4js.getLogger();
global.environment.api_hostname = require('os').hostname().split('.')[0];
// Include cartodb_windshaft only _after_ the "global" variable is set
// See https://github.com/Vizzuality/Windshaft-cartodb/issues/28
var cartodbWindshaft = require('./lib/cartodb/server');
@ -128,6 +128,40 @@ listener.on('listening', function() {
);
});
function getCPUUsage (oldUsage) {
let usage;
if (oldUsage && oldUsage._start) {
usage = Object.assign({}, process.cpuUsage(oldUsage._start.cpuUsage));
usage.time = Date.now() - oldUsage._start.time;
} else {
usage = Object.assign({}, process.cpuUsage());
usage.time = process.uptime() * 1000; // s to ms
}
usage.percent = (usage.system + usage.user) / (usage.time * 10);
Object.defineProperty(usage, '_start', {
value: {
cpuUsage: process.cpuUsage(),
time: Date.now()
}
});
return usage;
}
let previousCPUUsage = getCPUUsage();
setInterval(function cpuUsageMetrics () {
const CPUUsage = getCPUUsage(previousCPUUsage);
Object.keys(CPUUsage).forEach(property => {
global.statsClient.gauge(`windshaft.cpu.${property}`, CPUUsage[property]);
});
previousCPUUsage = CPUUsage;
}, 5000);
setInterval(function() {
var memoryUsage = process.memoryUsage();
Object.keys(memoryUsage).forEach(function(k) {
@ -154,9 +188,46 @@ if (global.gc) {
if (gcInterval > 0) {
setInterval(function gcForcedCycle() {
var start = Date.now();
global.gc();
global.statsClient.timing('windshaft.gc', Date.now() - start);
}, gcInterval);
}
}
const gcStats = require('gc-stats')();
gcStats.on('stats', function ({ pauseMS, gctype }) {
global.statsClient.timing('windshaft.gc', pauseMS);
global.statsClient.timing(`windshaft.gctype.${getGCTypeValue(gctype)}`, pauseMS);
});
function getGCTypeValue (type) {
// 1: Scavenge (minor GC)
// 2: Mark/Sweep/Compact (major GC)
// 4: Incremental marking
// 8: Weak/Phantom callback processing
// 15: All
let value;
switch (type) {
case 1:
value = 'Scavenge';
break;
case 2:
value = 'MarkSweepCompact';
break;
case 4:
value = 'IncrementalMarking';
break;
case 8:
value = 'ProcessWeakCallbacks';
break;
case 15:
value = 'All';
break;
default:
value = 'Unkown';
break;
}
return value;
}

17
carto-package.json Normal file
View File

@ -0,0 +1,17 @@
{
"name": "carto_windshaft",
"current_version": {
"requires": {
"node": "^10.15.1",
"npm": "^6.4.1",
"mapnik": "==3.0.15.9",
"crankshaft": "~0.8.1"
},
"works_with": {
"redis": ">=4.0.0",
"postgresql": ">=10.0.0",
"postgis": ">=2.4.4.5",
"carto_postgresql_ext": ">=0.24.1"
}
}
}

206
config/environments/development.js.example
View File

@ -15,26 +15,59 @@ var config = {
// Base URLs for the APIs
//
// See http://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/map/named|/user/:user/api/v1/map/named|/tiles/template)'
// Base url for the Detached Maps API
// "maps" is the the new API,
// "tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/map|/user/:user/api/v1/map|/tiles/layergroup)'
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
,routes: {
v1: {
paths: [
'/api/v1',
'/user/:user/api/v1',
],
// Base url for the Detached Maps API
// "/api/v1/map" is the new API,
map: {
paths: [
'/map',
]
},
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
template: {
paths: [
'/map/named'
]
}
},
// For compatibility with versions up to 1.6.x
v0: {
paths: [
'/tiles'
],
// Base url for the Detached Maps API
// "/tiles/layergroup" is for compatibility with versions up to 1.6.x
map: {
paths: [
'/layergroup'
]
},
// Base url for the Templated Maps API
// "/tiles/template" is for compatibility with versions up to 1.6.x
template: {
paths: [
'/template'
]
}
}
}
// Resource URLs expose endpoints to request/retrieve metadata associated to Maps: dataviews, analysis node status.
//
// This URLs depend on how `base_url_detached` and `user_from_host` are configured: the application can be
// This URLs depend on how `routes` and `user_from_host` are configured: the application can be
// configured to accept request with the {user} in the header host or in the request path.
// It also might depend on the configured cdn_url via `serverMetadata.cdn_url`.
//
// This template allows to make the endpoints generation more flexible, the template exposes the following params:
// 1. {{=it.cdn_url}}: will be used when `serverMetadata.cdn_url` exists.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `base_url_detached`.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `routes`.
// 3. {{=it.port}}: will use the `port` from this very same configuration file.
,resources_url_templates: {
http: 'http://{{=it.user}}.localhost.lan:{{=it.port}}/api/v1/map',
@ -54,12 +87,12 @@ var config = {
// idle socket timeout, in milliseconds
,socket_timeout: 600000
,enable_cors: true
,cache_enabled: false
,cache_enabled: true
,log_format: ':req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type] (:res[X-Tiler-Profiler]) (:res[X-Tiler-Errors])'
// If log_filename is given logs will be written
// there, in append mode. Otherwise stdout is used (default).
// Log file will be re-opened on receiving the HUP signal
,log_filename: undefined
,log_filename: 'logs/node-windshaft.log'
// Templated database username for authorized user
// Supported labels: 'user_id' (read from redis)
,postgres_auth_user: 'development_cartodb_user_<%= user_id %>'
@ -67,39 +100,25 @@ var config = {
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: '<%= user_password %>'
,postgres: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
type: "postgis",
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
/* experimental
geometry_field: "the_geom",
extent: "-180,-90,180,90",
srid: 4326,
*/
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
max_size: 500
pool: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
,mapnik_version: undefined
,mapnik_tile_format: 'png8:m=h'
,statsd: {
host: 'localhost',
port: 8125,
prefix: 'dev.',
prefix: 'dev.', // could be hostname, better not containing dots
cacheDns: true
// support all allowed node-statsd options
}
@ -108,18 +127,9 @@ var config = {
cache_ttl: 60000,
statsInterval: 5000, // milliseconds between each report to statsd about number of renderers and mapnik pool status
mvt: {
//If enabled, MVTs will be generated with PostGIS directly, instead of using Mapnik,
//PostGIS 2.4 is required for this to work
//If disabled it will use Mapnik MVT generation
usePostGIS: false,
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
//If enabled, MVTs will be generated with PostGIS directly
//If disabled, MVTs will be generated with Mapnik MVT
usePostGIS: true
},
mapnik: {
// The size of the pool of internal mapnik backend
@ -128,6 +138,10 @@ var config = {
// Important: check the configuration of uv_threadpool_size to use suitable value
poolSize: 8,
// The maximum number of waiting clients of the pool of internal mapnik backend
// This maximum number is per mapnik renderer created in Windshaft's RendererFactory
poolMaxWaitingClients: 64,
// Whether grainstore will use a child process or not to transform CartoCSS into Mapnik XML.
// This will prevent blocking the main thread.
useCartocssWorkers: false,
@ -171,7 +185,31 @@ var config = {
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
clipByBox2d: true,
postgis: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
max_size: 500,
twkb_encoding: true
},
limits: {
// Time in milliseconds a render request can take before it fails, some notes:
@ -186,31 +224,17 @@ var config = {
cacheOnTimeout: true
},
geojson: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
},
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
// geometries will be simplified using ST_RemoveRepeatedPoints
// which cost is no more expensive than snapping and results are
// much closer to the original geometry
removeRepeatedPoints: false // this requires postgis >=2.2
},
// If enabled Mapnik will reuse the features retrieved from the database
// instead of requesting them once per style inside a layer
'cache-features': true,
// Require metrics to the renderer
metrics: false
metrics: false,
// Options for markers attributes, ellipses and images caches
markers_symbolizer_caches: {
disabled: false
}
},
http: {
timeout: 2000, // the timeout in ms for a http tile request
@ -226,16 +250,7 @@ var config = {
src: __dirname + '/../../assets/default-placeholder.png'
}
},
torque: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
torque: {}
}
// anything analyses related
,analysis: {
@ -256,7 +271,7 @@ var config = {
// If filename is given logs comming from analysis client will be written
// there, in append mode. Otherwise 'log_filename' is used. Otherwise stdout is used (default).
// Log file will be re-opened on receiving the HUP signal
filename: '/tmp/analysis.log'
filename: 'logs/node-windshaft-analysis.log'
},
// Define max execution time in ms for analyses or tags
// If analysis or tag are not found in redis this values will be used as default.
@ -326,6 +341,12 @@ var config = {
// X-Tiler-Profile header containing elapsed timing for various
// steps taken for producing the response.
,useProfiler:true
,serverMetadata: {
cdn_url: {
http: undefined,
https: undefined
}
}
// Settings for the health check available at /health
,health: {
enabled: false,
@ -343,7 +364,28 @@ var config = {
// whether the affected tables for a given SQL must query directly postgresql or use the SQL API
cdbQueryTablesFromPostgres: true,
// whether in mapconfig is available stats & metadata for each layer
layerStats: true
layerStats: true,
// whether it should rate limit endpoints (global configuration)
rateLimitsEnabled: false,
// whether it should rate limit one or more endpoints (only if rateLimitsEnabled = true)
rateLimitsByEndpoint: {
anonymous: false,
static: false,
static_named: false,
dataview: false,
dataview_search: false,
analysis: false,
analysis_catalog: false,
tile: false,
attributes: false,
named_list: false,
named_create: false,
named_get: false,
named: false,
named_update: false,
named_delete: false,
named_tiles: false
}
}
};

190
config/environments/production.js.example
View File

@ -15,26 +15,59 @@ var config = {
// Base URLs for the APIs
//
// See http://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/map/named|/user/:user/api/v1/map/named|/tiles/template)'
// Base url for the Detached Maps API
// "maps" is the the new API,
// "tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/map|/user/:user/api/v1/map|/tiles/layergroup)'
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
,routes: {
v1: {
paths: [
'/api/v1',
'/user/:user/api/v1',
],
// Base url for the Detached Maps API
// "/api/v1/map" is the new API,
map: {
paths: [
'/map',
]
},
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
template: {
paths: [
'/map/named'
]
}
},
// For compatibility with versions up to 1.6.x
v0: {
paths: [
'/tiles'
],
// Base url for the Detached Maps API
// "/tiles/layergroup" is for compatibility with versions up to 1.6.x
map: {
paths: [
'/layergroup'
]
},
// Base url for the Templated Maps API
// "/tiles/template" is for compatibility with versions up to 1.6.x
template: {
paths: [
'/template'
]
}
}
}
// Resource URLs expose endpoints to request/retrieve metadata associated to Maps: dataviews, analysis node status.
//
// This URLs depend on how `base_url_detached` and `user_from_host` are configured: the application can be
// This URLs depend on how `routes` and `user_from_host` are configured: the application can be
// configured to accept request with the {user} in the header host or in the request path.
// It also might depend on the configured cdn_url via `serverMetadata.cdn_url`.
//
// This template allows to make the endpoints generation more flexible, the template exposes the following params:
// 1. {{=it.cdn_url}}: will be used when `serverMetadata.cdn_url` exists.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `base_url_detached`.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `routes`.
// 3. {{=it.port}}: will use the `port` from this very same configuration file.
,resources_url_templates: {
http: 'http://{{=it.cdn_url}}/{{=it.user}}/api/v1/map',
@ -67,26 +100,18 @@ var config = {
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: '<%= user_password %>'
,postgres: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 6432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
max_size: 500
port: 5432,
pool: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
,mapnik_version: undefined
,mapnik_tile_format: 'png8:m=h'
@ -102,18 +127,9 @@ var config = {
cache_ttl: 60000,
statsInterval: 5000, // milliseconds between each report to statsd about number of renderers and mapnik pool status
mvt: {
//If enabled, MVTs will be generated with PostGIS directly, instead of using Mapnik,
//PostGIS 2.4 is required for this to work
//If disabled it will use Mapnik MVT generation
usePostGIS: false,
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
//If enabled, MVTs will be generated with PostGIS directly
//If disabled, MVTs will be generated with Mapnik MVT
usePostGIS: true
},
mapnik: {
// The size of the pool of internal mapnik backend
@ -122,6 +138,10 @@ var config = {
// Important: check the configuration of uv_threadpool_size to use suitable value
poolSize: 8,
// The maximum number of waiting clients of the pool of internal mapnik backend
// This maximum number is per mapnik renderer created in Windshaft's RendererFactory
poolMaxWaitingClients: 64,
// Whether grainstore will use a child process or not to transform CartoCSS into Mapnik XML.
// This will prevent blocking the main thread.
useCartocssWorkers: false,
@ -165,7 +185,31 @@ var config = {
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
clipByBox2d: true,
postgis: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
max_size: 500,
twkb_encoding: true
},
limits: {
// Time in milliseconds a render request can take before it fails, some notes:
@ -180,33 +224,17 @@ var config = {
cacheOnTimeout: true
},
geojson: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
},
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
// geometries will be simplified using ST_RemoveRepeatedPoints
// which cost is no more expensive than snapping and results are
// much closer to the original geometry
removeRepeatedPoints: false // this requires postgis >=2.2
},
// If enabled Mapnik will reuse the features retrieved from the database
// instead of requesting them once per style inside a layer
'cache-features': true,
// Require metrics to the renderer
metrics: false
metrics: false,
// Options for markers attributes, ellipses and images caches
markers_symbolizer_caches: {
disabled: false
}
},
http: {
timeout: 2000, // the timeout in ms for a http tile request
@ -222,16 +250,7 @@ var config = {
src: __dirname + '/../../assets/default-placeholder.png'
}
},
torque: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
torque: {}
}
// anything analyses related
,analysis: {
@ -252,7 +271,7 @@ var config = {
// If filename is given logs comming from analysis client will be written
// there, in append mode. Otherwise 'log_filename' is used. Otherwise stdout is used (default).
// Log file will be re-opened on receiving the HUP signal
filename: 'logs/analysis.log'
filename: 'logs/node-windshaft-analysis.log'
},
// Define max execution time in ms for analyses or tags
// If analysis or tag are not found in redis this values will be used as default.
@ -345,7 +364,28 @@ var config = {
// whether the affected tables for a given SQL must query directly postgresql or use the SQL API
cdbQueryTablesFromPostgres: true,
// whether in mapconfig is available stats & metadata for each layer
layerStats: false
layerStats: false,
// whether it should rate limit endpoints (global configuration)
rateLimitsEnabled: false,
// whether it should rate limit one or more endpoints (only if rateLimitsEnabled = true)
rateLimitsByEndpoint: {
anonymous: false,
static: false,
static_named: false,
dataview: false,
dataview_search: false,
analysis: false,
analysis_catalog: false,
tile: false,
attributes: false,
named_list: false,
named_create: false,
named_get: false,
named: false,
named_update: false,
named_delete: false,
named_tiles: false
}
}
};

198
config/environments/staging.js.example
View File

@ -15,29 +15,62 @@ var config = {
// Base URLs for the APIs
//
// See http://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/maps/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/map/named|/user/:user/api/v1/map/named|/tiles/template)'
// Base url for the Detached Maps API
// "/api/v1/maps" is the the new API,
// "/tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/map|/user/:user/api/v1/map|/tiles/layergroup)'
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
,routes: {
v1: {
paths: [
'/api/v1',
'/user/:user/api/v1',
],
// Base url for the Detached Maps API
// "/api/v1/map" is the new API,
map: {
paths: [
'/map',
]
},
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
template: {
paths: [
'/map/named'
]
}
},
// For compatibility with versions up to 1.6.x
v0: {
paths: [
'/tiles'
],
// Base url for the Detached Maps API
// "/tiles/layergroup" is for compatibility with versions up to 1.6.x
map: {
paths: [
'/layergroup'
]
},
// Base url for the Templated Maps API
// "/tiles/template" is for compatibility with versions up to 1.6.x
template: {
paths: [
'/template'
]
}
}
}
// Resource URLs expose endpoints to request/retrieve metadata associated to Maps: dataviews, analysis node status.
//
// This URLs depend on how `base_url_detached` and `user_from_host` are configured: the application can be
// This URLs depend on how `routes` and `user_from_host` are configured: the application can be
// configured to accept request with the {user} in the header host or in the request path.
// It also might depend on the configured cdn_url via `serverMetadata.cdn_url`.
//
// This template allows to make the endpoints generation more flexible, the template exposes the following params:
// 1. {{=it.cdn_url}}: will be used when `serverMetadata.cdn_url` exists.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `base_url_detached`.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `routes`.
// 3. {{=it.port}}: will use the `port` from this very same configuration file.
,resources_url_templates: {
http: 'http://{{=it.user}}.localhost.lan:{{=it.port}}/api/v1/map',
http: 'http://{{=it.cdn_url}}/{{=it.user}}/api/v1/map',
https: 'https://{{=it.cdn_url}}/{{=it.user}}/api/v1/map'
}
@ -55,7 +88,7 @@ var config = {
,socket_timeout: 600000
,enable_cors: true
,cache_enabled: true
,log_format: ':req[X-Real-IP] :method :req[Host]:url :status :response-time ms (:res[X-Tiler-Profiler]) -> :res[Content-Type] (:res[X-Tiler-Errors])'
,log_format: ':req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type] (:res[X-Tiler-Profiler]) (:res[X-Tiler-Errors])'
// If log_filename is given logs will be written
// there, in append mode. Otherwise stdout is used (default).
// Log file will be re-opened on receiving the HUP signal
@ -67,33 +100,25 @@ var config = {
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: '<%= user_password %>'
,postgres: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 6432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
max_size: 500
port: 5432,
pool: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
,mapnik_version: undefined
,mapnik_tile_format: 'png8:m=h'
,statsd: {
host: 'localhost',
port: 8125,
prefix: 'stage.:host.',
prefix: 'stage.:host.', // could be hostname, better not containing dots
cacheDns: true
// support all allowed node-statsd options
}
@ -102,18 +127,9 @@ var config = {
cache_ttl: 60000,
statsInterval: 5000, // milliseconds between each report to statsd about number of renderers and mapnik pool status
mvt: {
//If enabled, MVTs will be generated with PostGIS directly, instead of using Mapnik,
//PostGIS 2.4 is required for this to work
//If disabled it will use Mapnik MVT generation
usePostGIS: false,
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
//If enabled, MVTs will be generated with PostGIS directly
//If disabled, MVTs will be generated with Mapnik MVT
usePostGIS: true
},
mapnik: {
// The size of the pool of internal mapnik backend
@ -122,6 +138,10 @@ var config = {
// Important: check the configuration of uv_threadpool_size to use suitable value
poolSize: 8,
// The maximum number of waiting clients of the pool of internal mapnik backend
// This maximum number is per mapnik renderer created in Windshaft's RendererFactory
poolMaxWaitingClients: 64,
// Whether grainstore will use a child process or not to transform CartoCSS into Mapnik XML.
// This will prevent blocking the main thread.
useCartocssWorkers: false,
@ -165,7 +185,31 @@ var config = {
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
clipByBox2d: true,
postgis: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
max_size: 500,
twkb_encoding: true
},
limits: {
// Time in milliseconds a render request can take before it fails, some notes:
@ -180,33 +224,17 @@ var config = {
cacheOnTimeout: true
},
geojson: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
},
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
// geometries will be simplified using ST_RemoveRepeatedPoints
// which cost is no more expensive than snapping and results are
// much closer to the original geometry
removeRepeatedPoints: false // this requires postgis >=2.2
},
// If enabled Mapnik will reuse the features retrieved from the database
// instead of requesting them once per style inside a layer
'cache-features': true,
// Require metrics to the renderer
metrics: false
metrics: false,
// Options for markers attributes, ellipses and images caches
markers_symbolizer_caches: {
disabled: false
}
},
http: {
timeout: 2000, // the timeout in ms for a http tile request
@ -222,16 +250,7 @@ var config = {
src: __dirname + '/../../assets/default-placeholder.png'
}
},
torque: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
torque: {}
}
// anything analyses related
,analysis: {
@ -252,7 +271,7 @@ var config = {
// If filename is given logs comming from analysis client will be written
// there, in append mode. Otherwise 'log_filename' is used. Otherwise stdout is used (default).
// Log file will be re-opened on receiving the HUP signal
filename: 'logs/analysis.log'
filename: 'logs/node-windshaft-analysis.log'
},
// Define max execution time in ms for analyses or tags
// If analysis or tag are not found in redis this values will be used as default.
@ -330,7 +349,7 @@ var config = {
}
// Settings for the health check available at /health
,health: {
enabled: false,
enabled: true,
username: 'localhost',
z: 0,
x: 0,
@ -345,7 +364,28 @@ var config = {
// whether the affected tables for a given SQL must query directly postgresql or use the SQL API
cdbQueryTablesFromPostgres: true,
// whether in mapconfig is available stats & metadata for each layer
layerStats: true
layerStats: true,
// whether it should rate limit endpoints (global configuration)
rateLimitsEnabled: false,
// whether it should rate limit one or more endpoints (only if rateLimitsEnabled = true)
rateLimitsByEndpoint: {
anonymous: false,
static: false,
static_named: false,
dataview: false,
dataview_search: false,
analysis: false,
analysis_catalog: false,
tile: false,
attributes: false,
named_list: false,
named_create: false,
named_get: false,
named: false,
named_update: false,
named_delete: false,
named_tiles: false
}
}
};

206
config/environments/test.js.example
View File

@ -16,28 +16,62 @@ var config = {
// Base URLs for the APIs
//
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/map/named|/user/:user/api/v1/map/named|/tiles/template)'
// Base url for the Detached Maps API
// "maps" is the the new API,
// "tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/map|/user/:user/api/v1/map|/tiles/layergroup)'
,routes: {
v1: {
paths: [
'/api/v1',
'/user/:user/api/v1',
],
// Base url for the Detached Maps API
// "/api/v1/map" is the new API,
map: {
paths: [
'/map',
]
},
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
template: {
paths: [
'/map/named'
]
}
},
// For compatibility with versions up to 1.6.x
v0: {
paths: [
'/tiles'
],
// Base url for the Detached Maps API
// "/tiles/layergroup" is for compatibility with versions up to 1.6.x
map: {
paths: [
'/layergroup'
]
},
// Base url for the Templated Maps API
// "/tiles/template" is for compatibility with versions up to 1.6.x
template: {
paths: [
'/template'
]
}
}
}
// Resource URLs expose endpoints to request/retrieve metadata associated to Maps: dataviews, analysis node status.
//
// This URLs depend on how `base_url_detached` and `user_from_host` are configured: the application can be
// This URLs depend on how `routes` and `user_from_host` are configured: the application can be
// configured to accept request with the {user} in the header host or in the request path.
// It also might depend on the configured cdn_url via `serverMetadata.cdn_url`.
//
// This template allows to make the endpoints generation more flexible, the template exposes the following params:
// 1. {{=it.cdn_url}}: will be used when `serverMetadata.cdn_url` exists.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `base_url_detached`.
// 2. {{=it.user}}: will use the username as extraced from `user_from_host` or `routes`.
// 3. {{=it.port}}: will use the `port` from this very same configuration file.
,resources_url_templates: {
http: 'http://{{=it.user}}.localhost.lan:{{=it.port}}/api/v1/map'
http: 'http://{{=it.user}}.localhost.lan:{{=it.port}}/api/v1/map',
https: 'https://{{=it.user}}.localhost.lan:{{=it.port}}/api/v1/map'
}
// Maximum number of connections for one process
@ -54,11 +88,11 @@ var config = {
,socket_timeout: 600000
,enable_cors: true
,cache_enabled: false
,log_format: '[:date] :req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type] (:res[X-Tiler-Profiler]) (:res[X-Tiler-Errors])'
,log_format: ':req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type] (:res[X-Tiler-Profiler]) (:res[X-Tiler-Errors])'
// If log_filename is given logs will be written
// there, in append mode. Otherwise stdout is used (default).
// Log file will be re-opened on receiving the HUP signal
//,log_filename: 'logs/node-windshaft.log'
,log_filename: '/tmp/node-windshaft.log'
// Templated database username for authorized user
// Supported labels: 'user_id' (read from redis)
,postgres_auth_user: 'test_windshaft_cartodb_user_<%= user_id %>'
@ -66,33 +100,25 @@ var config = {
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: 'test_windshaft_cartodb_user_<%= user_id %>_pass'
,postgres: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "test_windshaft_publicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
max_size: 500
pool: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
,mapnik_version: ''
,mapnik_version: undefined
,mapnik_tile_format: 'png8:m=h'
,statsd: {
host: 'localhost',
port: 8125,
prefix: 'test.:host.',
prefix: 'test.:host.', // could be hostname, better not containing dots
cacheDns: true
// support all allowed node-statsd options
}
@ -101,18 +127,9 @@ var config = {
cache_ttl: 60000,
statsInterval: 5000, // milliseconds between each report to statsd about number of renderers and mapnik pool status
mvt: {
//If enabled, MVTs will be generated with PostGIS directly, instead of using Mapnik,
//PostGIS 2.4 is required for this to work
//If disabled it will use Mapnik MVT generation
usePostGIS: false,
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
//If enabled, MVTs will be generated with PostGIS directly
//If disabled, MVTs will be generated with Mapnik MVT
usePostGIS: true
},
mapnik: {
// The size of the pool of internal mapnik backend
@ -121,6 +138,10 @@ var config = {
// Important: check the configuration of uv_threadpool_size to use suitable value
poolSize: 8,
// The maximum number of waiting clients of the pool of internal mapnik backend
// This maximum number is per mapnik renderer created in Windshaft's RendererFactory
poolMaxWaitingClients: 64,
// Whether grainstore will use a child process or not to transform CartoCSS into Mapnik XML.
// This will prevent blocking the main thread.
useCartocssWorkers: false,
@ -164,7 +185,31 @@ var config = {
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
clipByBox2d: true,
postgis: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
// max number of rows to return when querying data, 0 means no limit
row_limit: 65535,
/*
* Set persist_connection to false if you want
* database connections to be closed on renderer
* expiration (1 minute after last use).
* Setting to true (the default) would never
* close any connection for the server's lifetime
*/
persist_connection: false,
simplify_geometries: true,
use_overviews: true, // use overviews to retrieve raster
max_size: 500,
twkb_encoding: false
},
limits: {
// Time in milliseconds a render request can take before it fails, some notes:
@ -179,32 +224,17 @@ var config = {
cacheOnTimeout: true
},
geojson: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
},
// SQL queries will be wrapped with ST_ClipByBox2D
// Returning the portion of a geometry falling within a rectangle
// It will only work if snapToGrid is enabled
clipByBox2d: false, // this requires postgis >=2.2 and geos >=3.5
// geometries will be simplified using ST_RemoveRepeatedPoints
// which cost is no more expensive than snapping and results are
// much closer to the original geometry
removeRepeatedPoints: false // this requires postgis >=2.2
},
// If enabled Mapnik will reuse the features retrieved from the database
// instead of requesting them once per style inside a layer
'cache-features': true,
'cache-features': false,
// Require metrics to the renderer
metrics: false
metrics: false,
// Options for markers attributes, ellipses and images caches
markers_symbolizer_caches: {
disabled: false
}
},
http: {
timeout: 2000, // the timeout in ms for a http tile request
@ -222,16 +252,7 @@ var config = {
src: __dirname + '/../../assets/default-placeholder.png'
}
},
torque: {
dbPoolParams: {
// maximum number of resources to create at any given time
size: 16,
// max milliseconds a resource can go unused before it should be destroyed
idleTimeout: 3000,
// frequency to check for idle resources
reapInterval: 1000
}
}
torque: {}
}
// anything analyses related
,analysis: {
@ -252,7 +273,7 @@ var config = {
// If filename is given logs comming from analysis client will be written
// there, in append mode. Otherwise 'log_filename' is used. Otherwise stdout is used (default).
// Log file will be re-opened on receiving the HUP signal
filename: 'node-windshaft.log'
filename: '/tmp/node-windshaft-analysis.log'
},
// Define max execution time in ms for analyses or tags
// If analysis or tag are not found in redis this values will be used as default.
@ -322,6 +343,12 @@ var config = {
// X-Tiler-Profile header containing elapsed timing for various
// steps taken for producing the response.
,useProfiler:true
,serverMetadata: {
cdn_url: {
http: undefined,
https: undefined
}
}
// Settings for the health check available at /health
,health: {
enabled: false,
@ -339,7 +366,28 @@ var config = {
// whether the affected tables for a given SQL must query directly postgresql or use the SQL API
cdbQueryTablesFromPostgres: true,
// whether in mapconfig is available stats & metadata for each layer
layerStats: true
layerStats: true,
// whether it should rate limit endpoints (global configuration)
rateLimitsEnabled: false,
// whether it should rate limit one or more endpoints (only if rateLimitsEnabled = true)
rateLimitsByEndpoint: {
anonymous: false,
static: false,
static_named: false,
dataview: false,
dataview_search: false,
analysis: false,
analysis_catalog: false,
tile: false,
attributes: false,
named_list: false,
named_create: false,
named_get: false,
named: false,
named_update: false,
named_delete: false,
named_tiles: false
}
}
};

13
docker-bash.sh Executable file
View File

@ -0,0 +1,13 @@
#!/bin/bash
echo "*********************"
echo "To install Node.js, run:"
echo "/src/nodejs-install.sh"
echo "Use NODEJS_VERSION env var to select the Node.js version"
echo " "
echo "To start postgres, run:"
echo "/etc/init.d/postgresql start"
echo "*********************"
echo " "
docker run -it -v `pwd`:/srv carto/nodejs-xenial-pg101:latest bash

13
docker-test.sh Normal file → Executable file
View File

@ -1,11 +1,4 @@
export NPROCS=1 && export JOBS=1 && export CXX=g++-4.9 && export PGUSER=postgres
#!/bin/bash
npm install -g yarn@0.27.5
yarn
/etc/init.d/postgresql start
createdb template_postgis && createuser publicuser
psql -c "CREATE EXTENSION postgis" template_postgis
POSTGIS_VERSION=2.4 npm test
docker run -e "NODEJS_VERSION=${1}" -v `pwd`:/srv carto/nodejs-xenial-pg101:latest bash run_tests_docker.sh && \
docker ps --filter status=dead --filter status=exited -aq | xargs docker rm -v

92
docker/Dockerfile-nodejs-xenial-pg101:latest Normal file
View File

@ -0,0 +1,92 @@
FROM ubuntu:xenial
# Use UTF8 to avoid encoding problems with pgsql
ENV LANG C.UTF-8
ENV NPROCS 1
ENV JOBS 1
ENV CXX g++-4.9
ENV PGUSER postgres
# Add external repos
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
curl \
software-properties-common \
locales \
&& add-apt-repository -y ppa:ubuntu-toolchain-r/test \
&& add-apt-repository -y ppa:cartodb/postgresql-10 \
&& add-apt-repository -y ppa:cartodb/gis \
&& curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash \
&& . ~/.nvm/nvm.sh \
&& locale-gen en_US.UTF-8 \
&& update-locale LANG=en_US.UTF-8
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
g++-4.9 \
gcc-4.9 \
git \
libcairo2-dev \
libgdal-dev \
libgdal1i \
libgdal20 \
libgeos-dev \
libgif-dev \
libjpeg8-dev \
libjson-c-dev \
libpango1.0-dev \
libpixman-1-dev \
libproj-dev \
libprotobuf-c-dev \
libxml2-dev \
gdal-bin \
make \
nodejs \
protobuf-c-compiler \
pkg-config \
wget \
zip \
postgresql-10 \
postgresql-10-plproxy \
postgis=2.4.4.6+carto-1 \
postgresql-10-postgis-2.4=2.4.4.6+carto-1 \
postgresql-10-postgis-2.4-scripts=2.4.4.6+carto-1 \
postgresql-10-postgis-scripts=2.4.4.6+carto-1 \
postgresql-client-10 \
postgresql-client-common \
postgresql-common \
postgresql-contrib \
postgresql-plpython-10 \
postgresql-server-dev-10 \
&& wget http://download.redis.io/releases/redis-4.0.8.tar.gz \
&& tar xvzf redis-4.0.8.tar.gz \
&& cd redis-4.0.8 \
&& make \
&& make install \
&& cd .. \
&& rm redis-4.0.8.tar.gz \
&& rm -R redis-4.0.8 \
&& apt-get purge -y wget protobuf-c-compiler \
&& apt-get autoremove -y
# Configure PostgreSQL
RUN set -ex \
&& echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf \
&& echo "local all all trust" > /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all 0.0.0.0/0 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all ::1/128 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& /etc/init.d/postgresql start \
&& createdb template_postgis \
&& createuser publicuser \
&& psql -c "CREATE EXTENSION postgis" template_postgis \
&& /etc/init.d/postgresql stop
WORKDIR /srv
EXPOSE 5858
COPY ./scripts/nodejs-install.sh /src/nodejs-install.sh
RUN chmod 777 /src/nodejs-install.sh
CMD /src/nodejs-install.sh

88
docker/Dockerfile-nodejs10-xenial-pg101:postgis-2.4.4.5 Normal file
View File

@ -0,0 +1,88 @@
FROM ubuntu:xenial
# Use UTF8 to avoid encoding problems with pgsql
ENV LANG C.UTF-8
ENV NPROCS 1
ENV JOBS 1
ENV CXX g++-4.9
ENV PGUSER postgres
# Add external repos
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
curl \
software-properties-common \
locales \
&& add-apt-repository -y ppa:ubuntu-toolchain-r/test \
&& add-apt-repository -y ppa:cartodb/postgresql-10 \
&& add-apt-repository -y ppa:cartodb/gis \
&& curl -sL https://deb.nodesource.com/setup_10.x | bash \
&& locale-gen en_US.UTF-8 \
&& update-locale LANG=en_US.UTF-8
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
g++-4.9 \
gcc-4.9 \
git \
libcairo2-dev \
libgdal-dev \
libgdal1i \
libgdal20 \
libgeos-dev \
libgif-dev \
libjpeg8-dev \
libjson-c-dev \
libpango1.0-dev \
libpixman-1-dev \
libproj-dev \
libprotobuf-c-dev \
libxml2-dev \
gdal-bin \
make \
nodejs \
protobuf-c-compiler \
pkg-config \
wget \
zip \
postgresql-10 \
postgresql-10-plproxy \
postgis=2.4.4.5+carto-1 \
postgresql-10-postgis-2.4=2.4.4.5+carto-1 \
postgresql-10-postgis-2.4-scripts=2.4.4.5+carto-1 \
postgresql-10-postgis-scripts=2.4.4.5+carto-1 \
postgresql-client-10 \
postgresql-client-common \
postgresql-common \
postgresql-contrib \
postgresql-plpython-10 \
postgresql-server-dev-10 \
&& wget http://download.redis.io/releases/redis-4.0.8.tar.gz \
&& tar xvzf redis-4.0.8.tar.gz \
&& cd redis-4.0.8 \
&& make \
&& make install \
&& cd .. \
&& rm redis-4.0.8.tar.gz \
&& rm -R redis-4.0.8 \
&& apt-get purge -y wget protobuf-c-compiler \
&& apt-get autoremove -y
# Configure PostgreSQL
RUN set -ex \
&& echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf \
&& echo "local all all trust" > /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all 0.0.0.0/0 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all ::1/128 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& /etc/init.d/postgresql start \
&& createdb template_postgis \
&& createuser publicuser \
&& psql -c "CREATE EXTENSION postgis" template_postgis \
&& /etc/init.d/postgresql stop
WORKDIR /srv
EXPOSE 5858
CMD /etc/init.d/postgresql start

89
docker/Dockerfile-nodejs6-xenial-pg101:latest Normal file
View File

@ -0,0 +1,89 @@
FROM ubuntu:xenial
# Use UTF8 to avoid encoding problems with pgsql
ENV LANG C.UTF-8
ENV NPROCS 1
ENV JOBS 1
ENV CXX g++-4.9
ENV PGUSER postgres
# Add external repos
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
curl \
software-properties-common \
locales \
&& add-apt-repository -y ppa:ubuntu-toolchain-r/test \
&& add-apt-repository -y ppa:cartodb/postgresql-10 \
&& add-apt-repository -y ppa:cartodb/gis \
&& curl -sL https://deb.nodesource.com/setup_6.x | bash \
&& locale-gen en_US.UTF-8 \
&& update-locale LANG=en_US.UTF-8
# Install dependencies and PostGIS 2.4 from sources
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
g++-4.9 \
gcc-4.9 \
git \
libcairo2-dev \
libgdal-dev \
libgdal1i \
libgdal20 \
libgeos-dev \
libgif-dev \
libjpeg8-dev \
libjson-c-dev \
libpango1.0-dev \
libpixman-1-dev \
libproj-dev \
libprotobuf-c-dev \
libxml2-dev \
gdal-bin \
make \
nodejs \
protobuf-c-compiler \
pkg-config \
wget \
zip \
postgresql-10 \
postgresql-10-plproxy \
postgresql-10-postgis-2.4 \
postgresql-10-postgis-2.4-scripts \
postgresql-10-postgis-scripts \
postgresql-client-10 \
postgresql-client-common \
postgresql-common \
postgresql-contrib \
postgresql-plpython-10 \
postgresql-server-dev-10 \
postgis \
&& wget http://download.redis.io/releases/redis-4.0.8.tar.gz \
&& tar xvzf redis-4.0.8.tar.gz \
&& cd redis-4.0.8 \
&& make \
&& make install \
&& cd .. \
&& rm redis-4.0.8.tar.gz \
&& rm -R redis-4.0.8 \
&& apt-get purge -y wget protobuf-c-compiler \
&& apt-get autoremove -y
# Configure PostgreSQL
RUN set -ex \
&& echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf \
&& echo "local all all trust" > /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all 0.0.0.0/0 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all ::1/128 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& /etc/init.d/postgresql start \
&& createdb template_postgis \
&& createuser publicuser \
&& psql -c "CREATE EXTENSION postgis" template_postgis \
&& /etc/init.d/postgresql stop
WORKDIR /srv
EXPOSE 5858
CMD /etc/init.d/postgresql start

89
docker/Dockerfile-nodejs6-xenial-pg101:postgis-2.4 Normal file
View File

@ -0,0 +1,89 @@
FROM ubuntu:xenial
# Use UTF8 to avoid encoding problems with pgsql
ENV LANG C.UTF-8
ENV NPROCS 1
ENV JOBS 1
ENV CXX g++-4.9
ENV PGUSER postgres
# Add external repos
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
curl \
software-properties-common \
locales \
&& add-apt-repository -y ppa:ubuntu-toolchain-r/test \
&& add-apt-repository -y ppa:cartodb/postgresql-10 \
&& add-apt-repository -y ppa:cartodb/gis \
&& curl -sL https://deb.nodesource.com/setup_6.x | bash \
&& locale-gen en_US.UTF-8 \
&& update-locale LANG=en_US.UTF-8
# Install dependencies and PostGIS 2.4 from sources
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
g++-4.9 \
gcc-4.9 \
git \
libcairo2-dev \
libgdal-dev \
libgdal1i \
libgdal20 \
libgeos-dev \
libgif-dev \
libjpeg8-dev \
libjson-c-dev \
libpango1.0-dev \
libpixman-1-dev \
libproj-dev \
libprotobuf-c-dev \
libxml2-dev \
gdal-bin \
make \
nodejs \
protobuf-c-compiler \
pkg-config \
wget \
zip \
postgresql-10 \
postgresql-10-plproxy \
postgresql-10-postgis-2.4 \
postgresql-10-postgis-2.4-scripts \
postgresql-10-postgis-scripts \
postgresql-client-10 \
postgresql-client-common \
postgresql-common \
postgresql-contrib \
postgresql-plpython-10 \
postgresql-server-dev-10 \
postgis \
&& wget http://download.redis.io/releases/redis-4.0.8.tar.gz \
&& tar xvzf redis-4.0.8.tar.gz \
&& cd redis-4.0.8 \
&& make \
&& make install \
&& cd .. \
&& rm redis-4.0.8.tar.gz \
&& rm -R redis-4.0.8 \
&& apt-get purge -y wget protobuf-c-compiler \
&& apt-get autoremove -y
# Configure PostgreSQL
RUN set -ex \
&& echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf \
&& echo "local all all trust" > /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all 0.0.0.0/0 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all ::1/128 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& /etc/init.d/postgresql start \
&& createdb template_postgis \
&& createuser publicuser \
&& psql -c "CREATE EXTENSION postgis" template_postgis \
&& /etc/init.d/postgresql stop
WORKDIR /srv
EXPOSE 5858
CMD /etc/init.d/postgresql start

88
docker/Dockerfile-nodejs6-xenial-pg101:postgis-2.4.4.5 Normal file
View File

@ -0,0 +1,88 @@
FROM ubuntu:xenial
# Use UTF8 to avoid encoding problems with pgsql
ENV LANG C.UTF-8
ENV NPROCS 1
ENV JOBS 1
ENV CXX g++-4.9
ENV PGUSER postgres
# Add external repos
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
curl \
software-properties-common \
locales \
&& add-apt-repository -y ppa:ubuntu-toolchain-r/test \
&& add-apt-repository -y ppa:cartodb/postgresql-10 \
&& add-apt-repository -y ppa:cartodb/gis \
&& curl -sL https://deb.nodesource.com/setup_6.x | bash \
&& locale-gen en_US.UTF-8 \
&& update-locale LANG=en_US.UTF-8
RUN set -ex \
&& apt-get update \
&& apt-get install -y \
g++-4.9 \
gcc-4.9 \
git \
libcairo2-dev \
libgdal-dev \
libgdal1i \
libgdal20 \
libgeos-dev \
libgif-dev \
libjpeg8-dev \
libjson-c-dev \
libpango1.0-dev \
libpixman-1-dev \
libproj-dev \
libprotobuf-c-dev \
libxml2-dev \
gdal-bin \
make \
nodejs \
protobuf-c-compiler \
pkg-config \
wget \
zip \
postgresql-10 \
postgresql-10-plproxy \
postgis=2.4.4.5+carto-1 \
postgresql-10-postgis-2.4=2.4.4.5+carto-1 \
postgresql-10-postgis-2.4-scripts=2.4.4.5+carto-1 \
postgresql-10-postgis-scripts=2.4.4.5+carto-1 \
postgresql-client-10 \
postgresql-client-common \
postgresql-common \
postgresql-contrib \
postgresql-plpython-10 \
postgresql-server-dev-10 \
&& wget http://download.redis.io/releases/redis-4.0.8.tar.gz \
&& tar xvzf redis-4.0.8.tar.gz \
&& cd redis-4.0.8 \
&& make \
&& make install \
&& cd .. \
&& rm redis-4.0.8.tar.gz \
&& rm -R redis-4.0.8 \
&& apt-get purge -y wget protobuf-c-compiler \
&& apt-get autoremove -y
# Configure PostgreSQL
RUN set -ex \
&& echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf \
&& echo "local all all trust" > /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all 0.0.0.0/0 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& echo "host all all ::1/128 trust" >> /etc/postgresql/10/main/pg_hba.conf \
&& /etc/init.d/postgresql start \
&& createdb template_postgis \
&& createuser publicuser \
&& psql -c "CREATE EXTENSION postgis" template_postgis \
&& /etc/init.d/postgresql stop
WORKDIR /srv
EXPOSE 5858
CMD /etc/init.d/postgresql start

23
docker/reference.md Normal file
View File

@ -0,0 +1,23 @@
After running the tests with docker, you will need Docker installed and the docker image downloaded.
## Install docker
`sudo apt install docker.io && sudo usermod -aG docker $(whoami)`
## Download image
`docker pull carto/IMAGE`
## Carto account
https://hub.docker.com/r/carto/
## Update image
- Edit the docker image file with your desired changes
- Build image:
- `docker build -t carto/IMAGE -f docker/DOCKER_FILE docker/`
- Upload to docker hub:
- Login into docker hub:
- `docker login`
- Create tag:
- `docker tag carto/IMAGE carto/IMAGE`
- Upload:
- `docker push carto/IMAGE`

13
docker/scripts/nodejs-install.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
if [ -z $NODEJS_VERSION ]; then
NODEJS_VERSION="10"
NODEJS_VERSION_OPTIONS="--lts"
fi
nvm install $NODEJS_VERSION $NODEJS_VERSION_OPTIONS
nvm alias default $NODEJS_VERSION
nvm use default

2
docs/MapConfig-Aggregation-extension.md
View File

@ -29,7 +29,7 @@ The value of this attribute can be `false` to explicitly disable aggregation for
// object, defines the columns of the aggregated datasets. Each property corresponds to a columns name and
// should contain an object with two properties: "aggregate_function" (one of "sum", "max", "min", "avg", "mode" or "count"),
// and "aggregated_column" (the name of a column of the original layer query or "*")
// A column defined as `"_cdb_features_count": {"aggregate_function": "count", aggregated_column: "*"}`
// A column defined as `"_cdb_feature_count": {"aggregate_function": "count", aggregated_column: "*"}`
// is always generated in addition to the defined columns.
// The column names `cartodb_id`, `the_geom`, `the_geom_webmercator` and `_cdb_feature_count` cannot be used
// for aggregated columns, as they correspond to columns always present in the result.

85
docs/aggregation.md
View File

@ -10,7 +10,7 @@ Aggregation is available only for point geometries. During aggregation the point
When no placement or columns are specified a special default aggregation is performed.
This special mode performs only spatial aggregation (using a grid defined by the requested tile and the resolution, parameter, as all the other cases), and returns a _random_ record from each group (grid cell) with all its columns and an additional `_cdb_features_count` with the number of features in the group.
This special mode performs only spatial aggregation (using a grid defined by the requested tile and the resolution, parameter, as all the other cases), and returns a _random_ record from each group (grid cell) with all its columns and an additional `_cdb_feature_count` with the number of features in the group.
Regarding the randomness of the sample: currently we use the row with the minimum `cartodb_id` value in each group.
@ -18,7 +18,7 @@ The rationale behind having this special aggregation with all the original colum
### User defined aggregations
When either a explicit placement or columns are requested we no longer use the special, query; we use one determined by the placement (which will default to "centroid"), and it will have as columns only the aggregated columns specified, in addition to `_cdb_features_count`, which is always present.
When either a explicit placement or columns are requested we no longer use the special, query; we use one determined by the placement (which will default to "centroid"), and it will have as columns only the aggregated columns specified, in addition to `_cdb_feature_count`, which is always present.
We might decide in the future to allow sampling column values for any of the different placement modes.
@ -134,6 +134,10 @@ of the original dataset applying three different aggregate functions.
> Note that you can use the original column names as names of the result, but all the result column names must be unique. In particular, the names `cartodb_id`, `the_geom`, `the_geom_webmercator` and `_cdb_feature_count` cannot be used for aggregated columns, as they correspond to columns always present in the result.
#### Limitations:
* The iso text format does not admit `starting` or `count` parameters
* Cyclic units (day of the week, etc.) don't admit `count` or `starting` either.
### `resolution`
Defines the cell-size of the spatial aggregation grid. This is equivalent to the [CartoCSS `-torque-resolution`](https://carto.com/docs/carto-engine/cartocss/properties-for-torque/#-torque-resolution-float) property of Torque maps.
@ -185,3 +189,80 @@ This is the minimum number of (estimated) rows in the dataset (query results) fo
]
}
```
### `filters`
Aggregated data can be filtered by imposing filtering conditions on the aggregated columns.
Each condition is represented by one or more parameters:
* `{ "equal": V }` selects an specific value of the aggregated column.
* `{ "not_equal": V }` selects values different from the one specified.
* `{ "in": [v1, v2, v3] }` selects any value from a list.
* `{ "not_in": [v1, v2, v3] }` selects any value not in a list.
* `{ "less_than": v }` selects values strictly less than the one given.
* `{ "less_than_or_equal_to": v }` selects values less than or equal to the one given.
* `{ "greater_than": v }` selects values strictly greater than the one given.
* `{ "greater_than_or_equal_to": v }` selects values greater than or equal to the one given.
One of the *less* conditions can be combined with one of the *greater* conditions to select a range of values, for example:
* `{ "greater_than": v1, "less_than": v2 }`
* `{ "greater_than_or_equal_to": v1, "less_than": v2 }`
* `{ "greater_than": v1, "less_than_or_equal_to": v2 }`
* `{ "greater_than_or_equal_to": v1, "less_than_or_equal_to": v2 }`
For a given column, multiple conditions can be passed in an array; the conditions will logically ORed (any of the conditions have to be verifid for the value to be selected):
* `"myvalue": [ { "equal": 10 }, { "less_than": 0 }]` will select values of the column `myvalue` which are equal to 10 **or** less than 0.
In addition, the filters applied to different columns are logically combined with AND (all the conditions have to be satisfied for an element to be selected); for example with the following `filters` parameter we'll select aggregated records which have a `total_value` > 100 **and** a category equal to "a".
```json
{
"total_value": { "greater_than": 100 },
"category": { "equal": "a" }
}
```
Note that the filtered columns have to be defined with the `columns` parameter, except for `_cdb_feature_count`, which is always implicitly defined and can be filtered too.
#### Example
```json
{
"version": "1.7.0",
"extent": [-20037508.5, -20037508.5, 20037508.5, 20037508.5],
"srid": 3857,
"maxzoom": 18,
"minzoom": 3,
"layers": [
{
"type": "mapnik",
"options": {
"sql": "select * from table",
"cartocss": "#table { marker-width: [total]; marker-fill: ramp(value, (red, green, blue), jenks); }",
"cartocss_version": "2.3.0",
"aggregation": {
"placement": "centroid",
"columns": {
"total_value": {
"aggregate_function": "sum",
"aggregated_column": "value"
},
"category": {
"aggregate_function": "mode",
"aggregated_column": "category"
}
},
"filters" : {
"total_value": { "greater_than": 100 },
"category": { "equal": "a" }
},
"resolution": 2,
"threshold": 500000
}
}
}
]
}
```

5
docs/static_maps_api.md
View File

@ -11,7 +11,7 @@ Begin by instantiating either a Named or Anonymous Map using the `layergroupid t
#### Definition
```bash
GET /api/v1/map/static/center/{token}/{z}/{lat}/{lng}/{width}/{height}.{format}
GET /api/v1/map/static/center/{token}/{z}/{lat}/{lng}/{width}/{height}.{format}{{?}extra_options}
```
#### Params
@ -58,6 +58,9 @@ Note: you can see this endpoint as
GET /api/v1/map/static/bbox/{token}/{west},{south},{east},{north}/{width}/{height}.{format}`
```
#### Extra options
* Layer: List of layers to be shown in the image (by default `all`), for example `?layer=0,1`.
### Named Map
#### Definition

323
lib/cartodb/api/api-router.js Normal file
View File

@ -0,0 +1,323 @@
'use strict';
const { Router: router } = require('express');
const RedisPool = require('redis-mpool');
const cartodbRedis = require('cartodb-redis');
const windshaft = require('windshaft');
const PgConnection = require('../backends/pg_connection');
const AnalysisBackend = require('../backends/analysis');
const AnalysisStatusBackend = require('../backends/analysis-status');
const DataviewBackend = require('../backends/dataview');
const TemplateMaps = require('../backends/template_maps.js');
const PgQueryRunner = require('../backends/pg_query_runner');
const StatsBackend = require('../backends/stats');
const AuthBackend = require('../backends/auth');
const UserLimitsBackend = require('../backends/user-limits');
const OverviewsMetadataBackend = require('../backends/overviews-metadata');
const FilterStatsApi = require('../backends/filter-stats');
const TablesExtentBackend = require('../backends/tables-extent');
const LayergroupAffectedTablesCache = require('../cache/layergroup_affected_tables');
const SurrogateKeysCache = require('../cache/surrogate_keys_cache');
const VarnishHttpCacheBackend = require('../cache/backend/varnish_http');
const FastlyCacheBackend = require('../cache/backend/fastly');
const NamedMapProviderCache = require('../cache/named_map_provider_cache');
const NamedMapsCacheEntry = require('../cache/model/named_maps_entry');
const SqlWrapMapConfigAdapter = require('../models/mapconfig/adapter/sql-wrap-mapconfig-adapter');
const MapConfigNamedLayersAdapter = require('../models/mapconfig/adapter/mapconfig-named-layers-adapter');
const MapConfigBufferSizeAdapter = require('../models/mapconfig/adapter/mapconfig-buffer-size-adapter');
const AnalysisMapConfigAdapter = require('../models/mapconfig/adapter/analysis-mapconfig-adapter');
const MapConfigOverviewsAdapter = require('../models/mapconfig/adapter/mapconfig-overviews-adapter');
const TurboCartoAdapter = require('../models/mapconfig/adapter/turbo-carto-adapter');
const DataviewsWidgetsAdapter = require('../models/mapconfig/adapter/dataviews-widgets-adapter');
const AggregationMapConfigAdapter = require('../models/mapconfig/adapter/aggregation-mapconfig-adapter');
const MapConfigAdapter = require('../models/mapconfig/adapter');
const VectorMapConfigAdapter = require('../models/mapconfig/adapter/vector-mapconfig-adapter');
const ResourceLocator = require('../models/resource-locator');
const LayergroupMetadata = require('../utils/layergroup-metadata');
const RendererStatsReporter = require('../stats/reporter/renderer');
const initializeStatusCode = require('./middlewares/initialize-status-code');
const logger = require('./middlewares/logger');
const bodyParser = require('body-parser');
const servedByHostHeader = require('./middlewares/served-by-host-header');
const stats = require('./middlewares/stats');
const lzmaMiddleware = require('./middlewares/lzma');
const cors = require('./middlewares/cors');
const user = require('./middlewares/user');
const sendResponse = require('./middlewares/send-response');
const syntaxError = require('./middlewares/syntax-error');
const errorMiddleware = require('./middlewares/error-middleware');
const MapRouter = require('./map/map-router');
const TemplateRouter = require('./template/template-router');
module.exports = class ApiRouter {
constructor ({ serverOptions, environmentOptions }) {
this.serverOptions = serverOptions;
const redisOptions = Object.assign({
name: 'windshaft-server',
unwatchOnRelease: false,
noReadyCheck: true
}, environmentOptions.redis);
const redisPool = new RedisPool(redisOptions);
redisPool.on('status', function (status) {
var keyPrefix = 'windshaft.redis-pool.' + status.name + '.db' + status.db + '.';
global.statsClient.gauge(keyPrefix + 'count', status.count);
global.statsClient.gauge(keyPrefix + 'unused', status.unused);
global.statsClient.gauge(keyPrefix + 'waiting', status.waiting);
});
const metadataBackend = cartodbRedis({ pool: redisPool });
const pgConnection = new PgConnection(metadataBackend);
const mapStore = new windshaft.storage.MapStore({
pool: redisPool,
expire_time: serverOptions.grainstore.default_layergroup_ttl
});
const rendererFactory = createRendererFactory({ redisPool, serverOptions, environmentOptions });
const rendererCacheOpts = Object.assign({
ttl: 60000, // 60 seconds TTL by default
statsInterval: 60000 // reports stats every milliseconds defined here
}, serverOptions.renderCache || {});
const rendererCache = new windshaft.cache.RendererCache(rendererFactory, rendererCacheOpts);
const rendererStatsReporter = new RendererStatsReporter(rendererCache, rendererCacheOpts.statsInterval);
rendererStatsReporter.start();
const tileBackend = new windshaft.backend.Tile(rendererCache);
const attributesBackend = new windshaft.backend.Attributes();
const previewBackend = new windshaft.backend.Preview(rendererCache);
const mapValidatorBackend = new windshaft.backend.MapValidator(tileBackend, attributesBackend);
const mapBackend = new windshaft.backend.Map(rendererCache, mapStore, mapValidatorBackend);
const surrogateKeysCacheBackends = createSurrogateKeysCacheBackends(serverOptions);
const surrogateKeysCache = new SurrogateKeysCache(surrogateKeysCacheBackends);
const templateMaps = createTemplateMaps({ redisPool, surrogateKeysCache });
const analysisStatusBackend = new AnalysisStatusBackend();
const analysisBackend = new AnalysisBackend(metadataBackend, serverOptions.analysis);
const dataviewBackend = new DataviewBackend(analysisBackend);
const statsBackend = new StatsBackend();
const userLimitsBackend = new UserLimitsBackend(metadataBackend, {
limits: {
cacheOnTimeout: serverOptions.renderer.mapnik.limits.cacheOnTimeout || false,
render: serverOptions.renderer.mapnik.limits.render || 0,
rateLimitsEnabled: global.environment.enabledFeatures.rateLimitsEnabled
}
});
const authBackend = new AuthBackend(pgConnection, metadataBackend, mapStore, templateMaps);
const layergroupAffectedTablesCache = new LayergroupAffectedTablesCache();
if (process.env.NODE_ENV === 'test') {
this.layergroupAffectedTablesCache = layergroupAffectedTablesCache;
}
const pgQueryRunner = new PgQueryRunner(pgConnection);
const overviewsMetadataBackend = new OverviewsMetadataBackend(pgQueryRunner);
const filterStatsBackend = new FilterStatsApi(pgQueryRunner);
const tablesExtentBackend = new TablesExtentBackend(pgQueryRunner);
const mapConfigAdapter = new MapConfigAdapter(
new MapConfigNamedLayersAdapter(templateMaps, pgConnection),
new MapConfigBufferSizeAdapter(),
new SqlWrapMapConfigAdapter(),
new DataviewsWidgetsAdapter(),
new AnalysisMapConfigAdapter(analysisBackend),
new VectorMapConfigAdapter(pgConnection),
new AggregationMapConfigAdapter(pgConnection),
new MapConfigOverviewsAdapter(overviewsMetadataBackend, filterStatsBackend),
new TurboCartoAdapter()
);
const resourceLocator = new ResourceLocator(global.environment);
const layergroupMetadata = new LayergroupMetadata(resourceLocator);
const namedMapProviderCache = new NamedMapProviderCache(
templateMaps,
pgConnection,
metadataBackend,
userLimitsBackend,
mapConfigAdapter,
layergroupAffectedTablesCache
);
['update', 'delete'].forEach(function(eventType) {
templateMaps.on(eventType, namedMapProviderCache.invalidate.bind(namedMapProviderCache));
});
const collaborators = {
analysisStatusBackend,
attributesBackend,
dataviewBackend,
previewBackend,
tileBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache,
templateMaps,
mapBackend,
metadataBackend,
mapConfigAdapter,
statsBackend,
layergroupMetadata,
namedMapProviderCache,
tablesExtentBackend
};
this.mapRouter = new MapRouter({ collaborators });
this.templateRouter = new TemplateRouter({ collaborators });
}
register (app) {
// FIXME: we need a better way to reset cache while running tests
if (process.env.NODE_ENV === 'test') {
app.layergroupAffectedTablesCache = this.layergroupAffectedTablesCache;
}
Object.keys(this.serverOptions.routes).forEach(apiVersion => {
const routes = this.serverOptions.routes[apiVersion];
const apiRouter = router({ mergeParams: true });
apiRouter.use(logger(this.serverOptions));
apiRouter.use(initializeStatusCode());
apiRouter.use(bodyParser.json());
apiRouter.use(servedByHostHeader());
apiRouter.use(stats({
enabled: this.serverOptions.useProfiler,
statsClient: global.statsClient
}));
apiRouter.use(lzmaMiddleware());
apiRouter.use(cors());
apiRouter.use(user());
this.templateRouter.register(apiRouter, routes.template.paths);
this.mapRouter.register(apiRouter, routes.map.paths);
apiRouter.use(sendResponse());
apiRouter.use(syntaxError());
apiRouter.use(errorMiddleware());
const apiPaths = routes.paths;
apiPaths.forEach(path => app.use(path, apiRouter));
});
}
};
function createTemplateMaps ({ redisPool, surrogateKeysCache }) {
const templateMaps = new TemplateMaps(redisPool, {
max_user_templates: global.environment.maxUserTemplates
});
function invalidateNamedMap (owner, templateName) {
var startTime = Date.now();
surrogateKeysCache.invalidate(new NamedMapsCacheEntry(owner, templateName), function(err) {
var logMessage = JSON.stringify({
username: owner,
type: 'named_map_invalidation',
elapsed: Date.now() - startTime,
error: !!err ? JSON.stringify(err.message) : undefined
});
if (err) {
global.logger.warn(logMessage);
} else {
global.logger.info(logMessage);
}
});
}
['update', 'delete'].forEach(function(eventType) {
templateMaps.on(eventType, invalidateNamedMap);
});
return templateMaps;
}
function createSurrogateKeysCacheBackends(serverOptions) {
var cacheBackends = [];
if (serverOptions.varnish_purge_enabled) {
cacheBackends.push(
new VarnishHttpCacheBackend(serverOptions.varnish_host, serverOptions.varnish_http_port)
);
}
if (serverOptions.fastly &&
!!serverOptions.fastly.enabled && !!serverOptions.fastly.apiKey && !!serverOptions.fastly.serviceId) {
cacheBackends.push(
new FastlyCacheBackend(serverOptions.fastly.apiKey, serverOptions.fastly.serviceId)
);
}
return cacheBackends;
}
const timeoutErrorTilePath = __dirname + '/../../../assets/render-timeout-fallback.png';
const timeoutErrorTile = require('fs').readFileSync(timeoutErrorTilePath, {encoding: null});
function createRendererFactory ({ redisPool, serverOptions, environmentOptions }) {
var onTileErrorStrategy;
if (environmentOptions.enabledFeatures.onTileErrorStrategy !== false) {
onTileErrorStrategy = function onTileErrorStrategy$TimeoutTile(err, tile, headers, stats, format, callback) {
function isRenderTimeoutError (err) {
return err.message === 'Render timed out';
}
function isDatasourceTimeoutError (err) {
return err.message && err.message.match(/canceling statement due to statement timeout/i);
}
function isTimeoutError (err) {
return isRenderTimeoutError(err) || isDatasourceTimeoutError(err);
}
function isRasterFormat (format) {
return format === 'png' || format === 'jpg';
}
if (isTimeoutError(err) && isRasterFormat(format)) {
return callback(null, timeoutErrorTile, {
'Content-Type': 'image/png',
}, {});
} else {
return callback(err, tile, headers, stats);
}
};
}
const rendererFactory = new windshaft.renderer.Factory({
onTileErrorStrategy: onTileErrorStrategy,
mapnik: {
redisPool: redisPool,
grainstore: serverOptions.grainstore,
mapnik: serverOptions.renderer.mapnik
},
http: serverOptions.renderer.http,
mvt: serverOptions.renderer.mvt,
torque: serverOptions.renderer.torque
});
return rendererFactory;
}

138
lib/cartodb/api/auth_api.js
View File

@ -1,138 +0,0 @@
var assert = require('assert');
var step = require('step');
/**
*
* @param {PgConnection} pgConnection
* @param metadataBackend
* @param {MapStore} mapStore
* @param {TemplateMaps} templateMaps
* @constructor
* @type {AuthApi}
*/
function AuthApi(pgConnection, metadataBackend, mapStore, templateMaps) {
this.pgConnection = pgConnection;
this.metadataBackend = metadataBackend;
this.mapStore = mapStore;
this.templateMaps = templateMaps;
}
module.exports = AuthApi;
// Check if the user is authorized by a signer
//
// @param res express response object
// @param callback function(err, signed_by) signed_by will be
// null if the request is not signed by anyone
// or will be a string cartodb username otherwise.
//
AuthApi.prototype.authorizedBySigner = function(res, callback) {
if ( ! res.locals.token || ! res.locals.signer ) {
return callback(null, false); // no signer requested
}
var self = this;
var layergroup_id = res.locals.token;
var auth_token = res.locals.auth_token;
this.mapStore.load(layergroup_id, function(err, mapConfig) {
if (err) {
return callback(err);
}
var authorized = self.templateMaps.isAuthorized(mapConfig.obj().template, auth_token);
return callback(null, authorized);
});
};
// Check if a request is authorized by api_key
//
// @param user
// @param req express request object
// @param callback function(err, authorized)
// NOTE: authorized is expected to be 0 or 1 (integer)
//
AuthApi.prototype.authorizedByAPIKey = function(user, req, callback) {
var givenKey = req.query.api_key || req.query.map_key;
if ( ! givenKey && req.body ) {
// check also in request body
givenKey = req.body.api_key || req.body.map_key;
}
if ( ! givenKey ) {
return callback(null, 0); // no api key, no authorization...
}
var self = this;
step(
function () {
self.metadataBackend.getUserMapKey(user, this);
},
function checkApiKey(err, val){
assert.ifError(err);
return val && givenKey === val;
},
function finish(err, authorized) {
callback(err, authorized);
}
);
};
/**
* Check access authorization
*
* @param req - standard req object. Importantly contains table and host information
* @param res - standard res object. Contains the auth parameters in locals
* @param callback function(err, allowed) is access allowed not?
*/
AuthApi.prototype.authorize = function(req, res, callback) {
var self = this;
var user = res.locals.user;
step(
function () {
self.authorizedByAPIKey(user, req, this);
},
function checkApiKey(err, authorized){
req.profiler.done('authorizedByAPIKey');
assert.ifError(err);
// if not authorized by api_key, continue
if (!authorized) {
// not authorized by api_key, check if authorized by signer
return self.authorizedBySigner(res, this);
}
// authorized by api key, login as the given username and stop
self.pgConnection.setDBAuth(user, res.locals, function(err) {
callback(err, true); // authorized (or error)
});
},
function checkSignAuthorized(err, authorized) {
if (err) {
return callback(err);
}
if ( ! authorized ) {
// request not authorized by signer.
// if no signer name was given, let dbparams and
// PostgreSQL do the rest.
//
if ( ! res.locals.signer ) {
return callback(null, true); // authorized so far
}
// if signer name was given, return no authorization
return callback(null, false);
}
self.pgConnection.setDBAuth(user, res.locals, function(err) {
req.profiler.done('setDBAuth');
callback(err, true); // authorized (or error)
});
}
);
};

59
lib/cartodb/api/filter_stats_api.js
View File

@ -1,59 +0,0 @@
var _ = require('underscore');
var step = require('step');
var AnalysisFilter = require('../models/filter/analysis');
function FilterStatsApi(pgQueryRunner) {
this.pgQueryRunner = pgQueryRunner;
}
module.exports = FilterStatsApi;
function getEstimatedRows(pgQueryRunner, username, query, callback) {
pgQueryRunner.run(username, "EXPLAIN (FORMAT JSON)"+query, function(err, result_rows) {
if (err){
callback(err);
return;
}
var rows;
if ( result_rows[0] && result_rows[0]['QUERY PLAN'] &&
result_rows[0]['QUERY PLAN'][0] && result_rows[0]['QUERY PLAN'][0].Plan ) {
rows = result_rows[0]['QUERY PLAN'][0].Plan['Plan Rows'];
}
return callback(null, rows);
});
}
FilterStatsApi.prototype.getFilterStats = function (username, unfiltered_query, filters, callback) {
var stats = {};
var self = this;
step(
function getUnfilteredRows() {
getEstimatedRows(self.pgQueryRunner, username, unfiltered_query, this);
},
function receiveUnfilteredRows(err, rows) {
if (err){
callback(err);
return;
}
stats.unfiltered_rows = rows;
this(null, rows);
},
function getFilteredRows() {
if ( filters && !_.isEmpty(filters)) {
var analysisFilter = new AnalysisFilter(filters);
var query = analysisFilter.sql(unfiltered_query);
getEstimatedRows(self.pgQueryRunner, username, query, this);
} else {
this(null, null);
}
},
function receiveFilteredRows(err, rows) {
if (err){
callback(err);
return;
}
stats.filtered_rows = rows;
callback(null, stats);
}
);
};

117
lib/cartodb/controllers/analyses.js → lib/cartodb/api/map/analyses-catalog-controller.js
View File

@ -1,37 +1,54 @@
var PSQL = require('cartodb-psql');
var cors = require('../middleware/cors');
var userMiddleware = require('../middleware/user');
'use strict';
function AnalysesController(prepareContext) {
this.prepareContext = prepareContext;
}
const PSQL = require('cartodb-psql');
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const authorize = require('../middlewares/authorize');
const dbConnSetup = require('../middlewares/db-conn-setup');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
const cacheControlHeader = require('../middlewares/cache-control-header');
const dbParamsFromResLocals = require('../../utils/database-params');
module.exports = AnalysesController;
module.exports = class AnalysesController {
constructor (pgConnection, authBackend, userLimitsBackend) {
this.pgConnection = pgConnection;
this.authBackend = authBackend;
this.userLimitsBackend = userLimitsBackend;
}
AnalysesController.prototype.register = function (app) {
app.get(
`${app.base_url_mapconfig}/analyses/catalog`,
cors(),
userMiddleware,
this.prepareContext,
this.createPGClient(),
this.getDataFromQuery({ queryTemplate: catalogQueryTpl, key: 'catalog' }),
this.getDataFromQuery({ queryTemplate: tablesQueryTpl, key: 'tables' }),
this.prepareResponse(),
this.setCacheControlHeader(),
this.sendResponse(),
this.unathorizedError()
);
register (mapRouter) {
mapRouter.get('/analyses/catalog', this.middlewares());
}
middlewares () {
return [
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.ANALYSIS_CATALOG),
cleanUpQueryParams(),
createPGClient(),
getDataFromQuery({ queryTemplate: catalogQueryTpl, key: 'catalog' }),
getDataFromQuery({ queryTemplate: tablesQueryTpl, key: 'tables' }),
prepareResponse(),
cacheControlHeader({ ttl: 10, revalidate: true }),
unauthorizedError()
];
}
};
AnalysesController.prototype.createPGClient = function () {
function createPGClient () {
return function createPGClientMiddleware (req, res, next) {
res.locals.pg = new PSQL(dbParamsFromReqParams(res.locals));
const dbParams = dbParamsFromResLocals(res.locals);
res.locals.pg = new PSQL(dbParams);
next();
};
};
}
AnalysesController.prototype.getDataFromQuery = function ({ queryTemplate, key }) {
function getDataFromQuery({ queryTemplate, key }) {
const readOnlyTransactionOn = true;
return function getCatalogMiddleware(req, res, next) {
@ -48,9 +65,9 @@ AnalysesController.prototype.getDataFromQuery = function ({ queryTemplate, key }
next();
}, readOnlyTransactionOn);
};
};
}
AnalysesController.prototype.prepareResponse = function () {
function prepareResponse () {
return function prepareResponseMiddleware (req, res, next) {
const { catalog, tables } = res.locals;
@ -87,32 +104,14 @@ AnalysesController.prototype.prepareResponse = function () {
return -1;
});
res.statusCode = 200;
res.body = { catalog: analysisCatalog };
next();
};
};
}
AnalysesController.prototype.setCacheControlHeader = function () {
return function setCacheControlHeaderMiddleware (req, res, next) {
res.set('Cache-Control', 'public,max-age=10,must-revalidate');
next();
};
};
AnalysesController.prototype.sendResponse = function() {
return function sendResponseMiddleware (req, res) {
res.status(200);
if (req.query && req.query.callback) {
res.jsonp(res.body);
} else {
res.json(res.body);
}
};
};
AnalysesController.prototype.unathorizedError = function () {
function unauthorizedError () {
return function unathorizedErrorMiddleware(err, req, res, next) {
if (err.message.match(/permission\sdenied/)) {
err = new Error('Unauthorized');
@ -121,7 +120,7 @@ AnalysesController.prototype.unathorizedError = function () {
next(err);
};
};
}
const catalogQueryTpl = ctx => `
SELECT analysis_def->>'type' as type, * FROM cdb_analysis_catalog WHERE username = '${ctx._username}'
@ -145,23 +144,3 @@ var tablesQueryTpl = ctx => `
FROM analysis_tables
ORDER BY size DESC
`;
function dbParamsFromReqParams(params) {
var dbParams = {};
if ( params.dbuser ) {
dbParams.user = params.dbuser;
}
if ( params.dbpassword ) {
dbParams.pass = params.dbpassword;
}
if ( params.dbhost ) {
dbParams.host = params.dbhost;
}
if ( params.dbport ) {
dbParams.port = params.dbport;
}
if ( params.dbname ) {
dbParams.dbname = params.dbname;
}
return dbParams;
}

61
lib/cartodb/api/map/analysis-layergroup-controller.js Normal file
View File

@ -0,0 +1,61 @@
'use strict';
const layergroupToken = require('../middlewares/layergroup-token');
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
const dbParamsFromResLocals = require('../../utils/database-params');
module.exports = class AnalysisLayergroupController {
constructor (analysisStatusBackend, pgConnection, userLimitsBackend, authBackend) {
this.analysisStatusBackend = analysisStatusBackend;
this.pgConnection = pgConnection;
this.userLimitsBackend = userLimitsBackend;
this.authBackend = authBackend;
}
register (mapRouter) {
mapRouter.get('/:token/analysis/node/:nodeId', this.middlewares());
}
middlewares () {
return [
layergroupToken(),
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.ANALYSIS),
cleanUpQueryParams(),
analysisNodeStatus(this.analysisStatusBackend)
];
}
};
function analysisNodeStatus (analysisStatusBackend) {
return function analysisNodeStatusMiddleware(req, res, next) {
const { nodeId } = req.params;
const dbParams = dbParamsFromResLocals(res.locals);
analysisStatusBackend.getNodeStatus(nodeId, dbParams, (err, nodeStatus, stats = {}) => {
req.profiler.add(stats);
if (err) {
err.label = 'GET NODE STATUS';
return next(err);
}
res.set({
'Cache-Control': 'public,max-age=5',
'Last-Modified': new Date().toUTCString()
});
res.statusCode = 200;
res.body = nodeStatus;
next();
});
};
}

224
lib/cartodb/api/map/anonymous-map-controller.js Normal file
View File

@ -0,0 +1,224 @@
'use strict';
const windshaft = require('windshaft');
const MapConfig = windshaft.model.MapConfig;
const Datasource = windshaft.model.Datasource;
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const initProfiler = require('../middlewares/init-profiler');
const checkJsonContentType = require('../middlewares/check-json-content-type');
const incrementMapViewCount = require('../middlewares/increment-map-view-count');
const augmentLayergroupData = require('../middlewares/augment-layergroup-data');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
const lastUpdatedTimeLayergroup = require('../middlewares/last-updated-time-layergroup');
const layerStats = require('../middlewares/layer-stats');
const layergroupIdHeader = require('../middlewares/layergroup-id-header');
const layergroupMetadata = require('../middlewares/layergroup-metadata');
const mapError = require('../middlewares/map-error');
const CreateLayergroupMapConfigProvider = require('../../models/mapconfig/provider/create-layergroup-provider');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
module.exports = class AnonymousMapController {
/**
* @param {AuthBackend} authBackend
* @param {PgConnection} pgConnection
* @param {TemplateMaps} templateMaps
* @param {MapBackend} mapBackend
* @param metadataBackend
* @param {SurrogateKeysCache} surrogateKeysCache
* @param {UserLimitsBackend} userLimitsBackend
* @param {LayergroupAffectedTables} layergroupAffectedTables
* @param {MapConfigAdapter} mapConfigAdapter
* @param {StatsBackend} statsBackend
* @constructor
*/
constructor (
pgConnection,
templateMaps,
mapBackend,
metadataBackend,
surrogateKeysCache,
userLimitsBackend,
layergroupAffectedTables,
mapConfigAdapter,
statsBackend,
authBackend,
layergroupMetadata
) {
this.pgConnection = pgConnection;
this.templateMaps = templateMaps;
this.mapBackend = mapBackend;
this.metadataBackend = metadataBackend;
this.surrogateKeysCache = surrogateKeysCache;
this.userLimitsBackend = userLimitsBackend;
this.layergroupAffectedTables = layergroupAffectedTables;
this.mapConfigAdapter = mapConfigAdapter;
this.statsBackend = statsBackend;
this.authBackend = authBackend;
this.layergroupMetadata = layergroupMetadata;
}
register (mapRouter) {
mapRouter.options('/');
mapRouter.get('/', this.middlewares());
mapRouter.post('/', this.middlewares());
}
middlewares () {
const isTemplateInstantiation = false;
const useTemplateHash = false;
const includeQuery = true;
const label = 'ANONYMOUS LAYERGROUP';
const addContext = true;
return [
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.ANONYMOUS),
cleanUpQueryParams(['aggregation']),
initProfiler(isTemplateInstantiation),
checkJsonContentType(),
checkCreateLayergroup(),
prepareAdapterMapConfig(this.mapConfigAdapter),
createLayergroup (
this.mapBackend,
this.userLimitsBackend,
this.pgConnection,
this.layergroupAffectedTables
),
incrementMapViewCount(this.metadataBackend),
augmentLayergroupData(),
cacheControlHeader({ ttl: global.environment.varnish.layergroupTtl || 86400, revalidate: true }),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader(),
lastUpdatedTimeLayergroup(),
layerStats(this.pgConnection, this.statsBackend),
layergroupIdHeader(this.templateMaps, useTemplateHash),
layergroupMetadata(this.layergroupMetadata, includeQuery),
mapError({ label, addContext })
];
}
};
function checkCreateLayergroup () {
return function checkCreateLayergroupMiddleware (req, res, next) {
if (req.method === 'GET') {
const { config } = req.query;
if (!config) {
return next(new Error('layergroup GET needs a "config" parameter'));
}
try {
req.body = JSON.parse(config);
} catch (err) {
return next(err);
}
}
req.profiler.done('checkCreateLayergroup');
return next();
};
}
function prepareAdapterMapConfig (mapConfigAdapter) {
return function prepareAdapterMapConfigMiddleware(req, res, next) {
const requestMapConfig = req.body;
const { user, api_key } = res.locals;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const params = Object.assign({ dbuser, dbname, dbpassword, dbhost, dbport }, req.query);
const context = {
analysisConfiguration: {
user,
db: {
host: dbhost,
port: dbport,
dbname: dbname,
user: dbuser,
pass: dbpassword
},
batch: {
username: user,
apiKey: api_key
}
}
};
mapConfigAdapter.getMapConfig(user,
requestMapConfig,
params,
context,
(err, requestMapConfig, stats = { overviewsAddedToMapconfig : false }) => {
req.profiler.done('anonymous.getMapConfig');
stats.mapType = 'anonymous';
req.profiler.add(stats);
if (err) {
return next(err);
}
req.body = requestMapConfig;
res.locals.context = context;
next();
});
};
}
function createLayergroup (mapBackend, userLimitsBackend, pgConnection, affectedTablesCache) {
return function createLayergroupMiddleware (req, res, next) {
const requestMapConfig = req.body;
const { context } = res.locals;
const { user, cache_buster, api_key } = res.locals;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const params = {
cache_buster, api_key,
dbuser, dbname, dbpassword, dbhost, dbport
};
const datasource = context.datasource || Datasource.EmptyDatasource();
const mapConfig = new MapConfig(requestMapConfig, datasource);
const mapConfigProvider = new CreateLayergroupMapConfigProvider(
mapConfig,
user,
userLimitsBackend,
pgConnection,
affectedTablesCache,
params
);
res.locals.mapConfig = mapConfig;
res.locals.analysesResults = context.analysesResults;
const mapParams = { dbuser, dbname, dbpassword, dbhost, dbport };
mapBackend.createLayergroup(mapConfig, mapParams, mapConfigProvider, (err, layergroup, stats = {}) => {
req.profiler.add(stats);
if (err) {
return next(err);
}
res.statusCode = 200;
res.body = layergroup;
res.locals.mapConfigProvider = mapConfigProvider;
next();
});
};
}

91
lib/cartodb/api/map/attributes-layergroup-controller.js Normal file
View File

@ -0,0 +1,91 @@
'use strict';
const layergroupToken = require('../middlewares/layergroup-token');
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
const createMapStoreMapConfigProvider = require('../middlewares/map-store-map-config-provider');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
module.exports = class AttributesLayergroupController {
constructor (
attributesBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
) {
this.attributesBackend = attributesBackend;
this.pgConnection = pgConnection;
this.mapStore = mapStore;
this.userLimitsBackend = userLimitsBackend;
this.layergroupAffectedTablesCache = layergroupAffectedTablesCache;
this.authBackend = authBackend;
this.surrogateKeysCache = surrogateKeysCache;
}
register (mapRouter) {
mapRouter.get('/:token/:layer/attributes/:fid', this.middlewares());
}
middlewares () {
return [
layergroupToken(),
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.ATTRIBUTES),
cleanUpQueryParams(),
createMapStoreMapConfigProvider(
this.mapStore,
this.userLimitsBackend,
this.pgConnection,
this.layergroupAffectedTablesCache
),
getFeatureAttributes(this.attributesBackend),
cacheControlHeader(),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader()
];
}
};
function getFeatureAttributes (attributesBackend) {
return function getFeatureAttributesMiddleware (req, res, next) {
req.profiler.start('windshaft.maplayer_attribute');
const { mapConfigProvider } = res.locals;
const { token } = res.locals;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const { layer, fid } = req.params;
const params = {
token,
dbuser, dbname, dbpassword, dbhost, dbport,
layer, fid
};
attributesBackend.getFeatureAttributes(mapConfigProvider, params, false, (err, tile, stats = {}) => {
req.profiler.add(stats);
if (err) {
err.label = 'GET ATTRIBUTES';
return next(err);
}
res.statusCode = 200;
res.body = tile;
next();
});
};
}

144
lib/cartodb/api/map/dataview-layergroup-controller.js Normal file
View File

@ -0,0 +1,144 @@
'use strict';
const layergroupToken = require('../middlewares/layergroup-token');
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
const createMapStoreMapConfigProvider = require('../middlewares/map-store-map-config-provider');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
const ALLOWED_DATAVIEW_QUERY_PARAMS = [
'filters', // json
'own_filter', // 0, 1
'no_filters', // 0, 1
'bbox', // w,s,e,n
'start', // number
'end', // number
'column_type', // string
'bins', // number
'aggregation', //string
'offset', // number
'q', // widgets search
'categories', // number
];
module.exports = class DataviewLayergroupController {
constructor (
dataviewBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
) {
this.dataviewBackend = dataviewBackend;
this.pgConnection = pgConnection;
this.mapStore = mapStore;
this.userLimitsBackend = userLimitsBackend;
this.layergroupAffectedTablesCache = layergroupAffectedTablesCache;
this.authBackend = authBackend;
this.surrogateKeysCache = surrogateKeysCache;
}
register (mapRouter) {
// Undocumented/non-supported API endpoint methods.
// Use at your own peril.
mapRouter.get('/:token/dataview/:dataviewName', this.middlewares({
action: 'get',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.DATAVIEW
}));
mapRouter.get('/:token/:layer/widget/:dataviewName', this.middlewares({
action: 'get',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.DATAVIEW
}));
mapRouter.get('/:token/dataview/:dataviewName/search', this.middlewares({
action: 'search',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.DATAVIEW_SEARCH
}));
mapRouter.get('/:token/:layer/widget/:dataviewName/search', this.middlewares({
action: 'search',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.DATAVIEW_SEARCH
}));
}
middlewares ({ action, rateLimitGroup }) {
return [
layergroupToken(),
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, rateLimitGroup),
cleanUpQueryParams(ALLOWED_DATAVIEW_QUERY_PARAMS),
createMapStoreMapConfigProvider(
this.mapStore,
this.userLimitsBackend,
this.pgConnection,
this.layergroupAffectedTablesCache
),
action === 'search' ? dataviewSearch(this.dataviewBackend) : getDataview(this.dataviewBackend),
cacheControlHeader(),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader()
];
}
};
function getDataview (dataviewBackend) {
return function getDataviewMiddleware (req, res, next) {
const { user, mapConfigProvider } = res.locals;
const { dataviewName } = req.params;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const params = Object.assign({ dataviewName, dbuser, dbname, dbpassword, dbhost, dbport }, req.query);
dataviewBackend.getDataview(mapConfigProvider, user, params, (err, dataview, stats = {}) => {
req.profiler.add(stats);
if (err) {
err.label = 'GET DATAVIEW';
return next(err);
}
res.statusCode = 200;
res.body = dataview;
next();
});
};
}
function dataviewSearch (dataviewBackend) {
return function dataviewSearchMiddleware (req, res, next) {
const { user, mapConfigProvider } = res.locals;
const { dataviewName } = req.params;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const params = Object.assign({ dbuser, dbname, dbpassword, dbhost, dbport }, req.query);
dataviewBackend.search(mapConfigProvider, user, dataviewName, params, (err, searchResult, stats = {}) => {
req.profiler.add(stats);
if (err) {
err.label = 'GET DATAVIEW SEARCH';
return next(err);
}
res.statusCode = 200;
res.body = searchResult;
next();
});
};
}

131
lib/cartodb/api/map/map-router.js Normal file
View File

@ -0,0 +1,131 @@
'use strict';
const { Router: router } = require('express');
const AnalysisLayergroupController = require('./analysis-layergroup-controller');
const AttributesLayergroupController = require('./attributes-layergroup-controller');
const DataviewLayergroupController = require('./dataview-layergroup-controller');
const PreviewLayergroupController = require('./preview-layergroup-controller');
const TileLayergroupController = require('./tile-layergroup-controller');
const AnonymousMapController = require('./anonymous-map-controller');
const PreviewTemplateController = require('./preview-template-controller');
const AnalysesCatalogController = require('./analyses-catalog-controller');
module.exports = class MapRouter {
constructor ({ collaborators }) {
const {
analysisStatusBackend,
attributesBackend,
dataviewBackend,
previewBackend,
tileBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache,
templateMaps,
mapBackend,
metadataBackend,
mapConfigAdapter,
statsBackend,
layergroupMetadata,
namedMapProviderCache,
tablesExtentBackend
} = collaborators;
this.analysisLayergroupController = new AnalysisLayergroupController(
analysisStatusBackend,
pgConnection,
userLimitsBackend,
authBackend
);
this.attributesLayergroupController = new AttributesLayergroupController(
attributesBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
);
this.dataviewLayergroupController = new DataviewLayergroupController(
dataviewBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
);
this.previewLayergroupController = new PreviewLayergroupController(
previewBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
);
this.tileLayergroupController = new TileLayergroupController(
tileBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
);
this.anonymousMapController = new AnonymousMapController(
pgConnection,
templateMaps,
mapBackend,
metadataBackend,
surrogateKeysCache,
userLimitsBackend,
layergroupAffectedTablesCache,
mapConfigAdapter,
statsBackend,
authBackend,
layergroupMetadata
);
this.previewTemplateController = new PreviewTemplateController(
namedMapProviderCache,
previewBackend,
surrogateKeysCache,
tablesExtentBackend,
metadataBackend,
pgConnection,
authBackend,
userLimitsBackend
);
this.analysesController = new AnalysesCatalogController(
pgConnection,
authBackend,
userLimitsBackend
);
}
register (apiRouter, mapPaths) {
const mapRouter = router({ mergeParams: true });
this.analysisLayergroupController.register(mapRouter);
this.attributesLayergroupController.register(mapRouter);
this.dataviewLayergroupController.register(mapRouter);
this.previewLayergroupController.register(mapRouter);
this.tileLayergroupController.register(mapRouter);
this.anonymousMapController.register(mapRouter);
this.previewTemplateController.register(mapRouter);
this.analysesController.register(mapRouter);
mapPaths.forEach(path => apiRouter.use(path, mapRouter));
}
};

158
lib/cartodb/api/map/preview-layergroup-controller.js Normal file
View File

@ -0,0 +1,158 @@
'use strict';
const layergroupToken = require('../middlewares/layergroup-token');
const coordinates = require('../middlewares/coordinates');
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const noop = require('../middlewares/noop');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
const createMapStoreMapConfigProvider = require('../middlewares/map-store-map-config-provider');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
const checkStaticImageFormat = require('../middlewares/check-static-image-format');
module.exports = class PreviewLayergroupController {
constructor (
previewBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
) {
this.previewBackend = previewBackend;
this.pgConnection = pgConnection;
this.mapStore = mapStore;
this.userLimitsBackend = userLimitsBackend;
this.layergroupAffectedTablesCache = layergroupAffectedTablesCache;
this.authBackend = authBackend;
this.surrogateKeysCache = surrogateKeysCache;
}
register (mapRouter) {
mapRouter.get('/static/center/:token/:z/:lat/:lng/:width/:height.:format', this.middlewares({
validateZoom: true,
previewType: 'centered'
}));
mapRouter.get('/static/bbox/:token/:west,:south,:east,:north/:width/:height.:format', this.middlewares({
validateZoom: false,
previewType: 'bbox'
}));
}
middlewares ({ validateZoom, previewType }) {
const forcedFormat = 'png';
let getPreviewImage;
if (previewType === 'centered') {
getPreviewImage = getPreviewImageByCenter;
}
if (previewType === 'bbox') {
getPreviewImage = getPreviewImageByBoundingBox;
}
return [
layergroupToken(),
validateZoom ? coordinates({ z: true, x: false, y: false }) : noop(),
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.STATIC),
cleanUpQueryParams(['layer']),
checkStaticImageFormat(),
createMapStoreMapConfigProvider(
this.mapStore,
this.userLimitsBackend,
this.pgConnection,
this.layergroupAffectedTablesCache,
forcedFormat
),
getPreviewImage(this.previewBackend),
cacheControlHeader(),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader()
];
}
};
function getPreviewImageByCenter (previewBackend) {
return function getPreviewImageByCenterMiddleware (req, res, next) {
const width = +req.params.width;
const height = +req.params.height;
const zoom = +req.params.z;
const center = {
lng: +req.params.lng,
lat: +req.params.lat
};
const format = req.params.format === 'jpg' ? 'jpeg' : 'png';
const { mapConfigProvider: provider } = res.locals;
previewBackend.getImage(provider, format, width, height, zoom, center, (err, image, headers, stats = {}) => {
req.profiler.done(`render-${format}`);
req.profiler.add(stats);
if (err) {
err.label = 'STATIC_MAP';
return next(err);
}
if (headers) {
res.set(headers);
}
res.set('Content-Type', headers['Content-Type'] || `image/${format}`);
res.statusCode = 200;
res.body = image;
next();
});
};
}
function getPreviewImageByBoundingBox (previewBackend) {
return function getPreviewImageByBoundingBoxMiddleware (req, res, next) {
const width = +req.params.width;
const height = +req.params.height;
const bounds = {
west: +req.params.west,
north: +req.params.north,
east: +req.params.east,
south: +req.params.south
};
const format = req.params.format === 'jpg' ? 'jpeg' : 'png';
const { mapConfigProvider: provider } = res.locals;
previewBackend.getImage(provider, format, width, height, bounds, (err, image, headers, stats = {}) => {
req.profiler.done(`render-${format}`);
req.profiler.add(stats);
if (err) {
err.label = 'STATIC_MAP';
return next(err);
}
if (headers) {
res.set(headers);
}
res.set('Content-Type', headers['Content-Type'] || `image/${format}`);
res.statusCode = 200;
res.body = image;
next();
});
};
}

369
lib/cartodb/api/map/preview-template-controller.js Normal file
View File

@ -0,0 +1,369 @@
'use strict';
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const namedMapProvider = require('../middlewares/named-map-provider');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
const checkStaticImageFormat = require('../middlewares/check-static-image-format');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
const DEFAULT_ZOOM_CENTER = {
zoom: 1,
center: {
lng: 0,
lat: 0
}
};
function numMapper(n) {
return +n;
}
module.exports = class PreviewTemplateController {
constructor (
namedMapProviderCache,
previewBackend,
surrogateKeysCache,
tablesExtentBackend,
metadataBackend,
pgConnection,
authBackend,
userLimitsBackend
) {
this.namedMapProviderCache = namedMapProviderCache;
this.previewBackend = previewBackend;
this.surrogateKeysCache = surrogateKeysCache;
this.tablesExtentBackend = tablesExtentBackend;
this.metadataBackend = metadataBackend;
this.pgConnection = pgConnection;
this.authBackend = authBackend;
this.userLimitsBackend = userLimitsBackend;
}
register (mapRouter) {
mapRouter.get('/static/named/:template_id/:width/:height.:format', this.middlewares());
}
middlewares () {
return [
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.STATIC_NAMED),
cleanUpQueryParams(['layer', 'zoom', 'lon', 'lat', 'bbox']),
checkStaticImageFormat(),
namedMapProvider({
namedMapProviderCache: this.namedMapProviderCache,
label: 'STATIC_VIZ_MAP', forcedFormat: 'png'
}),
getTemplate({ label: 'STATIC_VIZ_MAP' }),
prepareLayerFilterFromPreviewLayers({
namedMapProviderCache: this.namedMapProviderCache,
label: 'STATIC_VIZ_MAP'
}),
getStaticImageOptions({ tablesExtentBackend: this.tablesExtentBackend }),
getImage({ previewBackend: this.previewBackend, label: 'STATIC_VIZ_MAP' }),
setContentTypeHeader(),
incrementMapViews({ metadataBackend: this.metadataBackend }),
cacheControlHeader(),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader()
];
}
};
function getTemplate ({ label }) {
return function getTemplateMiddleware (req, res, next) {
const { mapConfigProvider } = res.locals;
mapConfigProvider.getTemplate((err, template) => {
if (err) {
err.label = label;
return next(err);
}
res.locals.template = template;
next();
});
};
}
function prepareLayerFilterFromPreviewLayers ({ namedMapProviderCache, label }) {
return function prepareLayerFilterFromPreviewLayersMiddleware (req, res, next) {
const { template } = res.locals;
const { config, auth_token } = req.query;
if (!template || !template.view || !template.view.preview_layers) {
return next();
}
var previewLayers = template.view.preview_layers;
var layerVisibilityFilter = [];
template.layergroup.layers.forEach((layer, index) => {
if (previewLayers[''+index] !== false && previewLayers[layer.id] !== false) {
layerVisibilityFilter.push(''+index);
}
});
if (!layerVisibilityFilter.length) {
return next();
}
const { user, token, cache_buster, api_key } = res.locals;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const { template_id, format } = req.params;
const params = {
user, token, cache_buster, api_key,
dbuser, dbname, dbpassword, dbhost, dbport,
template_id, format
};
// overwrites 'all' default filter
params.layer = layerVisibilityFilter.join(',');
// recreates the provider
namedMapProviderCache.get(user, template_id, config, auth_token, params, (err, provider) => {
if (err) {
err.label = label;
return next(err);
}
res.locals.mapConfigProvider = provider;
next();
});
};
}
function getStaticImageOptions ({ tablesExtentBackend }) {
return function getStaticImageOptionsMiddleware(req, res, next) {
const { user, mapConfigProvider, template } = res.locals;
const { zoom, lon, lat, bbox } = req.query;
const params = { zoom, lon, lat, bbox };
const imageOpts = getImageOptions(params, template);
if (imageOpts) {
res.locals.imageOpts = imageOpts;
return next();
}
res.locals.imageOpts = DEFAULT_ZOOM_CENTER;
mapConfigProvider.createAffectedTables((err, affectedTables) => {
if (err) {
return next();
}
var tables = affectedTables.tables || [];
if (tables.length === 0) {
return next();
}
tablesExtentBackend.getBounds(user, tables, (err, bounds) => {
if (err) {
return next();
}
res.locals.imageOpts = bounds;
return next();
});
});
};
}
function getImageOptions (params, template) {
const { zoom, lon, lat, bbox } = params;
let imageOpts = getImageOptionsFromCoordinates(zoom, lon, lat);
if (imageOpts) {
return imageOpts;
}
imageOpts = getImageOptionsFromBoundingBox(bbox);
if (imageOpts) {
return imageOpts;
}
imageOpts = getImageOptionsFromTemplate(template, zoom);
if (imageOpts) {
return imageOpts;
}
}
function getImageOptionsFromCoordinates (zoom, lon, lat) {
if ([zoom, lon, lat].map(numMapper).every(Number.isFinite)) {
return {
zoom: zoom,
center: {
lng: lon,
lat: lat
}
};
}
}
function getImageOptionsFromTemplate (template, zoom) {
if (template.view) {
var zoomCenter = templateZoomCenter(template.view);
if (zoomCenter) {
if (Number.isFinite(+zoom)) {
zoomCenter.zoom = +zoom;
}
return zoomCenter;
}
var bounds = templateBounds(template.view);
if (bounds) {
return bounds;
}
}
}
function getImageOptionsFromBoundingBox (bbox = '') {
var _bbox = bbox.split(',').map(numMapper);
if (_bbox.length === 4 && _bbox.every(Number.isFinite)) {
return {
bounds: {
west: _bbox[0],
south: _bbox[1],
east: _bbox[2],
north: _bbox[3]
}
};
}
}
function getImage({ previewBackend, label }) {
return function getImageMiddleware (req, res, next) {
const { imageOpts, mapConfigProvider } = res.locals;
const { zoom, center, bounds } = imageOpts;
let { width, height } = req.params;
width = +width;
height = +height;
const format = req.params.format === 'jpg' ? 'jpeg' : 'png';
if (zoom !== undefined && center) {
return previewBackend.getImage(mapConfigProvider, format, width, height, zoom, center,
(err, image, headers, stats) => {
req.profiler.add(stats);
if (err) {
err.label = label;
return next(err);
}
if (headers) {
res.set(headers);
}
res.statusCode = 200;
res.body = image;
next();
});
}
previewBackend.getImage(mapConfigProvider, format, width, height, bounds, (err, image, headers, stats) => {
req.profiler.add(stats);
req.profiler.done('render-' + format);
if (err) {
err.label = label;
return next(err);
}
if (headers) {
res.set(headers);
}
res.statusCode = 200;
res.body = image;
next();
});
};
}
function setContentTypeHeader () {
return function setContentTypeHeaderMiddleware(req, res, next) {
res.set('Content-Type', res.get('content-type') || res.get('Content-Type') || 'image/png');
next();
};
}
function incrementMapViewsError (ctx) {
return `ERROR: failed to increment mapview count for user '${ctx.user}': ${ctx.err}`;
}
function incrementMapViews ({ metadataBackend }) {
return function incrementMapViewsMiddleware(req, res, next) {
const { user, mapConfigProvider } = res.locals;
mapConfigProvider.getMapConfig((err, mapConfig) => {
if (err) {
global.logger.log(incrementMapViewsError({ user, err }));
return next();
}
const statTag = mapConfig.obj().stat_tag;
metadataBackend.incMapviewCount(user, statTag, (err) => {
if (err) {
global.logger.log(incrementMapViewsError({ user, err }));
}
next();
});
});
};
}
function templateZoomCenter(view) {
if (view.zoom !== undefined && view.center) {
return {
zoom: view.zoom,
center: view.center
};
}
return false;
}
function templateBounds(view) {
if (view.bounds) {
var hasAllBounds = ['west', 'south', 'east', 'north'].every(prop => Number.isFinite(view.bounds[prop]));
if (hasAllBounds) {
return {
bounds: {
west: view.bounds.west,
south: view.bounds.south,
east: view.bounds.east,
north: view.bounds.north
}
};
} else {
return false;
}
}
return false;
}

168
lib/cartodb/api/map/tile-layergroup-controller.js Normal file
View File

@ -0,0 +1,168 @@
'use strict';
const layergroupToken = require('../middlewares/layergroup-token');
const coordinates = require('../middlewares/coordinates');
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
const createMapStoreMapConfigProvider = require('../middlewares/map-store-map-config-provider');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
const vectorError = require('../middlewares/vector-error');
const SUPPORTED_FORMATS = {
grid_json: true,
json_torque: true,
torque_json: true,
png: true,
png32: true,
mvt: true
};
module.exports = class TileLayergroupController {
constructor (
tileBackend,
pgConnection,
mapStore,
userLimitsBackend,
layergroupAffectedTablesCache,
authBackend,
surrogateKeysCache
) {
this.tileBackend = tileBackend;
this.pgConnection = pgConnection;
this.mapStore = mapStore;
this.userLimitsBackend = userLimitsBackend;
this.layergroupAffectedTablesCache = layergroupAffectedTablesCache;
this.authBackend = authBackend;
this.surrogateKeysCache = surrogateKeysCache;
}
register (mapRouter) {
// REGEXP: doesn't match with `val`
const not = (val) => `(?!${val})([^\/]+?)`;
// Sadly the path that matches 1 also matches with 2 so we need to tell to express
// that performs only the middlewares of the first path that matches
// for that we use one array to group all paths.
mapRouter.get([
`/:token/:z/:x/:y@:scale_factor?x.:format`, // 1
`/:token/:z/:x/:y.:format`, // 2
`/:token${not('static')}/:layer/:z/:x/:y.(:format)`
], this.middlewares());
}
middlewares () {
return [
layergroupToken(),
coordinates(),
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.TILE),
cleanUpQueryParams(),
createMapStoreMapConfigProvider(
this.mapStore,
this.userLimitsBackend,
this.pgConnection,
this.layergroupAffectedTablesCache
),
getTile(this.tileBackend),
cacheControlHeader(),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader(),
incrementSuccessMetrics(global.statsClient),
incrementErrorMetrics(global.statsClient),
tileError(),
vectorError()
];
}
};
function parseFormat (format = '') {
const prettyFormat = format.replace('.', '_');
return SUPPORTED_FORMATS[prettyFormat] ? prettyFormat : 'invalid';
}
function getStatusCode(tile, format){
return tile.length === 0 && format === 'mvt' ? 204 : 200;
}
function getTile (tileBackend) {
return function getTileMiddleware (req, res, next) {
req.profiler.start(`windshaft.${req.params.layer ? 'maplayer_tile' : 'map_tile'}`);
const { mapConfigProvider } = res.locals;
const { token } = res.locals;
const { layer, z, x, y, format } = req.params;
const params = { token, layer, z, x, y, format };
tileBackend.getTile(mapConfigProvider, params, (err, tile, headers, stats = {}) => {
req.profiler.add(stats);
if (err) {
return next(err);
}
if (headers) {
res.set(headers);
}
const formatStat = parseFormat(req.params.format);
res.statusCode = getStatusCode(tile, formatStat);
res.body = tile;
next();
});
};
}
function incrementSuccessMetrics (statsClient) {
return function incrementSuccessMetricsMiddleware (req, res, next) {
const formatStat = parseFormat(req.params.format);
statsClient.increment('windshaft.tiles.success');
statsClient.increment(`windshaft.tiles.${formatStat}.success`);
next();
};
}
function incrementErrorMetrics (statsClient) {
return function incrementErrorMetricsMiddleware (err, req, res, next) {
const formatStat = parseFormat(req.params.format);
statsClient.increment('windshaft.tiles.error');
statsClient.increment(`windshaft.tiles.${formatStat}.error`);
next(err);
};
}
function tileError () {
return function tileErrorMiddleware (err, req, res, next) {
// See https://github.com/Vizzuality/Windshaft-cartodb/issues/68
let errMsg = err.message ? ( '' + err.message ) : ( '' + err );
// Rewrite mapnik parsing errors to start with layer number
const matches = errMsg.match("(.*) in style 'layer([0-9]+)'");
if (matches) {
errMsg = `style${matches[2]}: ${matches[1]}`;
}
err.message = errMsg;
err.label = 'TILE RENDER';
next(err);
};
}

16
lib/cartodb/api/middlewares/augment-layergroup-data.js Normal file
View File

@ -0,0 +1,16 @@
'use strict';
const _ = require('underscore');
module.exports = function augmentLayergroupData () {
return function augmentLayergroupDataMiddleware (req, res, next) {
const layergroup = res.body;
// include in layergroup response the variables in serverMedata
// those variables are useful to send to the client information
// about how to reach this server or information about it
_.extend(layergroup, global.environment.serverMetadata);
next();
};
};

9
lib/cartodb/middleware/context/authorize.js → lib/cartodb/api/middlewares/authorize.js
View File

@ -1,9 +1,10 @@
module.exports = function authorizeMiddleware (authApi) {
return function (req, res, next) {
req.profiler.done('req2params.setup');
'use strict';
authApi.authorize(req, res, (err, authorized) => {
module.exports = function authorize (authBackend) {
return function authorizeMiddleware (req, res, next) {
authBackend.authorize(req, res, (err, authorized) => {
req.profiler.done('authorize');
if (err) {
return next(err);
}

26
lib/cartodb/api/middlewares/cache-channel-header.js Normal file
View File

@ -0,0 +1,26 @@
'use strict';
module.exports = function setCacheChannelHeader () {
return function setCacheChannelHeaderMiddleware (req, res, next) {
if (req.method !== 'GET') {
return next();
}
const { mapConfigProvider } = res.locals;
mapConfigProvider.getAffectedTables((err, affectedTables) => {
if (err) {
global.logger.warn('ERROR generating Cache Channel Header:', err);
return next();
}
if (!affectedTables) {
return next();
}
res.set('X-Cache-Channel', affectedTables.getCacheChannel());
next();
});
};
};

21
lib/cartodb/api/middlewares/cache-control-header.js Normal file
View File

@ -0,0 +1,21 @@
'use strict';
const ONE_YEAR_IN_SECONDS = 60 * 60 * 24 * 365;
module.exports = function setCacheControlHeader ({ ttl = ONE_YEAR_IN_SECONDS, revalidate = false } = {}) {
return function setCacheControlHeaderMiddleware (req, res, next) {
if (req.method !== 'GET') {
return next();
}
const directives = [ 'public', `max-age=${ttl}` ];
if (revalidate) {
directives.push('must-revalidate');
}
res.set('Cache-Control', directives.join(','));
next();
};
};

13
lib/cartodb/api/middlewares/check-json-content-type.js Normal file
View File

@ -0,0 +1,13 @@
'use strict';
module.exports = function checkJsonContentType () {
return function checkJsonContentTypeMiddleware(req, res, next) {
if (req.method === 'POST' && !req.is('application/json')) {
return next(new Error('POST data must be of type application/json'));
}
req.profiler.done('checkJsonContentTypeMiddleware');
next();
};
};

13
lib/cartodb/api/middlewares/check-static-image-format.js Normal file
View File

@ -0,0 +1,13 @@
'use strict';
const VALID_IMAGE_FORMATS = ['png', 'jpg'];
module.exports = function checkStaticImageFormat () {
return function checkStaticImageFormatMiddleware (req, res, next) {
if(!VALID_IMAGE_FORMATS.includes(req.params.format)) {
return next(new Error(`Unsupported image format "${req.params.format}"`));
}
next();
};
};

17
lib/cartodb/middleware/context/clean-up-query-params.js → lib/cartodb/api/middlewares/clean-up-query-params.js
View File

@ -1,3 +1,5 @@
'use strict';
const _ = require('underscore');
// Whitelist query parameters and attach format
@ -14,19 +16,16 @@ const REQUEST_QUERY_PARAMS_WHITELIST = [
'filters' // json
];
module.exports = function cleanUpQueryParamsMiddleware () {
return function cleanUpQueryParams (req, res, next) {
var allowedQueryParams = REQUEST_QUERY_PARAMS_WHITELIST;
module.exports = function cleanUpQueryParamsMiddleware (customQueryParams = []) {
if (!Array.isArray(customQueryParams)) {
throw new Error('customQueryParams must receive an Array of params');
}
if (Array.isArray(res.locals.allowedQueryParams)) {
allowedQueryParams = allowedQueryParams.concat(res.locals.allowedQueryParams);
}
return function cleanUpQueryParams (req, res, next) {
const allowedQueryParams = [...REQUEST_QUERY_PARAMS_WHITELIST, ...customQueryParams];
req.query = _.pick(req.query, allowedQueryParams);
// bring all query values onto res.locals object
_.extend(res.locals, req.query);
next();
};
};

43
lib/cartodb/api/middlewares/coordinates.js Normal file
View File

@ -0,0 +1,43 @@
'use strict';
const positiveIntegerNumberRegExp = /^\d+$/;
const integerNumberRegExp = /^-?\d+$/;
const invalidZoomMessage = function (zoom) {
return `Invalid zoom value (${zoom}). It should be an integer number greather than or equal to 0`;
};
const invalidCoordXMessage = function (x) {
return `Invalid coodinate 'x' value (${x}). It should be an integer number`;
};
const invalidCoordYMessage = function (y) {
return `Invalid coodinate 'y' value (${y}). It should be an integer number greather than or equal to 0`;
};
module.exports = function coordinates (validate = { z: true, x: true, y: true }) {
return function coordinatesMiddleware (req, res, next) {
const { z, x, y } = req.params;
if (validate.z && !positiveIntegerNumberRegExp.test(z)) {
const err = new Error(invalidZoomMessage(z));
err.http_status = 400;
return next(err);
}
// Negative values for x param are valid. The x param is wrapped
if (validate.x && !integerNumberRegExp.test(x)) {
const err = new Error(invalidCoordXMessage(x));
err.http_status = 400;
return next(err);
}
if (validate.y && !positiveIntegerNumberRegExp.test(y)) {
const err = new Error(invalidCoordYMessage(y));
err.http_status = 400;
return next(err);
}
next();
};
};

21
lib/cartodb/api/middlewares/cors.js Normal file
View File

@ -0,0 +1,21 @@
'use strict';
module.exports = function cors () {
return function corsMiddleware (req, res, next) {
const headers = [
'X-Requested-With',
'X-Prototype-Version',
'X-CSRF-Token',
'Authorization'
];
if (req.method === 'OPTIONS') {
headers.push('Content-Type');
}
res.set("Access-Control-Allow-Origin", "*");
res.set("Access-Control-Allow-Headers", headers.join(', '));
next();
};
};

87
lib/cartodb/api/middlewares/credentials.js Normal file
View File

@ -0,0 +1,87 @@
'use strict';
const basicAuth = require('basic-auth');
module.exports = function credentials () {
return function credentialsMiddleware(req, res, next) {
const apikeyCredentials = getApikeyCredentialsFromRequest(req);
res.locals.api_key = apikeyCredentials.token;
res.locals.basicAuthUsername = apikeyCredentials.username;
res.set('vary', 'Authorization'); //Honor Authorization header when caching.
return next();
};
};
function getApikeyCredentialsFromRequest(req) {
let apikeyCredentials = {
token: null,
username: null,
};
for (let getter of apikeyGetters) {
apikeyCredentials = getter(req);
if (apikeyTokenFound(apikeyCredentials)) {
break;
}
}
return apikeyCredentials;
}
const apikeyGetters = [
getApikeyTokenFromHeaderAuthorization,
getApikeyTokenFromRequestQueryString,
getApikeyTokenFromRequestBody,
];
function getApikeyTokenFromHeaderAuthorization(req) {
const credentials = basicAuth(req);
if (credentials) {
return {
username: credentials.username,
token: credentials.pass
};
} else {
return {
username: null,
token: null,
};
}
}
function getApikeyTokenFromRequestQueryString(req) {
let token = null;
if (req.query && req.query.api_key) {
token = req.query.api_key;
} else if (req.query && req.query.map_key) {
token = req.query.map_key;
}
return {
username: null,
token: token,
};
}
function getApikeyTokenFromRequestBody(req) {
let token = null;
if (req.body && req.body.api_key) {
token = req.body.api_key;
} else if (req.body && req.body.map_key) {
token = req.body.map_key;
}
return {
username: null,
token: token,
};
}
function apikeyTokenFound(apikey) {
return !!apikey && !!apikey.token;
}

21
lib/cartodb/middleware/context/db-conn-setup.js → lib/cartodb/api/middlewares/db-conn-setup.js
View File

@ -1,31 +1,32 @@
'use strict';
const _ = require('underscore');
module.exports = function dbConnSetupMiddleware(pgConnection) {
return function dbConnSetup(req, res, next) {
const user = res.locals.user;
module.exports = function dbConnSetup (pgConnection) {
return function dbConnSetupMiddleware (req, res, next) {
const { user } = res.locals;
pgConnection.setDBConn(user, res.locals, (err) => {
req.profiler.done('dbConnSetup');
if (err) {
if (err.message && -1 !== err.message.indexOf('name not found')) {
err.http_status = 404;
}
req.profiler.done('req2params');
return next(err);
}
// Add default database connection parameters
// if none given
_.defaults(res.locals, {
dbuser: global.environment.postgres.user,
dbpassword: global.environment.postgres.password,
dbhost: global.environment.postgres.host,
dbport: global.environment.postgres.port
});
res.set('X-Served-By-DB-Host', res.locals.dbhost);
req.profiler.done('req2params');
next(null);
next();
});
};
};

61
lib/cartodb/middleware/error-middleware.js → lib/cartodb/api/middlewares/error-middleware.js
View File

@ -1,3 +1,5 @@
'use strict';
const _ = require('underscore');
const debug = require('debug')('windshaft:cartodb:error-middleware');
@ -7,7 +9,7 @@ module.exports = function errorMiddleware (/* options */) {
// jshint maxcomplexity:9
var allErrors = Array.isArray(err) ? err : [err];
allErrors = populateTimeoutErrors(allErrors);
allErrors = populateLimitErrors(allErrors);
const label = err.label || 'UNKNOWN';
err = allErrors[0] || new Error(label);
@ -15,10 +17,6 @@ module.exports = function errorMiddleware (/* options */) {
var statusCode = findStatusCode(err);
if (err.message === 'Tile does not exist' && res.locals.format === 'mvt') {
statusCode = 204;
}
setErrorHeader(allErrors, statusCode, res);
debug('[%s ERROR] -- %d: %s, %s', label, statusCode, err, err.stack);
@ -50,26 +48,53 @@ function isDatasourceTimeoutError (err) {
return err.message && err.message.match(/canceling statement due to statement timeout/i);
}
function isTimeoutError (err) {
return isRenderTimeoutError(err) || isDatasourceTimeoutError(err);
function isTimeoutError (errorTypes) {
return errorTypes.renderTimeoutError || errorTypes.datasourceTimeoutError;
}
function populateTimeoutErrors (errors) {
function getErrorTypes(error) {
return {
renderTimeoutError: isRenderTimeoutError(error),
datasourceTimeoutError: isDatasourceTimeoutError(error),
};
}
function isMaxWaitingClientsError (err) {
return err.message === 'max waitingClients count exceeded';
}
function populateLimitErrors (errors) {
return errors.map(function (error) {
if (isRenderTimeoutError(error)) {
error.subtype = 'render';
if (isMaxWaitingClientsError(error)) {
error.message = 'You are over platform\'s limits: Max render capacity exceeded.' +
' Contact CARTO support for more details.';
error.type = 'limit';
error.subtype = 'render-capacity';
error.http_status = 429;
return error;
}
if (isDatasourceTimeoutError(error)) {
error.subtype = 'datasource';
}
const errorTypes = getErrorTypes(error);
if (isTimeoutError(error)) {
if (isTimeoutError(errorTypes)) {
error.message = 'You are over platform\'s limits. Please contact us to know more details';
error.type = 'limit';
error.http_status = 429;
}
if (errorTypes.datasourceTimeoutError) {
error.subtype = 'datasource';
error.message = 'You are over platform\'s limits: SQL query timeout error.' +
' Refactor your query before running again or contact CARTO support for more details.';
}
if (errorTypes.renderTimeoutError) {
error.subtype = 'render';
error.message = 'You are over platform\'s limits: Render timeout error.' +
' Contact CARTO support for more details.';
}
return error;
});
}
@ -186,15 +211,15 @@ function setErrorHeader(errors, statusCode, res) {
subtype: error.subtype
};
});
res.set('X-Tiler-Errors', stringifyForLogs(errorsLog));
}
/**
* Remove problematic nested characters
* Remove problematic nested characters
* from object for logs RegEx
*
* @param {Object} object
*
* @param {Object} object
*/
function stringifyForLogs(object) {
Object.keys(object).map(key => {

18
lib/cartodb/api/middlewares/increment-map-view-count.js Normal file
View File

@ -0,0 +1,18 @@
'use strict';
module.exports = function incrementMapViewCount (metadataBackend) {
return function incrementMapViewCountMiddleware(req, res, next) {
const { mapConfig, user } = res.locals;
// Error won't blow up, just be logged.
metadataBackend.incMapviewCount(user, mapConfig.obj().stat_tag, (err) => {
req.profiler.done('incMapviewCount');
if (err) {
global.logger.log(`ERROR: failed to increment mapview count for user '${user}': ${err.message}`);
}
next();
});
};
};

11
lib/cartodb/api/middlewares/init-profiler.js Normal file
View File

@ -0,0 +1,11 @@
'use strict';
module.exports = function initProfiler (isTemplateInstantiation) {
const operation = isTemplateInstantiation ? 'instance_template' : 'createmap';
return function initProfilerMiddleware (req, res, next) {
req.profiler.start(`windshaft-cartodb.${operation}_${req.method.toLowerCase()}`);
req.profiler.done(`${operation}.initProfilerMiddleware`);
next();
};
};

11
lib/cartodb/api/middlewares/initialize-status-code.js Normal file
View File

@ -0,0 +1,11 @@
'use strict';
module.exports = function initializeStatusCode () {
return function initializeStatusCodeMiddleware (req, res, next) {
if (req.method !== 'OPTIONS') {
res.statusCode = 404;
}
next();
};
};

40
lib/cartodb/api/middlewares/last-modified-header.js Normal file
View File

@ -0,0 +1,40 @@
'use strict';
module.exports = function setLastModifiedHeader () {
return function setLastModifiedHeaderMiddleware(req, res, next) {
if (req.method !== 'GET') {
return next();
}
const { mapConfigProvider, cache_buster } = res.locals;
if (cache_buster) {
const cacheBuster = parseInt(cache_buster, 10);
const lastModifiedDate = Number.isFinite(cacheBuster) ? new Date(cacheBuster) : new Date();
res.set('Last-Modified', lastModifiedDate.toUTCString());
return next();
}
mapConfigProvider.getAffectedTables((err, affectedTables) => {
if (err) {
global.logger.warn('ERROR generating Last Modified Header:', err);
return next();
}
if (!affectedTables) {
res.set('Last-Modified', new Date().toUTCString());
return next();
}
const lastUpdatedAt = affectedTables.getLastUpdatedAt();
const lastModifiedDate = Number.isFinite(lastUpdatedAt) ? new Date(lastUpdatedAt) : new Date();
res.set('Last-Modified', lastModifiedDate.toUTCString());
next();
});
};
};

41
lib/cartodb/api/middlewares/last-updated-time-layergroup.js Normal file
View File

@ -0,0 +1,41 @@
'use strict';
module.exports = function setLastUpdatedTimeToLayergroup () {
return function setLastUpdatedTimeToLayergroupMiddleware (req, res, next) {
const { mapConfigProvider, analysesResults } = res.locals;
const layergroup = res.body;
mapConfigProvider.createAffectedTables((err, affectedTables) => {
if (err) {
return next(err);
}
if (!affectedTables) {
return next();
}
var lastUpdateTime = affectedTables.getLastUpdatedAt();
lastUpdateTime = getLastUpdatedTime(analysesResults, lastUpdateTime) || lastUpdateTime;
// last update for layergroup cache buster
layergroup.layergroupid = layergroup.layergroupid + ':' + lastUpdateTime;
layergroup.last_updated = new Date(lastUpdateTime).toISOString();
next();
});
};
};
function getLastUpdatedTime(analysesResults, lastUpdateTime) {
if (!Array.isArray(analysesResults)) {
return lastUpdateTime;
}
return analysesResults.reduce(function(lastUpdateTime, analysis) {
return analysis.getNodes().reduce(function(lastNodeUpdatedAtTime, node) {
var nodeUpdatedAtDate = node.getUpdatedAt();
var nodeUpdatedTimeAt = (nodeUpdatedAtDate && nodeUpdatedAtDate.getTime()) || 0;
return nodeUpdatedTimeAt > lastNodeUpdatedAtTime ? nodeUpdatedTimeAt : lastNodeUpdatedAtTime;
}, lastUpdateTime);
}, lastUpdateTime);
}

28
lib/cartodb/api/middlewares/layer-stats.js Normal file
View File

@ -0,0 +1,28 @@
'use strict';
module.exports = function setLayerStats (pgConnection, statsBackend) {
return function setLayerStatsMiddleware(req, res, next) {
const { user, mapConfig } = res.locals;
const layergroup = res.body;
pgConnection.getConnection(user, (err, connection) => {
if (err) {
return next(err);
}
statsBackend.getStats(mapConfig, connection, function(err, layersStats) {
if (err) {
return next(err);
}
if (layersStats.length > 0) {
layergroup.metadata.layers.forEach(function (layer, index) {
layer.meta.stats = layersStats[index];
});
}
next();
});
});
};
};

17
lib/cartodb/api/middlewares/layergroup-id-header.js Normal file
View File

@ -0,0 +1,17 @@
'use strict';
module.exports = function setLayergroupIdHeader (templateMaps, useTemplateHash) {
return function setLayergroupIdHeaderMiddleware (req, res, next) {
const { user, template } = res.locals;
const layergroup = res.body;
if (useTemplateHash) {
var templateHash = templateMaps.fingerPrint(template).substring(0, 8);
layergroup.layergroupid = `${user}@${templateHash}@${layergroup.layergroupid}`;
}
res.set('X-Layergroup-Id', layergroup.layergroupid);
next();
};
};

17
lib/cartodb/api/middlewares/layergroup-metadata.js Normal file
View File

@ -0,0 +1,17 @@
'use strict';
module.exports = function setMetadataToLayergroup (layergroupMetadata, includeQuery) {
return function setMetadataToLayergroupMiddleware (req, res, next) {
const { user, mapConfig, analysesResults = [], context, api_key: userApiKey } = res.locals;
const layergroup = res.body;
layergroupMetadata.addDataviewsAndWidgetsUrls(user, layergroup, mapConfig.obj());
layergroupMetadata.addAnalysesMetadata(user, layergroup, analysesResults, includeQuery);
layergroupMetadata.addTurboCartoContextMetadata(layergroup, mapConfig.obj(), context);
layergroupMetadata.addAggregationContextMetadata(layergroup, mapConfig.obj(), context);
layergroupMetadata.addDateWrappingMetadata (layergroup, mapConfig.obj());
layergroupMetadata.addTileJsonMetadata(layergroup, user, mapConfig, userApiKey);
next();
};
};

30
lib/cartodb/api/middlewares/layergroup-token.js Normal file
View File

@ -0,0 +1,30 @@
'use strict';
const LayergroupToken = require('../../models/layergroup-token');
const authErrorMessageTemplate = function (signer, user) {
return `Cannot use map signature of user "${signer}" on db of user "${user}"`;
};
module.exports = function layergroupToken () {
return function layergroupTokenMiddleware (req, res, next) {
const user = res.locals.user;
const layergroupToken = LayergroupToken.parse(req.params.token);
res.locals.token = layergroupToken.token;
res.locals.cache_buster = layergroupToken.cacheBuster;
if (layergroupToken.signer) {
res.locals.signer = layergroupToken.signer;
if (res.locals.signer !== user) {
const err = new Error(authErrorMessageTemplate(res.locals.signer, user));
err.type = 'auth';
err.http_status = (req.query && req.query.callback) ? 200: 403;
return next(err);
}
}
return next();
};
};

24
lib/cartodb/api/middlewares/logger.js Normal file
View File

@ -0,0 +1,24 @@
'use strict';
module.exports = function logger (options) {
if (!global.log4js || !options.log_format) {
return function dummyLoggerMiddleware (req, res, next) {
next();
};
}
const opts = {
level: 'info',
// Allowing for unbuffered logging is mainly
// used to avoid hanging during unit testing.
// TODO: provide an explicit teardown function instead,
// releasing any event handler or timer set by
// this component.
buffer: !options.unbuffered_logging,
// optional log format
format: options.log_format
};
const logger = global.log4js.getLogger();
return global.log4js.connectLogger(logger, opts);
};

35
lib/cartodb/api/middlewares/lzma.js Normal file
View File

@ -0,0 +1,35 @@
'use strict';
const LZMA = require('lzma').LZMA;
module.exports = function lzma () {
const lzmaWorker = new LZMA();
return function lzmaMiddleware (req, res, next) {
if (!req.query.hasOwnProperty('lzma')) {
return next();
}
// Decode (from base64)
var lzma = new Buffer(req.query.lzma, 'base64')
.toString('binary')
.split('')
.map(function(c) {
return c.charCodeAt(0) - 128;
});
// Decompress
lzmaWorker.decompress(lzma, function(result) {
try {
delete req.query.lzma;
Object.assign(req.query, JSON.parse(result));
req.profiler.done('lzma');
next();
} catch (err) {
next(new Error('Error parsing lzma as JSON: ' + err));
}
});
};
};

37
lib/cartodb/api/middlewares/map-error.js Normal file
View File

@ -0,0 +1,37 @@
'use strict';
module.exports = function mapError (options) {
const { addContext = false, label = 'MAPS CONTROLLER' } = options;
return function mapErrorMiddleware (err, req, res, next) {
req.profiler.done('error');
const { mapConfig } = res.locals;
if (addContext) {
err = Number.isFinite(err.layerIndex) ? populateError(err, mapConfig) : err;
}
err.label = label;
next(err);
};
};
function populateError(err, mapConfig) {
var error = new Error(err.message);
error.http_status = err.http_status;
if (!err.http_status && err.message.indexOf('column "the_geom_webmercator" does not exist') >= 0) {
error.http_status = 400;
}
error.type = 'layer';
error.subtype = err.message.indexOf('Postgis Plugin') >= 0 ? 'query' : undefined;
error.layer = {
id: mapConfig.getLayerId(err.layerIndex),
index: err.layerIndex,
type: mapConfig.layerType(err.layerIndex)
};
return error;
}

40
lib/cartodb/api/middlewares/map-store-map-config-provider.js Normal file
View File

@ -0,0 +1,40 @@
'use strict';
const MapStoreMapConfigProvider = require('../../models/mapconfig/provider/map-store-provider');
module.exports = function createMapStoreMapConfigProvider (
mapStore,
userLimitsBackend,
pgConnection,
affectedTablesCache,
forcedFormat = null
) {
return function createMapStoreMapConfigProviderMiddleware (req, res, next) {
const { user, token, cache_buster, api_key } = res.locals;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const { layer: layerFromParams, z, x, y, scale_factor, format } = req.params;
const { layer: layerFromQuery } = req.query;
const params = {
user, token, cache_buster, api_key,
dbuser, dbname, dbpassword, dbhost, dbport,
layer: (layerFromQuery || layerFromParams), z, x, y, scale_factor, format
};
if (forcedFormat) {
params.format = forcedFormat;
params.layer = params.layer || 'all';
}
res.locals.mapConfigProvider = new MapStoreMapConfigProvider(
mapStore,
user,
userLimitsBackend,
pgConnection,
affectedTablesCache,
params
);
next();
};
};

34
lib/cartodb/api/middlewares/named-map-provider.js Normal file
View File

@ -0,0 +1,34 @@
'use strict';
module.exports = function getNamedMapProvider ({ namedMapProviderCache, label, forcedFormat = null }) {
return function getNamedMapProviderMiddleware (req, res, next) {
const { user, token, cache_buster, api_key } = res.locals;
const { dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const { template_id, layer: layerFromParams, z, x, y, format } = req.params;
const { layer: layerFromQuery } = req.query;
const params = {
user, token, cache_buster, api_key,
dbuser, dbname, dbpassword, dbhost, dbport,
template_id, layer: (layerFromQuery || layerFromParams), z, x, y, format
};
if (forcedFormat) {
params.format = forcedFormat;
params.layer = params.layer || 'all';
}
const { config, auth_token } = req.query;
namedMapProviderCache.get(user, template_id, config, auth_token, params, (err, namedMapProvider) => {
if (err) {
err.label = label;
return next(err);
}
res.locals.mapConfigProvider = namedMapProvider;
next();
});
};
};

7
lib/cartodb/api/middlewares/noop.js Normal file
View File

@ -0,0 +1,7 @@
'use strict';
module.exports = function noop () {
return function noopMiddleware (req, res, next) {
next();
};
};

72
lib/cartodb/api/middlewares/rate-limit.js Normal file
View File

@ -0,0 +1,72 @@
'use strict';
const RATE_LIMIT_ENDPOINTS_GROUPS = {
ANONYMOUS: 'anonymous',
STATIC: 'static',
STATIC_NAMED: 'static_named',
DATAVIEW: 'dataview',
DATAVIEW_SEARCH: 'dataview_search',
ANALYSIS: 'analysis',
ANALYSIS_CATALOG: 'analysis_catalog',
TILE: 'tile',
ATTRIBUTES: 'attributes',
NAMED_LIST: 'named_list',
NAMED_CREATE: 'named_create',
NAMED_GET: 'named_get',
NAMED: 'named',
NAMED_UPDATE: 'named_update',
NAMED_DELETE: 'named_delete',
NAMED_TILES: 'named_tiles'
};
function rateLimit(userLimitsBackend, endpointGroup = null) {
if (!isRateLimitEnabled(endpointGroup)) {
return function rateLimitDisabledMiddleware(req, res, next) { next(); };
}
return function rateLimitMiddleware(req, res, next) {
userLimitsBackend.getRateLimit(res.locals.user, endpointGroup, function (err, userRateLimit) {
if (err) {
return next(err);
}
if (!userRateLimit) {
return next();
}
const [isBlocked, limit, remaining, retry, reset] = userRateLimit;
res.set({
'Carto-Rate-Limit-Limit': limit,
'Carto-Rate-Limit-Remaining': remaining,
'Carto-Rate-Limit-Reset': reset
});
if (isBlocked) {
// retry is floor rounded in seconds by redis-cell
res.set('Retry-After', retry + 1);
let rateLimitError = new Error(
'You are over platform\'s limits: too many requests.' +
' Please contact us to know more details'
);
rateLimitError.http_status = 429;
rateLimitError.type = 'limit';
rateLimitError.subtype = 'rate-limit';
return next(rateLimitError);
}
return next();
});
};
}
function isRateLimitEnabled(endpointGroup) {
return global.environment.enabledFeatures.rateLimitsEnabled &&
endpointGroup &&
global.environment.enabledFeatures.rateLimitsByEndpoint[endpointGroup];
}
module.exports = rateLimit;
module.exports.RATE_LIMIT_ENDPOINTS_GROUPS = RATE_LIMIT_ENDPOINTS_GROUPS;

19
lib/cartodb/api/middlewares/send-response.js Normal file
View File

@ -0,0 +1,19 @@
'use strict';
module.exports = function sendResponse () {
return function sendResponseMiddleware (req, res) {
req.profiler.done('res');
res.status(res.statusCode);
if (Buffer.isBuffer(res.body)) {
return res.send(res.body);
}
if (req.query.callback) {
return res.jsonp(res.body);
}
res.json(res.body);
};
};

13
lib/cartodb/api/middlewares/served-by-host-header.js Normal file
View File

@ -0,0 +1,13 @@
'use strict';
const os = require('os');
module.exports = function servedByHostHeader () {
const hostname = os.hostname().split('.')[0];
return function servedByHostHeaderMiddleware (req, res, next) {
res.set('X-Served-By-Host', hostname);
next();
};
};

8
lib/cartodb/middleware/stats.js → lib/cartodb/api/middlewares/stats.js
View File

@ -1,11 +1,13 @@
const Profiler = require('../stats/profiler_proxy');
'use strict';
const Profiler = require('../../stats/profiler_proxy');
const debug = require('debug')('windshaft:cartodb:stats');
const onHeaders = require('on-headers');
module.exports = function statsMiddleware(options) {
module.exports = function stats (options) {
const { enabled = true, statsClient } = options;
return function stats(req, res, next) {
return function statsMiddleware (req, res, next) {
req.profiler = new Profiler({
statsd_client: statsClient,
profile: enabled

33
lib/cartodb/api/middlewares/surrogate-key-header.js Normal file
View File

@ -0,0 +1,33 @@
'use strict';
const NamedMapsCacheEntry = require('../../cache/model/named_maps_entry');
const NamedMapMapConfigProvider = require('../../models/mapconfig/provider/named-map-provider');
module.exports = function setSurrogateKeyHeader ({ surrogateKeysCache }) {
return function setSurrogateKeyHeaderMiddleware(req, res, next) {
const { user, mapConfigProvider } = res.locals;
if (mapConfigProvider instanceof NamedMapMapConfigProvider) {
surrogateKeysCache.tag(res, new NamedMapsCacheEntry(user, mapConfigProvider.getTemplateName()));
}
if (req.method !== 'GET') {
return next();
}
mapConfigProvider.getAffectedTables((err, affectedTables) => {
if (err) {
global.logger.warn('ERROR generating Surrogate Key Header:', err);
return next();
}
if (!affectedTables || !affectedTables.tables || affectedTables.tables.length === 0) {
return next();
}
surrogateKeysCache.tag(res, affectedTables);
next();
});
};
};

12
lib/cartodb/api/middlewares/syntax-error.js Normal file
View File

@ -0,0 +1,12 @@
'use strict';
module.exports = function syntaxError () {
return function syntaxErrorMiddleware (err, req, res, next) {
if (err.name === 'SyntaxError') {
err.http_status = 400;
err.message = `${err.name}: ${err.message}`;
}
next(err);
};
};

13
lib/cartodb/api/middlewares/user.js Normal file
View File

@ -0,0 +1,13 @@
'use strict';
const CdbRequest = require('../../models/cdb_request');
module.exports = function user () {
const cdbRequest = new CdbRequest();
return function userMiddleware(req, res, next) {
res.locals.user = cdbRequest.userByReq(req);
next();
};
};

11
lib/cartodb/middleware/vector-error.js → lib/cartodb/api/middlewares/vector-error.js
View File

@ -1,12 +1,13 @@
const fs = require('fs');
'use strict';
const timeoutErrorVectorTile = fs.readFileSync(__dirname + '/../../../assets/render-timeout-fallback.mvt');
const fs = require('fs');
const timeoutErrorVectorTile = fs.readFileSync(__dirname + '/../../../../assets/render-timeout-fallback.mvt');
module.exports = function vectorError() {
return function vectorErrorMiddleware(err, req, res, next) {
if(req.params.format === 'mvt') {
if (isTimeoutError(err)) {
if (isTimeoutError(err) || isRateLimitError(err)) {
res.set('Content-Type', 'application/x-protobuf');
return res.status(429).send(timeoutErrorVectorTile);
}
@ -28,3 +29,7 @@ function isDatasourceTimeoutError (err) {
function isTimeoutError (err) {
return isRenderTimeoutError(err) || isDatasourceTimeoutError(err);
}
function isRateLimitError (err) {
return err.type === 'limit' && err.subtype === 'rate-limit';
}

233
lib/cartodb/api/template/admin-template-controller.js Normal file
View File

@ -0,0 +1,233 @@
'use strict';
const { templateName } = require('../../backends/template_maps');
const credentials = require('../middlewares/credentials');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
module.exports = class AdminTemplateController {
/**
* @param {AuthBackend} authBackend
* @param {PgConnection} pgConnection
* @param {TemplateMaps} templateMaps
* @constructor
*/
constructor (authBackend, templateMaps, userLimitsBackend) {
this.authBackend = authBackend;
this.templateMaps = templateMaps;
this.userLimitsBackend = userLimitsBackend;
}
register (templateRouter) {
templateRouter.options(`/:template_id`);
templateRouter.post('/', this.middlewares({
action: 'create',
label: 'POST TEMPLATE',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.NAMED_CREATE
}));
templateRouter.put('/:template_id', this.middlewares({
action: 'update',
label: 'PUT TEMPLATE',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.NAMED_UPDATE
}));
templateRouter.get('/:template_id', this.middlewares({
action: 'get',
label: 'GET TEMPLATE',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.NAMED_GET
}));
templateRouter.delete('/:template_id', this.middlewares({
action: 'delete',
label: 'DELETE TEMPLATE',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.NAMED_DELETE
}));
templateRouter.get('/', this.middlewares({
action: 'list',
label: 'GET TEMPLATE LIST',
rateLimitGroup: RATE_LIMIT_ENDPOINTS_GROUPS.NAMED_LIST
}));
}
middlewares ({ action, label, rateLimitGroup }) {
let template;
if (action === 'create') {
template = createTemplate;
}
if (action === 'update') {
template = updateTemplate;
}
if (action === 'get') {
template = retrieveTemplate;
}
if (action === 'delete') {
template = destroyTemplate;
}
if (action === 'list') {
template = listTemplates;
}
return [
credentials(),
authorizedByAPIKey({ authBackend: this.authBackend, action, label }),
rateLimit(this.userLimitsBackend, rateLimitGroup),
checkContentType({ action: 'POST', label: 'POST TEMPLATE' }),
template({ templateMaps: this.templateMaps })
];
}
};
function checkContentType ({ label }) {
return function checkContentTypeMiddleware (req, res, next) {
if ((req.method === 'POST' || req.method === 'PUT') && !req.is('application/json')) {
const error = new Error(`${req.method} template data must be of type application/json`);
error.label = label;
return next(error);
}
next();
};
}
function authorizedByAPIKey ({ authBackend, action, label }) {
return function authorizedByAPIKeyMiddleware (req, res, next) {
const { user } = res.locals;
authBackend.authorizedByAPIKey(user, res, (err, authenticated, apikey) => {
if (err) {
return next(err);
}
if (!authenticated) {
const error = new Error(`Only authenticated users can ${action} templated maps`);
error.http_status = 403;
error.label = label;
return next(error);
}
if (apikey.type !== 'master') {
const error = new Error('Forbidden');
error.type = 'auth';
error.subtype = 'api-key-does-not-grant-access';
error.http_status = 403;
return next(error);
}
next();
});
};
}
function createTemplate ({ templateMaps }) {
return function createTemplateMiddleware (req, res, next) {
const { user } = res.locals;
const template = req.body;
templateMaps.addTemplate(user, template, (err, templateId) => {
if (err) {
return next(err);
}
res.statusCode = 200;
res.body = { template_id: templateId };
next();
});
};
}
function updateTemplate ({ templateMaps }) {
return function updateTemplateMiddleware (req, res, next) {
const { user } = res.locals;
const template = req.body;
const templateId = templateName(req.params.template_id);
templateMaps.updTemplate(user, templateId, template, (err) => {
if (err) {
return next(err);
}
res.statusCode = 200;
res.body = { template_id: templateId };
next();
});
};
}
function retrieveTemplate ({ templateMaps }) {
return function retrieveTemplateMiddleware (req, res, next) {
req.profiler.start('windshaft-cartodb.get_template');
const { user } = res.locals;
const templateId = templateName(req.params.template_id);
templateMaps.getTemplate(user, templateId, (err, template) => {
if (err) {
return next(err);
}
if (!template) {
const error = new Error(`Cannot find template '${templateId}' of user '${user}'`);
error.http_status = 404;
return next(error);
}
// auth_id was added by ourselves,
// so we remove it before returning to the user
delete template.auth_id;
res.statusCode = 200;
res.body = { template };
next();
});
};
}
function destroyTemplate ({ templateMaps }) {
return function destroyTemplateMiddleware (req, res, next) {
req.profiler.start('windshaft-cartodb.delete_template');
const { user } = res.locals;
const templateId = templateName(req.params.template_id);
templateMaps.delTemplate(user, templateId, (err/* , tpl_val */) => {
if (err) {
return next(err);
}
res.statusCode = 204;
res.body = '';
next();
});
};
}
function listTemplates ({ templateMaps }) {
return function listTemplatesMiddleware (req, res, next) {
req.profiler.start('windshaft-cartodb.get_template_list');
const { user } = res.locals;
templateMaps.listTemplates(user, (err, templateIds) => {
if (err) {
return next(err);
}
res.statusCode = 200;
res.body = { template_ids: templateIds };
next();
});
};
}

220
lib/cartodb/api/template/named-template-controller.js Normal file
View File

@ -0,0 +1,220 @@
'use strict';
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const initProfiler = require('../middlewares/init-profiler');
const checkJsonContentType = require('../middlewares/check-json-content-type');
const incrementMapViewCount = require('../middlewares/increment-map-view-count');
const augmentLayergroupData = require('../middlewares/augment-layergroup-data');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
const lastUpdatedTimeLayergroup = require('../middlewares/last-updated-time-layergroup');
const layerStats = require('../middlewares/layer-stats');
const layergroupIdHeader = require('../middlewares/layergroup-id-header');
const layergroupMetadata = require('../middlewares/layergroup-metadata');
const mapError = require('../middlewares/map-error');
const NamedMapMapConfigProvider = require('../../models/mapconfig/provider/named-map-provider');
const CreateLayergroupMapConfigProvider = require('../../models/mapconfig/provider/create-layergroup-provider');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
module.exports = class NamedMapController {
/**
* @param {PgConnection} pgConnection
* @param {TemplateMaps} templateMaps
* @param {MapBackend} mapBackend
* @param metadataBackend
* @param {SurrogateKeysCache} surrogateKeysCache
* @param {UserLimitsBackend} userLimitsBackend
* @param {LayergroupAffectedTables} layergroupAffectedTables
* @param {MapConfigAdapter} mapConfigAdapter
* @param {StatsBackend} statsBackend
* @param {AuthBackend} authBackend
* @param layergroupMetadata
* @constructor
*/
constructor (
pgConnection,
templateMaps,
mapBackend,
metadataBackend,
surrogateKeysCache,
userLimitsBackend,
layergroupAffectedTables,
mapConfigAdapter,
statsBackend,
authBackend,
layergroupMetadata
) {
this.pgConnection = pgConnection;
this.templateMaps = templateMaps;
this.mapBackend = mapBackend;
this.metadataBackend = metadataBackend;
this.surrogateKeysCache = surrogateKeysCache;
this.userLimitsBackend = userLimitsBackend;
this.layergroupAffectedTables = layergroupAffectedTables;
this.mapConfigAdapter = mapConfigAdapter;
this.statsBackend = statsBackend;
this.authBackend = authBackend;
this.layergroupMetadata = layergroupMetadata;
}
register (templateRouter) {
templateRouter.get('/:template_id/jsonp', this.middlewares());
templateRouter.post('/:template_id', this.middlewares());
}
middlewares () {
const isTemplateInstantiation = true;
const useTemplateHash = true;
const includeQuery = false;
const label = 'NAMED MAP LAYERGROUP';
const addContext = false;
return [
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.NAMED),
cleanUpQueryParams(['aggregation']),
initProfiler(isTemplateInstantiation),
checkJsonContentType(),
checkInstantiteLayergroup(),
getTemplate(
this.templateMaps,
this.pgConnection,
this.metadataBackend,
this.userLimitsBackend,
this.mapConfigAdapter,
this.layergroupAffectedTables
),
instantiateLayergroup(
this.mapBackend,
this.userLimitsBackend,
this.pgConnection,
this.layergroupAffectedTables
),
incrementMapViewCount(this.metadataBackend),
augmentLayergroupData(),
cacheControlHeader({ ttl: global.environment.varnish.layergroupTtl || 86400, revalidate: true }),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader(),
lastUpdatedTimeLayergroup(),
layerStats(this.pgConnection, this.statsBackend),
layergroupIdHeader(this.templateMaps ,useTemplateHash),
layergroupMetadata(this.layergroupMetadata, includeQuery),
mapError({ label, addContext })
];
}
};
function checkInstantiteLayergroup () {
return function checkInstantiteLayergroupMiddleware(req, res, next) {
if (req.method === 'GET') {
const { callback, config } = req.query;
if (callback === undefined || callback.length === 0) {
return next(new Error('callback parameter should be present and be a function name'));
}
if (config) {
try {
req.body = JSON.parse(config);
} catch(e) {
return next(new Error('Invalid config parameter, should be a valid JSON'));
}
}
}
req.profiler.done('checkInstantiteLayergroup');
return next();
};
}
function getTemplate (
templateMaps,
pgConnection,
metadataBackend,
userLimitsBackend,
mapConfigAdapter,
affectedTablesCache
) {
return function getTemplateMiddleware (req, res, next) {
const templateParams = req.body;
const { user, dbuser, dbname, dbpassword, dbhost, dbport } = res.locals;
const { template_id } = req.params;
const { auth_token } = req.query;
const params = Object.assign({ dbuser, dbname, dbpassword, dbhost, dbport }, req.query);
const mapConfigProvider = new NamedMapMapConfigProvider(
templateMaps,
pgConnection,
metadataBackend,
userLimitsBackend,
mapConfigAdapter,
affectedTablesCache,
user,
template_id,
templateParams,
auth_token,
params
);
mapConfigProvider.getMapConfig((err, mapConfig, rendererParams, context, stats = {}) => {
req.profiler.done('named.getMapConfig');
stats.mapType = 'named';
req.profiler.add(stats);
if (err) {
return next(err);
}
res.locals.mapConfig = mapConfig;
res.locals.rendererParams = rendererParams;
res.locals.mapConfigProvider = mapConfigProvider;
next();
});
};
}
function instantiateLayergroup (mapBackend, userLimitsBackend, pgConnection, affectedTablesCache) {
return function instantiateLayergroupMiddleware (req, res, next) {
const { user, mapConfig, rendererParams } = res.locals;
const mapConfigProvider = new CreateLayergroupMapConfigProvider(
mapConfig,
user,
userLimitsBackend,
pgConnection,
affectedTablesCache,
rendererParams
);
mapBackend.createLayergroup(mapConfig, rendererParams, mapConfigProvider, (err, layergroup, stats = {}) => {
req.profiler.add(stats);
if (err) {
return next(err);
}
res.statusCode = 200;
res.body = layergroup;
const { mapConfigProvider } = res.locals;
res.locals.analysesResults = mapConfigProvider.analysesResults;
res.locals.template = mapConfigProvider.template;
res.locals.context = mapConfigProvider.context;
next();
});
};
}

66
lib/cartodb/api/template/template-router.js Normal file
View File

@ -0,0 +1,66 @@
'use strict';
const { Router: router } = require('express');
const NamedMapController = require('./named-template-controller');
const AdminTemplateController = require('./admin-template-controller');
const TileTemplateController = require('./tile-template-controller');
module.exports = class TemplateRouter {
constructor ({ collaborators }) {
const {
pgConnection,
templateMaps,
mapBackend,
metadataBackend,
surrogateKeysCache,
userLimitsBackend,
layergroupAffectedTablesCache,
mapConfigAdapter,
statsBackend,
authBackend,
layergroupMetadata,
namedMapProviderCache,
tileBackend,
} = collaborators;
this.namedMapController = new NamedMapController(
pgConnection,
templateMaps,
mapBackend,
metadataBackend,
surrogateKeysCache,
userLimitsBackend,
layergroupAffectedTablesCache,
mapConfigAdapter,
statsBackend,
authBackend,
layergroupMetadata
);
this.tileTemplateController = new TileTemplateController(
namedMapProviderCache,
tileBackend,
surrogateKeysCache,
pgConnection,
authBackend,
userLimitsBackend
);
this.adminTemplateController = new AdminTemplateController(
authBackend,
templateMaps,
userLimitsBackend
);
}
register (apiRouter, templatePaths) {
const templateRouter = router({ mergeParams: true });
this.namedMapController.register(templateRouter);
this.tileTemplateController.register(templateRouter);
this.adminTemplateController.register(templateRouter);
templatePaths.forEach(path => apiRouter.use(path, templateRouter));
}
};

97
lib/cartodb/api/template/tile-template-controller.js Normal file
View File

@ -0,0 +1,97 @@
'use strict';
const coordinates = require('../middlewares/coordinates');
const cleanUpQueryParams = require('../middlewares/clean-up-query-params');
const credentials = require('../middlewares/credentials');
const dbConnSetup = require('../middlewares/db-conn-setup');
const authorize = require('../middlewares/authorize');
const namedMapProvider = require('../middlewares/named-map-provider');
const cacheControlHeader = require('../middlewares/cache-control-header');
const cacheChannelHeader = require('../middlewares/cache-channel-header');
const surrogateKeyHeader = require('../middlewares/surrogate-key-header');
const lastModifiedHeader = require('../middlewares/last-modified-header');
const vectorError = require('../middlewares/vector-error');
const rateLimit = require('../middlewares/rate-limit');
const { RATE_LIMIT_ENDPOINTS_GROUPS } = rateLimit;
module.exports = class TileTemplateController {
constructor (
namedMapProviderCache,
tileBackend,
surrogateKeysCache,
pgConnection,
authBackend,
userLimitsBackend
) {
this.namedMapProviderCache = namedMapProviderCache;
this.tileBackend = tileBackend;
this.surrogateKeysCache = surrogateKeysCache;
this.pgConnection = pgConnection;
this.authBackend = authBackend;
this.userLimitsBackend = userLimitsBackend;
}
register (templateRouter) {
templateRouter.get('/:template_id/:layer/:z/:x/:y.(:format)', this.middlewares());
}
middlewares () {
return [
coordinates(),
credentials(),
authorize(this.authBackend),
dbConnSetup(this.pgConnection),
rateLimit(this.userLimitsBackend, RATE_LIMIT_ENDPOINTS_GROUPS.NAMED_TILES),
cleanUpQueryParams(),
namedMapProvider({
namedMapProviderCache: this.namedMapProviderCache,
label: 'NAMED_MAP_TILE'
}),
getTile({
tileBackend: this.tileBackend,
label: 'NAMED_MAP_TILE'
}),
setContentTypeHeader(),
cacheControlHeader(),
cacheChannelHeader(),
surrogateKeyHeader({ surrogateKeysCache: this.surrogateKeysCache }),
lastModifiedHeader(),
vectorError()
];
}
};
function getTile ({ tileBackend, label }) {
return function getTileMiddleware (req, res, next) {
const { mapConfigProvider } = res.locals;
const { layer, z, x, y, format } = req.params;
const params = { layer, z, x, y, format };
tileBackend.getTile(mapConfigProvider, params, (err, tile, headers, stats) => {
req.profiler.add(stats);
req.profiler.done('render-' + format);
if (err) {
err.label = label;
return next(err);
}
if (headers) {
res.set(headers);
}
res.statusCode = 200;
res.body = tile;
next();
});
};
}
function setContentTypeHeader () {
return function setContentTypeHeaderMiddleware(req, res, next) {
res.set('Content-Type', res.get('content-type') || res.get('Content-Type') || 'image/png');
next();
};
}

79
lib/cartodb/api/user_limits_api.js
View File

@ -1,79 +0,0 @@
var step = require('step');
/**
*
* @param metadataBackend
* @param options
* @constructor
* @type {UserLimitsApi}
*/
function UserLimitsApi(metadataBackend, options) {
this.metadataBackend = metadataBackend;
this.options = options || {};
this.options.limits = this.options.limits || {};
}
module.exports = UserLimitsApi;
UserLimitsApi.prototype.getRenderLimits = function (username, apiKey, callback) {
var self = this;
var limits = {
cacheOnTimeout: self.options.limits.cacheOnTimeout || false,
render: self.options.limits.render || 0
};
self.getTimeoutRenderLimit(username, apiKey, function (err, timeoutRenderLimit) {
if (err) {
return callback(err);
}
if (timeoutRenderLimit && timeoutRenderLimit.render) {
if (Number.isFinite(timeoutRenderLimit.render)) {
limits.render = timeoutRenderLimit.render;
}
}
return callback(null, limits);
});
};
UserLimitsApi.prototype.getTimeoutRenderLimit = function (username, apiKey, callback) {
var self = this;
step(
function isAuthorized() {
var next = this;
if (!apiKey) {
return next(null, false);
}
self.metadataBackend.getUserMapKey(username, function (err, userApiKey) {
if (err) {
return next(err);
}
return next(null, userApiKey === apiKey);
});
},
function getUserTimeoutRenderLimits(err, authorized) {
var next = this;
if (err) {
return next(err);
}
self.metadataBackend.getUserTimeoutRenderLimits(username, function (err, timeoutRenderLimit) {
if (err) {
return next(err);
}
next(null, {
render: authorized ? timeoutRenderLimit.render : timeoutRenderLimit.renderPublic
});
});
},
callback
);
};

30
lib/cartodb/backends/analysis-status.js
View File

@ -1,3 +1,5 @@
'use strict';
var PSQL = require('cartodb-psql');
function AnalysisStatusBackend() {
@ -5,16 +7,14 @@ function AnalysisStatusBackend() {
module.exports = AnalysisStatusBackend;
AnalysisStatusBackend.prototype.getNodeStatus = function (params, callback) {
var nodeId = params.nodeId;
AnalysisStatusBackend.prototype.getNodeStatus = function (nodeId, dbParams, callback) {
var statusQuery = [
'SELECT node_id, status, updated_at, last_error_message as error_message',
'FROM cdb_analysis_catalog where node_id = \'' + nodeId + '\''
].join(' ');
var pg = new PSQL(dbParamsFromReqParams(params));
var pg = new PSQL(dbParams);
pg.query(statusQuery, function(err, result) {
if (err) {
return callback(err, result);
@ -36,23 +36,3 @@ AnalysisStatusBackend.prototype.getNodeStatus = function (params, callback) {
return callback(null, statusResponse);
}, true); // use read-only transaction
};
function dbParamsFromReqParams(params) {
var dbParams = {};
if ( params.dbuser ) {
dbParams.user = params.dbuser;
}
if ( params.dbpassword ) {
dbParams.pass = params.dbpassword;
}
if ( params.dbhost ) {
dbParams.host = params.dbhost;
}
if ( params.dbport ) {
dbParams.port = params.dbport;
}
if ( params.dbname ) {
dbParams.dbname = params.dbname;
}
return dbParams;
}

180
lib/cartodb/backends/auth.js Normal file
View File

@ -0,0 +1,180 @@
'use strict';
/**
*
* @param {PgConnection} pgConnection
* @param metadataBackend
* @param {MapStore} mapStore
* @param {TemplateMaps} templateMaps
* @constructor
* @type {AuthBackend}
*/
function AuthBackend(pgConnection, metadataBackend, mapStore, templateMaps) {
this.pgConnection = pgConnection;
this.metadataBackend = metadataBackend;
this.mapStore = mapStore;
this.templateMaps = templateMaps;
}
module.exports = AuthBackend;
// Check if the user is authorized by a signer
//
// @param res express response object
// @param callback function(err, signed_by) signed_by will be
// null if the request is not signed by anyone
// or will be a string cartodb username otherwise.
//
AuthBackend.prototype.authorizedBySigner = function(req, res, callback) {
if ( ! res.locals.token || ! res.locals.signer ) {
return callback(null, false); // no signer requested
}
var self = this;
var layergroup_id = res.locals.token;
var auth_token = req.query.auth_token;
this.mapStore.load(layergroup_id, function(err, mapConfig) {
if (err) {
return callback(err);
}
var authorized = self.templateMaps.isAuthorized(mapConfig.obj().template, auth_token);
return callback(null, authorized);
});
};
function isValidApiKey(apikey) {
return apikey.type &&
apikey.user &&
apikey.databasePassword &&
apikey.databaseRole;
}
// Check if a request is authorized by api_key
//
// @param user
// @param res express response object
// @param callback function(err, authorized)
// NOTE: authorized is expected to be 0 or 1 (integer)
//
AuthBackend.prototype.authorizedByAPIKey = function(user, res, callback) {
const apikeyToken = res.locals.api_key;
const basicAuthUsername = res.locals.basicAuthUsername;
if ( ! apikeyToken ) {
return callback(null, false); // no api key, no authorization...
}
this.metadataBackend.getApikey(user, apikeyToken, (err, apikey) => {
if (err) {
if (isNameNotFoundError(err)) {
err.http_status = 404;
}
return callback(err);
}
if ( !isValidApiKey(apikey)) {
const error = new Error('Unauthorized');
error.type = 'auth';
error.subtype = 'api-key-not-found';
error.http_status = 401;
return callback(error);
}
if (!usernameMatches(basicAuthUsername, res.locals.user)) {
const error = new Error('Forbidden');
error.type = 'auth';
error.subtype = 'api-key-username-mismatch';
error.http_status = 403;
return callback(error);
}
if (!apikey.grantsMaps) {
const error = new Error('Forbidden');
error.type = 'auth';
error.subtype = 'api-key-does-not-grant-access';
error.http_status = 403;
return callback(error);
}
return callback(null, true, apikey);
});
};
function isNameNotFoundError (err) {
return err.message && -1 !== err.message.indexOf('name not found');
}
function usernameMatches (basicAuthUsername, requestUsername) {
return !(basicAuthUsername && (basicAuthUsername !== requestUsername));
}
/**
* Check access authorization
*
* @param req - standard req object. Importantly contains table and host information
* @param res - standard res object. Contains the auth parameters in locals
* @param callback function(err, allowed) is access allowed not?
*/
AuthBackend.prototype.authorize = function(req, res, callback) {
var user = res.locals.user;
this.authorizedByAPIKey(user, res, (err, isAuthorizedByApikey) => {
if (err) {
return callback(err);
}
if (isAuthorizedByApikey) {
return this.pgConnection.setDBAuth(user, res.locals, 'regular', function (err) {
req.profiler.done('setDBAuth');
if (err) {
return callback(err);
}
callback(null, true);
});
}
this.authorizedBySigner(req, res, (err, isAuthorizedBySigner) => {
if (err) {
return callback(err);
}
if (isAuthorizedBySigner) {
return this.pgConnection.setDBAuth(user, res.locals, 'master', function (err) {
req.profiler.done('setDBAuth');
if (err) {
return callback(err);
}
callback(null, true);
});
}
// if no signer name was given, use default api key
if (!res.locals.signer) {
return this.pgConnection.setDBAuth(user, res.locals, 'default', function (err) {
req.profiler.done('setDBAuth');
if (err) {
return callback(err);
}
callback(null, true);
});
}
// if signer name was given, return no authorization
return callback(null, false);
});
});
};

204
lib/cartodb/backends/dataview.js
View File

@ -1,13 +1,11 @@
var assert = require('assert');
'use strict';
var _ = require('underscore');
var PSQL = require('cartodb-psql');
var step = require('step');
var BBoxFilter = require('../models/filter/bbox');
var DataviewFactory = require('../models/dataview/factory');
var DataviewFactoryWithOverviews = require('../models/dataview/overviews/factory');
const dbParamsFromReqParams = require('../utils/database-params');
var OverviewsQueryRewriter = require('../utils/overviews_query_rewriter');
var overviewsQueryRewriter = new OverviewsQueryRewriter({
zoom_level: 'CDB_ZoomFromScale(!scale_denominator!)'
@ -23,53 +21,76 @@ function DataviewBackend(analysisBackend) {
module.exports = DataviewBackend;
DataviewBackend.prototype.getDataview = function (mapConfigProvider, user, params, callback) {
const dataviewName = params.dataviewName;
var dataviewName = params.dataviewName;
step(
function getMapConfig() {
mapConfigProvider.getMapConfig(this);
},
function runDataviewQuery(err, mapConfig) {
assert.ifError(err);
mapConfigProvider.getMapConfig(function (err, mapConfig) {
if (err) {
return callback(err);
}
var dataviewDefinition = getDataviewDefinition(mapConfig.obj(), dataviewName);
if (!dataviewDefinition) {
throw new Error("Dataview '" + dataviewName + "' does not exists");
}
var dataviewDefinition = getDataviewDefinition(mapConfig.obj(), dataviewName);
if (!dataviewDefinition) {
const error = new Error(`Dataview '${dataviewName}' does not exist`);
error.type = 'dataview';
error.http_status = 400;
return callback(error);
}
if (!validFilterParams(params)) {
const error = new Error('Both own_filter and no_filters cannot be sent in the same request');
error.type = 'dataview';
error.http_status = 400;
return callback(error);
}
var pg;
var overrideParams;
var dataview;
try {
pg = new PSQL(dbParamsFromReqParams(params));
var query = getQueryWithFilters(dataviewDefinition, params);
var queryRewriteData = getQueryRewriteData(mapConfig, dataviewDefinition, params);
var dataviewFactory = DataviewFactoryWithOverviews.getFactory(overviewsQueryRewriter, queryRewriteData, {
bbox: params.bbox
});
dataview = dataviewFactory.getDataview(query, dataviewDefinition);
var ownFilter = +params.own_filter;
var noFilters = +params.no_filters;
if (Number.isFinite(ownFilter) && Number.isFinite(noFilters)) {
err = new Error();
err.message = 'Both own_filter and no_filters cannot be sent in the same request';
err.type = 'dataview';
err.http_status = 400;
overrideParams = getOverrideParams(params, !!ownFilter);
} catch (error) {
return callback(error);
}
dataview.getResult(pg, overrideParams, function (err, dataviewResult, stats = {}) {
if (err) {
return callback(err);
}
var pg = new PSQL(dbParamsFromReqParams(params));
var query = getDataviewQuery(dataviewDefinition, ownFilter, noFilters);
if (params.bbox) {
var bboxFilter = new BBoxFilter({column: 'the_geom_webmercator', srid: 3857}, {bbox: params.bbox});
query = bboxFilter.sql(query);
}
var queryRewriteData = getQueryRewriteData(mapConfig, dataviewDefinition, params);
var dataviewFactory = DataviewFactoryWithOverviews.getFactory(
overviewsQueryRewriter, queryRewriteData, { bbox: params.bbox }
);
var dataview = dataviewFactory.getDataview(query, dataviewDefinition);
dataview.getResult(pg, getOverrideParams(params, !!ownFilter), this);
},
function returnCallback(err, result) {
return callback(err, result);
}
);
return callback(null, dataviewResult, stats);
});
});
};
function validFilterParams (params) {
var ownFilter = +params.own_filter;
var noFilters = +params.no_filters;
return !(Number.isFinite(ownFilter) && Number.isFinite(noFilters));
}
function getQueryWithFilters (dataviewDefinition, params) {
var ownFilter = +params.own_filter;
var noFilters = +params.no_filters;
var query = getDataviewQuery(dataviewDefinition, ownFilter, noFilters);
if (params.bbox) {
var bboxFilter = new BBoxFilter({column: 'the_geom_webmercator', srid: 3857}, {bbox: params.bbox});
query = bboxFilter.sql(query);
}
return query;
}
function getDataviewQuery(dataviewDefinition, ownFilter, noFilters) {
if (noFilters) {
return dataviewDefinition.sql.no_filters;
@ -131,62 +152,57 @@ function getOverrideParams(params, ownFilter) {
}
DataviewBackend.prototype.search = function (mapConfigProvider, user, dataviewName, params, callback) {
step(
function getMapConfig() {
mapConfigProvider.getMapConfig(this);
},
function runDataviewSearchQuery(err, mapConfig) {
assert.ifError(err);
var dataviewDefinition = getDataviewDefinition(mapConfig.obj(), dataviewName);
if (!dataviewDefinition) {
throw new Error("Dataview '" + dataviewName + "' does not exists");
}
var pg = new PSQL(dbParamsFromReqParams(params));
var ownFilter = +params.own_filter;
ownFilter = !!ownFilter;
var query = (ownFilter) ? dataviewDefinition.sql.own_filter_on : dataviewDefinition.sql.own_filter_off;
if (params.bbox) {
var bboxFilter = new BBoxFilter({column: 'the_geom', srid: 4326}, {bbox: params.bbox});
query = bboxFilter.sql(query);
}
var userQuery = params.q;
var dataview = DataviewFactory.getDataview(query, dataviewDefinition);
dataview.search(pg, userQuery, this);
},
function returnCallback(err, result) {
return callback(err, result);
mapConfigProvider.getMapConfig(function (err, mapConfig) {
if (err) {
return callback(err);
}
);
var dataviewDefinition = getDataviewDefinition(mapConfig.obj(), dataviewName);
if (!dataviewDefinition) {
const error = new Error(`Dataview '${dataviewName}' does not exist`);
error.type = 'dataview';
error.http_status = 400;
return callback(error);
}
var pg;
var query;
var dataview;
var userQuery = params.q;
try {
pg = new PSQL(dbParamsFromReqParams(params));
query = getQueryWithOwnFilters(dataviewDefinition, params);
dataview = DataviewFactory.getDataview(query, dataviewDefinition);
} catch (error) {
return callback(error);
}
dataview.search(pg, userQuery, function (err, result) {
if (err) {
return callback(err);
}
return callback(null, result);
});
});
};
function getQueryWithOwnFilters (dataviewDefinition, params) {
var ownFilter = +params.own_filter;
ownFilter = !!ownFilter;
var query = (ownFilter) ? dataviewDefinition.sql.own_filter_on : dataviewDefinition.sql.own_filter_off;
if (params.bbox) {
var bboxFilter = new BBoxFilter({ column: 'the_geom', srid: 4326 }, { bbox: params.bbox });
query = bboxFilter.sql(query);
}
return query;
}
function getDataviewDefinition(mapConfig, dataviewName) {
var dataviews = mapConfig.dataviews || {};
return dataviews[dataviewName];
}
function dbParamsFromReqParams(params) {
var dbParams = {};
if ( params.dbuser ) {
dbParams.user = params.dbuser;
}
if ( params.dbpassword ) {
dbParams.pass = params.dbpassword;
}
if ( params.dbhost ) {
dbParams.host = params.dbhost;
}
if ( params.dbport ) {
dbParams.port = params.dbport;
}
if ( params.dbname ) {
dbParams.dbname = params.dbname;
}
return dbParams;
}

53
lib/cartodb/backends/filter-stats.js Normal file
View File

@ -0,0 +1,53 @@
'use strict';
var _ = require('underscore');
var AnalysisFilter = require('../models/filter/analysis');
function FilterStatsBackends(pgQueryRunner) {
this.pgQueryRunner = pgQueryRunner;
}
module.exports = FilterStatsBackends;
function getEstimatedRows(pgQueryRunner, username, query, callback) {
pgQueryRunner.run(username, "EXPLAIN (FORMAT JSON)"+query, function(err, result_rows) {
if (err){
callback(err);
return;
}
var rows;
if ( result_rows[0] && result_rows[0]['QUERY PLAN'] &&
result_rows[0]['QUERY PLAN'][0] && result_rows[0]['QUERY PLAN'][0].Plan ) {
rows = result_rows[0]['QUERY PLAN'][0].Plan['Plan Rows'];
}
return callback(null, rows);
});
}
FilterStatsBackends.prototype.getFilterStats = function (username, unfiltered_query, filters, callback) {
var stats = {};
getEstimatedRows(this.pgQueryRunner, username, unfiltered_query, (err, rows) => {
if (err){
return callback(err);
}
stats.unfiltered_rows = rows;
if (!filters || _.isEmpty(filters)) {
return callback(null, stats);
}
var analysisFilter = new AnalysisFilter(filters);
var query = analysisFilter.sql(unfiltered_query);
getEstimatedRows(this.pgQueryRunner, username, query, (err, rows) => {
if (err){
return callback(err);
}
stats.filtered_rows = rows;
return callback(null, stats);
});
});
};

2
lib/cartodb/backends/layer-stats/empty-layer-stats.js
View File

@ -1,3 +1,5 @@
'use strict';
function EmptyLayerStats(types) {
this._types = types || {};
}

2
lib/cartodb/backends/layer-stats/factory.js
View File

@ -1,3 +1,5 @@
'use strict';
var LayerStats = require('./layer-stats');
var EmptyLayerStats = require('./empty-layer-stats');
var MapnikLayerStats = require('./mapnik-layer-stats');

2
lib/cartodb/backends/layer-stats/layer-stats.js
View File

@ -1,3 +1,5 @@
'use strict';
var queue = require('queue-async');
function LayerStats(layerStatsIterator) {

304
lib/cartodb/backends/layer-stats/mapnik-layer-stats.js
View File

@ -1,4 +1,8 @@
var queryUtils = require('../../utils/query-utils');
'use strict';
const queryUtils = require('../../utils/query-utils');
const AggregationMapConfig = require('../../models/aggregation/aggregation-mapconfig');
const aggregationQuery = require('../../models/aggregation/aggregation-query');
function MapnikLayerStats () {
this._types = {
@ -11,17 +15,297 @@ MapnikLayerStats.prototype.is = function (type) {
return this._types[type] ? this._types[type] : false;
};
function columnAggregations(field) {
if (field.type === 'number') {
return ['min', 'max', 'avg', 'sum'];
}
if (field.type === 'date') { // TODO other types too?
return ['min', 'max'];
}
if (field.type === 'timeDimension') {
return ['min', 'max'];
}
return [];
}
function _getSQL(ctx, query, type='pre', zoom=0) {
let sql;
if (type === 'pre') {
sql = ctx.preQuery;
}
else {
sql = ctx.aggrQuery;
}
sql = queryUtils.subsituteTokensForZoom(sql, zoom || 0);
return query(sql);
}
function _estimatedFeatureCount(ctx) {
return queryUtils.queryPromise(ctx.dbConnection, _getSQL(ctx, queryUtils.getQueryRowEstimation))
.then(res => ({ estimatedFeatureCount: res.rows[0].rows }))
.catch(() => ({ estimatedFeatureCount: -1 }));
}
function _featureCount(ctx) {
if (ctx.metaOptions.featureCount) {
// TODO: if ctx.metaOptions.columnStats we can combine this with column stats query
return queryUtils.queryPromise(ctx.dbConnection, _getSQL(ctx, queryUtils.getQueryActualRowCount))
.then(res => ({ featureCount: res.rows[0].rows }));
}
return Promise.resolve();
}
function _aggrFeatureCount(ctx) {
if (ctx.metaOptions.hasOwnProperty('aggrFeatureCount')) {
// We expect as zoom level as the value of aggrFeatureCount
// TODO: it'd be nice to admit an array of zoom levels to
// return metadata for multiple levels.
return queryUtils.queryPromise(
ctx.dbConnection,
_getSQL(ctx, queryUtils.getQueryActualRowCount, 'post', ctx.metaOptions.aggrFeatureCount)
).then(res => ({ aggrFeatureCount: res.rows[0].rows }));
}
return Promise.resolve();
}
function _geometryType(ctx) {
if (ctx.metaOptions.geometryType) {
const geometryColumn = AggregationMapConfig.getAggregationGeometryColumn();
const sqlQuery = _getSQL(ctx, sql => queryUtils.getQueryGeometryType(sql, geometryColumn));
return queryUtils.queryPromise(ctx.dbConnection, sqlQuery)
.then(res => ({ geometryType: (res.rows[0] || {}).geom_type }));
}
return Promise.resolve();
}
function _columns(ctx) {
if (ctx.metaOptions.columns || ctx.metaOptions.columnStats || ctx.metaOptions.dimensions) {
// note: post-aggregation columns are in layer.options.columns when aggregation is present
return queryUtils.queryPromise(ctx.dbConnection, _getSQL(ctx, sql => queryUtils.getQueryLimited(sql, 0)))
.then(res => formatResultFields(ctx.dbConnection, res.fields));
}
return Promise.resolve();
}
// combine a list of results merging the properties of all the objects
// undefined results are admitted and ignored
function mergeResults(results) {
if (results) {
if (results.length === 0) {
return {};
}
return results.reduce((a, b) => {
if (a === undefined) {
return b;
}
if (b === undefined) {
return a;
}
return Object.assign({}, a, b);
});
}
}
// deeper (1 level) combination of a list of objects:
// mergeColumns([{ col1: { a: 1 }, col2: { a: 2 } }, { col1: { b: 3 } }]) => { col1: { a: 1, b: 3 }, col2: { a: 2 } }
function mergeColumns(results) {
if (results) {
if (results.length === 0) {
return {};
}
return results.reduce((a, b) => {
let c = Object.assign({}, b || {}, a || {});
Object.keys(c).forEach(key => {
if (b.hasOwnProperty(key)) {
c[key] = Object.assign(c[key], b[key]);
}
});
return c;
});
}
}
const SAMPLE_SEED = 0.5;
const DEFAULT_SAMPLE_ROWS = 100;
function _sample(ctx, numRows) {
if (ctx.metaOptions.sample) {
const sampleProb = Math.min(ctx.metaOptions.sample.num_rows / numRows, 1);
// We'll use a safety limit just in case numRows is a bad estimate
const requestedRows = ctx.metaOptions.sample.num_rows || DEFAULT_SAMPLE_ROWS;
const limit = Math.ceil(requestedRows * 1.5);
let columns = ctx.metaOptions.sample.include_columns;
return queryUtils.queryPromise(ctx.dbConnection, _getSQL(
ctx,
sql => queryUtils.getQuerySample(sql, sampleProb, limit, SAMPLE_SEED, columns)
)).then(res => ({ sample: res.rows }));
}
return Promise.resolve();
}
function _columnsMetadataRequired(options) {
// We need determine the columns of a query
// if either column stats or dimension stats are required,
// since we'll ultimately use the same query to fetch both
return options.columnStats || options.dimensions;
}
function _columnStats(ctx, columns, dimensions) {
if (!columns) {
return Promise.resolve();
}
if (_columnsMetadataRequired(ctx.metaOptions)) {
let queries = [];
let aggr = [];
if (ctx.metaOptions.columnStats) {
queries.push(new Promise(resolve => resolve({ columns }))); // add columns as first result
Object.keys(columns).forEach(name => {
aggr = aggr.concat(
columnAggregations(columns[name])
.map(fn => `${fn}("${name}") AS "${name}_${fn}"`)
);
if (columns[name].type === 'string') {
const topN = ctx.metaOptions.columnStats.topCategories || 1024;
const includeNulls = ctx.metaOptions.columnStats.hasOwnProperty('includeNulls') ?
ctx.metaOptions.columnStats.includeNulls :
true;
// TODO: ctx.metaOptions.columnStats.maxCategories
// => use PG stats to dismiss columns with more distinct values
queries.push(
queryUtils.queryPromise(
ctx.dbConnection,
_getSQL(ctx, sql => queryUtils.getQueryTopCategories(sql, name, topN, includeNulls))
).then(res => ({ columns: { [name]: { categories: res.rows } } }))
);
}
});
}
const dimensionsStats = {};
let dimensionsInfo = {};
if (ctx.metaOptions.dimensions && dimensions) {
dimensionsInfo = aggregationQuery.infoForOptions({ dimensions });
Object.keys(dimensionsInfo).forEach(dimName => {
const info = dimensionsInfo[dimName];
if (info.type === 'timeDimension') {
dimensionsStats[dimName] = {
params: info.params
};
aggr = aggr.concat(
columnAggregations(info).map(fn => `${fn}(${info.sql}) AS "${dimName}_${fn}"`)
);
}
});
}
queries.push(
queryUtils.queryPromise(
ctx.dbConnection,
_getSQL(ctx, sql => `SELECT ${aggr.join(',')} FROM (${sql}) AS __cdb_query`)
).then(res => {
let stats = { columns: {}, dimensions: {} };
Object.keys(columns).forEach(name => {
stats.columns[name] = {};
columnAggregations(columns[name]).forEach(fn => {
stats.columns[name][fn] = res.rows[0][`${name}_${fn}`];
});
});
Object.keys(dimensionsInfo).forEach(name => {
stats.dimensions[name] = stats.dimensions[name] || Object.assign({}, dimensionsStats[name]);
let type = null;
columnAggregations(dimensionsInfo[name]).forEach(fn => {
type = type ||
fieldTypeSafe(ctx.dbConnection, res.fields.find(f => f.name === `${name}_${fn}`));
stats.dimensions[name][fn] = res.rows[0][`${name}_${fn}`];
});
stats.dimensions[name].type = type;
});
return stats;
})
);
return Promise.all(queries).then(results => ({
columns: mergeColumns(results.map(r => r.columns)),
dimensions: mergeColumns(results.map( r => r.dimensions))
}));
}
return Promise.resolve({ columns });
}
// This is adapted from SQL API:
function fieldType(cname) {
let tname;
switch (true) {
case /bool/.test(cname):
tname = 'boolean';
break;
case /int|float|numeric/.test(cname):
tname = 'number';
break;
case /text|char|unknown/.test(cname):
tname = 'string';
break;
case /date|time/.test(cname):
tname = 'date';
break;
default:
tname = cname;
}
if ( tname && cname.match(/^_/) ) {
tname += '[]';
}
return tname;
}
function fieldTypeSafe(dbConnection, field) {
const cname = dbConnection.typeName(field.dataTypeID);
return cname ? fieldType(cname) : `unknown(${field.dataTypeID})`;
}
// columns are returned as an object { columnName1: { type1: ...}, ..}
// for consistency with SQL API
function formatResultFields(dbConnection, fields = []) {
let nfields = {};
for (let field of fields) {
nfields[field.name] = { type: fieldTypeSafe(dbConnection, field) };
}
return nfields;
}
MapnikLayerStats.prototype.getStats =
function (layer, dbConnection, callback) {
var queryRowCountSql = queryUtils.getQueryRowCount(layer.options.sql);
// This query would gather stats for postgresql table if not exists
dbConnection.query(queryRowCountSql, function (err, res) {
if (err) {
return callback(null, {estimatedFeatureCount: -1});
} else {
// We decided that the relation is 1 row == 1 feature
return callback(null, {estimatedFeatureCount: res.rows[0].rows});
}
let aggrQuery = layer.options.sql;
let preQuery = layer.options.sql_raw || aggrQuery;
let ctx = {
dbConnection,
preQuery,
aggrQuery,
metaOptions: layer.options.metadata || {},
};
// TODO: could save some queries if queryUtils.getAggregationMetadata() has been used and kept somewhere
// we would set queries.results.estimatedFeatureCount and queries.results.geometryType
// (if metaOptions.geometryType) from it.
// TODO: compute _sample with _featureCount when available
// TODO: add support for sample.exclude option by, in that case, forcing the columns query and
// passing the results to the sample query function.
const dimensions = (layer.options.aggregation || {}).dimensions;
Promise.all([
_estimatedFeatureCount(ctx).then(
({ estimatedFeatureCount }) => _sample(ctx, estimatedFeatureCount)
.then(sampleResults => mergeResults([sampleResults, { estimatedFeatureCount }]))
),
_featureCount(ctx),
_aggrFeatureCount(ctx),
_geometryType(ctx),
_columns(ctx).then(columns => _columnStats(ctx, columns, dimensions))
]).then(results => {
results = mergeResults(results);
callback(null, results);
}).catch(error => {
callback(error);
});
};

2
lib/cartodb/backends/layer-stats/torque-layer-stats.js
View File

@ -1,3 +1,5 @@
'use strict';
function TorqueLayerStats() {
this._types = {
torque: true

27
lib/cartodb/api/overviews_metadata_api.js → lib/cartodb/backends/overviews-metadata.js
View File

@ -1,25 +1,22 @@
var SubstitutionTokens = require('../utils/substitution-tokens');
'use strict';
function OverviewsMetadataApi(pgQueryRunner) {
const queryUtils = require('../utils/query-utils');
function OverviewsMetadataBackend(pgQueryRunner) {
this.pgQueryRunner = pgQueryRunner;
}
module.exports = OverviewsMetadataApi;
module.exports = OverviewsMetadataBackend;
function prepareSql(sql) {
return sql && SubstitutionTokens.replace(sql, {
bbox: 'ST_MakeEnvelope(0,0,0,0)',
scale_denominator: '0',
pixel_width: '1',
pixel_height: '1'
});
}
OverviewsMetadataApi.prototype.getOverviewsMetadata = function (username, sql, callback) {
OverviewsMetadataBackend.prototype.getOverviewsMetadata = function (username, sql, callback) {
// FIXME: Currently using internal function _cdb_schema_name
// CDB_Overviews should provide the schema information directly.
var query = 'SELECT *, _cdb_schema_name(base_table)' +
' FROM CDB_Overviews(CDB_QueryTablesText($windshaft$' + prepareSql(sql) + '$windshaft$))';
const query = `
SELECT *, _cdb_schema_name(base_table)
FROM CDB_Overviews(
CDB_QueryTablesText($windshaft$${queryUtils.substituteDummyTokens(sql)}$windshaft$)
);
`;
this.pgQueryRunner.run(username, query, function handleOverviewsRows(err, rows) {
if (err){
callback(err);

184
lib/cartodb/backends/pg_connection.js
View File

@ -1,7 +1,8 @@
var assert = require('assert');
var step = require('step');
'use strict';
var PSQL = require('cartodb-psql');
var _ = require('underscore');
const debug = require('debug')('cachechan');
function PgConnection(metadataBackend) {
this.metadataBackend = metadataBackend;
@ -20,45 +21,59 @@ module.exports = PgConnection;
//
// @param callback function(err)
//
PgConnection.prototype.setDBAuth = function(username, params, callback) {
var self = this;
var user_params = {};
var auth_user = global.environment.postgres_auth_user;
var auth_pass = global.environment.postgres_auth_pass;
step(
function getId() {
self.metadataBackend.getUserId(username, this);
},
function(err, user_id) {
assert.ifError(err);
user_params.user_id = user_id;
var dbuser = _.template(auth_user, user_params);
_.extend(params, {dbuser:dbuser});
// skip looking up user_password if postgres_auth_pass
// doesn't contain the "user_password" label
if (!auth_pass || ! auth_pass.match(/\buser_password\b/) ) {
return null;
PgConnection.prototype.setDBAuth = function(username, params, apikeyType, callback) {
if (apikeyType === 'master') {
this.metadataBackend.getMasterApikey(username, (err, apikey) => {
if (err) {
if (isNameNotFoundError(err)) {
err.http_status = 404;
}
return callback(err);
}
self.metadataBackend.getUserDBPass(username, this);
},
function(err, user_password) {
assert.ifError(err);
user_params.user_password = user_password;
if ( auth_pass ) {
var dbpass = _.template(auth_pass, user_params);
_.extend(params, {dbpassword:dbpass});
params.dbuser = apikey.databaseRole;
params.dbpassword = apikey.databasePassword;
return callback();
});
} else if (apikeyType === 'regular') { //Actually it can be any type of api key
this.metadataBackend.getApikey(username, params.api_key, (err, apikey) => {
if (err) {
if (isNameNotFoundError(err)) {
err.http_status = 404;
}
return callback(err);
}
return true;
},
function finish(err) {
callback(err);
}
);
params.dbuser = apikey.databaseRole;
params.dbpassword = apikey.databasePassword;
return callback();
});
} else if (apikeyType === 'default') {
this.metadataBackend.getApikey(username, 'default_public', (err, apikey) => {
if (err) {
if (isNameNotFoundError(err)) {
err.http_status = 404;
}
return callback(err);
}
params.dbuser = apikey.databaseRole;
params.dbpassword = apikey.databasePassword;
return callback();
});
} else {
return callback(new Error(`Invalid Apikey type: ${apikeyType}, valid ones: master, regular, default`));
}
};
function isNameNotFoundError (err) {
return err.message && -1 !== err.message.indexOf('name not found');
}
// Set db connection parameters to those for the given username
//
// @param dbowner cartodb username of database owner,
@ -71,36 +86,30 @@ PgConnection.prototype.setDBAuth = function(username, params, callback) {
// @param callback function(err)
//
PgConnection.prototype.setDBConn = function(dbowner, params, callback) {
var self = this;
// Add default database connection parameters
// if none given
_.defaults(params, {
dbuser: global.environment.postgres.user,
dbpassword: global.environment.postgres.password,
// dbuser: global.environment.postgres.user,
// dbpassword: global.environment.postgres.password,
dbhost: global.environment.postgres.host,
dbport: global.environment.postgres.port
});
step(
function getConnectionParams() {
self.metadataBackend.getUserDBConnectionParams(dbowner, this);
},
function extendParams(err, dbParams){
assert.ifError(err);
// we don't want null values or overwrite a non public user
if (params.dbuser !== 'publicuser' || !dbParams.dbuser) {
delete dbParams.dbuser;
}
if ( dbParams ) {
_.extend(params, dbParams);
}
return null;
},
function finish(err) {
callback(err);
}
);
};
this.metadataBackend.getUserDBConnectionParams(dbowner, (err, dbParams) => {
if (err) {
return callback(err);
}
// we don’t want null values or overwrite a non public user
if (params.dbuser !== 'publicuser' || !dbParams.dbuser) {
delete dbParams.dbuser;
}
if (dbParams) {
_.extend(params, dbParams);
}
callback();
});
};
/**
* Returns a `cartodb-psql` object for a given username.
@ -109,28 +118,37 @@ PgConnection.prototype.setDBConn = function(dbowner, params, callback) {
*/
PgConnection.prototype.getConnection = function(username, callback) {
var self = this;
debug("getConn1");
var params = {};
require('debug')('cachechan')("getConn1");
step(
function setAuth() {
self.setDBAuth(username, params, this);
},
function setConn(err) {
assert.ifError(err);
self.setDBConn(username, params, this);
},
function openConnection(err) {
assert.ifError(err);
return callback(err, new PSQL({
user: params.dbuser,
pass: params.dbpass,
host: params.dbhost,
port: params.dbport,
dbname: params.dbname
}));
this.getDatabaseParams(username, (err, databaseParams) => {
if (err) {
return callback(err);
}
);
return callback(err, new PSQL({
user: databaseParams.dbuser,
pass: databaseParams.dbpass,
host: databaseParams.dbhost,
port: databaseParams.dbport,
dbname: databaseParams.dbname
}));
});
};
PgConnection.prototype.getDatabaseParams = function(username, callback) {
const databaseParams = {};
this.setDBAuth(username, databaseParams, 'master', err => {
if (err) {
return callback(err);
}
this.setDBConn(username, databaseParams, err => {
if (err) {
return callback(err);
}
callback(null, databaseParams);
});
});
};

46
lib/cartodb/backends/pg_query_runner.js
View File

@ -1,6 +1,6 @@
var assert = require('assert');
'use strict';
var PSQL = require('cartodb-psql');
var step = require('step');
function PgQueryRunner(pgConnection) {
this.pgConnection = pgConnection;
@ -16,31 +16,23 @@ module.exports = PgQueryRunner;
* @param {Function} callback function({Error}, {Array}) second argument is guaranteed to be an array
*/
PgQueryRunner.prototype.run = function(username, query, callback) {
var self = this;
var params = {};
step(
function setAuth() {
self.pgConnection.setDBAuth(username, params, this);
},
function setConn(err) {
assert.ifError(err);
self.pgConnection.setDBConn(username, params, this);
},
function executeQuery(err) {
assert.ifError(err);
var psql = new PSQL({
user: params.dbuser,
pass: params.dbpass,
host: params.dbhost,
port: params.dbport,
dbname: params.dbname
});
psql.query(query, function(err, resultSet) {
resultSet = resultSet || {};
return callback(err, resultSet.rows || []);
});
this.pgConnection.getDatabaseParams(username, (err, databaseParams) => {
if (err) {
return callback(err);
}
);
const psql = new PSQL({
user: databaseParams.dbuser,
pass: databaseParams.dbpass,
host: databaseParams.dbhost,
port: databaseParams.dbport,
dbname: databaseParams.dbname
});
psql.query(query, function (err, resultSet) {
resultSet = resultSet || {};
return callback(err, resultSet.rows || []);
});
});
};

2
lib/cartodb/backends/stats.js
View File

@ -1,3 +1,5 @@
'use strict';
var layerStats = require('./layer-stats/factory');
function StatsBackend() {

8
lib/cartodb/api/tables_extent_api.js → lib/cartodb/backends/tables-extent.js
View File

@ -1,8 +1,10 @@
function TablesExtentApi(pgQueryRunner) {
'use strict';
function TablesExtentBackend(pgQueryRunner) {
this.pgQueryRunner = pgQueryRunner;
}
module.exports = TablesExtentApi;
module.exports = TablesExtentBackend;
/**
* Given a username and a list of tables it will return the estimated extent in SRID 4326 for all the tables based on
@ -13,7 +15,7 @@ module.exports = TablesExtentApi;
* `table_name` format as valid input
* @param {Function} callback function(err, result) {Object} result with `west`, `south`, `east`, `north`
*/
TablesExtentApi.prototype.getBounds = function (username, tables, callback) {
TablesExtentBackend.prototype.getBounds = function (username, tables, callback) {
var estimatedExtentSQLs = tables.map(function(table) {
return "ST_EstimatedExtent('" + table.schema_name + "', '" + table.table_name + "', 'the_geom_webmercator')";
});

257
lib/cartodb/backends/template_maps.js
View File

@ -1,7 +1,7 @@
var assert = require('assert');
'use strict';
var crypto = require('crypto');
var debug = require('debug')('windshaft:templates');
var step = require('step');
var _ = require('underscore');
var dot = require('dot');
@ -69,27 +69,19 @@ TemplateMaps.prototype._userTemplateLimit = function() {
* @param callback - function to pass results too.
*/
TemplateMaps.prototype._redisCmd = function(redisFunc, redisArgs, callback) {
var redisClient;
var that = this;
var db = that.db_signatures;
this.redis_pool.acquire(this.db_signatures, (err, redisClient) => {
if (err) {
return callback(err);
}
step(
function getRedisClient() {
that.redis_pool.acquire(db, this);
},
function executeQuery(err, data) {
assert.ifError(err);
redisClient = data;
redisArgs.push(this);
redisClient[redisFunc.toUpperCase()].apply(redisClient, redisArgs);
},
function releaseRedisClient(err, data) {
if ( ! _.isUndefined(redisClient) ) {
that.redis_pool.release(db, redisClient);
}
callback(err, data);
}
);
redisClient[redisFunc.toUpperCase()](...redisArgs, (err, data) => {
this.redis_pool.release(this.db_signatures, redisClient);
if (err) {
return callback(err);
}
return callback(null, data);
});
});
};
var _reValidNameIdentifier = /^[a-z0-9][0-9a-z_\-]*$/i;
@ -184,6 +176,37 @@ function templateDefaults(template) {
});
}
/**
* Checks if the if the user reaches the templetes limit
*
* @param userTemplatesKey user templat key in Redis
* @param owner cartodb username of the template owner
* @param callback returns error if the user reaches the limit
*/
TemplateMaps.prototype._checkUserTemplatesLimit = function(userTemplatesKey, owner, callback) {
const limit = this._userTemplateLimit();
if(!limit) {
return callback();
}
this._redisCmd('HLEN', [userTemplatesKey], (err, numberOfTemplates) => {
if (err) {
return callback(err);
}
if (numberOfTemplates >= limit) {
const limitReachedError = new Error(
`User '${owner}' reached limit on number of templates (${numberOfTemplates}/${limit})`
);
limitReachedError.http_status = 409;
return callback(limitReachedError);
}
return callback();
});
};
//--------------- PUBLIC API -------------------------------------
// Add a template
@ -199,52 +222,41 @@ function templateDefaults(template) {
// Return template identifier (only valid for given user)
//
TemplateMaps.prototype.addTemplate = function(owner, template, callback) {
var self = this;
template = templateDefaults(template);
var invalidError = this._checkInvalidTemplate(template);
if ( invalidError ) {
if (invalidError) {
return callback(invalidError);
}
var templateName = template.name;
var userTemplatesKey = this.key_usr_tpl({ owner:owner });
var limit = this._userTemplateLimit();
var userTemplatesKey = this.key_usr_tpl({ owner });
step(
function checkLimit() {
if ( ! limit ) {
return 0;
}
self._redisCmd('HLEN', [ userTemplatesKey ], this);
},
function installTemplateIfDoesNotExist(err, numberOfTemplates) {
assert.ifError(err);
if ( limit && numberOfTemplates >= limit ) {
var limitReachedError = new Error("User '" + owner + "' reached limit on number of templates (" +
numberOfTemplates + "/" + limit + ")");
limitReachedError.http_status = 409;
throw limitReachedError;
}
self._redisCmd('HSETNX', [ userTemplatesKey, templateName, JSON.stringify(template) ], this);
},
function validateInstallation(err, wasSet) {
assert.ifError(err);
if ( ! wasSet ) {
throw new Error("Template '" + templateName + "' of user '" + owner + "' already exists");
}
return true;
},
function finish(err) {
if (!err) {
self.emit('add', owner, templateName, template);
}
callback(err, templateName, template);
this._checkUserTemplatesLimit(userTemplatesKey, owner, err => {
if (err) {
return callback(err);
}
);
let templateString;
try {
templateString = JSON.stringify(template);
} catch (error) {
return callback(error);
}
this._redisCmd('HSETNX', [userTemplatesKey, template.name, templateString], (err, wasSet) => {
if (err) {
return callback(err);
}
if (!wasSet) {
var templateExistsError = new Error(`Template '${template.name}' of user '${owner}' already exists`);
return callback(templateExistsError);
}
this.emit('add', owner, template.name, template);
return callback(null, template.name, template);
});
});
};
// Delete a template
@ -257,26 +269,18 @@ TemplateMaps.prototype.addTemplate = function(owner, template, callback) {
// @param callback function(err)
//
TemplateMaps.prototype.delTemplate = function(owner, tpl_id, callback) {
var self = this;
step(
function deleteTemplate() {
self._redisCmd('HDEL', [ self.key_usr_tpl({ owner:owner }), tpl_id ], this);
},
function handleDeletion(err, deleted) {
assert.ifError(err);
if (!deleted) {
throw new Error("Template '" + tpl_id + "' of user '" + owner + "' does not exist");
}
return true;
},
function finish(err) {
if (!err) {
self.emit('delete', owner, tpl_id);
}
callback(err);
this._redisCmd('HDEL', [ this.key_usr_tpl({ owner:owner }), tpl_id ], (err, deleted) => {
if (err) {
return callback(err);
}
);
if (!deleted) {
return callback(new Error(`Template '${tpl_id}' of user '${owner}' does not exist`));
}
this.emit('delete', owner, tpl_id);
return callback();
});
};
// Update a template
@ -296,56 +300,58 @@ TemplateMaps.prototype.delTemplate = function(owner, tpl_id, callback) {
// @param callback function(err)
//
TemplateMaps.prototype.updTemplate = function(owner, tpl_id, template, callback) {
var self = this;
template = templateDefaults(template);
var invalidError = this._checkInvalidTemplate(template);
if ( invalidError ) {
if (invalidError) {
return callback(invalidError);
}
var templateName = template.name;
if ( tpl_id !== templateName ) {
return callback(new Error("Cannot update name of a map template ('" + tpl_id + "' != '" + templateName + "')"));
if (tpl_id !== template.name) {
return callback(new Error(`Cannot update name of a map template ('${tpl_id}' != '${template.name}')`));
}
var userTemplatesKey = this.key_usr_tpl({ owner:owner });
var userTemplatesKey = this.key_usr_tpl({ owner });
var previousTemplate = null;
this._redisCmd('HGET', [userTemplatesKey, tpl_id], (err, beforeUpdateTemplate) => {
if (err) {
return callback(err);
}
step(
function getExistingTemplate() {
self._redisCmd('HGET', [ userTemplatesKey, tpl_id ], this);
},
function updateTemplate(err, _currentTemplate) {
assert.ifError(err);
if (!_currentTemplate) {
throw new Error("Template '" + tpl_id + "' of user '" + owner + "' does not exist");
if (!beforeUpdateTemplate) {
return callback(new Error(`Template '${tpl_id}' of user '${owner}' does not exist`));
}
let templateString;
try {
templateString = JSON.stringify(template);
} catch (error) {
return callback(error);
}
this._redisCmd('HSET', [userTemplatesKey, template.name, templateString], (err, didSetNewField) => {
if (err) {
return callback(err);
}
previousTemplate = _currentTemplate;
self._redisCmd('HSET', [ userTemplatesKey, templateName, JSON.stringify(template) ], this);
},
function handleTemplateUpdate(err, didSetNewField) {
assert.ifError(err);
if (didSetNewField) {
debug('New template created on update operation');
}
return true;
},
function finish(err) {
if (!err) {
if (self.fingerPrint(JSON.parse(previousTemplate)) !== self.fingerPrint(template)) {
self.emit('update', owner, templateName, template);
}
let beforeUpdateTemplateObject;
try {
beforeUpdateTemplateObject = JSON.parse(beforeUpdateTemplate);
} catch (error) {
return callback(error);
}
callback(err, template);
}
);
if (this.fingerPrint(beforeUpdateTemplateObject) !== this.fingerPrint(template)) {
this.emit('update', owner, template.name, template);
}
return callback(null, template);
});
});
};
// List user templates
@ -370,19 +376,20 @@ TemplateMaps.prototype.listTemplates = function(owner, callback) {
// Return full template definition
//
TemplateMaps.prototype.getTemplate = function(owner, tpl_id, callback) {
var self = this;
step(
function getTemplate() {
self._redisCmd('HGET', [ self.key_usr_tpl({owner:owner}), tpl_id ], this);
},
function parseTemplate(err, tpl_val) {
assert.ifError(err);
return JSON.parse(tpl_val);
},
function finish(err, tpl) {
callback(err, tpl);
this._redisCmd('HGET', [this.key_usr_tpl({owner:owner}), tpl_id], (err, template) => {
if (err) {
return callback(err);
}
);
let templateObject;
try {
templateObject = JSON.parse(template);
} catch (error) {
return callback(error);
}
return callback(null, templateObject);
});
};
TemplateMaps.prototype.isAuthorized = function(template, authTokens) {

84
lib/cartodb/backends/user-limits.js Normal file
View File

@ -0,0 +1,84 @@
'use strict';
/**
*
* @param metadataBackend
* @param options
* @constructor
* @type {UserLimitsBackend}
*/
function UserLimitsBackend(metadataBackend, options) {
this.metadataBackend = metadataBackend;
this.options = options || {};
this.options.limits = this.options.limits || {};
this.preprareRateLimit();
}
module.exports = UserLimitsBackend;
UserLimitsBackend.prototype.getRenderLimits = function (username, apiKey, callback) {
var self = this;
var limits = {
cacheOnTimeout: self.options.limits.cacheOnTimeout || false,
render: self.options.limits.render || 0
};
self.getTimeoutRenderLimit(username, apiKey, function (err, timeoutRenderLimit) {
if (err) {
return callback(err);
}
if (timeoutRenderLimit && timeoutRenderLimit.render) {
if (Number.isFinite(timeoutRenderLimit.render)) {
limits.render = timeoutRenderLimit.render;
}
}
return callback(null, limits);
});
};
UserLimitsBackend.prototype.getTimeoutRenderLimit = function (username, apiKey, callback) {
isAuthorized(this.metadataBackend, username, apiKey, (err, authorized) => {
if (err) {
return callback(err);
}
this.metadataBackend.getUserTimeoutRenderLimits(username, (err, timeoutRenderLimit) => {
if (err) {
return callback(err);
}
return callback(
null,
{ render: authorized ? timeoutRenderLimit.render : timeoutRenderLimit.renderPublic }
);
});
});
};
function isAuthorized(metadataBackend, username, apiKey, callback) {
if (!apiKey) {
return callback(null, false);
}
metadataBackend.getUserMapKey(username, function (err, userApiKey) {
if (err) {
return callback(err);
}
return callback(null, userApiKey === apiKey);
});
}
UserLimitsBackend.prototype.preprareRateLimit = function () {
if (this.options.limits.rateLimitsEnabled) {
this.metadataBackend.loadRateLimitsScript();
}
};
UserLimitsBackend.prototype.getRateLimit = function (user, endpointGroup, callback) {
this.metadataBackend.getRateLimit(user, 'maps', endpointGroup, callback);
};

2
lib/cartodb/cache/backend/fastly.js vendored
View File

@ -1,3 +1,5 @@
'use strict';
var FastlyPurge = require('fastly-purge');
function FastlyCacheBackend(apiKey, serviceId) {

2
lib/cartodb/cache/backend/varnish_http.js vendored
View File

@ -1,3 +1,5 @@
'use strict';
var request = require('request');
function VarnishHttpCacheBackend(host, port) {

2
lib/cartodb/cache/layergroup_affected_tables.js vendored
View File

@ -1,3 +1,5 @@
'use strict';
var LruCache = require('lru-cache');
function LayergroupAffectedTables() {

2
lib/cartodb/cache/model/named_maps_entry.js vendored
View File

@ -1,3 +1,5 @@
'use strict';
var crypto = require('crypto');
function NamedMaps(owner, name) {

17
lib/cartodb/cache/named_map_provider_cache.js vendored
View File

@ -1,3 +1,5 @@
'use strict';
var _ = require('underscore');
var dot = require('dot');
var NamedMapMapConfigProvider = require('../models/mapconfig/provider/named-map-provider');
@ -6,12 +8,20 @@ var queue = require('queue-async');
var LruCache = require("lru-cache");
function NamedMapProviderCache(templateMaps, pgConnection, metadataBackend, userLimitsApi, mapConfigAdapter) {
function NamedMapProviderCache(
templateMaps,
pgConnection,
metadataBackend,
userLimitsBackend,
mapConfigAdapter,
affectedTablesCache
) {
this.templateMaps = templateMaps;
this.pgConnection = pgConnection;
this.metadataBackend = metadataBackend;
this.userLimitsApi = userLimitsApi;
this.userLimitsBackend = userLimitsBackend;
this.mapConfigAdapter = mapConfigAdapter;
this.affectedTablesCache = affectedTablesCache;
this.providerCache = new LruCache({ max: 2000 });
}
@ -28,8 +38,9 @@ NamedMapProviderCache.prototype.get = function(user, templateId, config, authTok
this.templateMaps,
this.pgConnection,
this.metadataBackend,
this.userLimitsApi,
this.userLimitsBackend,
this.mapConfigAdapter,
this.affectedTablesCache,
user,
templateId,
config,

2
lib/cartodb/cache/surrogate_keys_cache.js vendored
View File

@ -1,3 +1,5 @@
'use strict';
var queue = require('queue-async');
/**

8
lib/cartodb/controllers/index.js
View File

@ -1,8 +0,0 @@
module.exports = {
Analyses: require('./analyses'),
Layergroup: require('./layergroup'),
Map: require('./map'),
NamedMaps: require('./named_maps'),
NamedMapsAdmin: require('./named_maps_admin'),
ServerInfo: require('./server_info')
};

513
lib/cartodb/controllers/layergroup.js
View File

@ -1,513 +0,0 @@
var assert = require('assert');
var step = require('step');
var cors = require('../middleware/cors');
var userMiddleware = require('../middleware/user');
var allowQueryParams = require('../middleware/allow-query-params');
var vectorError = require('../middleware/vector-error');
var DataviewBackend = require('../backends/dataview');
var AnalysisStatusBackend = require('../backends/analysis-status');
var MapStoreMapConfigProvider = require('../models/mapconfig/provider/map-store-provider');
var QueryTables = require('cartodb-query-tables');
/**
* @param {AuthApi} authApi
* @param {PgConnection} pgConnection
* @param {MapStore} mapStore
* @param {TileBackend} tileBackend
* @param {PreviewBackend} previewBackend
* @param {AttributesBackend} attributesBackend
* @param {SurrogateKeysCache} surrogateKeysCache
* @param {UserLimitsApi} userLimitsApi
* @param {LayergroupAffectedTables} layergroupAffectedTables
* @param {AnalysisBackend} analysisBackend
* @constructor
*/
function LayergroupController(prepareContext, pgConnection, mapStore, tileBackend, previewBackend, attributesBackend,
surrogateKeysCache, userLimitsApi, layergroupAffectedTables, analysisBackend) {
this.pgConnection = pgConnection;
this.mapStore = mapStore;
this.tileBackend = tileBackend;
this.previewBackend = previewBackend;
this.attributesBackend = attributesBackend;
this.surrogateKeysCache = surrogateKeysCache;
this.userLimitsApi = userLimitsApi;
this.layergroupAffectedTables = layergroupAffectedTables;
this.dataviewBackend = new DataviewBackend(analysisBackend);
this.analysisStatusBackend = new AnalysisStatusBackend();
this.prepareContext = prepareContext;
}
module.exports = LayergroupController;
LayergroupController.prototype.register = function(app) {
app.get(
app.base_url_mapconfig + '/:token/:z/:x/:y@:scale_factor?x.:format',
cors(),
userMiddleware,
this.prepareContext,
this.tile.bind(this),
vectorError()
);
app.get(
app.base_url_mapconfig + '/:token/:z/:x/:y.:format',
cors(),
userMiddleware,
this.prepareContext,
this.tile.bind(this),
vectorError()
);
app.get(
app.base_url_mapconfig + '/:token/:layer/:z/:x/:y.(:format)',
cors(),
userMiddleware,
validateLayerRouteMiddleware,
this.prepareContext,
this.layer.bind(this),
vectorError()
);
app.get(
app.base_url_mapconfig + '/:token/:layer/attributes/:fid',
cors(),
userMiddleware,
this.prepareContext,
this.attributes.bind(this)
);
app.get(
app.base_url_mapconfig + '/static/center/:token/:z/:lat/:lng/:width/:height.:format',
cors(),
userMiddleware,
allowQueryParams(['layer']),
this.prepareContext,
this.center.bind(this)
);
app.get(
app.base_url_mapconfig + '/static/bbox/:token/:west,:south,:east,:north/:width/:height.:format',
cors(),
userMiddleware,
allowQueryParams(['layer']),
this.prepareContext,
this.bbox.bind(this)
);
// Undocumented/non-supported API endpoint methods.
// Use at your own peril.
var allowedDataviewQueryParams = [
'filters', // json
'own_filter', // 0, 1
'no_filters', // 0, 1
'bbox', // w,s,e,n
'start', // number
'end', // number
'column_type', // string
'bins', // number
'aggregation', //string
'offset', // number
'q', // widgets search
'categories', // number
];
app.get(
app.base_url_mapconfig + '/:token/dataview/:dataviewName',
cors(),
userMiddleware,
allowQueryParams(allowedDataviewQueryParams),
this.prepareContext,
this.dataview.bind(this)
);
app.get(
app.base_url_mapconfig + '/:token/:layer/widget/:dataviewName',
cors(),
userMiddleware,
allowQueryParams(allowedDataviewQueryParams),
this.prepareContext,
this.dataview.bind(this)
);
app.get(
app.base_url_mapconfig + '/:token/dataview/:dataviewName/search',
cors(),
userMiddleware,
allowQueryParams(allowedDataviewQueryParams),
this.prepareContext,
this.dataviewSearch.bind(this)
);
app.get(
app.base_url_mapconfig + '/:token/:layer/widget/:dataviewName/search',
cors(),
userMiddleware,
allowQueryParams(allowedDataviewQueryParams),
this.prepareContext,
this.dataviewSearch.bind(this)
);
app.get(
app.base_url_mapconfig + '/:token/analysis/node/:nodeId',
cors(),
userMiddleware,
this.prepareContext,
this.analysisNodeStatus.bind(this)
);
};
LayergroupController.prototype.analysisNodeStatus = function(req, res, next) {
var self = this;
step(
function retrieveNodeStatus() {
self.analysisStatusBackend.getNodeStatus(res.locals, this);
},
function finish(err, nodeStatus, stats) {
req.profiler.add(stats || {});
if (err) {
err.label = 'GET NODE STATUS';
next(err);
} else {
self.sendResponse(req, res, nodeStatus, 200, {
'Cache-Control': 'public,max-age=5',
'Last-Modified': new Date().toUTCString()
});
}
}
);
};
LayergroupController.prototype.dataview = function(req, res, next) {
var self = this;
step(
function retrieveDataview() {
var mapConfigProvider = new MapStoreMapConfigProvider(
self.mapStore, res.locals.user, self.userLimitsApi, res.locals
);
self.dataviewBackend.getDataview(
mapConfigProvider,
res.locals.user,
res.locals,
this
);
},
function finish(err, dataview, stats) {
req.profiler.add(stats || {});
if (err) {
err.label = 'GET DATAVIEW';
next(err);
} else {
self.sendResponse(req, res, dataview, 200);
}
}
);
};
LayergroupController.prototype.dataviewSearch = function(req, res, next) {
var self = this;
step(
function searchDataview() {
var mapConfigProvider = new MapStoreMapConfigProvider(
self.mapStore, res.locals.user, self.userLimitsApi, res.locals
);
self.dataviewBackend.search(mapConfigProvider, res.locals.user, req.params.dataviewName, res.locals, this);
},
function finish(err, searchResult, stats) {
req.profiler.add(stats || {});
if (err) {
err.label = 'GET DATAVIEW SEARCH';
next(err);
} else {
self.sendResponse(req, res, searchResult, 200);
}
}
);
};
LayergroupController.prototype.attributes = function(req, res, next) {
var self = this;
req.profiler.start('windshaft.maplayer_attribute');
step(
function retrieveFeatureAttributes() {
var mapConfigProvider = new MapStoreMapConfigProvider(
self.mapStore, res.locals.user, self.userLimitsApi, res.locals
);
self.attributesBackend.getFeatureAttributes(mapConfigProvider, res.locals, false, this);
},
function finish(err, tile, stats) {
req.profiler.add(stats || {});
if (err) {
err.label = 'GET ATTRIBUTES';
next(err);
} else {
self.sendResponse(req, res, tile, 200);
}
}
);
};
// Gets a tile for a given token and set of tile ZXY coords. (OSM style)
LayergroupController.prototype.tile = function(req, res, next) {
req.profiler.start('windshaft.map_tile');
this.tileOrLayer(req, res, next);
};
// Gets a tile for a given token, layer set of tile ZXY coords. (OSM style)
LayergroupController.prototype.layer = function(req, res, next) {
req.profiler.start('windshaft.maplayer_tile');
this.tileOrLayer(req, res, next);
};
LayergroupController.prototype.tileOrLayer = function (req, res, next) {
var self = this;
step(
function mapController$getTileOrGrid() {
self.tileBackend.getTile(
new MapStoreMapConfigProvider(self.mapStore, res.locals.user, self.userLimitsApi, res.locals),
req.params, this
);
},
function mapController$finalize(err, tile, headers, stats) {
req.profiler.add(stats);
self.finalizeGetTileOrGrid(err, req, res, tile, headers, next);
}
);
};
function getStatusCode(tile, format){
return tile.length===0 && format==='mvt'? 204:200;
}
// This function is meant for being called as the very last
// step by all endpoints serving tiles or grids
LayergroupController.prototype.finalizeGetTileOrGrid = function(err, req, res, tile, headers, next) {
var supportedFormats = {
grid_json: true,
json_torque: true,
torque_json: true,
png: true,
png32: true,
mvt: true
};
var formatStat = 'invalid';
if (req.params.format) {
var format = req.params.format.replace('.', '_');
if (supportedFormats[format]) {
formatStat = format;
}
}
if (err) {
// See https://github.com/Vizzuality/Windshaft-cartodb/issues/68
var errMsg = err.message ? ( '' + err.message ) : ( '' + err );
// Rewrite mapnik parsing errors to start with layer number
var matches = errMsg.match("(.*) in style 'layer([0-9]+)'");
if (matches) {
errMsg = 'style'+matches[2]+': ' + matches[1];
}
err.message = errMsg;
err.label = 'TILE RENDER';
next(err);
global.statsClient.increment('windshaft.tiles.error');
global.statsClient.increment('windshaft.tiles.' + formatStat + '.error');
} else {
this.sendResponse(req, res, tile, getStatusCode(tile, formatStat), headers);
global.statsClient.increment('windshaft.tiles.success');
global.statsClient.increment('windshaft.tiles.' + formatStat + '.success');
}
};
LayergroupController.prototype.bbox = function(req, res, next) {
this.staticMap(req, res, +req.params.width, +req.params.height, {
west: +req.params.west,
north: +req.params.north,
east: +req.params.east,
south: +req.params.south
}, null, next);
};
LayergroupController.prototype.center = function(req, res, next) {
this.staticMap(req, res, +req.params.width, +req.params.height, +req.params.z, {
lng: +req.params.lng,
lat: +req.params.lat
}, next);
};
LayergroupController.prototype.staticMap = function(req, res, width, height, zoom /* bounds */, center, next) {
var format = req.params.format === 'jpg' ? 'jpeg' : 'png';
// We force always the tile to be generated using PNG because
// is the only format we support by now
res.locals.format = 'png';
res.locals.layer = res.locals.layer || 'all';
var self = this;
step(
function getImage() {
if (center) {
self.previewBackend.getImage(
new MapStoreMapConfigProvider(self.mapStore, res.locals.user, self.userLimitsApi, res.locals),
format, width, height, zoom, center, this);
} else {
self.previewBackend.getImage(
new MapStoreMapConfigProvider(self.mapStore, res.locals.user, self.userLimitsApi, res.locals),
format, width, height, zoom /* bounds */, this);
}
},
function handleImage(err, image, headers, stats) {
req.profiler.done('render-' + format);
req.profiler.add(stats || {});
if (err) {
err.label = 'STATIC_MAP';
next(err);
} else {
res.set('Content-Type', headers['Content-Type'] || 'image/' + format);
self.sendResponse(req, res, image, 200);
}
}
);
};
LayergroupController.prototype.sendResponse = function(req, res, body, status, headers) {
var self = this;
req.profiler.done('res');
res.set('Cache-Control', 'public,max-age=31536000');
// Set Last-Modified header
var lastUpdated;
if (res.locals.cache_buster) {
// Assuming cache_buster is a timestamp
lastUpdated = new Date(parseInt(res.locals.cache_buster));
} else {
lastUpdated = new Date();
}
res.set('Last-Modified', lastUpdated.toUTCString());
var dbName = res.locals.dbname;
step(
function getAffectedTables() {
self.getAffectedTables(res.locals.user, dbName, res.locals.token, this);
},
function sendResponse(err, affectedTables) {
req.profiler.done('affectedTables');
if (err) {
global.logger.warn('ERROR generating cache channel: ' + err);
}
if (!!affectedTables) {
res.set('X-Cache-Channel', affectedTables.getCacheChannel());
self.surrogateKeysCache.tag(res, affectedTables);
}
if (headers) {
res.set(headers);
}
res.status(status);
if (!Buffer.isBuffer(body) && typeof body === 'object') {
if (req.query && req.query.callback) {
res.jsonp(body);
} else {
res.json(body);
}
} else {
res.send(body);
}
}
);
};
LayergroupController.prototype.getAffectedTables = function(user, dbName, layergroupId, callback) {
if (this.layergroupAffectedTables.hasAffectedTables(dbName, layergroupId)) {
return callback(null, this.layergroupAffectedTables.get(dbName, layergroupId));
}
var self = this;
step(
function extractSQL() {
step(
function loadFromStore() {
self.mapStore.load(layergroupId, this);
},
function getSQL(err, mapConfig) {
assert.ifError(err);
var queries = [];
mapConfig.getLayers().forEach(function(layer) {
queries.push(layer.options.sql);
if (layer.options.affected_tables) {
layer.options.affected_tables.map(function(table) {
queries.push('SELECT * FROM ' + table + ' LIMIT 0');
});
}
});
return queries.length ? queries.join(';') : null;
},
this
);
},
function findAffectedTables(err, sql) {
assert.ifError(err);
if ( ! sql ) {
throw new Error("this request doesn't need an X-Cache-Channel generated");
}
step(
function getConnection() {
self.pgConnection.getConnection(user, this);
},
function getAffectedTables(err, connection) {
assert.ifError(err);
QueryTables.getAffectedTablesFromQuery(connection, sql, this);
},
this
);
},
function buildCacheChannel(err, tables) {
assert.ifError(err);
self.layergroupAffectedTables.set(dbName, layergroupId, tables);
return tables;
},
callback
);
};
function validateLayerRouteMiddleware(req, res, next) {
if (req.params.token === 'static') {
return next('route');
}
next();
}

729
lib/cartodb/controllers/map.js
View File

@ -1,729 +0,0 @@
var _ = require('underscore');
var windshaft = require('windshaft');
var QueryTables = require('cartodb-query-tables');
var ResourceLocator = require('../models/resource-locator');
var cors = require('../middleware/cors');
var userMiddleware = require('../middleware/user');
const allowQueryParams = require('../middleware/allow-query-params');
var MapConfig = windshaft.model.MapConfig;
var Datasource = windshaft.model.Datasource;
var NamedMapsCacheEntry = require('../cache/model/named_maps_entry');
var NamedMapMapConfigProvider = require('../models/mapconfig/provider/named-map-provider');
var CreateLayergroupMapConfigProvider = require('../models/mapconfig/provider/create-layergroup-provider');
/**
* @param {AuthApi} authApi
* @param {PgConnection} pgConnection
* @param {TemplateMaps} templateMaps
* @param {MapBackend} mapBackend
* @param metadataBackend
* @param {SurrogateKeysCache} surrogateKeysCache
* @param {UserLimitsApi} userLimitsApi
* @param {LayergroupAffectedTables} layergroupAffectedTables
* @param {MapConfigAdapter} mapConfigAdapter
* @param {StatsBackend} statsBackend
* @constructor
*/
function MapController(prepareContext, pgConnection, templateMaps, mapBackend, metadataBackend,
surrogateKeysCache, userLimitsApi, layergroupAffectedTables, mapConfigAdapter,
statsBackend) {
this.pgConnection = pgConnection;
this.templateMaps = templateMaps;
this.mapBackend = mapBackend;
this.metadataBackend = metadataBackend;
this.surrogateKeysCache = surrogateKeysCache;
this.userLimitsApi = userLimitsApi;
this.layergroupAffectedTables = layergroupAffectedTables;
this.mapConfigAdapter = mapConfigAdapter;
this.resourceLocator = new ResourceLocator(global.environment);
this.statsBackend = statsBackend;
this.prepareContext = prepareContext;
}
module.exports = MapController;
MapController.prototype.register = function(app) {
const { base_url_mapconfig, base_url_templated } = app;
const useTemplate = true;
app.get(base_url_mapconfig, this.composeCreateMapMiddleware());
app.post(base_url_mapconfig, this.composeCreateMapMiddleware());
app.get(`${base_url_templated}/:template_id/jsonp`, this.composeCreateMapMiddleware(useTemplate));
app.post(`${base_url_templated}/:template_id`, this.composeCreateMapMiddleware(useTemplate));
app.options(app.base_url_mapconfig, cors('Content-Type'));
};
MapController.prototype.composeCreateMapMiddleware = function (useTemplate = false) {
const isTemplateInstantiation = useTemplate;
const useTemplateHash = useTemplate;
const includeQuery = !useTemplate;
const label = useTemplate ? 'NAMED MAP LAYERGROUP' : 'ANONYMOUS LAYERGROUP';
const addContext = !useTemplate;
return [
cors(),
userMiddleware,
allowQueryParams(['aggregation']),
this.prepareContext,
this.initProfiler(isTemplateInstantiation),
this.checkJsonContentType(),
useTemplate ? this.checkInstantiteLayergroup() : this.checkCreateLayergroup(),
useTemplate ? this.getTemplate() : this.prepareAdapterMapConfig(),
useTemplate ? this.instantiateLayergroup() : this.createLayergroup(),
this.incrementMapViewCount(),
this.augmentLayergroupData(),
this.getAffectedTables(),
this.setCacheChannel(),
this.setLastModified(),
this.setLastUpdatedTimeToLayergroup(),
this.setCacheControl(),
this.setLayerStats(),
this.setLayergroupIdHeader(useTemplateHash),
this.setDataviewsAndWidgetsUrlsToLayergroupMetadata(),
this.setAnalysesMetadataToLayergroup(includeQuery),
this.setTurboCartoMetadataToLayergroup(),
this.setAggregationMetadataToLayergroup(),
this.setTilejsonMetadataToLayergroup(),
this.setSurrogateKeyHeader(),
this.sendResponse(),
this.augmentError({ label, addContext })
];
};
MapController.prototype.initProfiler = function (isTemplateInstantiation) {
const operation = isTemplateInstantiation ? 'instance_template' : 'createmap';
return function initProfilerMiddleware (req, res, next) {
req.profiler.start(`windshaft-cartodb.${operation}_${req.method.toLowerCase()}`);
req.profiler.done(`${operation}.initProfilerMiddleware`);
next();
};
};
MapController.prototype.checkJsonContentType = function () {
return function checkJsonContentTypeMiddleware(req, res, next) {
if (req.method === 'POST' && !req.is('application/json')) {
return next(new Error('POST data must be of type application/json'));
}
req.profiler.done('checkJsonContentTypeMiddleware');
next();
};
};
MapController.prototype.checkInstantiteLayergroup = function () {
return function checkInstantiteLayergroupMiddleware(req, res, next) {
if (req.method === 'GET') {
const { callback, config } = req.query;
if (callback === undefined || callback.length === 0) {
return next(new Error('callback parameter should be present and be a function name'));
}
if (config) {
try {
req.body = JSON.parse(config);
} catch(e) {
return next(new Error('Invalid config parameter, should be a valid JSON'));
}
}
}
req.profiler.done('checkInstantiteLayergroup');
return next();
};
};
MapController.prototype.checkCreateLayergroup = function () {
return function checkCreateLayergroupMiddleware (req, res, next) {
if (req.method === 'GET') {
const { config } = res.locals;
if (!config) {
return next(new Error('layergroup GET needs a "config" parameter'));
}
try {
req.body = JSON.parse(config);
} catch (err) {
return next(err);
}
}
req.profiler.done('checkCreateLayergroup');
return next();
};
};
MapController.prototype.getTemplate = function () {
return function getTemplateMiddleware (req, res, next) {
const templateParams = req.body;
const { user } = res.locals;
const mapconfigProvider = new NamedMapMapConfigProvider(
this.templateMaps,
this.pgConnection,
this.metadataBackend,
this.userLimitsApi,
this.mapConfigAdapter,
user,
req.params.template_id,
templateParams,
res.locals.auth_token,
res.locals
);
mapconfigProvider.getMapConfig((err, mapconfig, rendererParams) => {
req.profiler.done('named.getMapConfig');
if (err) {
return next(err);
}
res.locals.mapconfig = mapconfig;
res.locals.rendererParams = rendererParams;
res.locals.mapconfigProvider = mapconfigProvider;
next();
});
}.bind(this);
};
MapController.prototype.prepareAdapterMapConfig = function () {
return function prepareAdapterMapConfigMiddleware(req, res, next) {
const requestMapConfig = req.body;
const { user, dbhost, dbport, dbname, dbuser, dbpassword, api_key } = res.locals;
const context = {
analysisConfiguration: {
user,
db: {
host: dbhost,
port: dbport,
dbname: dbname,
user: dbuser,
pass: dbpassword
},
batch: {
username: user,
apiKey: api_key
}
}
};
this.mapConfigAdapter.getMapConfig(user, requestMapConfig, res.locals, context, (err, requestMapConfig) => {
req.profiler.done('anonymous.getMapConfig');
if (err) {
return next(err);
}
req.body = requestMapConfig;
res.locals.context = context;
next();
});
}.bind(this);
};
MapController.prototype.createLayergroup = function () {
return function createLayergroupMiddleware (req, res, next) {
const requestMapConfig = req.body;
const { context, user } = res.locals;
const datasource = context.datasource || Datasource.EmptyDatasource();
const mapconfig = new MapConfig(requestMapConfig, datasource);
const mapconfigProvider =
new CreateLayergroupMapConfigProvider(mapconfig, user, this.userLimitsApi, res.locals);
res.locals.mapconfig = mapconfig;
res.locals.analysesResults = context.analysesResults;
this.mapBackend.createLayergroup(mapconfig, res.locals, mapconfigProvider, (err, layergroup) => {
req.profiler.done('createLayergroup');
if (err) {
return next(err);
}
res.locals.layergroup = layergroup;
next();
});
}.bind(this);
};
MapController.prototype.instantiateLayergroup = function () {
return function instantiateLayergroupMiddleware (req, res, next) {
const { user, mapconfig, rendererParams } = res.locals;
const mapconfigProvider =
new CreateLayergroupMapConfigProvider(mapconfig, user, this.userLimitsApi, rendererParams);
this.mapBackend.createLayergroup(mapconfig, rendererParams, mapconfigProvider, (err, layergroup) => {
req.profiler.done('createLayergroup');
if (err) {
return next(err);
}
res.locals.layergroup = layergroup;
const { mapconfigProvider } = res.locals;
res.locals.analysesResults = mapconfigProvider.analysesResults;
res.locals.template = mapconfigProvider.template;
res.locals.templateName = mapconfigProvider.getTemplateName();
res.locals.context = mapconfigProvider.context;
next();
});
}.bind(this);
};
MapController.prototype.incrementMapViewCount = function () {
return function incrementMapViewCountMiddleware(req, res, next) {
const { mapconfig, user } = res.locals;
// Error won't blow up, just be logged.
this.metadataBackend.incMapviewCount(user, mapconfig.obj().stat_tag, (err) => {
req.profiler.done('incMapviewCount');
if (err) {
global.logger.log(`ERROR: failed to increment mapview count for user '${user}': ${err.message}`);
}
next();
});
}.bind(this);
};
MapController.prototype.augmentLayergroupData = function () {
return function augmentLayergroupDataMiddleware (req, res, next) {
const { layergroup } = res.locals;
// include in layergroup response the variables in serverMedata
// those variables are useful to send to the client information
// about how to reach this server or information about it
_.extend(layergroup, global.environment.serverMetadata);
next();
};
};
function getTemplateUrl(url) {
return url.https || url.http;
}
function getTilejson(tiles, grids) {
const tilejson = {
tilejson: '2.2.0',
tiles: tiles.https || tiles.http
};
if (grids) {
tilejson.grids = grids.https || grids.http;
}
return tilejson;
}
MapController.prototype.setTilejsonMetadataToLayergroup = function () {
return function augmentLayergroupTilejsonMiddleware (req, res, next) {
const { layergroup, user, mapconfig } = res.locals;
const isVectorOnlyMapConfig = mapconfig.isVectorOnlyMapConfig();
let hasMapnikLayers = false;
layergroup.metadata.layers.forEach((layerMetadata, index) => {
const layerId = mapconfig.getLayerId(index);
const rasterResource = `${layergroup.layergroupid}/${layerId}/{z}/{x}/{y}.png`;
if (mapconfig.layerType(index) === 'mapnik') {
hasMapnikLayers = true;
const vectorResource = `${layergroup.layergroupid}/${layerId}/{z}/{x}/{y}.mvt`;
const layerTilejson = {
vector: getTilejson(this.resourceLocator.getTileUrls(user, vectorResource))
};
if (!isVectorOnlyMapConfig) {
let grids = null;
const layer = mapconfig.getLayer(index);
if (layer.options.interactivity) {
const gridResource = `${layergroup.layergroupid}/${layerId}/{z}/{x}/{y}.grid.json`;
grids = this.resourceLocator.getTileUrls(user, gridResource);
}
layerTilejson.raster = getTilejson(
this.resourceLocator.getTileUrls(user, rasterResource),
grids
);
}
layerMetadata.tilejson = layerTilejson;
} else {
layerMetadata.tilejson = {
raster: getTilejson(this.resourceLocator.getTileUrls(user, rasterResource))
};
}
});
const tilejson = {};
const url = {};
if (hasMapnikLayers) {
const vectorResource = `${layergroup.layergroupid}/{z}/{x}/{y}.mvt`;
tilejson.vector = getTilejson(
this.resourceLocator.getTileUrls(user, vectorResource)
);
url.vector = getTemplateUrl(this.resourceLocator.getTemplateUrls(user, vectorResource));
if (!isVectorOnlyMapConfig) {
const rasterResource = `${layergroup.layergroupid}/{z}/{x}/{y}.png`;
tilejson.raster = getTilejson(
this.resourceLocator.getTileUrls(user, rasterResource)
);
url.raster = getTemplateUrl(this.resourceLocator.getTemplateUrls(user, rasterResource));
}
}
layergroup.metadata.tilejson = tilejson;
layergroup.metadata.url = url;
next();
}.bind(this);
};
MapController.prototype.getAffectedTables = function () {
return function getAffectedTablesMiddleware (req, res, next) {
const { dbname, layergroup, user, mapconfig } = res.locals;
this.pgConnection.getConnection(user, (err, connection) => {
if (err) {
return next(err);
}
const sql = [];
mapconfig.getLayers().forEach(function(layer) {
sql.push(layer.options.sql);
if (layer.options.affected_tables) {
layer.options.affected_tables.map(function(table) {
sql.push('SELECT * FROM ' + table + ' LIMIT 0');
});
}
});
QueryTables.getAffectedTablesFromQuery(connection, sql.join(';'), (err, affectedTables) => {
req.profiler.done('getAffectedTablesFromQuery');
if (err) {
return next(err);
}
// feed affected tables cache so it can be reused from, for instance, layergroup controller
this.layergroupAffectedTables.set(dbname, layergroup.layergroupId, affectedTables);
res.locals.affectedTables = affectedTables;
next();
});
});
}.bind(this);
};
MapController.prototype.setCacheChannel = function () {
return function setCacheChannelMiddleware (req, res, next) {
const { affectedTables } = res.locals;
if (req.method === 'GET') {
res.set('X-Cache-Channel', affectedTables.getCacheChannel());
}
next();
};
};
MapController.prototype.setLastModified = function () {
return function setLastModifiedMiddleware (req, res, next) {
if (req.method === 'GET') {
res.set('Last-Modified', (new Date()).toUTCString());
}
next();
};
};
MapController.prototype.setLastUpdatedTimeToLayergroup = function () {
return function setLastUpdatedTimeToLayergroupMiddleware (req, res, next) {
const { affectedTables, layergroup, analysesResults } = res.locals;
var lastUpdateTime = affectedTables.getLastUpdatedAt();
lastUpdateTime = getLastUpdatedTime(analysesResults, lastUpdateTime) || lastUpdateTime;
// last update for layergroup cache buster
layergroup.layergroupid = layergroup.layergroupid + ':' + lastUpdateTime;
layergroup.last_updated = new Date(lastUpdateTime).toISOString();
next();
};
};
function getLastUpdatedTime(analysesResults, lastUpdateTime) {
if (!Array.isArray(analysesResults)) {
return lastUpdateTime;
}
return analysesResults.reduce(function(lastUpdateTime, analysis) {
return analysis.getNodes().reduce(function(lastNodeUpdatedAtTime, node) {
var nodeUpdatedAtDate = node.getUpdatedAt();
var nodeUpdatedTimeAt = (nodeUpdatedAtDate && nodeUpdatedAtDate.getTime()) || 0;
return nodeUpdatedTimeAt > lastNodeUpdatedAtTime ? nodeUpdatedTimeAt : lastNodeUpdatedAtTime;
}, lastUpdateTime);
}, lastUpdateTime);
}
MapController.prototype.setCacheControl = function () {
return function setCacheControlMiddleware (req, res, next) {
if (req.method === 'GET') {
var ttl = global.environment.varnish.layergroupTtl || 86400;
res.set('Cache-Control', 'public,max-age='+ttl+',must-revalidate');
}
next();
};
};
MapController.prototype.setLayerStats = function () {
return function setLayerStatsMiddleware(req, res, next) {
const { user, mapconfig, layergroup } = res.locals;
this.pgConnection.getConnection(user, (err, connection) => {
if (err) {
return next(err);
}
this.statsBackend.getStats(mapconfig, connection, function(err, layersStats) {
if (err) {
return next(err);
}
if (layersStats.length > 0) {
layergroup.metadata.layers.forEach(function (layer, index) {
layer.meta.stats = layersStats[index];
});
}
next();
});
});
}.bind(this);
};
MapController.prototype.setLayergroupIdHeader = function (useTemplateHash) {
return function setLayergroupIdHeaderMiddleware (req, res, next) {
const { layergroup, user, template } = res.locals;
if (useTemplateHash) {
var templateHash = this.templateMaps.fingerPrint(template).substring(0, 8);
layergroup.layergroupid = `${user}@${templateHash}@${layergroup.layergroupid}`;
}
res.set('X-Layergroup-Id', layergroup.layergroupid);
next();
}.bind(this);
};
MapController.prototype.setDataviewsAndWidgetsUrlsToLayergroupMetadata = function () {
return function setDataviewsAndWidgetsUrlsToLayergroupMetadataMiddleware (req, res, next) {
const { layergroup, user, mapconfig } = res.locals;
this.addDataviewsAndWidgetsUrls(user, layergroup, mapconfig.obj());
next();
}.bind(this);
};
// TODO this should take into account several URL patterns
MapController.prototype.addDataviewsAndWidgetsUrls = function(username, layergroup, mapConfig) {
this.addDataviewsUrls(username, layergroup, mapConfig);
this.addWidgetsUrl(username, layergroup, mapConfig);
};
MapController.prototype.addDataviewsUrls = function(username, layergroup, mapConfig) {
layergroup.metadata.dataviews = layergroup.metadata.dataviews || {};
var dataviews = mapConfig.dataviews || {};
Object.keys(dataviews).forEach(function(dataviewName) {
var resource = layergroup.layergroupid + '/dataview/' + dataviewName;
layergroup.metadata.dataviews[dataviewName] = {
url: this.resourceLocator.getUrls(username, resource)
};
}.bind(this));
};
MapController.prototype.addWidgetsUrl = function(username, layergroup, mapConfig) {
if (layergroup.metadata && Array.isArray(layergroup.metadata.layers) && Array.isArray(mapConfig.layers)) {
layergroup.metadata.layers = layergroup.metadata.layers.map(function(layer, layerIndex) {
var mapConfigLayer = mapConfig.layers[layerIndex];
if (mapConfigLayer.options && mapConfigLayer.options.widgets) {
layer.widgets = layer.widgets || {};
Object.keys(mapConfigLayer.options.widgets).forEach(function(widgetName) {
var resource = layergroup.layergroupid + '/' + layerIndex + '/widget/' + widgetName;
layer.widgets[widgetName] = {
type: mapConfigLayer.options.widgets[widgetName].type,
url: this.resourceLocator.getUrls(username, resource)
};
}.bind(this));
}
return layer;
}.bind(this));
}
};
MapController.prototype.setAnalysesMetadataToLayergroup = function (includeQuery) {
return function setAnalysesMetadataToLayergroupMiddleware (req, res, next) {
const { layergroup, user, analysesResults = [] } = res.locals;
this.addAnalysesMetadata(user, layergroup, analysesResults, includeQuery);
next();
}.bind(this);
};
MapController.prototype.addAnalysesMetadata = function(username, layergroup, analysesResults, includeQuery) {
includeQuery = includeQuery || false;
analysesResults = analysesResults || [];
layergroup.metadata.analyses = [];
analysesResults.forEach(function(analysis) {
var nodes = analysis.getNodes();
layergroup.metadata.analyses.push({
nodes: nodes.reduce(function(nodesIdMap, node) {
if (node.params.id) {
var nodeResource = layergroup.layergroupid + '/analysis/node/' + node.id();
var nodeRepr = {
status: node.getStatus(),
url: this.resourceLocator.getUrls(username, nodeResource)
};
if (includeQuery) {
nodeRepr.query = node.getQuery();
}
if (node.getStatus() === 'failed') {
nodeRepr.error_message = node.getErrorMessage();
}
nodesIdMap[node.params.id] = nodeRepr;
}
return nodesIdMap;
}.bind(this), {})
});
}.bind(this));
};
MapController.prototype.setTurboCartoMetadataToLayergroup = function () {
return function setTurboCartoMetadataToLayergroupMiddleware (req, res, next) {
const { layergroup, mapconfig, context } = res.locals;
addTurboCartoContextMetadata(layergroup, mapconfig.obj(), context);
next();
};
};
function addTurboCartoContextMetadata(layergroup, mapConfig, context) {
if (layergroup.metadata && Array.isArray(layergroup.metadata.layers) && Array.isArray(mapConfig.layers)) {
layergroup.metadata.layers = layergroup.metadata.layers.map(function(layer, layerIndex) {
if (context.turboCarto && Array.isArray(context.turboCarto.layers)) {
layer.meta.cartocss_meta = context.turboCarto.layers[layerIndex];
}
return layer;
});
}
}
// TODO: see how evolve this function, it's a good candidate to be refactored
MapController.prototype.setAggregationMetadataToLayergroup = function () {
return function setAggregationMetadataToLayergroupMiddleware (req, res, next) {
const { layergroup, mapconfig, context } = res.locals;
addAggregationContextMetadata(layergroup, mapconfig.obj(), context);
next();
};
};
function addAggregationContextMetadata(layergroup, mapConfig, context) {
if (layergroup.metadata && Array.isArray(layergroup.metadata.layers) && Array.isArray(mapConfig.layers)) {
layergroup.metadata.layers = layergroup.metadata.layers.map(function(layer, layerIndex) {
if (context.aggregation && Array.isArray(context.aggregation.layers)) {
layer.meta.aggregation = context.aggregation.layers[layerIndex];
}
return layer;
});
}
}
MapController.prototype.setSurrogateKeyHeader = function () {
return function setSurrogateKeyHeaderMiddleware(req, res, next) {
const { affectedTables, user, templateName } = res.locals;
if (req.method === 'GET' && affectedTables.tables && affectedTables.tables.length > 0) {
this.surrogateKeysCache.tag(res, affectedTables);
}
if (templateName) {
this.surrogateKeysCache.tag(res, new NamedMapsCacheEntry(user, templateName));
}
next();
}.bind(this);
};
MapController.prototype.sendResponse = function () {
return function sendResponseMiddleware (req, res) {
req.profiler.done('res');
const { layergroup } = res.locals;
res.status(200);
if (req.query && req.query.callback) {
res.jsonp(layergroup);
} else {
res.json(layergroup);
}
};
};
MapController.prototype.augmentError = function (options) {
const { addContext = false, label = 'MAPS CONTROLLER' } = options;
return function augmentErrorMiddleware (err, req, res, next) {
req.profiler.done('error');
const { mapconfig } = res.locals;
if (addContext) {
err = Number.isFinite(err.layerIndex) ? populateError(err, mapconfig) : err;
}
err.label = label;
next(err);
};
};
function populateError(err, mapConfig) {
var error = new Error(err.message);
error.http_status = err.http_status;
if (!err.http_status && err.message.indexOf('column "the_geom_webmercator" does not exist') >= 0) {
error.http_status = 400;
}
error.type = 'layer';
error.subtype = err.message.indexOf('Postgis Plugin') >= 0 ? 'query' : undefined;
error.layer = {
id: mapConfig.getLayerId(err.layerIndex),
index: err.layerIndex,
type: mapConfig.layerType(err.layerIndex)
};
return error;
}

Some files were not shown because too many files have changed in this diff Show More