cartodb/Windshaft-cartodb
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' into static-maps-layers-filter

Browse Source
This commit is contained in:
Raul Ochoa 2017-03-30 19:33:59 +02:00
commit 777ae31426
28 changed files with 3017 additions and 4347 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.travis.yml
View File

@ -2,6 +2,8 @@ dist: trusty
addons:
postgresql: "9.5"
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- postgresql-9.5-postgis-2.3
- postgresql-plpython-9.5
@ -10,6 +12,7 @@ addons:
- libjpeg8-dev
- libgif-dev
- libpango1.0-dev
- g++-4.9
before_install:
- createdb template_postgis
@ -17,8 +20,8 @@ before_install:
- psql -c "CREATE EXTENSION postgis" template_postgis
env:
- NPROCS=1 JOBS=1 PGUSER=postgres
- NPROCS=1 JOBS=1 PGUSER=postgres CXX=g++-4.9
language: node_js
node_js:
- "0.10"
- "6"

4
HOWTO_RELEASE
View File

@ -1,8 +1,8 @@
1. Test (make clean all check), fix if broken before proceeding
2. Ensure proper version in package.json
3. Ensure NEWS section exists for the new version, review it, add release date
4. Recreate npm-shrinkwrap.json with: `make shrinkwrap`
5. Commit package.json, npm-shrinwrap.json, NEWS
4. Recreate yarn.lock with: `yarn upgrade`
5. Commit package.json, yarn.lock, NEWS
6. git tag -a Major.Minor.Patch # use NEWS section as content
7. Stub NEWS/package for next version

12
INSTALL.md
View File

@ -4,11 +4,11 @@
Make sure that you have the requirements needed. These are
- Core
- Node.js >=0.8
- npm >=1.2.1 <2.0.0
- Node.js >=6.9.x
- yarn >=0.21.3
- PostgreSQL >8.3.x, PostGIS >1.5.x
- Redis >2.4.0 (http://www.redis.io)
- Mapnik 2.0.1, 2.0.2, 2.1.0, 2.2.0, 2.3.0. See [Installing Mapnik](https://github.com/CartoDB/Windshaft#installing-mapnik).
- Mapnik >3.x. See [Installing Mapnik](https://github.com/CartoDB/Windshaft#installing-mapnik).
- Windshaft: check [Windshaft dependencies and installation notes](https://github.com/CartoDB/Windshaft#dependencies)
- libcairo2-dev, libpango1.0-dev, libjpeg8-dev and libgif-dev for server side canvas support
@ -43,11 +43,11 @@ psql -d template_postgis -c 'CREATE EXTENSION postgis;'
To fetch and build all node-based dependencies, run:
```
npm install
yarn
```
Note that the ```npm install``` step will populate the node_modules/
Note that the ```yarn``` step will populate the node_modules/
directory with modules, some of which being compiled on demand. If you
happen to have startup errors you may need to force rebuilding those
modules. At any time just wipe out the node_modules/ directory and run
```npm install``` again.
```yarn``` again.

6
Makefile
View File

@ -9,12 +9,6 @@ all:
clean:
rm -rf node_modules/
shrinkwrap: clean
rm npm-shrinkwrap.json
npm install --no-shrinkwrap --production
npm prune
npm shrinkwrap
distclean: clean
rm config.status*

65
NEWS.md
View File

@ -1,9 +1,72 @@
# Changelog
## 2.88.5
## 3.2.1
Released 2017-mm-dd
## 3.2.0
Released 2017-03-30
Announcements:
- Upgrades windshaft to [3.1.0](https://github.com/CartoDB/windshaft/releases/tag/3.1.0).
- Active GC interval.
## 3.1.1
Released 2017-03-23
Bug fixes:
- Use crc32 instead of md5 for computing subdomain candidate #642
## 3.1.0
Released 2017-03-22
Features:
- Generate URLs for resources based on CDN and template rules
## 3.0.2
Released 2017-03-22
Bug fixes:
- Upgrade dependencies
- Improve docs: remove mentions to NPM and use yarn instead
- Remove script to generate npm-shrinkwrap file
## 3.0.1
Released 2017-03-21
Announcements:
- Upgrades windshaft to [3.0.1](https://github.com/CartoDB/windshaft/releases/tag/3.0.1).
## 3.0.0
Released 2017-03-21
Announcements:
- Supports Node v6.9.x
- Drops support for Node v0.10.x
- Upgrades windshaft to 3.0.0
- Upgrades cartodb-query-tables to 0.2.0
- Upgrades cartodb-redis to 0.13.2
- Upgrades redis-mpool to 0.4.1
**Note**: Due to this [issue](https://github.com/npm/npm/issues/15713), Windshaft-cartodb must be installed with `yarn` instead of `npm` providing just a `yarn.lock` to get consistent installs across machines.
## 2.89.0
Released 2017-03-17
**Deprecation warning**: v2.89.0 is the last release that supports Node v0.10.x. Next mayor release will support Node v6.9.x and further versions.
Announcements:
- Upgrades windshaft to [2.8.0](https://github.com/CartoDB/windshaft/releases/tag/2.8.0).
Bug fixes:
- Histogram column type discovery query uses non-filtered query #637
## 2.88.4
Released 2017-03-10

12
README.md
View File

@ -32,14 +32,14 @@ Upgrading
Checkout your commit/branch. If you need to reinstall dependencies (you can check [NEWS](NEWS.md)) do the following:
```
rm -rf node_modules; npm install
rm -rf node_modules; yarn
```
Run
---
```
node app.js <env>
node app.js <env>
```
Where <env> is the name of a configuration file under config/environments/.
@ -71,12 +71,12 @@ See [CONTRIBUTING.md](CONTRIBUTING.md).
### Developing with a custom windshaft version
If you plan or want to use a custom / not released yet version of windshaft (or any other dependency) the best option is
to use `npm link`. You can read more about it at [npm-link: Symlink a package folder](https://docs.npmjs.com/cli/link).
to use `yarn link`. You can read more about it at [yarn-link: Symlink a package folder](https://yarnpkg.com/en/docs/cli/link).
**Quick start**:
```shell
~/windshaft-directory $ npm install
~/windshaft-directory $ npm link
~/windshaft-cartodb-directory $ npm link windshaft
~/windshaft-directory $ yarn
~/windshaft-directory $ yarn link
~/windshaft-cartodb-directory $ yarn link windshaft
```

15
app.js
View File

@ -110,6 +110,7 @@ var listener = server.listen(serverOptions.bind.port, serverOptions.bind.host, b
var version = require("./package").version;
listener.on('listening', function() {
log("Using Node.js %s", process.version);
log('Using configuration file "%s"', configurationFile);
log(
"Windshaft tileserver %s started on %s:%s PID=%d (%s)",
@ -135,3 +136,17 @@ process.on('SIGHUP', function() {
process.on('uncaughtException', function(err) {
global.logger.error('Uncaught exception: ' + err.stack);
});
if (global.gc) {
var gcInterval = Number.isFinite(global.environment.gc_interval) ?
global.environment.gc_interval :
10000;
if (gcInterval > 0) {
setInterval(function gcForcedCycle() {
var start = Date.now();
global.gc();
global.statsClient.timing('windshaft.gc', Date.now() - start);
}, gcInterval);
}
}

8
docs/named_maps.md
View File

@ -124,7 +124,7 @@ view (optional) | extra keys to specify the view area for the map. It can be use
### Placeholder Format
Placeholders are variables that can be placed in your template.json file. Placeholders need to be defined with a `type` and a default value for MapConfigs. See details about defining a MapConfig `type` for [Layergoup configurations](http://docs.carto.com/carto-engine/maps-api/mapconfig/#layergroup-configurations).
Placeholders are variables that can be placed in your template.json file. Placeholders need to be defined with a `type` and a default value for MapConfigs. See details about defining a MapConfig `type` for [Layergroup configurations](http://docs.carto.com/carto-engine/maps-api/mapconfig/#layergroup-configurations).
Valid placeholder names start with a letter and can only contain letters, numbers, or underscores. They have to be written between the `<%=` and `%>` strings in order to be replaced inside the Named Maps API.
@ -527,11 +527,11 @@ If you are creating a Torque layer in a Named Map without using the Torque.js li
#### Examples of Named Maps created with CARTO.js
- [Named Map selectors with interaction](http://bl.ocks.org/ohasselblad/515a8af1f99d5e690484)
- [Named Map selectors with interaction](http://bl.ocks.org/andy-esch/515a8af1f99d5e690484)
- [Named Map with interactivity](http://bl.ocks.org/ohasselblad/d1a45b8ff5e7bd90cd68)
- [Named Map with interactivity](http://bl.ocks.org/andy-esch/d1a45b8ff5e7bd90cd68)
- [Toggling sublayers in a Named Map](http://bl.ocks.org/ohasselblad/c1a0f4913610eec53cd3)
- [Toggling sublayers in a Named Map](http://bl.ocks.org/andy-esch/c1a0f4913610eec53cd3)
## Fetching XYZ Tiles for Named Maps

68
lib/cartodb/controllers/map.js
View File

@ -1,11 +1,11 @@
var _ = require('underscore');
var dot = require('dot');
dot.templateSettings.strip = false;
var assert = require('assert');
var step = require('step');
var windshaft = require('windshaft');
var QueryTables = require('cartodb-query-tables');
var ResourceLocator = require('../models/resource-locator');
var util = require('util');
var BaseController = require('./base');
@ -46,20 +46,7 @@ function MapController(authApi, pgConnection, templateMaps, mapBackend, metadata
this.layergroupAffectedTables = layergroupAffectedTables;
this.mapConfigAdapter = mapConfigAdapter;
this.resourcesUrlTemplates = null;
if (global.environment.resources_url_templates) {
var templates = global.environment.resources_url_templates;
if (templates.http) {
this.resourcesUrlTemplates = this.resourcesUrlTemplates || {};
this.resourcesUrlTemplates.http = dot.template(templates.http + '/{{=it.resource}}');
}
if (templates.https) {
this.resourcesUrlTemplates = this.resourcesUrlTemplates || {};
this.resourcesUrlTemplates.https = dot.template(templates.https + '/{{=it.resource}}');
}
}
this.resourceLocator = new ResourceLocator(global.environment);
}
util.inherits(MapController, BaseController);
@ -399,7 +386,7 @@ MapController.prototype.addAnalysesMetadata = function(username, layergroup, ana
var nodeResource = layergroup.layergroupid + '/analysis/node/' + node.id();
var nodeRepr = {
status: node.getStatus(),
url: this.getUrls(username, nodeResource)
url: this.resourceLocator.getUrls(username, nodeResource)
};
if (includeQuery) {
nodeRepr.query = node.getQuery();
@ -429,7 +416,7 @@ MapController.prototype.addDataviewsUrls = function(username, layergroup, mapCon
Object.keys(dataviews).forEach(function(dataviewName) {
var resource = layergroup.layergroupid + '/dataview/' + dataviewName;
layergroup.metadata.dataviews[dataviewName] = {
url: this.getUrls(username, resource)
url: this.resourceLocator.getUrls(username, resource)
};
}.bind(this));
};
@ -444,7 +431,7 @@ MapController.prototype.addWidgetsUrl = function(username, layergroup, mapConfig
var resource = layergroup.layergroupid + '/' + layerIndex + '/widget/' + widgetName;
layer.widgets[widgetName] = {
type: mapConfigLayer.options.widgets[widgetName].type,
url: this.getUrls(username, resource)
url: this.resourceLocator.getUrls(username, resource)
};
}.bind(this));
}
@ -452,46 +439,3 @@ MapController.prototype.addWidgetsUrl = function(username, layergroup, mapConfig
}.bind(this));
}
};
MapController.prototype.getUrls = function(username, resource) {
if (this.resourcesUrlTemplates) {
return this.getUrlsFromTemplate(username, resource);
}
var cdnUrl = global.environment.serverMetadata && global.environment.serverMetadata.cdn_url;
if (cdnUrl) {
return {
http: 'http://' + cdnUrl.http + '/' + username + '/api/v1/map/' + resource,
https: 'https://' + cdnUrl.https + '/' + username + '/api/v1/map/' + resource
};
} else {
var port = global.environment.port;
return {
http: 'http://' + username + '.' + 'localhost.lan:' + port + '/api/v1/map/' + resource
};
}
};
MapController.prototype.getUrlsFromTemplate = function(username, resource) {
var urls = {};
var cdnUrl = global.environment.serverMetadata && global.environment.serverMetadata.cdn_url || {};
if (this.resourcesUrlTemplates.http) {
urls.http = this.resourcesUrlTemplates.http({
cdn_url: cdnUrl.http,
user: username,
port: global.environment.port,
resource: resource
});
}
if (this.resourcesUrlTemplates.https) {
urls.https = this.resourcesUrlTemplates.https({
cdn_url: cdnUrl.https,
user: username,
port: global.environment.port,
resource: resource
});
}
return urls;
};

2
lib/cartodb/models/dataview/factory.js
View File

@ -11,7 +11,7 @@ var DataviewFactory = {
if (!this.dataviews[type]) {
throw new Error('Invalid dataview type: "' + type + '"');
}
return new this.dataviews[type](query, dataviewDefinition.options);
return new this.dataviews[type](query, dataviewDefinition.options, dataviewDefinition.sql);
}
};

5
lib/cartodb/models/dataview/histogram.js
View File

@ -109,12 +109,13 @@ var TYPE = 'histogram';
}
}
*/
function Histogram(query, options) {
function Histogram(query, options, queries) {
if (!_.isString(options.column)) {
throw new Error('Histogram expects `column` in widget options');
}
this.query = query;
this.queries = queries;
this.column = options.column;
this.bins = options.bins;
@ -143,7 +144,7 @@ Histogram.prototype.sql = function(psql, override, callback) {
var _column = this.column;
var columnTypeQuery = columnTypeQueryTpl({
column: _column, query: this.query
column: _column, query: this.queries.no_filters
});
if (this._columnType === null) {

3
lib/cartodb/models/dataview/overviews/factory.js
View File

@ -14,7 +14,8 @@ OverviewsDataviewFactory.prototype.getDataview = function(query, dataviewDefinit
return parentFactory.getDataview(query, dataviewDefinition);
}
return new dataviews[type](
query, dataviewDefinition.options, this.queryRewriter, this.queryRewriteData, this.options
query, dataviewDefinition.options, this.queryRewriter, this.queryRewriteData, this.options,
dataviewDefinition.sql
);
};

5
lib/cartodb/models/dataview/overviews/histogram.js
View File

@ -96,10 +96,11 @@ var histogramQueryTpl = dot.template([
'ORDER BY bin'
].join('\n'));
function Histogram(query, options, queryRewriter, queryRewriteData, params) {
function Histogram(query, options, queryRewriter, queryRewriteData, params, queries) {
BaseOverviewsDataview.call(this, query, options, BaseDataview, queryRewriter, queryRewriteData, params);
this.query = query;
this.queries = queries;
this.column = options.column;
this.bins = options.bins;
@ -129,7 +130,7 @@ Histogram.prototype.sql = function(psql, override, callback) {
var _column = this.column;
var columnTypeQuery = columnTypeQueryTpl({
column: _column, query: this.rewrittenQuery(this.query)
column: _column, query: this.rewrittenQuery(this.queries.no_filters)
});
if (this._columnType === null) {

119
lib/cartodb/models/resource-locator.js Normal file
View File

@ -0,0 +1,119 @@
var dot = require('dot');
dot.templateSettings.strip = false;
function ResourceLocator(environment) {
this.environment = environment;
this.resourcesUrlTemplates = null;
if (this.environment.resources_url_templates) {
var templates = environment.resources_url_templates;
if (templates.http) {
this.resourcesUrlTemplates = this.resourcesUrlTemplates || {};
this.resourcesUrlTemplates.http = dot.template(templates.http + '/{{=it.resource}}');
}
if (templates.https) {
this.resourcesUrlTemplates = this.resourcesUrlTemplates || {};
this.resourcesUrlTemplates.https = dot.template(templates.https + '/{{=it.resource}}');
}
}
}
module.exports = ResourceLocator;
ResourceLocator.prototype.getUrls = function(username, resource) {
if (this.resourcesUrlTemplates) {
return this.getUrlsFromTemplate(username, resource);
}
var cdnDomain = getCdnDomain(this.environment.serverMetadata, resource);
if (cdnDomain) {
return {
http: 'http://' + cdnDomain.http + '/' + username + '/api/v1/map/' + resource,
https: 'https://' + cdnDomain.https + '/' + username + '/api/v1/map/' + resource
};
} else {
var port = this.environment.port;
return {
http: 'http://' + username + '.' + 'localhost.lan:' + port + '/api/v1/map/' + resource
};
}
};
ResourceLocator.prototype.getUrlsFromTemplate = function(username, resource) {
var urls = {};
var cdnDomain = getCdnDomain(this.environment.serverMetadata, resource) || {};
if (this.resourcesUrlTemplates.http) {
urls.http = this.resourcesUrlTemplates.http({
cdn_url: cdnDomain.http,
user: username,
port: this.environment.port,
resource: resource
});
}
if (this.resourcesUrlTemplates.https) {
urls.https = this.resourcesUrlTemplates.https({
cdn_url: cdnDomain.https,
user: username,
port: this.environment.port,
resource: resource
});
}
return urls;
};
function getCdnDomain(serverMetadata, resource) {
if (serverMetadata && serverMetadata.cdn_url) {
var cdnUrl = serverMetadata.cdn_url;
var http = cdnUrl.http;
var https = cdnUrl.https;
if (cdnUrl.templates) {
var templates = cdnUrl.templates;
var httpUrlTemplate = templates.http.url;
var httpsUrlTemplate = templates.https.url;
http = httpUrlTemplate
.replace(/^(http[s]*:\/\/)/, '')
.replace('{s}', subdomain(templates.http.subdomains, resource));
https = httpsUrlTemplate
.replace(/^(http[s]*:\/\/)/, '')
.replace('{s}', subdomain(templates.https.subdomains, resource));
}
return {
http: http,
https: https,
};
}
return null;
}
// ref https://jsperf.com/js-crc32
function crcTable() {
var c;
var table = [];
for (var n = 0; n < 256; n++) {
c = n;
for (var k = 0; k < 8; k++) {
c = ((c & 1) ? (0xEDB88320 ^ (c >>> 1)) : (c >>> 1));
}
table[n] = c;
}
return table;
}
var CRC_TABLE = crcTable();
function crc32(str) {
var crc = 0 ^ (-1);
for (var i = 0; i < str.length; i++) {
crc = (crc >>> 8) ^ CRC_TABLE[(crc ^ str.charCodeAt(i)) & 0xFF];
}
return (crc ^ (-1)) >>> 0;
}
function subdomain(subdomains, resource) {
var index = crc32(resource) % subdomains.length;
return subdomains[index];
}
module.exports.subdomain = subdomain;

4123
npm-shrinkwrap.json generated
View File

File diff suppressed because it is too large Load Diff

19
package.json
View File

@ -1,7 +1,7 @@
{
"private": true,
"name": "windshaft-cartodb",
"version": "2.88.5",
"version": "3.2.1",
"description": "A map tile server for CartoDB",
"keywords": [
"cartodb"
@ -22,8 +22,8 @@
"body-parser": "~1.14.0",
"camshaft": "0.50.3",
"cartodb-psql": "~0.7.1",
"cartodb-query-tables": "~0.1.0",
"cartodb-redis": "0.13.1",
"cartodb-query-tables": "0.2.0",
"cartodb-redis": "0.13.2",
"debug": "~2.2.0",
"dot": "~1.0.2",
"express": "~4.13.3",
@ -33,22 +33,21 @@
"lzma": "~2.3.2",
"node-statsd": "~0.0.7",
"queue-async": "~1.0.7",
"redis-mpool": "~0.4.0",
"redis-mpool": "0.4.1",
"request": "~2.79.0",
"step": "~0.0.6",
"step-profiler": "~0.3.0",
"turbo-carto": "0.19.0",
"underscore": "~1.6.0",
"windshaft": "2.7.0",
"yargs": "~5.0.0",
"zipfile": "cartodb/node-zipfile#0.5.11-cdb1"
"windshaft": "3.1.0",
"yargs": "~5.0.0"
},
"devDependencies": {
"istanbul": "~0.4.3",
"jshint": "~2.6.0",
"mocha": "~1.21.4",
"nock": "~2.11.0",
"redis": "~0.8.6",
"redis": "~0.12.1",
"semver": "~1.1.4",
"strftime": "~0.8.2"
},
@ -57,7 +56,7 @@
"test": "make test-all"
},
"engines": {
"node": ">=0.8 <0.11",
"npm": ">=2.14.16"
"node": ">=6.9",
"yarn": "^0.21.3"
}
}

2
scripts/install.sh
View File

@ -4,4 +4,4 @@ if [[ "$OSTYPE" == "darwin"* ]]; then
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/opt/X11/lib/pkgconfig
fi
npm install
yarn

91
test/acceptance/dataviews/overviews.js
View File

@ -144,6 +144,22 @@ describe('dataviews using tables with overviews', function() {
aggregationColumn: 'name',
}
},
test_histogram: {
type: 'histogram',
source: {id: 'data-source'},
options: {
column: 'value',
bins: 2
}
},
test_histogram_date: {
type: 'histogram',
source: {id: 'data-source'},
options: {
column: 'updated_at',
bins: 2
}
},
test_avg: {
type: 'formula',
source: {id: 'data-source'},
@ -265,8 +281,83 @@ describe('dataviews using tables with overviews', function() {
});
});
it("should expose a histogram", function (done) {
var testClient = new TestClient(overviewsMapConfig);
testClient.getDataview('test_histogram', function (err, histogram) {
if (err) {
return done(err);
}
assert.ok(histogram);
assert.equal(histogram.type, 'histogram');
assert.ok(Array.isArray(histogram.bins));
testClient.drain(done);
});
});
describe('filters', function() {
describe('histogram', function () {
it("should expose a filtered histogram", function (done) {
var params = {
filters: {
dataviews: { test_histogram: { min: 2 } }
}
};
var testClient = new TestClient(overviewsMapConfig);
testClient.getDataview('test_histogram', params, function (err, histogram) {
if (err) {
return done(err);
}
assert.ok(histogram);
assert.equal(histogram.type, 'histogram');
assert.ok(Array.isArray(histogram.bins));
assert.equal(histogram.bins.length, 4);
testClient.drain(done);
});
});
it("should expose a filtered histogram with no results", function (done) {
var params = {
filters: {
dataviews: { test_histogram: { max: -1 } }
}
};
var testClient = new TestClient(overviewsMapConfig);
testClient.getDataview('test_histogram', params, function (err, histogram) {
if (err) {
return done(err);
}
assert.ok(histogram);
assert.equal(histogram.type, 'histogram');
assert.ok(Array.isArray(histogram.bins));
assert.equal(histogram.bins.length, 0);
testClient.drain(done);
});
});
it("should expose a filtered date histogram with no results", function (done) {
// This most likely works because the overviews will pass
// the responsibility to the normal dataviews.
var params = {
filters: {
dataviews: { test_histogram_date: { max: -1 } }
}
};
var testClient = new TestClient(overviewsMapConfig);
testClient.getDataview('test_histogram_date', params, function (err, histogram) {
if (err) {
return done(err);
}
assert.ok(histogram);
assert.equal(histogram.type, 'histogram');
assert.ok(Array.isArray(histogram.bins));
assert.equal(histogram.bins.length, 0);
testClient.drain(done);
});
});
});
describe('category', function () {
var params = {

131
test/acceptance/multilayer.js
View File

@ -5,6 +5,9 @@ var step = require('step');
var strftime = require('strftime');
var redis_stats_db = 5;
var mapnik = require('windshaft').mapnik;
var semver = require('semver');
var helper = require(__dirname + '/../support/test_helper');
var LayergroupToken = require('../support/layergroup-token');
@ -574,7 +577,7 @@ describe(suiteName, function() {
if ( err ) {
return done(err);
}
// trip epoch
// strip epoch
expected_token = expected_token.split(':')[0];
keysToDelete['map_cfg|' + expected_token] = 0;
keysToDelete['user:localhost:mapviews:global'] = 5;
@ -973,70 +976,72 @@ describe(suiteName, function() {
});
// See https://github.com/CartoDB/Windshaft-cartodb/issues/93
it("accepts unused directives", function(done) {
var layergroup = {
version: '1.0.0',
layers: [
{ options: {
sql: "select 'SRID=3857;POINT(0 0)'::geometry as the_geom_webmercator",
cartocss: '#layer { point-transform:"scale(20)"; }',
cartocss_version: '2.0.1'
} }
]
};
var expected_token; // = "e34dd7e235138a062f8ba7ad051aa3a7";
step(
function do_post()
{
var next = this;
assert.response(server, {
url: layergroup_url,
method: 'POST',
headers: {host: 'localhost', 'Content-Type': 'application/json' },
data: JSON.stringify(layergroup)
}, {}, function(res) {
assert.equal(res.statusCode, 200, res.body);
var parsedBody = JSON.parse(res.body);
if ( expected_token ) {
assert.equal(parsedBody.layergroupid, expected_token + ':' + expected_last_updated_epoch);
assert.equal(res.headers['x-layergroup-id'], parsedBody.layergroupid);
}
else {
var token_components = parsedBody.layergroupid.split(':');
expected_token = token_components[0];
expected_last_updated_epoch = token_components[1];
}
next(null, res);
});
},
function do_get_tile(err)
{
assert.ifError(err);
var next = this;
assert.response(server, {
url: layergroup_url + "/" + expected_token + ':cb0/0/0/0.png',
method: 'GET',
headers: {host: 'localhost' },
encoding: 'binary'
}, {}, function(res) {
assert.equal(res.statusCode, 200, res.body);
assert.equal(res.headers['content-type'], "image/png");
assert.imageBufferIsSimilarToFile(res.body, windshaft_fixtures + '/test_default_mapnik_point.png',
IMAGE_EQUALS_TOLERANCE_PER_MIL, function(err/*, similarity*/) {
next(err);
if (semver.satisfies(mapnik.versions.mapnik, '2.3.x')) {
it("accepts unused directives", function(done) {
var layergroup = {
version: '1.0.0',
layers: [
{ options: {
sql: "select 'SRID=3857;POINT(0 0)'::geometry as the_geom_webmercator",
cartocss: '#layer { point-transform:"scale(20)"; }',
cartocss_version: '2.0.1'
} }
]
};
var expected_token; // = "e34dd7e235138a062f8ba7ad051aa3a7";
step(
function do_post()
{
var next = this;
assert.response(server, {
url: layergroup_url,
method: 'POST',
headers: {host: 'localhost', 'Content-Type': 'application/json' },
data: JSON.stringify(layergroup)
}, {}, function(res) {
assert.equal(res.statusCode, 200, res.body);
var parsedBody = JSON.parse(res.body);
if ( expected_token ) {
assert.equal(parsedBody.layergroupid, expected_token + ':' + expected_last_updated_epoch);
assert.equal(res.headers['x-layergroup-id'], parsedBody.layergroupid);
}
);
});
},
function finish(err) {
keysToDelete['user:localhost:mapviews:global'] = 5;
keysToDelete['map_cfg|' + expected_token] = 0;
done(err);
}
);
});
else {
var token_components = parsedBody.layergroupid.split(':');
expected_token = token_components[0];
expected_last_updated_epoch = token_components[1];
}
next(null, res);
});
},
function do_get_tile(err)
{
assert.ifError(err);
var next = this;
assert.response(server, {
url: layergroup_url + "/" + expected_token + ':cb0/0/0/0.png',
method: 'GET',
headers: {host: 'localhost' },
encoding: 'binary'
}, {}, function(res) {
assert.equal(res.statusCode, 200, res.body);
assert.equal(res.headers['content-type'], "image/png");
assert.imageBufferIsSimilarToFile(res.body, windshaft_fixtures + '/test_default_mapnik_point.png',
IMAGE_EQUALS_TOLERANCE_PER_MIL, function(err/*, similarity*/) {
next(err);
}
);
});
},
function finish(err) {
keysToDelete['user:localhost:mapviews:global'] = 5;
keysToDelete['map_cfg|' + expected_token] = 0;
done(err);
}
);
});
}
// See https://github.com/CartoDB/Windshaft-cartodb/issues/91
// and https://github.com/CartoDB/Windshaft-cartodb/issues/38
it("tiles for private tables can be fetched with api_key", function(done) {

4
test/acceptance/multilayer_server.js
View File

@ -228,7 +228,9 @@ describe('tests from old api translated to multilayer', function() {
},
function(res) {
var parsed = JSON.parse(res.body);
assert.deepEqual(parsed.errors, [ 'Unexpected token W' ]);
assert.ok(parsed.errors);
assert.equal(parsed.errors.length, 1);
assert.ok(parsed.errors[0].match(/^Unexpected token W/));
done();
}

4
test/acceptance/ported/multilayer_error_cases.js
View File

@ -419,7 +419,9 @@ describe('multilayer error cases', function() {
},
function(res) {
var parsedBody = JSON.parse(res.body);
assert.deepEqual(parsedBody, { errors: ['SyntaxError: Unexpected token {'] });
assert.ok(parsedBody.errors);
assert.equal(parsedBody.errors.length, 1);
assert.ok(parsedBody.errors[0].match(/^SyntaxError: Unexpected token {/));
done();
}
);

96
test/acceptance/ported/server_gettile.js
View File

@ -139,13 +139,15 @@ describe('server_gettile', function() {
13, 4011, 3088, imageCompareFn('test_table_13_4011_3088_styled_black.png', done));
});
// See http://github.com/CartoDB/Windshaft/issues/99
it("unused directives are tolerated", function(done){
var style = "#test_table{point-transform: 'scale(100)';}";
var sql = "SELECT 1 as cartodb_id, 'SRID=4326;POINT(0 0)'::geometry as the_geom";
testClient.getTile(testClient.singleLayerMapConfig(sql, style), 0, 0, 0,
imageCompareFn('test_default_mapnik_point.png', done));
});
if ( semver.satisfies(mapnik.versions.mapnik, '2.3.x') ) {
// See http://github.com/CartoDB/Windshaft/issues/99
it("unused directives are tolerated", function(done){
var style = "#test_table{point-transform: 'scale(100)';}";
var sql = "SELECT 1 as cartodb_id, 'SRID=4326;POINT(0 0)'::geometry as the_geom";
testClient.getTile(testClient.singleLayerMapConfig(sql, style), 0, 0, 0,
imageCompareFn('test_default_mapnik_point.png', done));
});
}
// See http://github.com/CartoDB/Windshaft/issues/100
var test_strictness = function(done) {
@ -168,51 +170,53 @@ describe('server_gettile', function() {
// Strictness handling changed in 2.3.x, possibly a bug: see http://github.com/mapnik/mapnik/issues/2301
it.skip('[skipped due to http://github.com/mapnik/mapnik/issues/2301]' + test_strict_lbl, test_strictness);
}
else {
else if (!semver.satisfies(mapnik.versions.mapnik, '3.0.x')) {
it(test_strict_lbl, test_strictness);
}
it('high cpu regression with mapnik <2.3.x', function(done) {
var sql = [
"SELECT 'my polygon name here' AS name,",
"st_envelope(st_buffer(st_transform(",
"st_setsrid(st_makepoint(-26.6592894004,49.7990296995),4326),3857),10000000)) AS the_geom",
"FROM generate_series(-6,6) x",
"UNION ALL",
"SELECT 'my marker name here' AS name,",
" st_transform(st_setsrid(st_makepoint(49.6042060319,-49.0522997372),4326),3857) AS the_geom",
"FROM generate_series(-6,6) x"
].join(' ');
if ( semver.satisfies(mapnik.versions.mapnik, '2.3.x') ) {
var style = [
'#test_table {marker-fill:#ff7;',
' marker-max-error:0.447492761618;',
' marker-line-opacity:0.659371340628;',
' marker-allow-overlap:true;',
' polygon-fill:green;',
' marker-spacing:0.0;',
' marker-width:4.0;',
' marker-height:18.0;',
' marker-opacity:0.942312062822;',
' line-color:green;',
' line-gamma:0.945973211092;',
' line-cap:square;',
' polygon-opacity:0.12576055992;',
' marker-type:arrow;',
' polygon-gamma:0.46354913107;',
' line-dasharray:33,23;',
' line-join:bevel;',
' marker-placement:line;',
' line-width:1.0;',
' marker-line-color:#ff7;',
' line-opacity:0.39403752154;',
' marker-line-width:3.0;',
'}'
].join('');
it('high cpu regression with mapnik <2.3.x', function(done) {
var sql = [
"SELECT 'my polygon name here' AS name,",
"st_envelope(st_buffer(st_transform(",
"st_setsrid(st_makepoint(-26.6592894004,49.7990296995),4326),3857),10000000)) AS the_geom",
"FROM generate_series(-6,6) x",
"UNION ALL",
"SELECT 'my marker name here' AS name,",
" st_transform(st_setsrid(st_makepoint(49.6042060319,-49.0522997372),4326),3857) AS the_geom",
"FROM generate_series(-6,6) x"
].join(' ');
testClient.getTile(testClient.singleLayerMapConfig(sql, style), 13, 4011, 3088, done);
});
var style = [
'#test_table {marker-fill:#ff7;',
' marker-max-error:0.447492761618;',
' marker-line-opacity:0.659371340628;',
' marker-allow-overlap:true;',
' polygon-fill:green;',
' marker-spacing:0.0;',
' marker-width:4.0;',
' marker-height:18.0;',
' marker-opacity:0.942312062822;',
' line-color:green;',
' line-gamma:0.945973211092;',
' line-cap:square;',
' polygon-opacity:0.12576055992;',
' marker-type:arrow;',
' polygon-gamma:0.46354913107;',
' line-dasharray:33,23;',
' line-join:bevel;',
' marker-placement:line;',
' line-width:1.0;',
' marker-line-color:#ff7;',
' line-opacity:0.39403752154;',
' marker-line-width:3.0;',
'}'
].join('');
testClient.getTile(testClient.singleLayerMapConfig(sql, style), 13, 4011, 3088, done);
});
}
// https://github.com/CartoDB/Windshaft-cartodb/issues/316
it('should return errors with better formatting', function(done) {
var mapConfig = {

2
test/acceptance/ported/torque.js
View File

@ -370,7 +370,7 @@ describe('torque', function() {
{
var next = this;
assert.response(server, {
url: '/database/windshaft_test/layergroup?dbport=1234567',
url: '/database/windshaft_test/layergroup?dbport=54777',
method: 'POST',
headers: {'Content-Type': 'application/json' },
data: JSON.stringify(mapconfig)

31
test/acceptance/widgets/ported/histogram.js
View File

@ -304,6 +304,37 @@ describe('widgets', function() {
});
});
it('can use a datetime filtered column with no results', function(done) {
this.testClient = new TestClient(histogramsMapConfig({
updated_at: {
type: 'histogram',
options: {
column: 'updated_at'
}
}
}));
var params = {
own_filter: 1,
filters: {
layers: [{
updated_at: {
// this will remove all results
max: -1
}
}]
}
};
this.testClient.getWidget('updated_at', params, function (err, res, histogram) {
assert.ok(!err, err);
assert.ok(histogram);
assert.equal(histogram.type, 'histogram');
assert.equal(histogram.bins.length, 0);
done();
});
});
it('can getTile with datetime filtered column', function(done) {
this.testClient = new TestClient(histogramsMapConfig({
updated_at: {

11
test/integration/mapconfig_named_layers_datasource.js
View File

@ -145,7 +145,7 @@ describe('named_layers datasources', function() {
assert.equal(layers[0].type, 'cartodb');
assert.equal(layers[0].options.sql, wadusSql);
assert.equal(datasource.getLayerDatasource(0), undefined);
assert.deepEqual(datasource.getLayerDatasource(0), {});
done();
}
@ -160,7 +160,7 @@ describe('named_layers datasources', function() {
assert.equal(layers[0].type, 'cartodb');
assert.equal(layers[0].options.sql, wadusSql);
assert.equal(datasource.getLayerDatasource(0), undefined);
assert.deepEqual(datasource.getLayerDatasource(0), {});
assert.equal(layers[1].type, 'cartodb');
assert.equal(layers[1].options.sql, wadusTemplateSql);
@ -181,7 +181,7 @@ describe('named_layers datasources', function() {
assert.equal(layers[0].type, 'cartodb');
assert.equal(layers[0].options.sql, wadusSql);
assert.equal(datasource.getLayerDatasource(0), undefined);
assert.deepEqual(datasource.getLayerDatasource(0), {});
assert.equal(layers[1].type, 'mapnik');
assert.equal(layers[1].options.sql, wadusMapnikSql);
@ -263,7 +263,7 @@ describe('named_layers datasources', function() {
assert.equal(layers[3].type, 'cartodb');
assert.equal(layers[3].options.sql, wadusSql);
assert.equal(datasource.getLayerDatasource(3), undefined);
assert.deepEqual(datasource.getLayerDatasource(3), {});
assert.equal(layers[4].type, 'cartodb');
assert.equal(layers[4].options.sql, wadusTemplateSql);
@ -273,7 +273,7 @@ describe('named_layers datasources', function() {
assert.equal(layers[5].type, 'cartodb');
assert.equal(layers[5].options.sql, wadusSql);
assert.equal(datasource.getLayerDatasource(5), undefined);
assert.deepEqual(datasource.getLayerDatasource(5), {});
assert.equal(layers[6].type, 'mapnik');
assert.equal(layers[6].options.sql, wadusMapnikSql);
@ -298,6 +298,7 @@ describe('named_layers datasources', function() {
var context = {};
mapConfigNamedLayersAdapter.getMapConfig(username, testScenario.config, params, context,
function(err, mapConfig) {
assert.ifError(err);
testScenario.test(err, mapConfig.layers, context.datasource, done);
}
);

132
test/unit/cartodb/model/resource-locator.test.js Normal file
View File

@ -0,0 +1,132 @@
require('../../../support/test_helper');
var assert = require('../../../support/assert');
var ResourceLocator = require('../../../../lib/cartodb/models/resource-locator');
describe('ResourceLocator.getUrls', function() {
var USERNAME = 'username';
var RESOURCE = 'wadus';
var HTTP_SUBDOMAINS = ['1', '2', '3', '4'];
var HTTPS_SUBDOMAINS = ['a', 'b', 'c', 'd'];
it('should return default urls when no serverMetadata is in environment', function() {
var resourceLocator = new ResourceLocator({});
var urls = resourceLocator.getUrls(USERNAME, RESOURCE);
assert.ok(urls);
});
var BASIC_ENVIRONMENT = {
serverMetadata: {
cdn_url: {
http: 'cdn.carto.com',
https: 'cdn.ssl.carto.com'
}
}
};
it('should return default urls when basic http and https domains are provided', function() {
var resourceLocator = new ResourceLocator(BASIC_ENVIRONMENT);
var urls = resourceLocator.getUrls(USERNAME, RESOURCE);
assert.ok(urls);
assert.equal(urls.http, ['http://cdn.carto.com', USERNAME, 'api/v1/map', RESOURCE].join('/'));
assert.equal(urls.https, ['https://cdn.ssl.carto.com', USERNAME, 'api/v1/map', RESOURCE].join('/'));
});
var RESOURCE_TEMPLATES_ENVIRONMENT = {
serverMetadata: {
cdn_url: {
http: 'cdn.carto.com',
https: 'cdn.ssl.carto.com'
}
},
resources_url_templates: {
http: 'http://{{=it.user}}.localhost.lan/api/v1/map',
https: 'https://{{=it.user}}.ssl.localhost.lan/api/v1/map'
}
};
it('resources_url_templates should take precedence over http and https domains', function() {
var resourceLocator = new ResourceLocator(RESOURCE_TEMPLATES_ENVIRONMENT);
var urls = resourceLocator.getUrls(USERNAME, RESOURCE);
assert.ok(urls);
assert.equal(urls.http, ['http://' + USERNAME + '.localhost.lan', 'api/v1/map', RESOURCE].join('/'));
assert.equal(urls.https, ['https://' + USERNAME + '.ssl.localhost.lan', 'api/v1/map', RESOURCE].join('/'));
});
var CDN_TEMPLATES_ENVIRONMENT = {
serverMetadata: {
cdn_url: {
http: 'cdn.carto.com',
https: 'cdn.ssl.carto.com',
templates: {
http: {
url: "http://{s}.cdn.carto.com",
subdomains: HTTP_SUBDOMAINS
},
https: {
url: "https://cdn_{s}.ssl.cdn.carto.com",
subdomains: HTTPS_SUBDOMAINS
}
}
}
}
};
it('cdn_url templates should take precedence over http and https domains', function() {
var resourceLocator = new ResourceLocator(CDN_TEMPLATES_ENVIRONMENT);
var urls = resourceLocator.getUrls(USERNAME, RESOURCE);
assert.ok(urls);
var httpSubdomain = ResourceLocator.subdomain(HTTP_SUBDOMAINS, RESOURCE);
var httpsSubdomain = ResourceLocator.subdomain(HTTPS_SUBDOMAINS, RESOURCE);
assert.equal(
urls.http,
['http://' + httpSubdomain + '.cdn.carto.com', USERNAME, 'api/v1/map', RESOURCE].join('/')
);
assert.equal(
urls.https,
['https://cdn_' + httpsSubdomain + '.ssl.cdn.carto.com', USERNAME, 'api/v1/map', RESOURCE].join('/')
);
});
var CDN_URL_AND_RESOURCE_TEMPLATES_ENVIRONMENT = {
serverMetadata: {
cdn_url: {
http: 'cdn.carto.com',
https: 'cdn.ssl.carto.com',
templates: {
http: {
url: "http://{s}.cdn.carto.com",
subdomains: HTTP_SUBDOMAINS
},
https: {
url: "https://cdn_{s}.ssl.cdn.carto.com",
subdomains: HTTPS_SUBDOMAINS
}
}
}
},
resources_url_templates: {
http: 'http://{{=it.cdn_url}}/u/{{=it.user}}/api/v1/map',
https: 'https://{{=it.cdn_url}}/u/{{=it.user}}/api/v1/map'
}
};
it('should mix cdn_url templates and resources_url_templates', function() {
var resourceLocator = new ResourceLocator(CDN_URL_AND_RESOURCE_TEMPLATES_ENVIRONMENT);
var urls = resourceLocator.getUrls(USERNAME, RESOURCE);
assert.ok(urls);
var httpSubdomain = ResourceLocator.subdomain(HTTP_SUBDOMAINS, RESOURCE);
var httpsSubdomain = ResourceLocator.subdomain(HTTPS_SUBDOMAINS, RESOURCE);
assert.equal(
urls.http,
['http://' + httpSubdomain + '.cdn.carto.com', 'u', USERNAME, 'api/v1/map', RESOURCE].join('/')
);
assert.equal(
urls.https,
['https://cdn_' + httpsSubdomain + '.ssl.cdn.carto.com', 'u', USERNAME, 'api/v1/map', RESOURCE].join('/')
);
});
});

2
test/unit/cartodb/ported/windshaft_server.test.js
View File

@ -19,7 +19,7 @@ describe('windshaft', function() {
it('can spawn a new server on the global listen port', function(done){
var ws = cartodbServer(serverOptions);
var server = ws.listen(global.environment.windshaft_port, function() {
var server = ws.listen(global.environment.port, function() {
assert.ok(ws);
server.close(done); /* allow proper tear down */
});

2385
yarn.lock Normal file
View File

File diff suppressed because it is too large Load Diff