cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #637 from CartoDB/no-makefile

Browse Source 
No makefile
This commit is contained in:
Daniel G. Aubert 2019-12-30 11:19:27 +01:00 committed by GitHub
commit c621e10d73
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
18 changed files with 1411 additions and 770 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -13,3 +13,4 @@ coverage/
npm-debug.log
log/*.log
yarn.lock
.nyc_output

2
.travis.yml
View File

@ -9,4 +9,4 @@ env:
services:
- docker
before_install: docker pull ${DOCKER_IMAGE}
script: npm run docker-test -- ${DOCKER_IMAGE} ${NODE_VERSION}
script: npm run test:docker

11
CONTRIBUTING.md
View File

@ -1,11 +0,0 @@
Contributing
---
The issue tracker is at [github.com/CartoDB/CartoDB-SQL-API](https://github.com/CartoDB/CartoDB-SQL-API).
We love pull requests from everyone, see [Contributing to Open Source on GitHub](https://guides.github.com/activities/contributing-to-open-source/#contributing).
## Submitting Contributions
* You will need to sign a Contributor License Agreement (CLA) before making a submission. [Learn more here](https://carto.com/contributions).

17
HOWTO_RELEASE
View File

@ -1,17 +0,0 @@
1. Test (make clean all check), fix if broken before proceeding
2. Ensure proper version in package.json and package-lock.json
3. Ensure NEWS section exists for the new version, review it, add release date
4. Commit package.json, package.json and, NEWS
5. git tag -a Major.Minor.Patch # use NEWS section as content
6. Stub NEWS/package for next version
Versions:
Bugfix releases increment Patch component of version.
Feature releases increment Minor and set Patch to zero.
If backward compatibility is broken, increment Major and
set to zero Minor and Patch.
Branches named 'b<Major>.<Minor>' are kept for any critical
fix that might need to be shipped before next feature release
is ready.

15
HOWTO_RELEASE.md Normal file
View File

@ -0,0 +1,15 @@
# How to release
1. Test (npm test), fix if broken before proceeding.
2. Ensure proper version in `package.json` and `package-lock.json`.
3. Ensure NEWS section exists for the new version, review it, add release date.
4. Commit `package.json`, `package-lock.json`, NEWS.
5. Run `git tag -a Major.Minor.Patch`. Use NEWS section as content.
6. Stub NEWS/package for next version.
## Versions
* Bugfix releases increment Patch component of version.
* Feature releases increment Minor and set Patch to zero.
* If backward compatibility is broken, increment Major and set to zero Minor and Patch.
* Branches named 'b<Major>.<Minor>' are kept for any critical fix that might need to be shipped before next feature release is ready.

47
Makefile
View File

@ -1,47 +0,0 @@
SHELL=/bin/bash
all:
npm install
clean:
rm -rf node_modules/*
check:
npm test
eslint:
@echo "***eslint***"
@./node_modules/.bin/eslint lib/**/*.js test/**/*.js app.js
TEST_SUITE := $(shell find test/{unit,integration,acceptance} -name "*.js")
TEST_SUITE_UNIT := $(shell find test/unit -name "*.js")
TEST_SUITE_INTEGRATION := $(shell find test/integration -name "*.js")
TEST_SUITE_ACCEPTANCE := $(shell find test/acceptance -name "*.js")
TEST_SUITE_BATCH := $(shell find test/*/batch -name "*.js")
test:
@echo "***tests***"
@$(SHELL) test/run_tests.sh ${RUNTESTFLAGS} $(TEST_SUITE)
test-unit:
@echo "***unit tests***"
@$(SHELL) test/run_tests.sh ${RUNTESTFLAGS} $(TEST_SUITE_UNIT)
test-integration:
@echo "***integration tests***"
@$(SHELL) test/run_tests.sh ${RUNTESTFLAGS} $(TEST_SUITE_INTEGRATION)
test-acceptance:
@echo "***acceptance tests***"
@$(SHELL) test/run_tests.sh ${RUNTESTFLAGS} $(TEST_SUITE_ACCEPTANCE)
test-batch:
@echo "***batch queries tests***"
@$(SHELL) test/run_tests.sh ${RUNTESTFLAGS} $(TEST_SUITE_BATCH)
test-all: test eslint
coverage:
@RUNTESTFLAGS=--with-coverage make test
.PHONY: test coverage

12
NEWS.md
View File

@ -3,6 +3,18 @@
## 5.0.1
Released 2019-mm-dd
- Stop using two different tools for package management, testing, and any other developer workflow.
- Removes Makefile and related bash scripts
- Use npm scripts as the only tool for testing, CI and linting.
- Simplified CI configuration.
- Improved documentation:
- Centralized several documents into README.md
- Remove outdated sections
- Update old sections
- Added missing sections.
- Remove deprecated coverage tool istanbul, using nyc instead.
## 5.0.0
Released 2019-11-13

136
README.md
View File

@ -1,72 +1,112 @@
SQL API for carto.com
========================
# CartoDB-SQL-API [![Build Status](https://travis-ci.org/CartoDB/CartoDB-SQL-API.svg?branch=master)](https://travis-ci.org/CartoDB/CartoDB-SQL-API)
[![Build Status](https://travis-ci.org/CartoDB/CartoDB-SQL-API.png?branch=master)](https://travis-ci.org/CartoDB/CartoDB-SQL-API)
The [`CARTO’s SQL API`](https://carto.com/developers/sql-api/) allows you to interact with your data inside CARTO, as if you were running SQL statements against a normal database.
Provides a node.js based API for running SQL queries against CartoDB.
* Run queries with fine-grained permissions through [`Auth API`](https://carto.com/developers/auth-api/).
* Export data in multiple geospatial formats (CVS, geopackage, KML, SHP, spatialite, geojson, topojson, etc).
* Schedule jobs using [`Batch Queries`](https://carto.com/developers/sql-api/guides/batch-queries/).
* [`Copy queries`](https://carto.com/developers/sql-api/guides/copy-queries/) allows you to use the [`PostgreSQL copy command`](https://www.postgresql.org/docs/10/static/sql-copy.html) for efficient streaming of data to and from CARTO.
* Users are authenticated over OAuth or via an API KEY.
* Authenticated requests to this API should always be made over SSL.
## Build
Requirements:
## Requirements
* Node 10.x
* npm 6.x
* PostgreSQL >= 10.0
* PostGIS >= 2.4
* CARTO Postgres Extension >= 0.24.1
* Redis >= 4
* [`Node 10.x (npm 6.x)`](https://nodejs.org/dist/latest-v10.x/)
* [`PostgreSQL >= 10.0`](https://www.postgresql.org/download/)
* [`PostGIS >= 2.4`](https://postgis.net/install/)
* [`CARTO Postgres Extension >= 0.24.1`](https://github.com/CartoDB/cartodb-postgresql)
* [`Redis >= 4`](https://redis.io/download)
* GDAL `1.11.0` (bin utils). See [installing GDAL](http://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries)
* zip commandline tool.
* C++11 (to build internal dependencies if needed)
* `C++11` to build internal dependencies. When there's no pre-built binaries for your OS/architecture distribution.
Optional:
## Install dependencies
* [`Varnish`](http://www.varnish-cache.org)
* [`Statsd`](https://github.com/statsd/statsd)
```sh
### PostGIS setup
A `template_postgis` database is expected. One can be set up with
```shell
$ createdb --owner postgres --template template0 template_postgis
$ psql -d template_postgis -c 'CREATE EXTENSION postgis;'
```
### Install
To fetch and build all node-based dependencies, run:
```shell
$ npm install
```
## Usage
### Run
Create the `./config/environments/<env>.js` file (there are `.example` files to start from). Look at `./lib/server-options.js` for more on config.
Create and edit config/environments/<environment>.js from .js.example files.
You may find the ./configure script useful to make an edited copy for you,
see ```./configure --help``` for a list of supported switches.
Make sure redis is running and knows about active cartodb user.
Make sure your PostgreSQL server is running, is accessible on
the host and port specified in the <environment> file, has
a 'publicuser' role (or whatever you set ``db_pubuser`` configuration
directive to) and trusts user authentication from localhost
connections.
```sh
node app.js <environment>
```shell
$ node app.js <env>
```
Supported <environment> values are development, test, production
Where `<env>` is the name of a configuration file under `./config/environments/`.
See doc/API.md for API documentation.
For examples of use, see under test/.
### Test
## Tests
Run with:
```sh
```shell
$ npm test
```
If any issue arise see test/README.md
### Coverage
Note that the environment should be set to ensure the default
PostgreSQL user is superuser (PGUSER=postgres make check).
```shell
$ npm run cover
```
Contributing
---
Open `./coverage/lcov-report/index.html`.
See [CONTRIBUTING.md](CONTRIBUTING.md).
### Docker support
We provide docker images just for testing and continuous integration purposes:
* [`nodejs-xenial-pg1121`](https://hub.docker.com/r/carto/nodejs-xenial-pg1121/tags)
* [`nodejs-xenial-pg101`](https://hub.docker.com/r/carto/nodejs-xenial-pg101/tags)
You can find instructions to install Docker, download, and update images [here](https://github.com/CartoDB/Windshaft-cartodb/blob/master/docker/reference.md).
### Useful `npm` scripts
Run test in a docker image with a specific Node.js version:
```shell
$ DOCKER_IMAGE=<docker-image-tag> NODE_VERSION=<nodejs-version> npm run test:docker
```
Where:
* `<docker-image-tag>`: the tag of required docker image, e.g. `carto/nodejs-xenial-pg1121:latest`
* `<nodejs-version>`: the Node.js version, e.g. `10.15.1`
In case you need to debug:
```shell
$ DOCKER_IMAGE=<docker-image-tag> npm run docker:bash
```
## Documentation
You can find an overview, guides, full reference, and support in [`CARTO's developer center`](https://carto.com/developers/sql-api/). The [docs directory](https://github.com/CartoDB/CartoDB-SQL-API/tree/master/docs) contains different documentation resources, from a higher level to more detailed ones.
## Contributing
* The issue tracker: [`Github`](https://github.com/CartoDB/CartoDB-SQL-API/issues).
* We love Pull Requests from everyone, see [contributing to Open Source on GitHub](https://guides.github.com/activities/contributing-to-open-source/#contributing).
* You'll need to sign a Contributor License Agreement (CLA) before submitting a Pull Request. [Learn more here](https://carto.com/contributions).
## Versioning
We follow [`SemVer`](http://semver.org/) for versioning. For available versions, see the [tags on this repository](https://github.com/CartoDB/CartoDB-SQL-API/tags).
## License
This project is licensed under the BSD 3-clause "New" or "Revised" License. See the [LICENSE](LICENSE) file for details.

36
configure vendored
View File

@ -1,36 +0,0 @@
#!/bin/sh
usage() {
echo "Usage: $0 [OPTION]"
echo
echo "Configuration:"
echo " --help display this help and exit"
echo " --with-pgport=NUM access PostgreSQL server on TCP port NUM"
}
PGPORT=5432
while test -n "$1"; do
case "$1" in
--help|-h)
usage
exit 0
;;
--with-pgport=*)
PGPORT=`echo "$1" | cut -d= -f2`
;;
*)
echo "Unused option '$1'" >&2
;;
esac
shift
done
echo "PGPORT: $PGPORT"
# TODO: allow specifying configuration settings !
for f in config/environments/*.example; do
o=`dirname "$f"`/`basename "$f" .example`
echo "Writing $o"
sed "s/\( *module.exports.db_port[ \t]*= *'\?\)[^';]*\('\?;\)/\1$PGPORT\2/" < "$f" > "$o"
done

13
docker-bash.sh
View File

@ -1,13 +0,0 @@
#!/bin/bash
echo "*********************"
echo "To install Node.js, run:"
echo "/src/nodejs-install.sh"
echo "Use NODEJS_VERSION env var to select the Node.js version"
echo " "
echo "To start postgres, run:"
echo "/etc/init.d/postgresql start"
echo "*********************"
echo " "
docker run -it -v `pwd`:/srv carto/nodejs-xenial-pg101:latest bash

4
docker-test.sh
View File

@ -1,4 +0,0 @@
#!/bin/bash
docker run -e "NODEJS_VERSION=${2}" -v `pwd`:/srv ${1} bash test/run_tests_docker.sh && \
docker ps --filter status=dead --filter status=exited -aq | xargs docker rm -v

12
test/run_tests_docker.sh → docker/scripts/test-setup.sh
View File

@ -4,21 +4,11 @@
source /src/nodejs-install.sh
# Configure
./configure
# Install cartodb-postgresql extension
git clone https://github.com/CartoDB/cartodb-postgresql.git
cd cartodb-postgresql && make && make install && cd ..
echo "Node.js version: "
node -v
cp config/environments/test.js.example config/environments/test.js
echo "npm version:"
npm -v
# install dependencies
npm ci
npm ls
npm test

1273
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

38
package.json
View File

@ -1,22 +1,30 @@
{
"private": true,
"name": "cartodb_sql_api",
"description": "high speed SQL api for cartodb",
"keywords": [
"cartodb"
],
"name": "cartodb-sql-api",
"version": "5.0.1",
"description": "High speed SQL API for CARTO",
"keywords": [
"carto",
"sql",
"batch",
"import",
"export",
"geospatial"
],
"url": "https://github.com/CartoDB/CartoDB-SQL-API",
"license": "BSD-3-Clause",
"repository": {
"type": "git",
"url": "git://github.com/CartoDB/CartoDB-SQL-API.git"
},
"license": "BSD-3-Clause",
"author": "Vizzuality <contact@vizzuality.com> (http://vizzuality.com)",
"author": "CARTO (https://carto.com)",
"contributors": [
"Simon Tokumine <simon@vizzuality.com>",
"Sandro Santilli <strk@vizzuality.com>",
"Raúl Ochoa <rochoa@carto.com>",
"Daniel García Aubert <dgaubert@carto.com>"
],
"main": "app.js",
"dependencies": {
"@carto/fqdn-sync": "0.2.2",
"basic-auth": "^2.0.0",
@ -54,10 +62,10 @@
"eslint-plugin-node": "^10.0.0",
"eslint-plugin-promise": "^4.2.1",
"eslint-plugin-standard": "^4.0.1",
"istanbul": "0.4.5",
"libxmljs": "0.19.5",
"mocha": "^5.2.0",
"mockdate": "^2.0.2",
"nyc": "^15.0.0",
"shapefile": "0.3.0",
"sqlite3": "^4.0.6",
"zipfile": "0.5.12"
@ -65,11 +73,15 @@
"scripts": {
"lint:fix": "eslint --fix app.js \"lib/**/*.js\" \"test/**/*.js\"",
"lint": "eslint app.js \"lib/**/*.js\" \"test/**/*.js\"",
"test": "make test-all",
"test:unit": "mocha test/unit/**/*.js --exit",
"test:unit:watch": "npm run test:unit -- -w",
"docker-test": "./docker-test.sh",
"docker-bash": "./docker-bash.sh"
"pretest:setup": "npm run lint",
"test:setup": "NODE_ENV=test node test setup",
"pretest": "npm run test:setup",
"test": "NODE_ENV=test TZ='Europe/Rome' mocha -t 5000 --exit --recursive test/acceptance test/integration test/unit",
"posttest": "npm run test:teardown",
"test:teardown": "NODE_ENV=test node test teardown",
"cover": "nyc --reporter=lcov npm test",
"test:docker": "docker run -e \"NODEJS_VERSION=$NODE_VERSION\" -v `pwd`:/srv $DOCKER_IMAGE bash docker/scripts/test-setup.sh && docker ps --filter status=dead --filter status=exited -aq | xargs docker rm -v",
"docker:bash": "docker run -it -v `pwd`:/srv $DOCKER_IMAGE bash"
},
"engines": {
"node": "^10.15.1",

21
test/README.md
View File

@ -1,21 +0,0 @@
cartodb-sql-api tests
---------------------
Tests require you create a test database and set some redis keys before,
you can execute prepare_db.sh script, it will create database, users
and redis stuff for you. Be sure postgres and redis are running.
> cd test && ./prepare_db.sh
Note that "make check" from top-level dir will try to do everything
needed to prepare & run the tests.
Acceptance tests (need ctrl-C to exit)
--------------------------------------
> mocha -u tdd test/acceptance/app.test.js
> mocha -u tdd test/acceptance/app.auth.test.js
Unit tests
--------------------------------
> mocha -u tdd test/unit/*.js (or run the tests individually)

175
test/index.js Normal file
View File

@ -0,0 +1,175 @@
'use strict';
const util = require('util');
const path = require('path');
const exec = util.promisify(require('child_process').exec);
if (!process.env.NODE_ENV) {
console.error('Please set "NODE_ENV" variable, e.g.: "NODE_ENV=test"');
process.exit(1);
}
const environment = require(`../config/environments/${process.env.NODE_ENV}.js`);
const REDIS_PORT = environment.redis_port;
const REDIS_CELL_PATH = path.resolve(
process.platform === 'darwin'
? './test/support/libredis_cell.dylib'
: './test/support/libredis_cell.so'
);
const TEST_USER_ID = 1;
const TEST_USER = environment.db_user.replace('<%= user_id %>', TEST_USER_ID);
const TEST_PASSWORD = environment.db_user_pass.replace('<%= user_id %>', TEST_USER_ID);
const PUBLIC_USER = environment.db_pubuser;
const PUBLIC_USER_PASSWORD = environment.db_pubuser_pass;
const TEST_DB = environment.db_base_name.replace('<%= user_id %>', TEST_USER_ID);
const PGHOST = environment.db_host;
async function startRedis () {
await exec(`redis-server --port ${REDIS_PORT} --loadmodule ${REDIS_CELL_PATH} --logfile ${__dirname}/redis-server.log --daemonize yes`);
}
async function stopRedis () {
await exec(`redis-cli -p ${REDIS_PORT} shutdown`);
}
async function dropDatabase () {
await exec(`dropdb --if-exists ${TEST_DB}`, {
env: Object.assign({ PGUSER: 'postgres' }, process.env)
});
}
async function createDatabase () {
await exec(`createdb -T template_postgis -EUTF8 "${TEST_DB}"`, {
env: Object.assign({ PGUSER: 'postgres' }, process.env)
});
}
async function createDatabaseExtension () {
await exec(`psql -c "CREATE EXTENSION IF NOT EXISTS cartodb CASCADE;" ${TEST_DB}`, {
env: Object.assign({ PGUSER: 'postgres' }, process.env)
});
}
async function populateDatabase () {
const filenames = [
'test',
'populated_places_simple_reduced',
'py_sleep',
'quota_mock'
].map(filename => `${__dirname}/support/sql/${filename}.sql`);
const populateDatabaseCmd = `
cat ${filenames.join(' ')} |
sed -e "s/:PUBLICUSER/${PUBLIC_USER}/g" |
sed -e "s/:PUBLICPASS/${PUBLIC_USER_PASSWORD}/g" |
sed -e "s/:TESTUSER/${TEST_USER}/g" |
sed -e "s/:TESTPASS/${TEST_PASSWORD}/g" |
PGOPTIONS='--client-min-messages=WARNING' psql -q -v ON_ERROR_STOP=1 ${TEST_DB}
`;
await exec(populateDatabaseCmd, {
env: Object.assign({ PGUSER: 'postgres' }, process.env)
});
}
async function populateRedis () {
const commands = `
HMSET rails:users:vizzuality \
id ${TEST_USER_ID} \
database_name "${TEST_DB}" \
database_host "${PGHOST}" \
map_key 1234
HMSET rails:users:cartodb250user \
id ${TEST_USER_ID} \
database_name "${TEST_DB}" \
database_host "${PGHOST}" \
database_password "${TEST_PASSWORD}" \
map_key 1234
HMSET api_keys:vizzuality:1234 \
user "vizzuality" \
type "master" \
grants_sql "true" \
database_role "${TEST_USER}" \
database_password "${TEST_PASSWORD}"
HMSET api_keys:vizzuality:default_public \
user "vizzuality" \
type "default" \
grants_sql "true" \
database_role "${PUBLIC_USER}" \
database_password "${PUBLIC_USER_PASSWORD}"
HMSET api_keys:vizzuality:regular1 \
user "vizzuality" \
type "regular" \
grants_sql "true" \
database_role "regular_1" \
database_password "regular1"
HMSET api_keys:vizzuality:regular2 \
user "vizzuality" \
type "regular" \
grants_sql "true" \
database_role "regular_2" \
database_password "regular2"
HMSET api_keys:cartodb250user:1234 \
user "cartodb250user" \
type "master" \
grants_sql "true" \
database_role "${TEST_USER}" \
database_password "${TEST_PASSWORD}"
HMSET api_keys:cartodb250user:default_public \
user "cartodb250user" \
type "default" \
grants_sql "true" \
database_role "${PUBLIC_USER}" \
database_password "${PUBLIC_USER_PASSWORD}"
`;
const oauthCommands = `
HMSET rails:oauth_access_tokens:l0lPbtP68ao8NfStCiA3V3neqfM03JKhToxhUQTR \
consumer_key fZeNGv5iYayvItgDYHUbot1Ukb5rVyX6QAg8GaY2 \
consumer_secret IBLCvPEefxbIiGZhGlakYV4eM8AbVSwsHxwEYpzx \
access_token_token l0lPbtP68ao8NfStCiA3V3neqfM03JKhToxhUQTR \
access_token_secret 22zBIek567fMDEebzfnSdGe8peMFVFqAreOENaDK \
user_id ${TEST_USER_ID} \
time sometime
`;
await exec(`echo "${commands}" | redis-cli -p ${REDIS_PORT} -n 5`);
await exec(`echo "${oauthCommands}" | redis-cli -p ${REDIS_PORT} -n 3`);
}
async function main (args) {
let code = 0;
try {
switch (args[0]) {
case 'setup':
await startRedis();
await populateRedis();
await dropDatabase();
await createDatabase();
await createDatabaseExtension();
await populateDatabase();
break;
case 'teardown':
await stopRedis();
break;
default:
throw new Error('Missing "mode" argument. Valid ones: "setup" or "teardown"');
}
} catch (err) {
console.error(err);
code = 1;
} finally {
process.exit(code);
}
}
main(process.argv.slice(2));

223
test/prepare_db.sh
View File

@ -1,223 +0,0 @@
#!/bin/sh
# this script prepare database and redis instance to run acceptance test
#
# NOTE: assumes existance of a "template_postgis" loaded with
# compatible version of postgis (legacy.sql included)
PREPARE_REDIS=yes
PREPARE_PGSQL=yes
while [ -n "$1" ]; do
if test "$1" = "--skip-pg"; then
PREPARE_PGSQL=no
shift; continue
elif test "$1" = "--skip-redis"; then
PREPARE_REDIS=no
shift; continue
fi
done
die() {
msg=$1
echo "${msg}" >&2
exit 1
}
# This is where postgresql connection parameters are read from
TESTENV=../config/environments/test.js
# Extract postgres configuration
PGHOST=`node -e "console.log(require('${TESTENV}').db_host || '')"`
echo "PGHOST: [$PGHOST]"
PGPORT=`node -e "console.log(require('${TESTENV}').db_port || '')"`
echo "PGPORT: [$PGPORT]"
PUBLICUSER=`node -e "console.log(require('${TESTENV}').db_pubuser || 'xxx')"`
PUBLICPASS=`node -e "console.log(require('${TESTENV}').db_pubuser_pass || 'xxx')"`
echo "PUBLICUSER: [${PUBLICUSER}]"
echo "PUBLICPASS: [${PUBLICPASS}]"
TESTUSERID=1
TESTUSER=`node -e "console.log(require('${TESTENV}').db_user || '')"`
if test -z "$TESTUSER"; then
echo "Missing db_user from ${TESTENV}" >&2
exit 1
fi
TESTUSER=`echo ${TESTUSER} | sed "s/<%= user_id %>/${TESTUSERID}/"`
echo "TESTUSER: [${TESTUSER}]"
TESTPASS=`node -e "console.log(require('${TESTENV}').db_user_pass || '')"`
TESTPASS=`echo ${TESTPASS} | sed "s/<%= user_id %>/${TESTUSERID}/"`
echo "TESTPASS: [${TESTPASS}]"
TEST_DB=`node -e "console.log(require('${TESTENV}').db_base_name || '')"`
if test -z "$TEST_DB"; then
echo "Missing db_base_name from ${TESTENV}" >&2
exit 1
fi
TEST_DB=`echo ${TEST_DB} | sed "s/<%= user_id %>/${TESTUSERID}/"`
export PGHOST PGPORT
if test x"$PREPARE_PGSQL" = xyes; then
echo "preparing postgres..."
echo "PostgreSQL server version: `psql -A -t -c 'select version()'`"
echo "PAUSE; RESUME;" | psql pgbouncer 2>/dev/null # make sure there are no connections pgbouncer -> test_db
dropdb --if-exists ${TEST_DB} || die "Could not drop test database ${TEST_DB}"
createdb -Ttemplate_postgis -EUTF8 ${TEST_DB} || die "Could not create test database ${TEST_DB}"
psql -c 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp";' ${TEST_DB}
psql -c "CREATE EXTENSION IF NOT EXISTS cartodb CASCADE;" ${TEST_DB}
LOCAL_SQL_SCRIPTS='test populated_places_simple_reduced py_sleep quota_mock'
for i in ${LOCAL_SQL_SCRIPTS}
do
cat support/sql/${i}.sql |
sed -e 's/cartodb\./public./g' -e "s/''cartodb''/''public''/g" |
sed "s/:PUBLICUSER/${PUBLICUSER}/" |
sed "s/:PUBLICPASS/${PUBLICPASS}/" |
sed "s/:TESTUSER/${TESTUSER}/" |
sed "s/:TESTPASS/${TESTPASS}/" |
psql -q -v ON_ERROR_STOP=1 ${TEST_DB} > /dev/null || exit 1
done
fi
if test x"$PREPARE_REDIS" = xyes; then
REDIS_HOST=`node -e "console.log(require('${TESTENV}').redis_host || '127.0.0.1')"`
REDIS_PORT=`node -e "console.log(require('${TESTENV}').redis_port || '6336')"`
echo "preparing redis..."
# delete previous publicuser
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HDEL rails:users:vizzuality database_host
HDEL rails:users:vizzuality database_publicuser
EOF
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HMSET rails:users:vizzuality \
id 1 \
database_name ${TEST_DB} \
database_host ${PGHOST} \
map_key 1234
SADD rails:users:vizzuality:map_key 1235
EOF
# A user configured as with cartodb-2.5.0+
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HMSET rails:users:cartodb250user \
id ${TESTUSERID} \
database_name ${TEST_DB} \
database_host ${PGHOST} \
database_password ${TESTPASS} \
map_key 1234
SADD rails:users:cartodb250user:map_key 1234
EOF
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 3
HMSET rails:oauth_access_tokens:l0lPbtP68ao8NfStCiA3V3neqfM03JKhToxhUQTR \
consumer_key fZeNGv5iYayvItgDYHUbot1Ukb5rVyX6QAg8GaY2 \
consumer_secret IBLCvPEefxbIiGZhGlakYV4eM8AbVSwsHxwEYpzx \
access_token_token l0lPbtP68ao8NfStCiA3V3neqfM03JKhToxhUQTR \
access_token_secret 22zBIek567fMDEebzfnSdGe8peMFVFqAreOENaDK \
user_id 1 \
time sometime
EOF
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HMSET rails:users:cartofante \
id 2 \
database_name ${TEST_DB} \
database_host ${PGHOST} \
database_password test_cartodb_user_2_pass \
map_key 4321
SADD rails:users:fallback_1:map_key 4321
EOF
# delete previous jobs
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
EVAL "return redis.call('del', unpack(redis.call('keys', ARGV[1])))" 0 batch:jobs:*
EOF
# delete job queue
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
DEL batch:queues:localhost
EOF
# delete user index
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
DEL batch:users:vizzuality
EOF
# User: vizzuality
# API Key Default public
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HMSET api_keys:vizzuality:default_public \
user "vizzuality" \
type "default" \
grants_sql "true" \
database_role "testpublicuser" \
database_password "public"
EOF
# API Key Master
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HMSET api_keys:vizzuality:1234 \
user "vizzuality" \
type "master" \
grants_sql "true" \
database_role "${TESTUSER}" \
database_password "${TESTPASS}"
EOF
# API Key Regular1
cat <<EOF | redis-cli -p ${REDIS_PORT} -n 5
HMSET api_keys:vizzuality:regular1 \
user "vizzuality" \
type "regular" \
grants_sql "true" \
database_role "regular_1" \
database_password "regular1"
EOF
# API Key Regular1
cat <<EOF | redis-cli -p ${REDIS_PORT} -n 5
HMSET api_keys:vizzuality:regular2 \
user "vizzuality" \
type "regular" \
grants_sql "true" \
database_role "regular_2" \
database_password "regular2"
EOF
# User: cartodb250user
# API Key Default public
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HMSET api_keys:cartodb250user:default_public \
user "cartodb250user" \
type "default" \
grants_sql "true" \
database_role "testpublicuser" \
database_password "public"
EOF
# API Key Master
cat <<EOF | redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} -n 5
HMSET api_keys:cartodb250user:1234 \
user "cartodb250user" \
type "master" \
grants_sql "true" \
database_role "${TESTUSER}" \
database_password "${TESTPASS}"
EOF
fi
echo "ok, you can run test now"

151
test/run_tests.sh
View File

@ -1,151 +0,0 @@
#!/bin/bash
# To make output dates deterministic
export TZ='Europe/Rome'
export PGAPPNAME='cartodb_sqlapi_tester'
# In case PGUSER env variable does not exist we attempt to use `postgres`
if test x"${PGUSER}" = x; then
echo "PGUSER not found"
PGUSER=postgres
else
echo "PGUSER found = ${PGUSER}"
fi
OPT_CREATE_PGSQL=yes # create/prepare the postgresql test database
OPT_CREATE_REDIS=yes # create/prepare the redis test databases
OPT_DROP_PGSQL=yes # drop the postgreql test environment
OPT_DROP_REDIS=yes # drop the redis test environment
OPT_COVERAGE=no # run tests with coverage
cd $(dirname $0)
BASEDIR=$(pwd)
cd -
REDIS_PORT=`node -e "console.log(require('${BASEDIR}/../config/environments/test.js').redis_port)"`
export REDIS_PORT
echo "REDIS_PORT: [${REDIS_PORT}]"
cleanup() {
if test x"$OPT_DROP" = xyes; then
if test x"$PID_REDIS" = x; then
PID_REDIS=$(cat ${BASEDIR}/redis.pid)
if test x"$PID_REDIS" = x; then
echo "Could not find a test redis pid to kill it"
return;
fi
fi
echo "Cleaning up"
kill ${PID_REDIS}
fi
}
cleanup_and_exit() {
cleanup
exit
}
die() {
msg=$1
echo "${msg}" >&2
cleanup
exit 1
}
trap 'cleanup_and_exit' 1 2 3 5 9 13
while [ -n "$1" ]; do
if test "$1" = "--nodrop"; then
OPT_DROP_REDIS=no
OPT_DROP_PGSQL=no
shift
continue
elif test "$1" = "--nodrop-pg"; then
OPT_DROP_PGSQL=no
shift
continue
elif test "$1" = "--nodrop-redis"; then
OPT_DROP_REDIS=no
shift
continue
elif test "$1" = "--nocreate"; then
OPT_CREATE_REDIS=no
OPT_CREATE_PGSQL=no
shift
continue
elif test "$1" = "--nocreate-pg"; then
OPT_CREATE_PGSQL=no
shift
continue
elif test "$1" = "--nocreate-redis"; then
OPT_CREATE_REDIS=no
shift
continue
elif test "$1" = "--with-coverage"; then
OPT_COVERAGE=yes
shift
continue
else
break
fi
done
if [ -z "$1" ]; then
echo "Usage: $0 [<options>] <test> [<test>]" >&2
echo "Options:" >&2
echo " --nocreate do not create the test environment on start" >&2
echo " --nocreate-pg do not create the pgsql test environment" >&2
echo " --nocreate-redis do not create the redis test environment" >&2
echo " --nodrop do not drop the test environment on exit" >&2
echo " --nodrop-pg do not drop the pgsql test environment" >&2
echo " --nodrop-redis do not drop the redis test environment" >&2
echo " --with-coverage use istanbul to determine code coverage" >&2
exit 1
fi
TESTS=$@
if test x"$OPT_CREATE_REDIS" = xyes; then
echo "Starting redis on port ${REDIS_PORT}"
REDIS_CELL_PATH="${BASEDIR}/support/libredis_cell.so"
if [[ "$OSTYPE" == "darwin"* ]]; then
REDIS_CELL_PATH="${BASEDIR}/support/libredis_cell.dylib"
fi
echo "port ${REDIS_PORT}" | redis-server - --loadmodule ${REDIS_CELL_PATH} > ${BASEDIR}/test.log &
PID_REDIS=$!
echo ${PID_REDIS} > ${BASEDIR}/redis.pid
fi
PREPARE_DB_OPTS=
if test x"$OPT_CREATE_PGSQL" != xyes; then
PREPARE_DB_OPTS="$PREPARE_DB_OPTS --skip-pg"
fi
if test x"$OPT_CREATE_REDIS" != xyes; then
PREPARE_DB_OPTS="$PREPARE_DB_OPTS --skip-redis"
fi
echo "Preparing the environment"
cd ${BASEDIR}
sh prepare_db.sh ${PREPARE_DB_OPTS} || die "database preparation failure"
cd -
PATH=node_modules/.bin/:node_modules/mocha/bin:$PATH
echo
echo "Environment:"
echo
echo " ogr2ogr version: "`ogr2ogr --version`
echo
if test x"$OPT_COVERAGE" = xyes; then
echo "Running tests with coverage"
./node_modules/.bin/istanbul cover node_modules/.bin/_mocha -- -u tdd --trace -t 5000 --exit ${TESTS}
else
echo "Running tests"
mocha -u tdd -t 5000 --exit ${TESTS}
fi
ret=$?
cleanup || exit 1
exit $ret