CartoDB-SQL-API/NEWS.md

# Changelog

**Deprecation warning**: Next major release will drop support for `Node.js 6 LTS` and `npm 3.x`. You'll be able to use the latest ES features as soon as we release 3.0.0. In the meantime, as a developer, you should keep compatibility with Node.js 6 LTS and keep updated both `package-lock.json` and `npm-shrinkwrap.json` files.

## 2.4.0
Released 2019-01-16

Announcements:
 * Update docs: compatible Node.js and npm versions
 * Set platform limits message also on streaming responses
 * Consider cancelled queries as platform limits.
 * Report fine-grained Garbage Collector stats
 * Both query endpoints as the same one in rate limits terms
 * Adding Authorization to Access-Control-Allow-Headers (https://github.com/CartoDB/CartoDB-SQL-API/issues/534)


## 2.3.1
Released 2018-12-23

Bug fixes:
* Update carto-package.json


## 2.3.0
Released 2018-12-26

Announcements:
 * Support Node.js 10
 * Add package-lock.json
 * Configure Travis CI to run docker tests against Node.js 6 & 10 versions
 * Update cartodb-psql to 0.13.1 (type cache depends now on db host)

Bug fixes:
 * Do not use `assert` to throw erros as in Node.js > 6 wraps the original error, the keyword 'throw' does the trick and it's backwards compatible
 * Make all modules to use strict mode semantics.
 * Avoid too long messages in `X-SQLAPI-Errors` header #543


## 2.2.1
Released 2018-09-18

Bug fixes:
  * Errors from zlib while gunzipping (`/sql/copyfrom` compressed requests) are now handled correctly.
  * Ensure exports temporal folder
  * Fix an issue with COPY TO returning paused DB connections to the pool #537


## 2.2.0
Released 2018-07-25

Announcements:
  * Improve error message when the DB query is over the user's limits
  * Updated cartodb-redis to 2.0.1
  * Modify the COPY query limits:
      - Instead of the generic timeout, it now uses a 5h timeout.
      - For COPY FROM, the limit is size-based, up to the remaining DB quota
      - The largest COPY FROM that can be made in a single POST request is limited to 2GB


## 2.1.0
Released 2018-06-13

Notice:
- This release changes the way that authentication works internally. You'll need to run `bundle exec rake carto:api_key:create_default` in your development environment to keep working.

New features:
 * CI tests with Ubuntu Xenial + PostgreSQL 10.1 and Ubuntu Precise + PostgreSQL 9.5
 * Making version 2.0.0 configuration parameters backwards compatible
 * New endpoint for COPY commands

Announcements:
 * Updated carto-psql to 0.12.0
 * [Test] Update sqlite3 to 4.0.0


## 2.0.0
Released 2018-03-22

Breaking changes:
 * Needs Redis v4

Features:
 * Implemented middleware to authenticate users throug the new Authorization System.
 * Upgrades cartodb-redis to 1.0.0
 * Rate limit feature (disabled by default)


## 1.48.1
Released 2018-02-27

Announcements:
 * Added `RESIZE=yes` param to gdal shapefile driver, wich optimizes size of exported shapefiles [#462](https://github.com/CartoDB/CartoDB-SQL-API/pull/462)


## 1.48.0
Released 2018-02-12

Announcements:
 * Change work in progress jobs endpoint from `[..]/job/wip` to `[..]/jobs-wip`
 * Documentation updates for Docs repo issue #840, GPKG Export.
 * Fix SHP exports, now it uses "the_geom" column by default when a dataset has more than one geometry column.
 * Logging all errors
 * Fix Postgres version in travis
 * Fix Python timeout error
 * Upgrades redis-mpool to 0.5.0
 * Upgrades cartodb-redis to 0.15.0


## 1.47.1
Released 2017-08-13

Announcements:
 * Upgrade cartodb-psql to 0.10.1.
 * Content edits to doc/version.md.


## 1.47.0
Released 2017-08-10

Announcements:
 * Now export and query APIs respond with `429 You are over the limits` when a query or export command overcomes the pre-configured user's timeout.


## 1.46.1
Released 2017-07-01

Announcements:
 * Now tableCache evicts keys based on their set time #244


## 1.46.0
Released 2017-06-27

Announcements:
 * Disable tableCache in-memory LRU by default [#422](https://github.com/CartoDB/CartoDB-SQL-API/issues/422)

## 1.45.1
Released 2017-06-27

Bug fixes:
 * Support special float values (NaN and ±Infinity) in query responses


## 1.45.0
Released 2017-04-18

Bug fixes:
 * Add error callback to ogr command while spawning #419

Announcements:
 * Make the zip command configurable #418


## 1.44.2
Released 2017-04-05

Bug fixes:
 * Update queue index while enqueueing jobs to the top of queue.


## 1.44.1
Released 2017-04-04

Bug fixes:
 * Avoid to scan the whole meta-database to discover active job queues. Now Batch Queries uses a set as index to know what queues are being processed #415


## 1.44.0
Released 2017-03-30

Announcements:
 * Active GC interval for Node.js >=v6.


## 1.43.1
Released 2017-01-16

Announcements:
 * Upgrade cartodb-psql to 0.7.1.


## 1.43.0
Released 2017-01-16

Announcements:
 * Upgrade cartodb-psql to 0.7.0.


## 1.42.7
Released 2017-01-12

Enhancements:
 * Avoid gpkg fid column #404.


## 1.42.6
Released 2016-12-19

Announcements:
 * Upgrade cartodb-redis to 0.13.2.
 * Upgrade redis-mpool to 0.4.1.


## 1.42.5
Released 2016-12-12

Enhancements:
 * Improvements in testing environment/tests.


## 1.42.4
Released 2016-11-30

Enhancements:
 * Include query status in batch queries log entries.


## 1.42.3
Released 2016-11-07

Announcements:
 * Raise payload limit for batch-queries to 16kb.


## 1.42.2
Released 2016-11-07

Bug fixes:
 * Improve error handling while registering jobs to be tracked.


## 1.42.1
Released 2016-11-03

Bug fixes:
 * Avoid to use SCAN command to find work-in-progress queues.


## 1.42.0
Released 2016-11-02

Announcements:
 * Adds endpoint to check running batch queries


## 1.41.0
Released 2016-10-21

Announcements:
 * Stop migrating old queues by default.

Bug fixes:
 * Fix some scenarios where batch queries got stuck waiting for available slots.


## 1.40.0
Released 2016-10-20

New features:
 * Batch queries are handled per db host.
   - There is an scheduler controlling how many queries and in what order they are run.
     - Priority is based on: number of queries already ran, and oldest user in queue.
 * Batch queries capacity: allow to configure how many jobs to run per db host.


## 1.39.1
Released 2016-10-17

Enhancements:
 * Log creation and waiting time for fallback jobs' queries.


## 1.39.0
Released 2016-10-17

Enhancements:
 * Use just one Redis pool across the whole application.

New features:
 * Batch queries use per user-queues.
 * Batch queries queues can limit the number of queued jobs per user.
   - Default is 64 jobs.
   - Configuration key `batch_max_queued_jobs` allows to modify the limit.


## 1.38.2
Released 2016-10-13

Bug fixes:
 * Batch queries: release redis clients to pool from locker and seeker.


## 1.38.1
Released 2016-10-13

Enhancements:
 * Batch queries: improvements over leader locking.


## 1.38.0
Released 2016-10-11

Announcements:
 * Allow to set statement timeout per query in multi query batch queries.
 * Batch queries default statement timeout set to 12 hours.
 * Multiple queries jobs pushed as first job between queries.


## 1.37.1
Released 2016-10-05

Bug fixes:
 * Body parser accepting multipart requests.


## 1.37.0
Released 2016-10-04

Enhancements:
 * Migrate to Express.js 4.x series.


## 1.36.2
Released 2016-10-03

Bug fixes:
 - Batch Queries logs: use path instead of stream to be able to reopen FD.

## 1.36.1
Released 2016-09-30

Enhancements:
 * Tag fallback jobs logs.


## 1.36.0
Released 2016-09-30

New features:
 * Log queries from batch fallback jobs.

Enhancements:
 * assert.response following callback(err, obj) pattern.


## 1.35.0
Released 2016-09-15

New features:
 * Allow to use `--config /path/to/config.js` to specify configuration file.
   - Environment will be loaded from config file if `environment` key is present, otherwise it keeps current behaviour.

Bug fixes:
 * Allow to use absolute paths for log files.

Announcements:
 * Removes support for optional rollbar logging.


## 1.34.2
Released 2016-08-30

Announcements:
 * Upgrades cartodb-redis to 0.13.1.
 * Set TTL of finished job to 2h


## 1.34.1
Released 2016-07-11

Bug fixes:
 * Fixed issue with redis connections in Batch API #326


## 1.34.0
Released 2016-07-11

New features:
 * Skip tables with no updated_at registered in cdb_tablemetadata.
 * Allow to setup more than one domain to validate oauth against.


## 1.33.0
Released 2016-07-01

New features:
 * Add `<%= job_id %>` template support for onerror and onsuccess fallback queries.


## 1.32.0
Released 2016-06-30

New features:
 * Broadcast after enqueueing jobs to improve query distribution load.
 * Batch pub-sub channel handles its connections using `redis-mpool`.


## 1.31.0
Released 2016-06-29

New features:
 * Adds start and end time for batch queries with fallback.
 * Add `<%= error_message %>` template support for onerror fallback queries.


## 1.30.1
Released 2016-06-23

Bug fixes:
 * Fixed issue with profiling in Batch API #318


## 1.30.0
Released 2016-06-14

Announcements:
 * Now Batch API sends stats metrics to statsd server #312
 * Now Batch API sets "skipped" instead of "pending" to queries that won't be performed #311

 Bug fixes:
  * Fixed issue with error handling in Batch API #316


## 1.29.2
Released 2016-05-25

Bug fixes:
 * Fixed issue with status transition in fallback jobs #308


## 1.29.1
Released 2016-05-24

Announcements:
 * Change Batch API size limit: 8kb per job.


## 1.29.0
Released 2016-05-24

New features:
 * Add support for fallback-jobs in Batch API #296

Bug fixes:
 * Fix issue in Batch API when a 'no longer running' job reports as 'running' before and after a job cancel #293


## 1.28.1
Released 2016-05-12

Bug fixes:
 * OGR with _needSRS=true_ fails for empty tables #299


## 1.28.0
Released 2016-05-11

Announcements:
 - Upgrades step-profiler to 0.3.0 to avoid dots in json keys #294

 New features:
  * Add support for geopackage format (`format=gpkg` at the URL) #291


## 1.27.1
Released 2016-04-18

Bug fixes:
  * Size of queries in Batch API is limited to 4kb per job


## 1.27.0
Released 2016-04-05

New features:
 * Add support for multiquery-jobs in Batch API #280
 * Add queue discovering for Batch processing at service startup #282

Bug fixes:
 * Fix issue in Batch API when after a period of inactivity it does not process jobs


## 1.26.0
Released 2016-03-08

New features:
 * Add [Surrogate-Key](https://github.com/CartoDB/cartodb/wiki/CartoDB-Surrogate-Keys) headers to responses

Enhancements:
 * Use new `node-cartodb-query-tables` library to obtain affected tables in queries


## 1.25.3
Released 2016-02-02

Enhancements:
 * QueryTablesApi caches affected tables and retrieves last modification #269


## 1.25.2
Released 2016-02-01

Bug fixes:
 * Skip query-tables-api for authenticated requests


## 1.25.1
Released 2016-01-28

Bug fixes:
 * Fix X-Cache-Channel generation when request are not authenticated #266


## 1.25.0
Released 2016-01-26

Bug fixes:
 * Stop adding X-Cache-Channel header when no tables involved #250

New features:
 * Set `Last-Modified` header based on affected tables (#101)
 * Batch API (#261):
   - New endpoint to create, read, update and delete long-running queries (jobs).
   - Batch service to process jobs.
 * Set Last-Modified header based on affected tables #247

Announcements:
 * Upgrades cartodb-psql to [0.6.1](https://github.com/CartoDB/node-cartodb-psql/releases/tag/0.6.1)


## 1.24.0
Released 2015-08-04

New features:
 * Client is removed from pool after error happens. This help to avoid issues with transactions (#241).

Announcements:
 * Upgrades cartodb-psql to [0.6.0](https://github.com/CartoDB/node-cartodb-psql/releases/tag/0.6.0)


## 1.23.0
Released 2015-06-16

Announcements:
 * Reverts tables=<fake> for ogr2ogr commands
   Ref https://github.com/CartoDB/CartoDB-SQL-API/commit/b6e53f732672369d7f9be26555ef412edc202727

Enhancements:
 * Makes ogr2ogr command configurable so it's possible to change path/bin
 * Uses :remote-addr instead of :req[X-Real-IP] \(#197)

New features:
 * Adds SpatiaLite as export format (#226)


## 1.22.2
Released 2015-05-26

Bug fixes:
 * Reintroduces tables= param in ogr2ogr exports (#204)
   This will avoid running a heavy ogr2ogr query when the pg catalog is big
   Ref https://github.com/CartoDB/CartoDB-SQL-API/commit/84c422c505391ef0e743aed2204214d4286d7e30


## 1.22.1
Released 2015-05-14

Bug fixes:
 * Close stream responses on error (#219)

Enhancements:
 * Format files split into pg and ogr directories


## 1.22.0
Released 2015-04-09

Announcements:
 * Now health check only validates against a disabling file
 * Supports user extraction from request params via base_url config


## 1.21.1
Released 2015-03-02

Enhancements:
 * Improve row size limit error message


## 1.21.0
Released 2015-03-02

New features:
 * Logs with console.error too large row errors


## 1.20.0
Released 2015-02-26

Announcements:
 * Upgrades cartodb-psql to 0.5.1 for keep alive configuration
 * Dependencies from npm registry when available


## 1.19.1
Released 2014-12-15

Bug fixes:
 * Closes stream responses on error (#188)
 * Closes fd for log files on `kill -HUP` (#187)


## 1.19.0
Released 2014-11-21

New features:
 * Add more fields to error responses with hint, detail and context for SQL errors.

Enhancements:
 * Don't loop twice over svg rows
 * Improve statement timeout error messages
 * Improve topojson output by streaming json


## 1.18.0
Released 2014-10-14

Announcements:
 * Dropping support for npm <1.2.1
   npm-shrinkwrap.json is incompatible when generated with npm >=1.2.1 and consumed by npm <1.2.1


## 1.17.1
Released 2014-09-23

Enhancements:
 * Removes tables=fake wadus param in ogr2ogr command so it can go to geometry
   columns view to retrieve the column data type. This requires to grant select
   permission on geometry_columns and geography_columns to the public user.
 * Removes query tables console.log

Bug fixes:
 * Fixes "make check" on systems with non-default PostgreSQL superuser (#152)


## 1.17.0
Released 2014-09-17

Bug fixes:
 * Returns 401 Unauthorized for queries without permission

New features:
 * New header for database host serving the request
 * Health check endpoint

Enhancements:
 * Upgrades dependencies:
    * cartodb-redis
    * cartodb-psql
    * log4js


## 1.16.0
Released 2014-08-19

Enhancements:
 * Metrics revamp: removes and adds some metrics

## 1.15.0
Released 2014-08-18

Enhancements:
 * Upgrades cartodb-redis
 * Upgrades underscore, removes underscore.string dependency
 * Uses https endpoints for dependencies


## 1.14.1
Released 2014-08-08

Other changes:
 * Constraint for pg_ queries if request is non authenticated

## 1.14.0
Released 2014-08-07

Other changes:
 * Removes sql statements restriction on pg_ queries


## 1.13.0
Released 2014-08-07

New features:
 * New authentication mechanism: checks in advance if credentials are provided
   in order to do a single request to redis to retrieve the required database
   connection parameters.
 * Retrieves OAuth hash values using new cartodb-redis method so it will reuse
   the redis pool from cartodb-redis instead of using just one pool for oauth.


## 1.12.1
Released 2014-08-05

Bug fixes:
 * Fixes GeoJSON stream error responses
 * Fixes GeoJSON stream empty responses
 * JSONP callbacks return with 200 status error code

Enhancements:
 * Re-enables tests

## 1.12.0
Released 2014-08-04

New features:
 * Add header for host serving the request
 * Stream JSON/GeoJSON responses

## 1.11.0
Released 2014-07-30

New features:
 * Support for different schemas and different public users

Enhancements:
 * Profiler header sent as JSON string

Other changes:
 * Revamped documentation

Bug fixes:
 * Pick redis pool configuration values

## 1.10.1
Released 2014-06-05

Bug fixes:
 * Backing out Stream JSON responses

## 1.10.0
Released 2014-06-04

New features:
 * Order by and sort order through http query params
 * Cancelling queries in Postgresql when HTTP request is aborted/closed

Enhancements:
 * Stream JSON responses
 * Pre-compiling may write regex
 * Set default PostgreSQL application name to "cartodb_sqlapi"

Bug fixes:
 * Support trailing semicolons (#147)

## 1.9.1
Released 2014-03-27

Bug fixes:
 * Fix paging with queries starting with comments (#144)

## 1.9.0
Released 2014-03-20

New features:
 * Add optional support for rollbar (#137)
 * Add '/version' endpoint (#138)
 * Add profiler support (#142)
 * Add statsd support (#133)

Enhancements:
 * Allow configuring log_format (#131)
 * Use log4js for logging (#136)
 * Include version in startup log
 * Allow passing environment configuration name via NODE_ENV to app.js
 * Print environment configuration name on app start
 * Upgrade node-zipfile to ~0.5.0
 * Add support for node-0.10 (#132)
 * Fix lack of response on backend crash (#135)
 * Reduce work on aborted requests (#129)

Other changes:
 * Switch to 3-clause BSD license (#143)

## 1.8.3
Released 2014-02-10

Bug fixes:
 * Honour the 'node_socket_timeout' configuration directive (#128)

Enhancements:
 * Add support for error handling in assert.request
 * Stop using ANSI colors in the logs (#130)

## 1.8.2
Released 2014-01-20

Bug fixes:
 * Restore compatibility with 1.6.x configuration
 * Use db_port in ogr2ogr

## 1.8.1
Released 2014-01-10

Bug fixes:
 * Fix use of "SELECT .. INTO" with windowing params (#127)


## 1.8.0
Released 2013-12-18

New features:
* Add 'user_from_host' directive to generalize username extraction (#124)

Improvements:
* Enhance error message on unknown cartodb username (#126)

## 1.7.1
Released 2013-12-02

* Fix documentation for CSV export format: geoms are in hexewkb, not ewkt.
* Fix field types names lookup after PSQL model refactoring
  NOTE: fixes missing .prj in shapefile export regression (#122)

## 1.7.0
Released 2013-11-19

New features:
* Optionally read user-specific database_host and database_password
  from redis, as per CartoDB-2.5.0 model (#120, #121)
* Add warnings and notices to JSON response (#104)

Other changes:
* CartoDB redis interaction delegated to "cartodb-redis" module

## 1.6.3
Released 2013-11-10

* JSON format: correctly recognize "numeric" type columns (#119)

## 1.6.2
Released 2013-11-07

* JSON format: correctly recognize "date" type columns (#117)
* Allow access to tables whose name contains (but does not start with)
  the "pg_" substring (#118)

## 1.6.1
Released 2013-11-05

* Still set a meaningful X-Cache-Channel with cache_policy=persist (#105)
* Fix wrong projection in KML exports for manually altered tables (#116)
* Set KML folder name to the requested filename (#115)
* Make public PostgreSQL user name a configuration parameter (#56)

## 1.6.0
Released 2013-10-02

* Fix shapefile export for non-linestring results starting with NULLs (#111)
* Fix missing .prj in shapefile export (#110)
* Improve recognition of non-standard field types names by db lookup (#112)
* Upgrade node-pg dependency to 2.6.2
* Drop support for cluster

## 1.5.4
Released 2013-10-01

* Honour skipfields in JSON schema response (#109)

## 1.5.3
Released yyyy-mm-dd

* Set Last-Modified header with cache_policy=persist
* Raise max-age to one year for all cacheable queries
* Set max-age to 0 for uncacheable (mutating) queries
* Add REINDEX to the list of uncacheable queries
* Support all parameters with POST as well as GET
* Ensure testsuite passes with both GDAL-0.9 and GDAL-0.10
* JSON output: report boolean types as boolean, not string (#106)

## 1.5.2
Released yyyy-mm-dd

* Keep numbers as such in JSON output (#100)
* Revert max-age=0 in Cache-Control when using no-cache

## 1.5.1
Released yyyy-mm-dd

* Improve cacheability of queries selecting "updated_at" fields (#99)

## 1.5.0
Released yyyy-mm-dd

* Add "fields" member in JSON return (#97)
* Add --skipfields switch to cdbsql
* Fix windowing with CTE
* Retain UTC offset in JSON date output
* Set max-age=0 in Cache-Control when using no-cache

## 1.4.1
Released yyyy-mm-dd

* Fix windowing support for non-uppercased SELECT queries
* Fix oAuth testcase

## 1.4.0
Released yyyy-mm-dd

* Add arraybuffer format
* Fix filesystem access conflict among clustered processes
* Fix discard of queued export requests on error
* Really fix problem identifying OAuth requests

## 1.3.10
Released yyyy-mm-dd

* Fixed problem identifying OAuth request protocol
* Make base url configurable
* Update underscore dependency
* Add munin plugin
* Make PostgreSQL client pooling settings configurable (#47)
* Do not execute queries on OPTIONS (#94)
* Survive postgresql connection losses (#95)

## 1.3.9
Released yyyy-mm-dd

* Do not choke on multiple `skipfields` parameter
* Do not request caching of TRUNCATE queries

## 1.3.8
Released yyyy-mm-dd

* Make using SET or querying system catalogues harder
* Allow sql queries to end with a semicolon (#90)
* Testsuite fixes, jenkins ready :)

## 1.3.7
Released yyyy-mm-dd

* Fix parsing of numeric arrays (#88)
* node-pool upgraded to 2.0.3
* Reduce memory use on KML export
* Fix concurrent request for KML and Shapefile exports
* Send an empty-like KML when exporting zero-rows queries
* Make temporary dir a configuration setting
* Use OGR for CSV output, reducing memory requirement

## 1.3.6
Released yyyy-mm-dd

* Do not confuse warnings with errors on shapefile output (#87)

## 1.3.5
Released 2013-02-19

* Fix skipfields use with SHP output format (#81)
* Fix Content-Disposition for error responses (#82)
* Add pid to /cachestatus (#83)
* Check CDB_QueryTable response before saving into cache (#83)
* Use an expiring LRU cache for query tables (#83)
* Fix X-Cache-Channel computation with paging parameters (#85)

## 1.3.4
Released 2013-01-21

* Improve mixed-geometry export error message (#78)
* Remove NULL the_geom features from topojson output (#80)
* Fix crash when issuing SQL "COPY" command
* Return an error when "the_geom" is in skipfield for SVG output (#73)

## 1.3.3
Released 2013-01-11

* Fix Date format in CSV output (#77)
* Add TopoJSON output format (#79)

## 1.3.2
Released 2012-11-30

* Fix KML export truncation (#70)
* Fix UTF8 in shapefile export (#66)

## 1.3.1
Released yyyy-mm-dd

* Support 'format' and 'filename' params in POST
* Fix oAuth bug introduced by 'skipfields' param in 1.3.0 (#69)

## 1.3.0
Released yyyy-mm-dd

* Support for specifying a filename for exports (#64)
* Support for specifying a list of fields to skip from output (#63)
* Add 'cache_policy' parameter (#62)

## 1.2.1
Released yyyy-mm-dd

* Added timeout default to 600 miliseconds in cluster.js

## 1.2.0
Released yyyy-mm-dd

* New output formats:
  * ESRI Shapefile (format=shp)
  * SVG (format=svg)
  * KML (format=kml)
* Advertise header presence in CSV Content-Type
* Fix CSV output with no rows (#60)
* Use "attachment" Content-Disposition for all output formats (#61)
* Only use last format parameter when multiple are requested
* Return a 400 response on unsupported format request
* Added X-Prototype-Version, X-CSRF-Token to Access-Control-Allow-Headers

## 1.1.0
Released 2012-10-30

* Fixed problem in cluster2 with pidfile name
* SVG output format
* Enhancement to the cdbsql tool:
  - New switches: --format, --key, --dp
  - Interactive mode
* API documentation
* ./configure script
* Restrict listening to a node host

## 1.0.0
Released 2012-10-03

* Migrated to node 0.8 version

## 0.9.0
Released 2012-09-18

* Fix INSERT and UPDATE with RETURNING clause