CartoDB-SQL-API/NEWS.md

1.40.0 - 2016-mm-dd

New features:

• Batch queries are handled per db host.
  • There is an scheduler controlling how many queries and in what order they are run.
    • Priority is based on: # of queries already run, last execution time, and oldest user in queue.
• Batch queries capacity: allow to configure how many jobs to run per db host.

1.39.1 - 2016-10-17

Enhancements:

• Log creation and waiting time for fallback jobs' queries.

1.39.0 - 2016-10-17

Enhancements:

• Use just one Redis pool across the whole application.

New features:

• Batch queries use per user-queues.
• Batch queries queues can limit the number of queued jobs per user.
  • Default is 64 jobs.
  • Configuration key batch_max_queued_jobs allows to modify the limit.

1.38.2 - 2016-10-13

Bug fixes:

• Batch queries: release redis clients to pool from locker and seeker.

1.38.1 - 2016-10-13

Enhancements:

• Batch queries: improvements over leader locking.

1.38.0 - 2016-10-11

Announcements:

• Allow to set statement timeout per query in multi query batch queries.
• Batch queries default statement timeout set to 12 hours.
• Multiple queries jobs pushed as first job between queries.

1.37.1 - 2016-10-05

Bug fixes:

• Body parser accepting multipart requests.

1.37.0 - 2016-10-04

Enhancements:

• Migrate to Express.js 4.x series.

1.36.2 - 2016-10-03

Bug fixes:

• Batch Queries logs: use path instead of stream to be able to reopen FD.

1.36.1 - 2016-09-30

Enhancements:

• Tag fallback jobs logs.

1.36.0 - 2016-09-30

New features:

• Log queries from batch fallback jobs.

Enhancements:

• assert.response following callback(err, obj) pattern.

1.35.0 - 2016-09-15

New features:

• Allow to use --config /path/to/config.js to specify configuration file.
  • Environment will be loaded from config file if environment key is present, otherwise it keeps current behaviour.

Bug fixes:

• Allow to use absolute paths for log files.

Announcements:

• Removes support for optional rollbar logging.

1.34.2 - 2016-08-30

Announcements:

• Upgrades cartodb-redis to 0.13.1.
• Set TTL of finished job to 2h

1.34.1 - 2016-07-11

Bug fixes:

• Fixed issue with redis connections in Batch API #326

1.34.0 - 2016-07-11

New features:

• Skip tables with no updated_at registered in cdb_tablemetadata.
• Allow to setup more than one domain to validate oauth against.

1.33.0 - 2016-07-01

New features:

• Add <%= job_id %> template support for onerror and onsuccess fallback queries.

1.32.0 - 2016-06-30

New features:

• Broadcast after enqueueing jobs to improve query distribution load.
• Batch pub-sub channel handles its connections using redis-mpool.

1.31.0 - 2016-06-29

New features:

• Adds start and end time for batch queries with fallback.
• Add <%= error_message %> template support for onerror fallback queries.

1.30.1 - 2016-06-23

Bug fixes:

• Fixed issue with profiling in Batch API #318

1.30.0 - 2016-06-14

Announcements:

• Now Batch API sends stats metrics to statsd server #312
• Now Batch API sets "skipped" instead of "pending" to queries that won't be performed #311

Bug fixes:

• Fixed issue with error handling in Batch API #316

1.29.2 - 2016-05-25

Bug fixes:

• Fixed issue with status transition in fallback jobs #308

1.29.1 - 2016-05-24

Announcements:

• Change Batch API size limit: 8kb per job.

1.29.0 - 2016-05-24

New features:

• Add support for fallback-jobs in Batch API #296

Bug fixes:

• Fix issue in Batch API when a 'no longer running' job reports as 'running' before and after a job cancel #293

1.28.1 - 2016-05-12

Bug fixes:

• OGR with needSRS=true fails for empty tables #299

1.28.0 - 2016-05-11

Announcements:

• Upgrades step-profiler to 0.3.0 to avoid dots in json keys #294

New features:

• Add support for geopackage format (format=gpkg at the URL) #291

1.27.1 - 2016-04-18

Bug fixes:

• Size of queries in Batch API is limited to 4kb per job

1.27.0 - 2016-04-05

New features:

• Add support for multiquery-jobs in Batch API #280
• Add queue discovering for Batch processing at service startup #282

Bug fixes:

• Fix issue in Batch API when after a period of inactivity it does not process jobs

1.26.0 - 2016-03-08

New features:

• Add Surrogate-Key headers to responses

Enhancements:

• Use new node-cartodb-query-tables library to obtain affected tables in queries

1.25.3 - 2016-02-02

Enhancements:

• QueryTablesApi caches affected tables and retrieves last modification #269

1.25.2 - 2016-02-01

Bug fixes:

• Skip query-tables-api for authenticated requests

1.25.1 - 2016-01-28

Bug fixes:

• Fix X-Cache-Channel generation when request are not authenticated #266

1.25.0 - 2016-01-26

Bug fixes:

• Stop adding X-Cache-Channel header when no tables involved #250

New features:

• Set Last-Modified header based on affected tables (#101)
• Batch API (#261):
  • New endpoint to create, read, update and delete long-running queries (jobs).
  • Batch service to process jobs.
• Set Last-Modified header based on affected tables #247

Announcements:

• Upgrades cartodb-psql to 0.6.1

1.24.0 - 2015-08-04

New features:

• Client is removed from pool after error happens. This help to avoid issues with transactions (#241).

Announcements:

• Upgrades cartodb-psql to 0.6.0

1.23.0 - 2015-06-16

Announcements:

• Reverts tables= for ogr2ogr commands Ref b6e53f7326

Enhancements:

• Makes ogr2ogr command configurable so it's possible to change path/bin
• Uses :remote-addr instead of :req[X-Real-IP] (#197)

New features:

• Adds SpatiaLite as export format (#226)

1.22.2 - 2015-05-26

Bug fixes:

• Reintroduces tables= param in ogr2ogr exports (#204) This will avoid running a heavy ogr2ogr query when the pg catalog is big Ref 84c422c505

1.22.1 - 2015-05-14

Bug fixes:

• Close stream responses on error (#219)

Enhancements:

• Format files split into pg and ogr directories

1.22.0 - 2015-04-09

Announcements:

• Now health check only validates against a disabling file
• Supports user extraction from request params via base_url config

1.21.1 - 2015-03-02

Enhancements:

• Improve row size limit error message

1.21.0 - 2015-03-02

New features:

• Logs with console.error too large row errors

1.20.0 - 2015-02-26

Announcements:

• Upgrades cartodb-psql to 0.5.1 for keep alive configuration
• Dependencies from npm registry when available

1.19.1 - 2014-12-15

Bug fixes:

• Closes stream responses on error (#188)
• Closes fd for log files on kill -HUP (#187)

1.19.0 - 2014-11-21

New features:

• Add more fields to error responses with hint, detail and context for SQL errors.

Enhancements:

• Don't loop twice over svg rows
• Improve statement timeout error messages
• Improve topojson output by streaming json

1.18.0 - 2014-10-14

Announcements:

• Dropping support for npm <1.2.1 npm-shrinkwrap.json is incompatible when generated with npm >=1.2.1 and consumed by npm <1.2.1

1.17.1 - 2014-09-23

Enhancements:

• Removes tables=fake wadus param in ogr2ogr command so it can go to geometry columns view to retrieve the column data type. This requires to grant select permission on geometry_columns and geography_columns to the public user.
• Removes query tables console.log

Bug fixes:

• Fixes "make check" on systems with non-default PostgreSQL superuser (#152)

1.17.0 - 2014-09-17

Bug fixes:

• Returns 401 Unauthorized for queries without permission

New features:

• New header for database host serving the request
• Health check endpoint

Enhancements:

• Upgrades dependencies:
  • cartodb-redis
  • cartodb-psql
  • log4js

1.16.0 - 2014-08-19

Enhancements:

• Metrics revamp: removes and adds some metrics

1.15.0 - 2014-08-18

Enhancements:

• Upgrades cartodb-redis
• Upgrades underscore, removes underscore.string dependency
• Uses https endpoints for dependencies

1.14.1 - 2014-08-08

Other changes:

• Constraint for pg_ queries if request is non authenticated

1.14.0 - 2014-08-07

Other changes:

• Removes sql statements restriction on pg_ queries

1.13.0 - 2014-08-07

New features:

• New authentication mechanism: checks in advance if credentials are provided in order to do a single request to redis to retrieve the required database connection parameters.
• Retrieves OAuth hash values using new cartodb-redis method so it will reuse the redis pool from cartodb-redis instead of using just one pool for oauth.

1.12.1 - 2014-08-05

Bug fixes:

• Fixes GeoJSON stream error responses
• Fixes GeoJSON stream empty responses
• JSONP callbacks return with 200 status error code

Enhancements:

• Re-enables tests

1.12.0 - 2014-08-04

New features:

• Add header for host serving the request
• Stream JSON/GeoJSON responses

1.11.0 - 2014-07-30

New features:

• Support for different schemas and different public users

Enhancements:

• Profiler header sent as JSON string

Other changes:

• Revamped documentation

Bug fixes:

• Pick redis pool configuration values

1.10.1 - 2014-06-05

Bug fixes:

• Backing out Stream JSON responses

1.10.0 - 2014-06-04

New features:

• Order by and sort order through http query params
• Cancelling queries in Postgresql when HTTP request is aborted/closed

Enhancements:

• Stream JSON responses
• Pre-compiling may write regex
• Set default PostgreSQL application name to "cartodb_sqlapi"

Bug fixes:

• Support trailing semicolons (#147)

1.9.1 - 2014-03-27

Bug fixes:

• Fix paging with queries starting with comments (#144)

1.9.0 - 2014-03-20

New features:

• Add optional support for rollbar (#137)
• Add '/version' endpoint (#138)
• Add profiler support (#142)
• Add statsd support (#133)

Enhancements:

• Allow configuring log_format (#131)
• Use log4js for logging (#136)
• Include version in startup log
• Allow passing environment configuration name via NODE_ENV to app.js
• Print environment configuration name on app start
• Upgrade node-zipfile to ~0.5.0
• Add support for node-0.10 (#132)
• Fix lack of response on backend crash (#135)
• Reduce work on aborted requests (#129)

Other changes:

• Switch to 3-clause BSD license (#143)

1.8.3 - 2014-02-10

Bug fixes:

• Honour the 'node_socket_timeout' configuration directive (#128)

Enhancements:

• Add support for error handling in assert.request
• Stop using ANSI colors in the logs (#130)

1.8.2 - 2014-01-20

Bug fixes:

• Restore compatibility with 1.6.x configuration
• Use db_port in ogr2ogr

1.8.1 - 2014-01-10

Bug fixes:

• Fix use of "SELECT .. INTO" with windowing params (#127)

1.8.0 - 2013-12-18

New features:

• Add 'user_from_host' directive to generalize username extraction (#124)

Improvements:

• Enhance error message on unknown cartodb username (#126)

1.7.1 - 2013-12-02

• Fix documentation for CSV export format: geoms are in hexewkb, not ewkt.
• Fix field types names lookup after PSQL model refactoring NOTE: fixes missing .prj in shapefile export regression (#122)

1.7.0 - 2013-11-19

New features:

• Optionally read user-specific database_host and database_password from redis, as per CartoDB-2.5.0 model (#120, #121)
• Add warnings and notices to JSON response (#104)

Other changes:

• CartoDB redis interaction delegated to "cartodb-redis" module

1.6.3 - 2013-11-10

• JSON format: correctly recognize "numeric" type columns (#119)

1.6.2 - 2013-11-07

• JSON format: correctly recognize "date" type columns (#117)
• Allow access to tables whose name contains (but does not start with) the "pg_" substring (#118)

1.6.1 - 2013-11-05

• Still set a meaningful X-Cache-Channel with cache_policy=persist (#105)
• Fix wrong projection in KML exports for manually altered tables (#116)
• Set KML folder name to the requested filename (#115)
• Make public PostgreSQL user name a configuration parameter (#56)

1.6.0 - 2013-10-02

• Fix shapefile export for non-linestring results starting with NULLs (#111)
• Fix missing .prj in shapefile export (#110)
• Improve recognition of non-standard field types names by db lookup (#112)
• Upgrade node-pg dependency to 2.6.2
• Drop support for cluster

1.5.4 - 2013-10-01

• Honour skipfields in JSON schema response (#109)

1.5.3

• Set Last-Modified header with cache_policy=persist
• Raise max-age to one year for all cacheable queries
• Set max-age to 0 for uncacheable (mutating) queries
• Add REINDEX to the list of uncacheable queries
• Support all parameters with POST as well as GET
• Ensure testsuite passes with both GDAL-0.9 and GDAL-0.10
• JSON output: report boolean types as boolean, not string (#106)

1.5.2

• Keep numbers as such in JSON output (#100)
• Revert max-age=0 in Cache-Control when using no-cache

1.5.1

• Improve cacheability of queries selecting "updated_at" fields (#99)

1.5.0

• Add "fields" member in JSON return (#97)
• Add --skipfields switch to cdbsql
• Fix windowing with CTE
• Retain UTC offset in JSON date output
• Set max-age=0 in Cache-Control when using no-cache

1.4.1

• Fix windowing support for non-uppercased SELECT queries
• Fix oAuth testcase

1.4.0

• Add arraybuffer format
• Fix filesystem access conflict among clustered processes
• Fix discard of queued export requests on error
• Really fix problem identifying OAuth requests

1.3.10

• Fixed problem identifying OAuth request protocol
• Make base url configurable
• Update underscore dependency
• Add munin plugin
• Make PostgreSQL client pooling settings configurable (#47)
• Do not execute queries on OPTIONS (#94)
• Survive postgresql connection losses (#95)

1.3.9

• Do not choke on multiple skipfields parameter
• Do not request caching of TRUNCATE queries

1.3.8

• Make using SET or querying system catalogues harder
• Allow sql queries to end with a semicolon (#90)
• Testsuite fixes, jenkins ready :)

1.3.7

• Fix parsing of numeric arrays (#88)
• node-pool upgraded to 2.0.3
• Reduce memory use on KML export
• Fix concurrent request for KML and Shapefile exports
• Send an empty-like KML when exporting zero-rows queries
• Make temporary dir a configuration setting
• Use OGR for CSV output, reducing memory requirement

1.3.6 (DD/MM/YY)

• Do not confuse warnings with errors on shapefile output (#87)

1.3.5 (19/02/13)

• Fix skipfields use with SHP output format (#81)
• Fix Content-Disposition for error responses (#82)
• Add pid to /cachestatus (#83)
• Check CDB_QueryTable response before saving into cache (#83)
• Use an expiring LRU cache for query tables (#83)
• Fix X-Cache-Channel computation with paging parameters (#85)

1.3.4 (21/01/13)

• Improve mixed-geometry export error message (#78)
• Remove NULL the_geom features from topojson output (#80)
• Fix crash when issuing SQL "COPY" command
• Return an error when "the_geom" is in skipfield for SVG output (#73)

1.3.3 (11/01/13)

• Fix Date format in CSV output (#77)
• Add TopoJSON output format (#79)

1.3.2 (30/11/12)

• Fix KML export truncation (#70)
• Fix UTF8 in shapefile export (#66)

1.3.1 (DD/MM/YY)

• Support 'format' and 'filename' params in POST
• Fix oAuth bug introduced by 'skipfields' param in 1.3.0 (#69)

1.3.0 (DD/MM/YY)

• Support for specifying a filename for exports (#64)
• Support for specifying a list of fields to skip from output (#63)
• Add 'cache_policy' parameter (#62)

1.2.1 (DD/MM/YY)

• Added timeout default to 600 miliseconds in cluster.js

1.2.0 (DD/MM/YY)

• New output formats:
  • ESRI Shapefile (format=shp)
  • SVG (format=svg)
  • KML (format=kml)
• Advertise header presence in CSV Content-Type
• Fix CSV output with no rows (#60)
• Use "attachment" Content-Disposition for all output formats (#61)
• Only use last format parameter when multiple are requested
• Return a 400 response on unsupported format request
• Added X-Prototype-Version, X-CSRF-Token to Access-Control-Allow-Headers

1.1.0 (30/10/12)

• Fixed problem in cluster2 with pidfile name
• SVG output format
• Enhancement to the cdbsql tool:
  • New switches: --format, --key, --dp
  • Interactive mode
• API documentation
• ./configure script
• Restrict listening to a node host

1.0.0 (03/10/12)

• Migrated to node 0.8 version

0.9.0 (18/09/12)

• Fix INSERT and UPDATE with RETURNING clause