- Log system revamp:
- Logs to stdout, disabled while testing
- Use header `X-Request-Id`, or create a new `uuid` when no present, to identyfy log entries
- Be able to set log level from env variable `LOG_LEVEL`, useful while testing: `LOG_LEVEL=info npm test`; even more human-readable: `LOG_LEVEL=info npm t | ./node_modules/.bin/pino-pretty`
- Be able to reduce the footprint in the final log file depending on the environment
- Use one logger for every service: Queries, Batch Queries (Jobs), and Data Ingestion (CopyTo/CopyFrom)
- Stop using headers such as: `X-SQL-API-Log`, `X-SQL-API-Profiler`, and `X-SQL-API-Errors` as a way to log info.
- Be able to tag requests with labels as an easier way to provide business metrics
- Metro: Add log-collector utility (`metro`), it will be moved to its own repository. Attaching it here fro development purposes. Try it with the following command `LOG_LEVEL=info npm t | node metro`
- Metro: Creates `metrics-collector.js` a stream to update Prometheus' counters and histograms and exposes them via Express' app (`:9145/metrics`). Use the ones defined in `grok_exporter`
Announcements:
- Profiler is always set. No need to check its existence anymore
- Unify profiler usage for every endpoint
Bug fixes:
- Avoid hung requests while fetching user identifier
- Update `gc-stats` to version 1.4.0
- Replace `zipfile` -> `adm-zip`
- Update `libxmljs` to version 0.19.7
- Update `sqlite` to version 4.2.0
- Adapted pool acquires to the new version of `cartodb-redis`
- Adapted test to use `adm-zip`
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Fix tests
* Fix typos
* Checks if pubsub config exists to enable the service
* Fix typo
* Normalize headers values for pubsub
* Trim fields when normalizing
* Trim fields when normalizing
* Tests: Work with re-runs
* Test: Adapt copyfrom test
- Reduce timeout from 10 to 1 ms.
- Increase the size of the file from 6 to 2016 rows.
* Use node-cartodb-psql 0.14.0
Ref: https://github.com/CartoDB/CartoDB-SQL-API/pull/612
Alberto Hernández