- Log system revamp:
- Logs to stdout, disabled while testing
- Use header `X-Request-Id`, or create a new `uuid` when no present, to identyfy log entries
- Be able to set log level from env variable `LOG_LEVEL`, useful while testing: `LOG_LEVEL=info npm test`; even more human-readable: `LOG_LEVEL=info npm t | ./node_modules/.bin/pino-pretty`
- Be able to reduce the footprint in the final log file depending on the environment
- Use one logger for every service: Queries, Batch Queries (Jobs), and Data Ingestion (CopyTo/CopyFrom)
- Stop using headers such as: `X-SQL-API-Log`, `X-SQL-API-Profiler`, and `X-SQL-API-Errors` as a way to log info.
- Be able to tag requests with labels as an easier way to provide business metrics
- Metro: Add log-collector utility (`metro`), it will be moved to its own repository. Attaching it here fro development purposes. Try it with the following command `LOG_LEVEL=info npm t | node metro`
- Metro: Creates `metrics-collector.js` a stream to update Prometheus' counters and histograms and exposes them via Express' app (`:9145/metrics`). Use the ones defined in `grok_exporter`
Announcements:
- Profiler is always set. No need to check its existence anymore
- Unify profiler usage for every endpoint
Bug fixes:
- Avoid hung requests while fetching user identifier
- Update `gc-stats` to version 1.4.0
- Replace `zipfile` -> `adm-zip`
- Update `libxmljs` to version 0.19.7
- Update `sqlite` to version 4.2.0
- Adapted pool acquires to the new version of `cartodb-redis`
- Adapted test to use `adm-zip`
If the final name is going to be too big, use the hash over the proposed
path to create an unique name that fits the requirements of path length
under Linux
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Fix tests
* Fix typos
* Checks if pubsub config exists to enable the service
* Fix typo
* Normalize headers values for pubsub
* Trim fields when normalizing
* Trim fields when normalizing
This introduces a couple of changes in the parameter interpretation:
copy_from_minimum_input_speed is now the total number of bytes in the interval
(so we could multiply the currently configured values by the interval to preserve min. throughput.
the minimum speed is now inclusive (only smaller values will cause an exception).
