- Log system revamp:
- Logs to stdout, disabled while testing
- Use header `X-Request-Id`, or create a new `uuid` when no present, to identyfy log entries
- Be able to set log level from env variable `LOG_LEVEL`, useful while testing: `LOG_LEVEL=info npm test`; even more human-readable: `LOG_LEVEL=info npm t | ./node_modules/.bin/pino-pretty`
- Be able to reduce the footprint in the final log file depending on the environment
- Use one logger for every service: Queries, Batch Queries (Jobs), and Data Ingestion (CopyTo/CopyFrom)
- Stop using headers such as: `X-SQL-API-Log`, `X-SQL-API-Profiler`, and `X-SQL-API-Errors` as a way to log info.
- Be able to tag requests with labels as an easier way to provide business metrics
- Metro: Add log-collector utility (`metro`), it will be moved to its own repository. Attaching it here fro development purposes. Try it with the following command `LOG_LEVEL=info npm t | node metro`
- Metro: Creates `metrics-collector.js` a stream to update Prometheus' counters and histograms and exposes them via Express' app (`:9145/metrics`). Use the ones defined in `grok_exporter`
Announcements:
- Profiler is always set. No need to check its existence anymore
- Unify profiler usage for every endpoint
Bug fixes:
- Avoid hung requests while fetching user identifier
- Update `gc-stats` to version 1.4.0
- Replace `zipfile` -> `adm-zip`
- Update `libxmljs` to version 0.19.7
- Update `sqlite` to version 4.2.0
- Adapted pool acquires to the new version of `cartodb-redis`
- Adapted test to use `adm-zip`
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Fix tests
* Fix typos
* Checks if pubsub config exists to enable the service
* Fix typo
* Normalize headers values for pubsub
* Trim fields when normalizing
* Trim fields when normalizing
- Extract from job subcriber everything related to queue dicovering
- Job Queue is responsible of queue dicovering and exposes functions to scan user queues.
- Batch creates an interval to consume queues
- Now Batch.start() does:
* Finds existing queues from metadata, adding to queue index the ones that are not indexed yet
* Binds callback to process jobs for every queue found.
* Inits queue consumer interval
* Subscribes to PUB/SUB channel
- Existing jobs are moved before start processing them.
- Uses a new queue prefix to avoid collisions.
- Pub/Sub also changes communication channel.
- Job subscriber emits user+host on new jobs.
- Batch processor is faulty. See TODO in batch.js.
