- Log system revamp:
- Logs to stdout, disabled while testing
- Use header `X-Request-Id`, or create a new `uuid` when no present, to identyfy log entries
- Be able to set log level from env variable `LOG_LEVEL`, useful while testing: `LOG_LEVEL=info npm test`; even more human-readable: `LOG_LEVEL=info npm t | ./node_modules/.bin/pino-pretty`
- Be able to reduce the footprint in the final log file depending on the environment
- Use one logger for every service: Queries, Batch Queries (Jobs), and Data Ingestion (CopyTo/CopyFrom)
- Stop using headers such as: `X-SQL-API-Log`, `X-SQL-API-Profiler`, and `X-SQL-API-Errors` as a way to log info.
- Be able to tag requests with labels as an easier way to provide business metrics
- Metro: Add log-collector utility (`metro`), it will be moved to its own repository. Attaching it here fro development purposes. Try it with the following command `LOG_LEVEL=info npm t | node metro`
- Metro: Creates `metrics-collector.js` a stream to update Prometheus' counters and histograms and exposes them via Express' app (`:9145/metrics`). Use the ones defined in `grok_exporter`
Announcements:
- Profiler is always set. No need to check its existence anymore
- Unify profiler usage for every endpoint
Bug fixes:
- Avoid hung requests while fetching user identifier
- Update `gc-stats` to version 1.4.0
- Replace `zipfile` -> `adm-zip`
- Update `libxmljs` to version 0.19.7
- Update `sqlite` to version 4.2.0
- Adapted pool acquires to the new version of `cartodb-redis`
- Adapted test to use `adm-zip`
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Create middleware and service for pubsub metrics
* Use pubsub middleware for all request
* Replace isEnabled with isDisabled to avoid negation
* Use new headers names
* Add acceptance and unit tests
* Remove commented log calls
* Remove only filters in tests
Remove typo
* Refactor service to ease integration test
* Fix interaction with query controller
* Use middleware at api-router and test all controllers
* Rename user middleware function
* Use sinon latest version
* Fix tests
* Fix typos
* Checks if pubsub config exists to enable the service
* Fix typo
* Normalize headers values for pubsub
* Trim fields when normalizing
* Trim fields when normalizing
This is needed to avoid a "test race" and get it closer to a real unit
test.
When running it along with others it passes:
```
$ make test-unit
...
48 passing (54ms)
12 pending
```
(note it relies on the ordering produced by `find`)
when running it along with some other test that reads the config, it
works as well:
```
$ test/run_tests.sh test/unit/apikeyauth.test.js test/unit/error_handler.test.js
...
3 passing (9ms)
12 pending
```
but when run in isolation, it fails:
```
$ test/run_tests.sh test/unit/error_handler.test.js
...
0 passing (15ms)
3 failing
1) error-handler should return a header with errors:
TypeError: Cannot read property 'environment' of undefined
at errorMiddleware (app/middlewares/error.js:10:28)
at Context.<anonymous> (test/unit/error_handler.test.js:53:26)
2) error-handler JSONP should return a header with error statuscode:
TypeError: Cannot read property 'environment' of undefined
at errorMiddleware (app/middlewares/error.js:10:28)
at Context.<anonymous> (test/unit/error_handler.test.js:79:26)
3) error-handler should escape chars that broke logs regex:
TypeError: Cannot read property 'environment' of undefined
at errorMiddleware (app/middlewares/error.js:10:28)
at Context.<anonymous> (test/unit/error_handler.test.js:108:26)
```
