CartoDB-SQL-API/batch
Daniel García Aubert 1905a2bd28 Please, jshint
2019-04-04 15:16:44 +02:00
..
leader Make leader locker to emit on renewal errors 2016-10-18 20:34:22 +02:00
maintenance Re-use redis pool as much as possible 2016-10-17 15:02:34 +02:00
models Fix property names to not expose user's credential info 2018-02-15 13:12:40 +01:00
pubsub Queue seeker refactor: 2017-04-03 19:26:21 +02:00
scheduler Re-insert into the tree if there was a user in done state that gets a new task 2016-10-21 11:42:27 +02:00
util Moved forever module 2016-05-18 11:55:58 +02:00
batch-logger.js reopenFileStreams to parent class 2018-06-18 18:47:12 +02:00
batch.js Please, jshint 2019-04-04 15:16:44 +02:00
index.js Use 'bunyan' logger instead of 'debug' to log debug messages 2019-04-04 14:31:41 +02:00
job_backend.js Use 'bunyan' logger instead of 'debug' to log debug messages 2019-04-04 14:31:41 +02:00
job_canceller.js use job configuration instead of user metadata service in test 2018-06-05 17:56:29 +02:00
job_queue.js Use 'bunyan' logger instead of 'debug' to log debug messages 2019-04-04 14:31:41 +02:00
job_runner.js Pass user's database credentiasl to query runner 2018-02-15 12:26:36 +01:00
job_service.js Use 'bunyan' logger instead of 'debug' to log debug messages 2019-04-04 14:31:41 +02:00
job_status.js Allow users to set max statement_timeout for their queries 2016-10-10 12:01:36 +02:00
query_runner.js Use 'bunyan' logger instead of 'debug' to log debug messages 2019-04-04 14:31:41 +02:00
README.md Add some notes about redis data structures for batch queries 2016-10-17 16:00:30 +02:00
user_database_metadata_service.js remove functionality from parseMetadataToDatabase. DB user and pass not longer needed 2018-06-05 17:58:55 +02:00

Batch Queries

This document describes features from Batch Queries, it also details some internals that might be useful for maintainers and developers.

Redis data structures

Jobs definition

Redis Hash: batch:jobs:{UUID}.

Redis DB: 5.

It stores the job definition, the user, and some metadata like the final status, the failure reason, and so.

Job queues

Redis List: batch:queue:{username}.

Redis DB: 5.

It stores a pending list of jobs per user. It points to a job definition with the {UUID}.

Job notifications

Redis Pub/Sub channel: batch:users.

Redis DB: 0.

In order to notify new jobs, it uses a Pub/Sub channel were the username for the queued job is published.

Job types

Format for the currently supported query types, and what they are missing in terms of features.

Simple

{
    "query": "update ..."
}

Does not support main fallback queries. Ideally it should support something like:

{
    "query": "update ...",
    "onsuccess": "select 'general success fallback'",
    "onerror": "select 'general error fallback'"
}

Multiple

{
    "query": [
        "update ...",
        "select ... into ..."
    ]
}

Does not support main fallback queries. Ideally it should support something like:

{
    "query": [
        "update ...",
        "select ... into ..."
    ],
    "onsuccess": "select 'general success fallback'",
    "onerror": "select 'general error fallback'"
}

Fallback

{
    "query": {
        "query": [
            {
                "query": "select 1",
                "onsuccess": "select 'success fallback query 1'",
                "onerror": "select 'error fallback query 1'"
            },
            {
                "query": "select 2",
                "onerror": "select 'error fallback query 2'"
            }
        ],
        "onsuccess": "select 'general success fallback'",
        "onerror": "select 'general error fallback'"
    }
}

It's weird to have two nested query attributes. Also, it's not possible to mix plain with fallback ones. Ideally it should support something like:

{
    "query": [
        {
            "query": "select 1",
            "onsuccess": "select 'success fallback query 1'",
            "onerror": "select 'error fallback query 1'"
        },
        "select 2"
    ],
    "onsuccess": "select 'general success fallback'",
    "onerror": "select 'general error fallback'"
    }
}

Where you don't need a nested query attribute, it's just an array as in Multiple job type, and you can mix objects and plain queries.