cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Changed and reorganized batch public doc

Browse Source
This commit is contained in:
Daniel García Aubert 2016-04-26 15:43:15 +02:00
parent 7899985678
commit 437e340deb
1 changed files with 41 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

70
doc/sql_batch_api.md
View File

@ -2,9 +2,9 @@
The SQL Batch API enables you to request queries with long-running CPU processing times. Typically, these kind of requests raise timeout errors when using the SQL API. In order to avoid timeouts, you can use the SQL Batch API to [create](#create-a-job), [read](#read-a-job), [list](#list-jobs), [update](#update-a-job) and [cancel](#cancel-a-job) queries. You can also run [multiple](#multi-query-batch-jobs) SQL queries in one job. The SQL Batch API schedules the incoming jobs and allows you to request the job status for each query.
_The Batch API is not intended to be used for large query payloads than contain over 4096 characters. If you are inserting a large number of rows (over 4kb) into your table, you still need to use the [Import API](http://docs.cartodb.com/cartodb-platform/import-api/) or [SQL API](http://docs.cartodb.com/cartodb-platform/sql-api/) for this type of data management. The Batch API is specific to queries and CPU usage._
_The Batch API is not intended to be used for large query payloads than contain over 4096 characters (4kb). For instance, if you are inserting a large number of rows into your table, you still need to use the [Import API](http://docs.cartodb.com/cartodb-platform/import-api/) or [SQL API](http://docs.cartodb.com/cartodb-platform/sql-api/) for this type of data management. The Batch API is specific to queries and CPU usage._
**Note:** In order to use the SQL Batch API, your table must be public, or you must be [authenticated](http://docs.cartodb.com/cartodb-platform/sql-api/authentication/#authentication) using API keys. **Review [Best Practices](#best-practices) for usage notes regarding the SQL Batch API.**
**Note:** In order to use the SQL Batch API, your table must be public, or you must be [authenticated](http://docs.cartodb.com/cartodb-platform/sql-api/authentication/#authentication) using API keys. **Review [Private Datasets](#private-datasets) for manipulate private datasets with SQL Batch API.**
## SQL Batch API Job Schema
@ -171,7 +171,7 @@ BODY: [{
}, {
"job_id": "ba25ed54-75b4-431b-af27-eb6b9e5428ff",
"user": "cartofante"
"query": "DELETE FROM user_dataset",
"query": "CREATE TABLE world_airports AS SELECT a.cartodb_id, a.the_geom, a.the_geom_webmercator, a.name airport, b.name country FROM world_borders b JOIN airports a ON ST_Contains(b.the_geom, a.the_geom)",
"status": "pending",
"created_at": "2015-12-15T07:43:12Z",
"updated_at": "2015-12-15T07:43:12Z"
@ -210,7 +210,7 @@ To update an SQL Batch API job, make a PUT request with the following parameters
```bash
HEADERS: PUT /api/v2/sql/job/de305d54-75b4-431b-adb2-eb6b9e546014
BODY: {
"query": "SELECT cartodb_id FROM user_dataset"
"query": "UPDATE airports SET type = 'military'"
}
```
@ -221,7 +221,7 @@ HEADERS: 200 OK; application/json
BODY: {
"job_id": "de305d54-75b4-431b-adb2-eb6b9e546014",
"user": "cartofante"
"query": "SELECT cartodb_id FROM user_dataset",
"query": "UPDATE airports SET type = 'military'",
"status": "pending",
"created_at": "2015-12-15T07:36:25Z",
"updated_at": "2015-12-17T15:45:56Z"
@ -242,7 +242,7 @@ If you are using the Batch API update operation for cURL PUT request, use the fo
```bash
curl -X PUT -H "Content-Type: application/json" -d '{
"query": "CREATE TABLE world_airports AS SELECT a.cartodb_id, a.the_geom, a.the_geom_webmercator, a.name airport, b.name country FROM world_borders b JOIN airports a ON ST_Contains(b.the_geom, a.the_geom)"
"query": "UPDATE airports SET type = 'military'"
}' "http://{username}.cartodb.com/api/v2/sql/job/{job_id}"
```
@ -257,7 +257,7 @@ var options = {
headers: {
"content-type": "application/json"
},
body: { query: "select the_geom from my_public_dataset" },
body: { query: "UPDATE airports SET type = 'military'" },
json: true
};
@ -457,45 +457,43 @@ request(options, function (error, response, body) {
});
```
## Retrieving Job Results
## Fetching Job Results
In some scenarios, you may need to collect the job results. If that is the case, wrap the query with SELECT * INTO, or CREATE TABLE AS. The results are stored in a new table in your database. For example:
In some scenarios, you may need to fetch the output of a job. If that is the case, wrap the query with `SELECT * INTO`, or `CREATE TABLE AS`. The output is stored in a new table in your database. Given the query `SELECT * FROM airports`:
1. A job query, `SELECT * FROM user_dataset;`
1. Wrap the query `SELECT * INTO job_result FROM (SELECT * FROM airports) AS job`
2. Wrap the query, `SELECT * INTO job_result FROM (SELECT * FROM user_dataset) AS job;`
2. [Create a job](#create-a-job) as described previously.
3. Once the table is created, retrieve the results through the CartoDB SQL API, `SELECT * FROM job_result;`
3. Once the job is done, fetch the results through the [CartoDB SQL API](http://docs.cartodb.com/cartodb-platform/sql-api/), `SELECT * FROM job_result`
Alternatively, use `UPDATE` as there is no need to wrap the query. For example, `UPDATE airports SET type = 'international'`.
**Note**: if you need to create a map or analysis with the new table you should use [CDB_CartodbfyTable function](https://github.com/CartoDB/cartodb-postgresql/blob/master/doc/cartodbfy-requirements.rst).
## Best Practices
For best practices, ensure that you are following these recommended usage notes when using the SQL Batch API:
## Private Datasets
- The Batch API is not intended for large INSERT jobs, use the [Import API](http://docs.cartodb.com/cartodb-platform/import-api/) for this type of data management
- Only the `query` element of the job scheme can be modified. All other elements of the job schema are defined by the SQL Batch API and are read-only
For all access to private tables, and for write access to public tables, Batch API requires API Key to [authorize your queries](http://docs.cartodb.com/cartodb-platform/sql-api/authentication/#authentication). The following error message appears if you are using private tables and are not authenticated:
- If you are creating a new table from a query without wrapping the query (via INTO or CREATE TABLE AS), the result will create ghost tables that are not configured for CartoDB. Ensure you wrap the query correctly, or use UPDATE instead
```bash
{
"error": [
"permission denied"
]
}
```
- There is a limit of 4kb per job, or 4096 characters per query. The following limit error message appears if your job exceeds this size:
In order to get full access, you must use your API Key.
`Your payload is too large (4097). Max size allowed is 4096 (4kb)`
- Your table must be public, or you must be [authenticated](http://docs.cartodb.com/cartodb-platform/sql-api/authentication/#authentication) using API keys. The following error message appears if you are using private tables, or are not authenticated:
`{"error":["permission denied"]}`
For all access to private tables, and for write access to public tables, CartoDB enforces secure API access that requires you to authorize your queries. In order to authorize queries, you can use an API Key or a Consumer Key, as shown in the following example.
Using cURL tool:
```bash
curl -X POST -H "Content-Type: application/json" -d '{
"query": "{your-expensive-query}"
"query": "{query}"
}' "http://{username}.cartodb.com/api/v2/sql/job?api_key={api_key}"
```
If you are using the Batch API create operation for a Node.js client POST request with private tables, use the following code:
Using Node.js request client:
```bash
var request = require("request");
@ -510,7 +508,7 @@ var options = {
"content-type": "application/json"
},
body: {
query: "{your-expensive-query}"
query: "{query}"
},
json: true
};
@ -521,3 +519,17 @@ request(options, function (error, response, body) {
console.log(body);
});
```
## Best Practices
For best practices, ensure that you are following these recommended usage notes when using the SQL Batch API:
- The Batch API is not intended for large query payloads (e.g: inserting thousands of rows), use the [Import API](http://docs.cartodb.com/cartodb-platform/import-api/) for this type of data management
- There is a limit of 4kb per job. The following error message appears if your job exceeds this size:
`Your payload is too large. Max size allowed is 4096 (4kb)`
- Only the `query` element of the job scheme can be modified. All other elements of the job schema are defined by the SQL Batch API and are read-only