cartodb/CartoDB-SQL-API
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' of https://github.com/CartoDB/CartoDB-SQL-API into multiuser

Browse Source 
Conflicts:
	app/controllers/app.js
This commit is contained in:
Raul Ochoa 2014-07-30 12:02:17 +02:00
commit abe273caa6
4 changed files with 259 additions and 158 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
app/controllers/app.js
View File

@ -34,7 +34,10 @@ var express = require('express')
, StatsD = require('node-statsd').StatsD
, Meta = require('cartodb-redis')({
host: global.settings.redis_host,
port: global.settings.redis_port
port: global.settings.redis_port,
max: global.settings.redisPool,
idleTimeoutMillis: global.settings.redisIdleTimeoutMillis,
reapIntervalMillis: global.settings.redisReapIntervalMillis
})
// global.settings.app_root + '/app/models/metadata')
, oAuth = require(global.settings.app_root + '/app/models/oauth')
@ -474,8 +477,7 @@ function handleQuery(req, res) {
opts.profiler = req.profiler;
opts.beforeSink = function() {
req.profiler.done('sendResponse');
var report = req.profiler.toString();
res.header('X-SQLAPI-Profiler', report);
res.header('X-SQLAPI-Profiler', req.profiler.toJSONString());
};
}
@ -549,8 +551,7 @@ function handleException(err, res){
if ( res.req && res.req.profiler ) {
res.req.profiler.done('finish');
var report = res.req.profiler.toString();
res.header('X-SQLAPI-Profiler', report);
res.header('X-SQLAPI-Profiler', res.req.profiler.toJSONString());
}
// if the exception defines a http status code, use that, else a 400

364
doc/API.md
View File

@ -1,108 +1,142 @@
SQL API
=======
## SQL API
Request format
--------------
CartoDB's SQL API allows you to interact with your tables and data inside CartoDB as if you were running SQL statements against a normal database. The database behind CartoDB is PostgreSQL so if you need help with specific SQL statements or you want to learn more about it, visit the [official documentation](http://www.postgresql.org/docs/9.1/static/sql.html).
Supported query string parameters:
There are two main situations in which you would want to use the SQL API:
'q': Specifies the SQL query to run
Example:
'http://entrypoint?q=SELECT count(*) FROM mytable'
- You want to **insert, update** or **delete** data. For example, you would like to insert a new column with a latitude and longitude data.
'format': Specifies which format to use for the response.
Supported formats: JSON (the default), GeoJSON,
TopoJSON, CSV, SVG, SHP, arraybuffer
- You want to **select** data from public tables in order to use it on your website or in your app. For example, you need to find the 10 closest records to a particular location.
'filename': Sets the filename to use for the query result
file attachment
Remember that in order to access, read or modify data in private tables, you will need to authenticate your requests. When a table is public, you can do non-authenticated queries that read data, but you cannot write or modify data without authentication.
'skipfields':
Comma separate list of fields that are not wanted
in output. Only useful with "SELECT *" queries.
## Authentication
'dp': Number of digits after the decimal point.
Only affects format GeoJSON, TopoJSON, SVG.
By default this is 6.
For all access to private tables and for write access to public tables, CartoDB enforces secure API access that requires you to authorize your queries. In order to authorize queries, you can use an API key or a Consumer Key and Secret from OAuth.
'api_key': Needed to authenticate in order to modify the database.
### API Key
'cache_policy':
Set to "persist" to have the server send an Cache-Control
header requesting caching devices to keep the response
cached as much as possible. This is best used with a
timestamp value in cache_buster for manual control of
updates.
The API key offers the simplest way to access private data or perform writes and updates to your public data. Remember that your API key protects access to your data, so keep it confidential and only share it if you want others to have this access. If necessary, you can reset your API key in your admin dashboard.
'rows_per_page':
Limit the number of rows in output. Affects all formats,
see also 'page'.
To find your API key:
'page':
When 'rows_per_page' is used to limit rows, 'page' can be
used to specify which page to start returning rows from.
First page has index 0.
- Go to your dashboard.
- Click on your username in the top right corner, and select "Your API keys."
- Here, you can copy your API key, see use examples, and reset your API key.
'order_by':
Causes the result rows to be sorted according to the specified
case sensitive column name. See also 'sort_order'.
To use your API key, pass it as a parameter in an URL call to the CartoDB API. For example, to perform an insert into your table, you would use the following URL structure.
'sort_order':
Optional param combined with 'order_by' one. Values are limited
to ASC (ascending) and DESC (descending), case insensitive. If
not specified or wrongly specified, ASC is assumed by default.
<div class="code-title code-request">Query example with the api_key parameter</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q={SQL statement}&api_key={Your API key}
{% endhighlight %}
### OAuth
Response formats
----------------
OAuth is an authentication protocol that enables users to give permission to an application to act on their behalf without sharing their password. More information can be found at the [OAuth website](http://oauth.net/) or in the [Beginner’s Guide to OAuth](http://hueniverse.com/oauth/) from Hueniverse.
For an easier route, check out the CartoDB clients [below](#libraries-in-different-languages).
### JSON
Getting OAuth keys - For secure access to your application you will need to generate a consumer key in your CartoDB dashboard.
The standard response from the CartoDB SQL API is JSON. If you are
building a web-application, the lightweight JSON format allows you to
quickly integrate data from the SQL API.
- Go to your dashboard.
- Click on your username in the top right corner, and select "Your API keys."
- Here, you can view and copy your OAuth Keys and Tokens, and you can request new OAuth Keys. Remember that requesting new OAuth Keys will affect all applications using OAuth for your CartDB application, and that your old keys will immediately become invalid.
The JSON response is as follows:
```
There are many other resources to help you authenticate access via OAuth. For further reading, take a look at [this list](http://oauth.net/code/) or at the libraries available for [integrating CartoDB](#libraries-in-different-languages) with several programming languages.
## Making calls to the SQL API
CartoDB is based on the rock solid PostgreSQL database. All of your tables reside a single database, which means you can perform complex queries joining tables or carrying out geospatial operations. The best place to learn about PostgreSQL's SQL language is the [official documentation](http://www.postgresql.org/docs/9.1/static/).
CartoDB is also based on PostGIS, so take a look at the [official PostGIS reference](http://postgis.refractions.net/docs/) to know what functionality we support in terms of geospatial operations. All of our tables include a column called *the_geom,* which is a geometry field that indexes geometries in the EPSG:4326 (WGS 1984) coordinate system. All tables also have an automatically generated and updated column called *the_geom_webmercator*. We use the column internally to quickly create tiles for maps.
### URL endpoints
All SQL API requests to your CartoDB account should follow this general pattern:
<div class="code-title code-request">SQL QUERY EXAMPLE</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q={SQL statement}
{% endhighlight %}
If you encounter errors, double-check that you are using the correct account name, and that your SQL statement is valid. A simple example of this pattern is conducting a count of all the records in your table:
<div class="code-title code-request with-result">SQL QUERY COUNT EXAMPLE</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=SELECT count(*) FROM {table_name}
{% endhighlight %}
<div class="code-title">RESULT</div>
{% highlight javascript %}
{
time: 0.007,
total_rows: 1,
rows: [
{
count: 4994
}
]
}
{% endhighlight %}
Finally, remember that in order to use the SQL API, either your table must be public, or you must be authenticated using API Keys or OAuth, as discussed above.
### POST and GET
The CartoDB SQL API is setup to handle both GET and POST requests. You can test the GET method directly in your browser. Below is an example of a JQuery SQL API request to CartoDB:
<div class="code-title">JQUERY</div>
{% highlight javascript %}
$.getJSON('http://'+your_account_name+'.cartodb.com/api/v2/sql/?q='+sql_statement, function(data) {
$.each(data.rows, function(key, val) {
// do something!
});
});
{% endhighlight %}
By default, GET requests work from anywhere. In CartoDB, POST requests work from any website as well. We achieve this by hosting a cross-domain policy file at the root of all of our servers. This allows you the greatest level of flexibility when developing your application.
### Response formats
The standard response from the CartoDB SQL API is JSON. If you are building a web-application, the lightweight JSON format allows you to quickly integrate data from the SQL API.
<div class="code-title code-request with-result">JSON</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=SELECT * FROM {table_name} LIMIT 1
{% endhighlight %}
<div class="code-title">RESULT</div>
{% highlight javascript %}
{
time: 0.006,
total_rows: 1,
fields: {
'year': { type:'string' },
'the_geom': { type:'geometry' },
'cartodb_id': { type:'number' },
'created_at': { type:'date' },
'updated_at': { type:'date' }
'valid': { type:'boolean' }
},
rows: [
{
year: " 2011",
month: 10,
day: "11",
the_geom: "0101000020E610...",
cartodb_id: 1,
created_at: "2012-02-06T22:50:35.778Z",
updated_at: "2012-02-12T21:34:08.193Z",
valid: true
the_geom_webmercator: "0101000020110F000..."
}
],
notices: [ 'notice1', 'notice2' ], // optional
warnings: [ 'warning1', 'warning2' ] // optional
}
```
]
}
{% endhighlight %}
Note that values of numerical fields will be rounded to at most 16 significant
digits due to a limit in the JSON format itself. If more precision is needed
the caller can cast those fields to text using SQL constructs.
Alternatively, you can use the [GeoJSON specification](http://www.geojson.org/geojson-spec.html) to return data from the API. To do so, simply supply the format parameter as GeoJSON:
### GeoJSON
<div class="code-title code-request with-result">GEOJSON</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?format=GeoJSON&q=SELECT * FROM {table_name} LIMIT 1
{% endhighlight %}
Alternatively, you can use the GeoJSON specification for returning data
from the API. To do so, simply supply the format parameter as GeoJSON.
The GeoJSON response is follows:
```
{
<div class="code-title">RESULT</div>
{% highlight javascript %}
{
type: "FeatureCollection",
features: [
{
@ -124,87 +158,153 @@ The GeoJSON response is follows:
}
}
]
}
```
}
{% endhighlight %}
### CSV
The SQL API accepts other output formats that can be useful to export data. Right now you can use the following formats: CSV, SHP, SVG, KML, GeoJSON.
CSV export format has field names in first line (header),
uses CR/NL line endings and HEXEWKB format for geometry fields.
### Getting table information
See https://github.com/postgis/postgis/blob/svn-trunk/doc/ZMSgeoms.txt#L42
Currently, there is no public method to access your table schemas. The simplest way to retrieve table structure is to access the first row of the data,
### KML
<div class="code-title code-request">COLUMNS</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=SELECT * FROM {table_name} LIMIT 1
{% endhighlight %}
See http://en.wikipedia.org/wiki/KML
### Response errors
### SVG
To help you debug your SQL queries, the CartoDB SQL API returns errors as part of the JSON response. Errors come back as follows,
See http://en.wikipedia.org/wiki/SVG
### TopoJSON
See http://en.wikipedia.org/wiki/Topojson
### SHP
Shapefile output composed by .shp, .shx, .dbf and .prj files
compressed in a .zip.
See http://en.wikipedia.org/wiki/Shapefile
### Arraybuffer
An octet stream binary-encoding each field as a typed array of value.
TODO: write more about this format
See http://github.com/CartoDB/CartoDB-SQL-API/issues/96
Response errors
---------------
To help you debug your SQL queries, the CartoDB SQL API returns errors
as part of the JSON response. Errors come back as follows,
```
{
<div class="code-title">RESULT</div>
{% highlight javascript %}
{
error: [
"syntax error at or near "LIMIT""
]
}
{% endhighlight %}
You can use these errors to help understand your SQL. For more complete documentation see the Error Codes and Solutions section of this Users Guide.
### Write data to your CartoDB account
Performing inserts or updates on your data is simple using your [API key](#authentication). All you need to do is supply a correct SQL [INSERT](http://www.postgresql.org/docs/9.1/static/sql-insert.html) or [UPDATE](http://www.postgresql.org/docs/9.1/static/sql-update.html) statement for your table along with the api_key parameter for your account. Be sure to keep these requests private, as anyone with your API key will be able to modify your tables. A correct SQL insert statement means that all the columns you want to insert into already exist in your table, and all the values for those columns are the right type (quoted string, unquoted string for geoms and dates, or numbers).
<div class="code-title code-request">COLUMNS</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=INSERT INTO test_table (column_name, column_name_2, the_geom) VALUES ('this is a string', 11, ST_SetSRID(ST_Point(-110, 43),4326))&api_key={Your API key}
{% endhighlight %}
Updates are just as simple. Here is an example, updating a row based on the value of the cartodb_id column.
<div class="code-title code-request">COLUMNS</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=UPDATE test_table SET column_name = 'my new string value' WHERE cartodb_id = 1 &api_key={Your API key}
{% endhighlight %}
## Handling geospatial data
Handling geospatial data through the SQL API is easy! By default, *the_geom* is returned straight from the database, in a format called Well-Known Binary. There are a handful of ways you can transform your geometries into more useful formats.
The first, is to use the format=GeoJSON method described above. Others can be handled through your SQL statements directly. For example, enclosing your the_geom in a function called [ST_AsGeoJSON](http://www.postgis.org/documentation/manual-svn/ST_AsGeoJSON.html) will allow you to use JSON for your data but a GeoJSON string for your geometry column only. Alternatively, using a the [ST_AsText](http://www.postgis.org/documentation/manual-svn/ST_AsGeoJSON.html) function will return your geometry as Well-Known Text.
<div class="code-title code-request with-result">ASGEOJSON</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=SELECT cartodb_id,ST_AsGeoJSON(the_geom) as the_geom FROM {table_name} LIMIT 1
{% endhighlight %}
<div class="code-title">RESULT</div>
{% highlight javascript %}
{
time: 0.003,
total_rows: 1,
rows: [
{
cartodb_id: 1,
the_geom: "{"type":"Point","coordinates":[-97.3349,35.4979]}"
}
```
You can use these errors to help understand your SQL.
]
}
{% endhighlight %}
Getting table information
-------------------------
<div class="code-title code-request with-result">ASTEXT</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=SELECT cartodb_id,ST_AsText(the_geom) FROM {table_name} LIMIT 1
{% endhighlight %}
Currently, there is no public method for accessing your table schemas. The
simplest way to get table structure is to access the first row of the data:
<div class="code-title">RESULT</div>
{% highlight javascript %}
{
time: 0.003,
total_rows: 1,
rows: [
{
cartodb_id: 1,
the_geom: "POINT(-74.0004162 40.6920918)",
}
]
}
{% endhighlight %}
http://entrypoint?q=SELECT * FROM mytable LIMIT 1
More advanced methods exist in the PostGIS library to extract meaningful data from your geometry. Explore the PostGIS documentation and get familiar with functions such as, [ST_XMin](http://www.postgis.org/docs/ST_XMin.html), [ST_XMax](http://www.postgis.org/docs/ST_XMax.html), [ST_AsText](http://www.postgis.org/docs/ST_AsText.html), and more.
Write data to your CartoDB account
----------------------------------
All data returned from *the_geom* column is in WGS 83 (EPSG:4326). You can change this quickly and easily on the fly using SQL. For example, if you desire geometries in the Hanoi 1972 (EPSG:4147) projection, you could [ST_Transform](http://www.postgis.org/docs/ST_Transform.html),
Perform inserts or updates on your data is simple now using your API
key. All you need to do, is supply a correct SQL INSERT or UPDATE
statement for your table along with the api_key parameter for your
account. Be sure to keep these requests private, as anyone with your API
key will be able to modify your tables. A correct SQL insert statement
means that all the columns you want to insert into already exist in
your table, and all the values for those columns are the right type
(quoted string, unquoted string for geoms and dates, or numbers).
<div class="code-title code-request">ASTEXT</div>
{% highlight bash %}
http://{account}.cartodb.com/api/v2/sql?q=SELECT ST_Transform(the_geom,4147) FROM {table_name} LIMIT 1
{% endhighlight %}
INSERT
CartoDB also stores a second geometry column, *the_geom_webmercator*. We use this internally to build your map tiles as fast as we can. In the user-interface it is hidden, but it is visible and available for use. In this column we store a reprojected version of all your geometries using Web Mercator (EPSG:3857).
http://entrypoint?q=INSERT INTO test_table (column_name, column_name_2, the_geom) VALUES ('this is a string', 11, ST_SetSRID(ST_Point(-110, 43),4326))&api_key={Your API key}
## Query optimizations
Updates are just as simple. Here is an example, updating a row based on
the value of the cartodb_id column.
There are some tricks to consider when using the SQL API that might make your application a little faster.
UPDATE
* Only request the fields you need. Selecting all columns will return a full version of your geometry in *the_geom* as well as a reprojected version in *the_geom_webmercator*.
http://entrypoint?q=UPDATE test_table SET column_name = 'my new string value' WHERE cartodb_id = 1 &api_key={Your API key}
* Use PostGIS functions to simplify and filter out unneeded geometries when possible. One very handy function is, [ST_Simplify](http://www.postgis.org/docs/ST_Simplify.html).
* Remember to build indexes that will speed up some of your more common queries.
* Use *cartodb_id* to retrieve specific rows of your data, this is the unique key column added to every CartoDB table.
<!-- TODO: Link to http://blog.cartodb.com/post/53301057653/faster-data-updates-with-cartodb -->
## API version number
All CartoDB applications use **Version 2** of our APIs. All other APIs are deprecated and will not be maintained or supported. You can check that you are using **Version 2** of our APIs by looking at your request URLS. They should all begin contain **/v2/** in the URLs as follows `http://{account}.cartodb.com/api/v2/`
## Libraries in different languages
To make things easier for developers we provide client libraries for different programming languages. These clients take care of handling OAuth to CartoDB and some of them provide some caching functionalities.
- **RUBY**
The Ruby library is our most mature CartoDB client and is being used in CartoSet. [Fork it on GitHub!](https://github.com/Vizzuality/cartodb-rb-client)
- **R**
To help more researchers use CartoDB to drive their geospatial data, we have released the R client library. [Fork it on GitHub!](https://github.com/Vizzuality/cartodb-r)
- **NODE.js**
This demo app authenticates with your CartoDB over OAuth/XAuth and shows how to perform read and write queries using the SQL API. [Fork it on GitHub!](https://github.com/Vizzuality/cartodb-nodejs)
- **PHP**
The PHP library handles basic OAuth and provides a wrapper around the SQL API to get PHP objects straight from SQL calls to CartoDB. [Fork it on GitHub!](https://github.com/Vizzuality/cartodbclient-php)
- **PYTHON**
Provides API Key and xAuth access to SQL API. [Fork it on GitHub!](https://github.com/vizzuality/cartodb-python)
- **JAVA**
Very basic example of how to access CartoDB SQL API using OAuth. [Fork it on GitHub!](https://github.com/Vizzuality/cartodb-java-client)
- **NET**
.NET library for authenticating with CartoDB using OAuth based on work started by [The Data Republic](http://www.thedatarepublic.com/). [Fork it on GitHub!](https://github.com/thedatarepublic/CartoDBClientDotNET)
- **Clojure**
Clojure library for authenticating with CartoDB using OAuth, maintained by [REDD Metrics](http://www.reddmetrics.com/). [Fork it on GitHub!](https://github.com/reddmetrics/cartodb-clj)
- **iOS**
Objective-C library for interacting with CartoDB in native iOS applications. [Fork it on GitHub!](https://github.com/jmnavarro/cartodb-objectivec-client)

4
npm-shrinkwrap.json generated
View File

@ -142,8 +142,8 @@
}
},
"step-profiler": {
"version": "0.0.1",
"from": "git://github.com/CartoDB/node-step-profiler.git#0.0.1"
"version": "0.1.0",
"from": "git://github.com/CartoDB/node-step-profiler.git#0.1.0"
},
"node-statsd": {
"version": "0.0.7"

2
package.json
View File

@ -27,7 +27,7 @@
"lru-cache":"~2.2.2",
"log4js": "~0.6.10",
"rollbar": "~0.3.1",
"step-profiler": "git://github.com/CartoDB/node-step-profiler.git#0.0.1",
"step-profiler": "git://github.com/CartoDB/node-step-profiler.git#0.1.0",
"node-statsd": "~0.0.7"
},
"devDependencies": {