commit
a9dbde7877
@ -4,8 +4,8 @@ Copy queries allow you to use the [PostgreSQL copy command](https://www.postgres
|
|||||||
|
|
||||||
The support for copy is split across two API end points:
|
The support for copy is split across two API end points:
|
||||||
|
|
||||||
* `http://{username}.carto.com/api/v2/sql/copyfrom` for uploading data to CARTO
|
* `https://{username}.carto.com/api/v2/sql/copyfrom` for uploading data to CARTO
|
||||||
* `http://{username}.carto.com/api/v2/sql/copyto` for exporting data out of CARTO
|
* `https://{username}.carto.com/api/v2/sql/copyto` for exporting data out of CARTO
|
||||||
|
|
||||||
## Copy From
|
## Copy From
|
||||||
|
|
||||||
@ -29,7 +29,9 @@ Composing a chunked POST is moderately complicated, so most developers will use
|
|||||||
|
|
||||||
### Example
|
### Example
|
||||||
|
|
||||||
For a table to be readable by CARTO, it must have a minimum of three columns with specific data types:
|
#### Table preparation
|
||||||
|
|
||||||
|
First of all, you'll need to have **a table with the right schema** to copy your data into. For a table to be readable by CARTO, it must have a minimum of three columns with specific data types:
|
||||||
|
|
||||||
* `cartodb_id`, a `serial` primary key
|
* `cartodb_id`, a `serial` primary key
|
||||||
* `the_geom`, a `geometry` in the ESPG:4326 projection (aka long/lat)
|
* `the_geom`, a `geometry` in the ESPG:4326 projection (aka long/lat)
|
||||||
@ -37,20 +39,29 @@ For a table to be readable by CARTO, it must have a minimum of three columns wit
|
|||||||
|
|
||||||
Creating a new CARTO table with all the right triggers and columns can be tricky, so here is an example:
|
Creating a new CARTO table with all the right triggers and columns can be tricky, so here is an example:
|
||||||
|
|
||||||
-- create the table using the *required* columns and a
|
```sql
|
||||||
-- couple more
|
-- Create the table using the *required* columns and a couple more
|
||||||
CREATE TABLE upload_example (
|
CREATE TABLE upload_example (
|
||||||
the_geom geometry,
|
the_geom geometry,
|
||||||
name text,
|
name text,
|
||||||
age integer
|
age integer
|
||||||
);
|
);
|
||||||
|
|
||||||
|
-- adds the 'cartodb_id' and 'the_geom_webmercator'
|
||||||
|
-- adds the required triggers and indexes
|
||||||
|
SELECT CDB_CartodbfyTable(current_schema, 'upload_example');
|
||||||
|
```
|
||||||
|
|
||||||
|
you can get these queries executed directly through the SQL API this way, just by replacing the `{username}` and `{api_key}` by yours:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Table creation
|
||||||
|
curl -v "https://{user}.carto.com/api/v2/sql?api_key={api_key}&q=CREATE+TABLE+upload_example+(the_geom+geometry,name+text,age+integer)"
|
||||||
|
|
||||||
|
# Cartodbfy table (optional, but allows for table registartion in Builder dashboard)
|
||||||
|
curl -v "http://{user}.carto.com/api/v2/sql?api_key={api_key}&q=SELECT+CDB_CartodbfyTable(current_schema,'upload_example')"
|
||||||
|
```
|
||||||
|
|
||||||
-- adds the 'cartodb_id' and 'the_geom_webmercator'
|
|
||||||
-- adds the required triggers and indexes
|
|
||||||
SELECT CDB_CartodbfyTable('upload_example');
|
|
||||||
|
|
||||||
-- Note that CDB_CartodbfyTable is called differently if you have an organization user
|
|
||||||
-- SELECT CDB_CartodbfyTable('your_org_username', 'upload_example');
|
|
||||||
|
|
||||||
Now you are ready to upload your file. Suppose you have a CSV file like this:
|
Now you are ready to upload your file. Suppose you have a CSV file like this:
|
||||||
|
|
||||||
@ -68,7 +79,7 @@ The `FROM STDIN` option tells the database that the input will come from a data
|
|||||||
|
|
||||||
To actually run upload, you will need a tool or script that can generate a chunked POST, so here are a few examples in different languages.
|
To actually run upload, you will need a tool or script that can generate a chunked POST, so here are a few examples in different languages.
|
||||||
|
|
||||||
### CURL Example
|
#### CURL Example
|
||||||
|
|
||||||
The [curl](https://curl.haxx.se/) utility makes it easy to run web requests from the command-line, and supports chunked POST upload, so it can feed the `copyfrom` end point.
|
The [curl](https://curl.haxx.se/) utility makes it easy to run web requests from the command-line, and supports chunked POST upload, so it can feed the `copyfrom` end point.
|
||||||
|
|
||||||
@ -78,7 +89,7 @@ Assuming that you have already created the table, and that the CSV file is named
|
|||||||
-H "Transfer-Encoding: chunked" \
|
-H "Transfer-Encoding: chunked" \
|
||||||
-H "Content-Type: application/octet-stream" \
|
-H "Content-Type: application/octet-stream" \
|
||||||
--data-binary @upload_example.csv \
|
--data-binary @upload_example.csv \
|
||||||
"http://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
|
"https://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
|
||||||
|
|
||||||
To upload a larger file, using compression for a faster transfer, first compress the file, and then upload it with the content encoding set:
|
To upload a larger file, using compression for a faster transfer, first compress the file, and then upload it with the content encoding set:
|
||||||
|
|
||||||
@ -87,10 +98,10 @@ To upload a larger file, using compression for a faster transfer, first compress
|
|||||||
-H "Transfer-Encoding: chunked" \
|
-H "Transfer-Encoding: chunked" \
|
||||||
-H "Content-Type: application/octet-stream" \
|
-H "Content-Type: application/octet-stream" \
|
||||||
--data-binary @upload_example.csv.gz \
|
--data-binary @upload_example.csv.gz \
|
||||||
"http://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
|
"https://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
|
||||||
|
|
||||||
|
|
||||||
### Python Example
|
#### Python Example
|
||||||
|
|
||||||
The [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library for HTTP makes doing a file upload relatively terse.
|
The [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library for HTTP makes doing a file upload relatively terse.
|
||||||
|
|
||||||
@ -102,7 +113,7 @@ username = {api_key}
|
|||||||
upload_file = 'upload_example.csv'
|
upload_file = 'upload_example.csv'
|
||||||
q = "COPY upload_example (the_geom, name, age) FROM STDIN WITH (FORMAT csv, HEADER true)"
|
q = "COPY upload_example (the_geom, name, age) FROM STDIN WITH (FORMAT csv, HEADER true)"
|
||||||
|
|
||||||
url = "http://%s.carto.com/api/v2/sql/copyfrom" % username
|
url = "https://%s.carto.com/api/v2/sql/copyfrom" % username
|
||||||
with open(upload_file, 'rb') as f:
|
with open(upload_file, 'rb') as f:
|
||||||
r = requests.post(url, params={'api_key': api_key, 'q': q}, data=f, stream=True)
|
r = requests.post(url, params={'api_key': api_key, 'q': q}, data=f, stream=True)
|
||||||
|
|
||||||
|
Loading…
Reference in New Issue
Block a user