Merge pull request #526 from CartoDB/fix-copy-docs

Fix copy docs
This commit is contained in:
Rafa de la Torre 2018-07-31 16:52:40 +02:00 committed by GitHub
commit a9dbde7877
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -4,8 +4,8 @@ Copy queries allow you to use the [PostgreSQL copy command](https://www.postgres
The support for copy is split across two API end points:
* `http://{username}.carto.com/api/v2/sql/copyfrom` for uploading data to CARTO
* `http://{username}.carto.com/api/v2/sql/copyto` for exporting data out of CARTO
* `https://{username}.carto.com/api/v2/sql/copyfrom` for uploading data to CARTO
* `https://{username}.carto.com/api/v2/sql/copyto` for exporting data out of CARTO
## Copy From
@ -29,7 +29,9 @@ Composing a chunked POST is moderately complicated, so most developers will use
### Example
For a table to be readable by CARTO, it must have a minimum of three columns with specific data types:
#### Table preparation
First of all, you'll need to have **a table with the right schema** to copy your data into. For a table to be readable by CARTO, it must have a minimum of three columns with specific data types:
* `cartodb_id`, a `serial` primary key
* `the_geom`, a `geometry` in the ESPG:4326 projection (aka long/lat)
@ -37,8 +39,8 @@ For a table to be readable by CARTO, it must have a minimum of three columns wit
Creating a new CARTO table with all the right triggers and columns can be tricky, so here is an example:
-- create the table using the *required* columns and a
-- couple more
```sql
-- Create the table using the *required* columns and a couple more
CREATE TABLE upload_example (
the_geom geometry,
name text,
@ -47,10 +49,19 @@ Creating a new CARTO table with all the right triggers and columns can be tricky
-- adds the 'cartodb_id' and 'the_geom_webmercator'
-- adds the required triggers and indexes
SELECT CDB_CartodbfyTable('upload_example');
SELECT CDB_CartodbfyTable(current_schema, 'upload_example');
```
you can get these queries executed directly through the SQL API this way, just by replacing the `{username}` and `{api_key}` by yours:
```sh
# Table creation
curl -v "https://{user}.carto.com/api/v2/sql?api_key={api_key}&q=CREATE+TABLE+upload_example+(the_geom+geometry,name+text,age+integer)"
# Cartodbfy table (optional, but allows for table registartion in Builder dashboard)
curl -v "http://{user}.carto.com/api/v2/sql?api_key={api_key}&q=SELECT+CDB_CartodbfyTable(current_schema,'upload_example')"
```
-- Note that CDB_CartodbfyTable is called differently if you have an organization user
-- SELECT CDB_CartodbfyTable('your_org_username', 'upload_example');
Now you are ready to upload your file. Suppose you have a CSV file like this:
@ -68,7 +79,7 @@ The `FROM STDIN` option tells the database that the input will come from a data
To actually run upload, you will need a tool or script that can generate a chunked POST, so here are a few examples in different languages.
### CURL Example
#### CURL Example
The [curl](https://curl.haxx.se/) utility makes it easy to run web requests from the command-line, and supports chunked POST upload, so it can feed the `copyfrom` end point.
@ -78,7 +89,7 @@ Assuming that you have already created the table, and that the CSV file is named
-H "Transfer-Encoding: chunked" \
-H "Content-Type: application/octet-stream" \
--data-binary @upload_example.csv \
"http://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
"https://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
To upload a larger file, using compression for a faster transfer, first compress the file, and then upload it with the content encoding set:
@ -87,10 +98,10 @@ To upload a larger file, using compression for a faster transfer, first compress
-H "Transfer-Encoding: chunked" \
-H "Content-Type: application/octet-stream" \
--data-binary @upload_example.csv.gz \
"http://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
"https://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
### Python Example
#### Python Example
The [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library for HTTP makes doing a file upload relatively terse.
@ -102,7 +113,7 @@ username = {api_key}
upload_file = 'upload_example.csv'
q = "COPY upload_example (the_geom, name, age) FROM STDIN WITH (FORMAT csv, HEADER true)"
url = "http://%s.carto.com/api/v2/sql/copyfrom" % username
url = "https://%s.carto.com/api/v2/sql/copyfrom" % username
with open(upload_file, 'rb') as f:
r = requests.post(url, params={'api_key': api_key, 'q': q}, data=f, stream=True)