Merge pull request #526 from CartoDB/fix-copy-docs

Fix copy docs
This commit is contained in:
Rafa de la Torre 2018-07-31 16:52:40 +02:00 committed by GitHub
commit a9dbde7877
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -4,8 +4,8 @@ Copy queries allow you to use the [PostgreSQL copy command](https://www.postgres
The support for copy is split across two API end points: The support for copy is split across two API end points:
* `http://{username}.carto.com/api/v2/sql/copyfrom` for uploading data to CARTO * `https://{username}.carto.com/api/v2/sql/copyfrom` for uploading data to CARTO
* `http://{username}.carto.com/api/v2/sql/copyto` for exporting data out of CARTO * `https://{username}.carto.com/api/v2/sql/copyto` for exporting data out of CARTO
## Copy From ## Copy From
@ -29,7 +29,9 @@ Composing a chunked POST is moderately complicated, so most developers will use
### Example ### Example
For a table to be readable by CARTO, it must have a minimum of three columns with specific data types: #### Table preparation
First of all, you'll need to have **a table with the right schema** to copy your data into. For a table to be readable by CARTO, it must have a minimum of three columns with specific data types:
* `cartodb_id`, a `serial` primary key * `cartodb_id`, a `serial` primary key
* `the_geom`, a `geometry` in the ESPG:4326 projection (aka long/lat) * `the_geom`, a `geometry` in the ESPG:4326 projection (aka long/lat)
@ -37,20 +39,29 @@ For a table to be readable by CARTO, it must have a minimum of three columns wit
Creating a new CARTO table with all the right triggers and columns can be tricky, so here is an example: Creating a new CARTO table with all the right triggers and columns can be tricky, so here is an example:
-- create the table using the *required* columns and a ```sql
-- couple more -- Create the table using the *required* columns and a couple more
CREATE TABLE upload_example ( CREATE TABLE upload_example (
the_geom geometry, the_geom geometry,
name text, name text,
age integer age integer
); );
-- adds the 'cartodb_id' and 'the_geom_webmercator'
-- adds the required triggers and indexes
SELECT CDB_CartodbfyTable(current_schema, 'upload_example');
```
you can get these queries executed directly through the SQL API this way, just by replacing the `{username}` and `{api_key}` by yours:
```sh
# Table creation
curl -v "https://{user}.carto.com/api/v2/sql?api_key={api_key}&q=CREATE+TABLE+upload_example+(the_geom+geometry,name+text,age+integer)"
# Cartodbfy table (optional, but allows for table registartion in Builder dashboard)
curl -v "http://{user}.carto.com/api/v2/sql?api_key={api_key}&q=SELECT+CDB_CartodbfyTable(current_schema,'upload_example')"
```
-- adds the 'cartodb_id' and 'the_geom_webmercator'
-- adds the required triggers and indexes
SELECT CDB_CartodbfyTable('upload_example');
-- Note that CDB_CartodbfyTable is called differently if you have an organization user
-- SELECT CDB_CartodbfyTable('your_org_username', 'upload_example');
Now you are ready to upload your file. Suppose you have a CSV file like this: Now you are ready to upload your file. Suppose you have a CSV file like this:
@ -68,7 +79,7 @@ The `FROM STDIN` option tells the database that the input will come from a data
To actually run upload, you will need a tool or script that can generate a chunked POST, so here are a few examples in different languages. To actually run upload, you will need a tool or script that can generate a chunked POST, so here are a few examples in different languages.
### CURL Example #### CURL Example
The [curl](https://curl.haxx.se/) utility makes it easy to run web requests from the command-line, and supports chunked POST upload, so it can feed the `copyfrom` end point. The [curl](https://curl.haxx.se/) utility makes it easy to run web requests from the command-line, and supports chunked POST upload, so it can feed the `copyfrom` end point.
@ -78,7 +89,7 @@ Assuming that you have already created the table, and that the CSV file is named
-H "Transfer-Encoding: chunked" \ -H "Transfer-Encoding: chunked" \
-H "Content-Type: application/octet-stream" \ -H "Content-Type: application/octet-stream" \
--data-binary @upload_example.csv \ --data-binary @upload_example.csv \
"http://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)" "https://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
To upload a larger file, using compression for a faster transfer, first compress the file, and then upload it with the content encoding set: To upload a larger file, using compression for a faster transfer, first compress the file, and then upload it with the content encoding set:
@ -87,10 +98,10 @@ To upload a larger file, using compression for a faster transfer, first compress
-H "Transfer-Encoding: chunked" \ -H "Transfer-Encoding: chunked" \
-H "Content-Type: application/octet-stream" \ -H "Content-Type: application/octet-stream" \
--data-binary @upload_example.csv.gz \ --data-binary @upload_example.csv.gz \
"http://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)" "https://{username}.carto.com/api/v2/sql/copyfrom?api_key={api_key}&q=COPY+upload_example+(the_geom,+name,+age)+FROM+STDIN+WITH+(FORMAT+csv,+HEADER+true)"
### Python Example #### Python Example
The [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library for HTTP makes doing a file upload relatively terse. The [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library for HTTP makes doing a file upload relatively terse.
@ -102,7 +113,7 @@ username = {api_key}
upload_file = 'upload_example.csv' upload_file = 'upload_example.csv'
q = "COPY upload_example (the_geom, name, age) FROM STDIN WITH (FORMAT csv, HEADER true)" q = "COPY upload_example (the_geom, name, age) FROM STDIN WITH (FORMAT csv, HEADER true)"
url = "http://%s.carto.com/api/v2/sql/copyfrom" % username url = "https://%s.carto.com/api/v2/sql/copyfrom" % username
with open(upload_file, 'rb') as f: with open(upload_file, 'rb') as f:
r = requests.post(url, params={'api_key': api_key, 'q': q}, data=f, stream=True) r = requests.post(url, params={'api_key': api_key, 'q': q}, data=f, stream=True)