Commit Graph

32 Commits

Author SHA1 Message Date
Nick Ballenger
f0b5b7e79f Updated Varnish to cache SQL API / Windshaft
The previous version of this file was enough to cache requests for the
SQL API, but unfortunately no traffic was ever reaching Varnish to be
cached. Nginx was proxying directly to the SQL API port, and Varnish was
set to listen on 6081, so it wasn't able to intercept those requests. I
updated the Nginx proxy config to aim at 6081 for requests to both SQL
API and Windshaft, so now Varnish is receiving traffic. However, in
order to know which backend to send traffic to, I had to add a custom
HTTP header in the Nginx proxy pass. That header is picked up in the
`vcl_recv` varnish subroutine and used to switch between backends.

Additionally I've added logic for controlling what hosts can issue an
HTTP PURGE command--in this case just localhost, since everything is on
a single image. The purges will typically come from a Postgres trigger.

As an overview of the purge related changes, see the Varnish docs here:

https://varnish-cache.org/docs/3.0/tutorial/purging.html#http-purges
2019-07-18 16:27:25 -07:00
Nick Ballenger
d02a917f38 Commented out 'invalidation_service' section
I commented out the entire 'invalidation_service' section from
app_config.yml. It _should_ be sufficient to set 'enabled' to false in
that block, in order to prevent the Redis/Resque based invalidation
service from being used inside the postgres trigger for invalidating
cache items, but it's actually easier to just comment out the whole
block. See this portion of the Carto code for reference:

05a05fd695/app/models/user/db_service.rb (L1062-L1070)

The branch we want to go down in that code is the middle one--we want to
end up with `create_function_invalidate_varnish_http` running. That will
create a postgres trigger based on hitting the Varnish server's HTTP
listener, which is running on 6081. (You could have it hit the telnet
port by taking the third branch of that code, but given that telnet
isn't included in later Varnish versions, best not to.)

You want to avoid the first branch of that code, `create_function_invalidate_varnish_invalidation_service`,
because it includes this line:

05a05fd695/app/models/user/db_service.rb (L1601)

That's calling a custom Redis command, `TCH`, which is defined in a repo
that Carto has not open sourced--meaning the 'invalidation service' (as
a Redis job queue for the Resque job runner) can't be used in open
source Carto (unless you reverse engineer the Redis commands it uses.)
2019-07-18 16:17:28 -07:00
Nick Ballenger
5f0eb86bed Changing the Nginx config to a single file
I've combined the core nginx.conf with the proxy config, which all goes
into /etc/nginx/nginx.conf.

I've made a number of changes:

* Nginx now proxies both SQL API and Windshaft requests through Varnish.
* Nginx adds a custom HTTP header, X-Carto-Service, so that Varnish can
differentiate between backends (since it can't do so based on incoming
port).
* I've modified the primary Nginx log format to include more information
on how requests are being proxied--you can now see the upstream address
for proxied requests.
* I've added the `proxy_no_cache` and `proxy_cache_bypass` directives to
the Windshaft and SQL API proxy sections. Without those directives,
Nginx may attempt to act as a cache, returning 304 Not Modified for
resources that more accurately should be cached by Varnish (whose cache
is invalidated via a Postgres trigger for updated metadata).
2019-07-18 16:08:53 -07:00
Stefan Verhoeven
65e9a7b253 Upgrade node + repin scikit-learn + sync config with sample 2019-04-09 15:20:17 +02:00
Stefan Verhoeven
397ffca212 Disable the request size max limit
Refs #63
2018-09-20 08:43:20 +02:00
Stefan Verhoeven
c32b1e1cd2 Rename active record adaptor 2018-07-27 09:11:15 +02:00
Stefan Verhoeven
943c32ff59 Enable new dashboard flag
Back to using master branch of cartodb repo
2018-07-04 11:10:16 +02:00
Stefan Verhoeven
0faa6b44e5 Using OS ogr2ogr 2018-07-02 23:07:35 +02:00
Stefan Verhoeven
5e721f0cac Sync config file 2018-06-22 10:29:42 +02:00
Stefan Verhoeven
8ff9dcd952 Sync config files
Refs #29
2018-04-04 22:23:54 +02:00
Stefan Verhoeven
6b0e3c6e2f Set client_max_body_size to 150MB
Fixes #9
2017-08-01 07:55:56 +02:00
Stefan Verhoeven
bc71a1bf83 Use -h to specify hostname + disabled hubspot + fix missing locales from upstream 2017-04-13 21:14:05 +02:00
Stefan Verhoeven
793ceb5373 Dataviews did not show due to Windshaft opening dataviews on own port instead of reverse proxy 2017-04-13 14:32:49 +02:00
Stefan Verhoeven
94395dfbb0 Widgets did not work due to blocked port 8181
Made widgets fetch data from port 80
2017-04-05 15:38:22 +02:00
Stefan Verhoeven
bbf2c54325 * Embed nginx inside Docker image so all services are reversed proxied behind port 80
* Use Varnish 3 from source tarball, newer and OS Varnish could not disable telnet authentication required by Carto
* Remove sources of varnish, gdal, /cartodb/.git when no longer needed in order to make image smaller
* Dropped Nokia and GMAP basemaps, they did not work
* Dropped schema trigger installation, it is no longer needed
* Configure Carto in Python and NodeJS
* Build assets
* Use self as asset_host, assets are hosted by nginx instead of rails
2017-04-03 22:55:12 +02:00
Stefan Verhoeven
3397aed11d Update to latest config of db sql api and windshaft 2017-04-03 22:45:30 +02:00
Stefan Verhoeven
455d578800 Merged changes of sample configs 2016-09-17 09:52:55 +02:00
Stefan Verhoeven
ee78eecdee Merge pull request #1 from SkyTruth/bugfix-nginx-api-ver
Bugfixes for api version/url and regexps
2016-03-25 21:20:21 +01:00
Stefan Verhoeven
e8085d3e2f Corrected avatar base url. 2016-03-25 18:35:46 +01:00
Stefan Verhoeven
885a51f40b Updated url for data library. 2016-03-25 18:35:09 +01:00
Stefan Verhoeven
90124b61d6 Merge new config 2016-02-07 21:59:11 +01:00
Egil Moeller
a22bb577e1 Bugfixes for api version/url and regexps 2015-12-04 02:41:21 +00:00
Stefan Verhoeven
b71f0ea0d7 Switched to subdomainless urls.
- Allow avatar to be uploaded to local fs.
- Renamed star avatar
- Allow uploads up to 100Mb file size
2015-07-08 16:48:20 +02:00
Stefan Verhoeven
9d841eda1f Use Maps and SQL api behind nginx. 2015-05-18 14:37:45 +02:00
Stefan Verhoeven
d7ab8b6dac Put cartodb db services behind nginx.
Updated configs to master branch of Cartodb repos.
2015-05-16 13:01:49 +02:00
Stefan Verhoeven
7bb64f62d8 Merge remote-tracking branch 'origin/fixlogin' into fixlogin 2015-05-12 08:11:18 +02:00
Stefan Verhoeven
0b474e34ed Added missing config 2015-05-12 08:09:35 +02:00
Stefan Verhoeven
5e698ad813 Add Nokia basemaps + use avatars from cdn 2015-02-18 09:50:20 +01:00
Stefan Verhoeven
5210a06218 Expose sql api and tile webservice. 2015-02-13 14:50:03 +01:00
Stefan Verhoeven
b22b3bc171 Fix NoMethodError by adding missing config key.
See https://github.com/CartoDB/cartodb/issues/1698
2015-02-13 14:47:46 +01:00
Stefan Verhoeven
40b37d6e6d Host in subdomain. Fixes WARNING: Can't verify CSRF token authenticity error. 2015-02-13 14:45:42 +01:00
fleu
21cc7911c1 Populate docker-cartodb with required files.
Here I commit all the required files for a development cartodb container
to be successfully build and run with Docker.
2014-11-15 16:32:19 +01:00