Edge nginx
Certificates, the real client IP, timeouts and the request body limit all live at this layer. Get it wrong here and the cluster looks perfectly healthy while users see 413s, stream cutoffs and 502s.
← Documentation index · Previous: Configure database · Back to: Deployment overview
1. Why this layer exists
Istio's ingressgateway has only one NodePort on a single host, and the reference environment's Gateway declares port 80 only —
TLS does not terminate inside the cluster. So you need one nginx layer in front of the cluster, and it does four things:
| Responsibility | What happens if you skip it |
|---|---|
| Certificates and HTTPS | You can only reach http://<IP>:<NodePort>, and OIDC callbacks and Secure cookies do not work |
| Request body limit | nginx defaults to client_max_body_size 1m; uploading 12 images at once returns 413 immediately |
| Long-connection timeout | nginx defaults to proxy_read_timeout 60s; conversations and SSE are cut off at 60 seconds |
| Real client IP | Logs, rate limiting and per-IP troubleshooting all become useless (you only see the node address) |
2. Installation and certificates
cat /etc/os-release # confirm the distribution
sudo apt update
sudo apt install nginx
sudo systemctl enable --now nginx
One certificate for every domain
A single host usually has 4–8 subdomains pointing at one entry point (site, gateway, identity, consoles), all under one root domain — a private deployment only registers that one root domain, never the subdomains separately. Signing a single multi-domain (SAN) certificate is less work than one
certificate per domain: renewal covers one lineage, and every server block refers to the same pair of files.
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx \
--cert-name <cert-name> \
-d titanides.com -d gateway.titanides.com -d accounts.titanides.com -d console.titanides.com
Three things to settle up front, so adding domains later does not get messy:
- Always pass
--cert-name. Without it, certbot names the lineage after the first-ddomain (that is,/etc/letsencrypt/live/titanides.com/). Add a domain carelessly later and you end up with a second lineage while nginx still points at the first. Give it a name unrelated to any domain (such as<product-name>) and that cannot happen. - The certificate path and
server_nameare independent. Severalserverblocks referring to the same pair ofssl_certificatefiles is perfectly normal; the directory name does not have to match a domain. - The
--nginxplugin edits your configuration directly: it adds thessl_certificatelines and the 80→443 redirect blocks inside the files undersites-enabled, and leaves# managed by Certbotmarkers.
Adding a domain later: --expand
When a new domain appears (say an entry point for kubectl later), do not sign a new certificate — merge it into the existing one:
sudo certbot --nginx --cert-name <cert-name> --expand \
-d titanides.com -d gateway.titanides.com -d accounts.titanides.com -d console.titanides.com \
-d k8s.titanides.com
- List every old domain plus the new one on the command line. List only the new domain and you get a certificate that
covers nothing but that domain — the business domains immediately fail with
NET::ERR_CERT_COMMON_NAME_INVALID; --expandskips the "create new or replace" prompt and merges the new domain into the lineage named by--cert-name;- Forgetting
--expandis harmless — certbot asks. The failure that actually bites is the bullet above: leaving out an old domain.
Confirm afterwards:
sudo certbot certificates # domains, expiry and certificate path for each lineage
When you do not want certbot to edit the configuration
Teams that maintain sites-enabled by hand should use the certificate-only modes and write the certificate path into the server block themselves:
# Validate over port 80; nginx must be stopped first
sudo systemctl stop nginx
sudo certbot certonly --standalone -d titanides.com
sudo systemctl start nginx
# Or keep nginx running and use webroot
sudo certbot certonly --webroot -w /var/www/html -d titanides.com
Do not let part of the certificates on one host be managed by certbot and part by hand — the renewal log does not show who renews which, so afterwards you can only go by memory. Either give everything to
--nginx, or usecertonlyeverywhere and write the configuration yourself.
Renewal
apt install certbot installs a systemd timer (visible with systemctl list-timers | grep certbot). It renews 30 days before
expiry and then runs the deploy hook — under --nginx that is a reload of nginx by default. Before going live, run:
sudo certbot renew --dry-run
Renewal uses HTTP-01, which reaches back from the public network to
/.well-known/acme-challenge/. If a CDN or a redirect rule sits in front, make sure that path is not blocked — see section 7 of Cloudflare in front.
3. Reverse proxy to Istio
Add a site under /etc/nginx/sites-enabled/. Get the NodePort first:
kubectl -n istio-system get svc istio-ingressgateway \
-o jsonpath='{.spec.ports[?(@.port==80)].nodePort}{"\n"}'
A site file is usually the four blocks below. The map must live at the http level — files under sites-enabled
are already included inside http {}, so writing it at the top level of the file is enough; its position in the file does
not matter, because nginx reads the whole configuration before resolving references.
# 1. WebSocket upgrade decision: one per host, shared by every server block
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# 2. The business entry point: every domain that points at the cluster shares one server block
server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
server_name titanides.com gateway.titanides.com accounts.titanides.com console.titanides.com;
# One multi-domain certificate; other server blocks may point at the same pair of files
ssl_certificate /etc/letsencrypt/live/<cert-name>/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/<cert-name>/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
proxy_pass http://127.0.0.1:<http-nodeport>;
# Host must pass through unchanged: Istio's VirtualService matches on the Host header
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Signature header for payment callbacks; do not drop it
proxy_set_header Stripe-Signature $http_stripe_signature;
# WebSocket (GraphQL subscriptions)
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
# Request body: see the parameter table below
client_max_body_size 50M;
# Long tasks: the default is 60s, which cuts conversations and exports off on schedule
proxy_connect_timeout 5s;
proxy_send_timeout 86400s;
proxy_read_timeout 86400s;
proxy_cache_bypass $http_upgrade;
}
# 3. A separate path for event streams (optional, see the end of this section)
location ^~ /stream {
proxy_pass http://127.0.0.1:<http-nodeport>;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Connection ''; # SSE is not WebSocket: clear the Upgrade header
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
add_header Cache-Control no-cache;
}
}
# 4. 80 → 443
server {
listen 80;
listen [::]:80;
server_name titanides.com gateway.titanides.com accounts.titanides.com console.titanides.com;
return 301 https://$host$request_uri;
}
After changing it, run sudo nginx -t first, then sudo nginx -s reload.
certbot fragments block 4. For each domain it adds, it starts another
listen 80server block containingif ($host = titanides.com) { return 301 https://$host$request_uri; }. A block that contains severalifstatements is normal and equivalent to block 4 above; behaviour is unaffected, so there is no need to merge them by hand. The reverse is also true: if you write block 4 by hand and then run--nginxagain, you get both — still equivalent, just harder to read.
The two levels of client_max_body_size
nginx takes the innermost matching value. The reference implementation sets 8m at server level and 50M inside
location /, so 50M is what applies; the server-level value only affects paths that do not override it in their own
location. Having both makes later readers misjudge it — set it in one place.
Why block 3 gives /stream its own location
proxy_buffering defaults to on, so nginx buffers the upstream response before sending it. SSE has to be delivered
incrementally, so:
- if the upstream sends
X-Accel-Buffering: no, the main location streams fine without disabling buffering; - if it does not, or you are not sure, you must set
proxy_buffering off; proxy_cache off;on the location.
The reference implementation puts event streams on a separate path precisely so that path can carry the full set of parameters without changing the behaviour of the main location (which keeps buffering, and is faster for static responses).
Why these parameters
| Directive | Reference value | Rationale | Symptom if misconfigured |
|---|---|---|---|
Host pass-through |
$host |
VirtualService matches on Host | Every domain lands on the default service, or returns 404 outright |
client_max_body_size |
Reference 50M; must be ≥ the gateway's upload limit | The gateway has its own UPLOAD_MAX_FILE_SIZE layer |
413 when the upload reaches 100% (the default is only 1m) |
proxy_read_timeout |
Reference 86400s; at least ≥ AJAX_TIMEOUT_MS (900s) |
Gateway-to-backend forwarding timeout is 15 minutes | Long tasks / conversation streams cut off at 60 seconds (the default) |
proxy_send_timeout |
Same order as read | Default 60s; a slow origin is disconnected mid-transfer | A large file is cut off part-way through |
proxy_buffering off |
Only on /stream |
SSE (/events) |
Events are held in the buffer and the frontend stops updating |
Connection $connection_upgrade |
map result | GraphQL subscriptions use WebSocket | Handshake 400 / the connection closes immediately |
Stripe-Signature |
pass through | The gateway verifies the signature after forwarding the callback to the payment service | Top-ups never arrive; the log shows a signature verification failure |
client_max_body_sizeis the outer of two layers: the gateway also hasUPLOAD_MAX_FILE_SIZEandGRAPHQL_BODY_LIMIT. Tune only nginx and not the gateway and you still get 413; tune only the gateway and not nginx and nginx rejects it first. When the two disagree, the smaller one wins — in the reference environment nginx is 50M while the gateway default is higher, so 50M is the real ceiling. To allow larger files, raise both.
4. A separate server for the cluster API
kubectl and the tethys console both need to reach the API Server from outside (MicroK8s defaults to 16443). Do not
share one server block with business traffic — kubectl's exec / logs use WebSocket, and their buffering and timeout requirements differ:
server {
listen 443 ssl;
server_name k8s.titanides.com;
location / {
proxy_pass https://127.0.0.1:16443;
# Skip verification with a self-signed certificate; in production configure proxy_ssl_trusted_certificate
proxy_ssl_verify off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade; # required for kubectl exec / logs
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
proxy_buffering off;
}
}
Exposing the API Server to the public network is exposing the cluster control plane. Do this only from an internal network or VPN, and make sure it is at least client-certificate or token authentication; being able to connect means being able to operate.
5. Verify the installation
# 1. The entry point responds and Host matching is correct
curl -sS -o /dev/null -w '%{http_code}\n' https://titanides.com/
curl -sS -o /dev/null -w '%{http_code}\n' https://gateway.titanides.com/graphql -X POST
# 2. One certificate, every domain present, complete chain (edge config must use fullchain.pem, not cert.pem)
openssl s_client -connect titanides.com:443 -servername titanides.com </dev/null 2>/dev/null \
| openssl x509 -noout -text | grep -A1 'Subject Alternative Name'
openssl s_client -connect titanides.com:443 -servername titanides.com </dev/null 2>/dev/null \
| openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout
# 3. Upload limit (use a file slightly larger than client_max_body_size; it should return 200, not 413)
head -c 52000000 /dev/zero > /tmp/big.bin
curl -sS -o /dev/null -w '%{http_code}\n' -F 'file=@/tmp/big.bin' https://titanides.com/<upload-endpoint>
# 4. The stream is not cut off (hold it for more than 5 minutes without an error)
curl -N -H 'Accept: text/event-stream' https://gateway.titanides.com/<sse-path>
The Subject Alternative Name line in check 2 is the one to read after adding domains: a certificate with only some of
the domains still signs successfully, and only the missing domain shows a certificate error. Checks 3 and 4 are the easiest
to miss: everyday clicks all work; the failure only appears on a real upload or a genuine long-running task.
6. Putting a CDN in front
A CDN such as Cloudflare and nginx are two independent TLS segments. It overrides part of what this layer does — the real client IP, caching, the 100-second origin timeout, the 100 MB request body limit. The full configuration is in Cloudflare in front.