TLS all the way to the crab
Part of my Krabber series, a Twitter clone in Go. The full source is on GitHub.
Intro
Krabber runs on a single Elastic Beanstalk instance, with no Application Load Balancer in front of it. That’s a deliberate cost choice, since an ALB would roughly double the fixed monthly bill for a site that fits on one box. But it does take away the easy button for HTTPS, and this post is about getting that button back by hand.
The usual pattern is to put an ALB in front, attach an ACM certificate, and let the ALB terminate TLS. ACM certificates normally can’t be exported, but with an ALB you never need to, because the ALB uses them internally. Drop the ALB, though, and that convenience disappears. Now I’ve got a bare EC2 instance that has to present a trusted certificate directly to CloudFront, and ACM doesn’t usually hand a certificate to a box. Here’s how I got encryption all the way to the origin anyway. (This, by the way, is the setup I wrongly suspected during the outage.)
I. The shape
viewer ──HTTPS──▶ CloudFront ──HTTPS (TLS 1.2)──▶ nginx :443 ──▶ Go app :5000
origin: origin.krabber.net
security group: CloudFront prefix list only
CloudFront’s origin protocol policy is https-only, so there is no plaintext path to the app. The whole trick is getting a valid certificate onto nginx and then keeping it fresh.
II. Exportable ACM, which is the part most people miss
A standard ACM public certificate cannot be exported. But ACM now issues exportable public certificates, where you opt in at request time and then you can pull the private key out. That’s the keystone of this whole thing. So origin.krabber.net gets an exportable cert, and a deploy hook exports it onto the instance.
aws acm describe-certificate --certificate-arn <arn> \
--query 'Certificate.Options.Export'
# "ENABLED"
If that comes back DISABLED, nothing else here works, and the bad news is you won’t find out until CloudFront throws a 502 at you.
III. The deploy hook
Elastic Beanstalk runs any .platform/hooks/prebuild scripts before each deploy. Mine exports the certificate, decrypts the key, drops both on disk, and writes the nginx server block. The key comes back from ACM encrypted under a one-off passphrase, so it’s never sitting around in plaintext longer than it has to be:
# one-time passphrase, used only to move the key from ACM to disk
openssl rand -hex 32 > "$tmp/pass"
aws acm export-certificate --certificate-arn "$arn" \
--passphrase "fileb://$tmp/pass" --output json > "$tmp/export.json"
# ...split out Certificate + CertificateChain, decrypt the key...
install -m 600 "$tmp/origin.key" /etc/pki/krabber/origin.key
install -m 644 "$tmp/origin.crt" /etc/pki/krabber/origin.crt
Then it writes an nginx server block listening on 443, pointed at those two files, proxying to the Go app on 127.0.0.1:5000. And here’s a small touch I like: without ORIGIN_CERT_ARN set, the hook just no-ops and the origin stays HTTP-only. So the whole feature is one environment variable away from off, which is handy in local dev.
IV. The chain is the part that bites you
CloudFront validates the origin certificate against a trusted root, and unlike curl -k it will not accept an incomplete chain. The export gives you the leaf and the CertificateChain, and you have to concatenate both into the file nginx serves. So I check the chain the way CloudFront sees it:
openssl crl2pkcs7 -nocrl -certfile /etc/pki/krabber/origin.crt \
| openssl pkcs7 -print_certs -noout
# subject=CN=origin.krabber.net
# issuer=Amazon RSA 2048 M04
# subject=Amazon RSA 2048 M04
# issuer=Amazon Root CA 1
# Verify return code: 0 (ok)
Now here’s the trap. If you serve the leaf alone, a local curl -k still succeeds, because -k skips verification, and you high-five yourself and move on. Then CloudFront can’t build a path to the root and returns a fast 502. During my outage I burned real time re-checking this exact chain, convinced it was the culprit. It was spotless; the actual bug was DNS. But the reason I could rule it out quickly is that the check above is unambiguous: Verify return code: 0 (ok) means CloudFront will be happy too.
V. ACM renews, so the box has to re-pull
ACM rotates certificates automatically, which is wonderful right up until you remember the key is a static file sitting on a box. So the hook also installs a systemd timer that re-runs the export once a day and reloads nginx. When ACM renews, about 45 days out, the instance picks up the new cert within a day, with no deploy and no 3 a.m. expiry page. The deploy writes the cert the first time; the timer keeps it alive after that.
VI. Locking the door: the prefix list
TLS encrypts the path, but it doesn’t stop someone from talking to the box directly if they find its address. So the instance’s security group allows inbound 443 only from CloudFront’s managed origin-facing prefix list:
# ingress 443 from the CloudFront origin-facing prefix list only
prefix_list_ids = ["pl-xxxxxxxx"] # com.amazonaws.global.cloudfront.origin-facing
A scraper that resolves the origin and tries to connect directly just times out, because the firewall drops it. Everything has to arrive through CloudFront, which means everything passes WAF and the rate limits first. (This is also why, when I was debugging from some random host, I couldn’t reach the origin either. The control was doing its job on me, which was humbling.) And as a second layer, CloudFront injects a secret X-Origin-Verify header that the origin can check, so even a request that somehow reached the box without coming through the edge can be turned away.
Conclusion
Was it worth skipping the ALB? For Krabber, yes. The ALB would have made TLS trivial and roughly doubled the fixed cost of a one-box site. Instead I get end-to-end encryption, automatic renewal, and an origin that’s unreachable except through the edge, all for the price of a prebuild hook and a systemd timer.
It is more moving parts than “attach cert to ALB,” and I learned the sharp edges (exportable certs, the full chain, the prefix list) the way I learn most things, by shipping it and then staring at a 502. But for the constraint I set, it’s the right amount of complexity. Encrypted all the way to the crab, no load balancer, known bill. If you’re ever boxed into serving TLS without an ALB, this path works. Thanks for reading, and may your chains always verify.