8.5.3 — Headers, Proxy Trust, Smuggling and Bots

An application sits behind a CDN, a load balancer and a reverse proxy. Rate limiting is keyed on the client IP address, taken from X-Forwarded-For.

An attacker sends:

POST /login HTTP/1.1
X-Forwarded-For: 1.2.3.4

Their real address is appended by the proxies, so the header arrives as 1.2.3.4, <real>, <lb>. The application takes the first value, because that is "the original client". The attacker changes the fake value on every request and the rate limiter never counts twice. The same trick bypasses IP allow-lists and poisons every log line.

This is the layer nobody owns. The application team assumes the infrastructure handles it; the infrastructure team assumes the application validates. This page is that gap.

1. The security headers, and what each one actually does

Content-Security-Policy — the highest-value header, and Chapter 6.10 covers it in full. One sentence here: it does not prevent injection, it makes a successful injection much less useful, and connect-src is the directive that stops exfiltration.

HSTS — Strict-Transport-Security: max-age=31536000; includeSubDomains. Converts plain-HTTP requests to HTTPS before they leave the browser, closing the SSL-stripping window (Chapter 5.7). Set it only on HTTPS responses, and treat includeSubDomains plus preload as close to irreversible: a subdomain that cannot serve HTTPS becomes unreachable for up to a year, and removal from the preload list is slow.

X-Content-Type-Options: nosniff — stops the browser from guessing a content type. Without it, a user-uploaded text file containing HTML can be interpreted as HTML and become stored XSS. Set it globally.

frame-ancestors in CSP, with X-Frame-Options: DENY for older clients — clickjacking, per Chapter 8.5.2.

Referrer-Policy: strict-origin-when-cross-origin — sends only the origin to other sites, so URLs containing identifiers or tokens do not leak. Now the browser default; set it explicitly anyway.

Permissions-Policy — disables features you do not use: camera=(), microphone=(), geolocation=(). Its main value is limiting what a compromised third-party script can request.

X-XSS-Protection — remove it. The legacy browser filter it enabled was itself exploitable and has been removed from browsers. Sending X-XSS-Protection: 0 is the modern recommendation where anything sets it.

And a header that is a security control without looking like one: Cache-Control: no-store on authenticated responses. A CDN or proxy that caches a personalised page serves one user's data to the next. This is a genuinely common and serious leak, and the rule is simple: any response whose body depends on who asked must be private, no-store, and Vary must be correct where caching is intentional (Chapter 5.6.2).

Use a library — Helmet or its equivalent — so the defaults are current, and then verify what actually reaches the browser. A proxy or framework further out can strip or override headers, and the only reliable check is looking at a real response.

2. Trusting a proxy header

X-Forwarded-For is append-only: each proxy adds the address it received the connection from.

X-Forwarded-For: <client>, <proxy1>, <proxy2>

A client can send the header itself, and everything after it is real. So the values you can trust are counted from the right, one per proxy you actually operate.

app.set('trust proxy', 2);   // (1)

(1) "Two proxies of mine are in front" — Express then takes the third from the right as the client address. trust proxy: true trusts the entire chain including whatever the client injected, and that is the vulnerability in the opening. Chapter 9.9.2 works through the four silent failures this produces.

If you are behind a CDN, prefer its own header. CF-Connecting-IP, True-Client-IP and the like are set by the CDN and overwrite any client-supplied value — but only if requests cannot reach your origin directly. Lock the origin to the CDN's address ranges, or an attacker bypasses the CDN and sets the header themselves.

X-Forwarded-Proto has its own failure. If your application decides "is this HTTPS" from it and trusts it wrongly, an attacker can claim HTTPS over plain HTTP — or, in the reverse case, a proxy that fails to set it makes your application think every request is insecure, so Secure cookies are never set and login silently fails.

The Host header is the one that reaches furthest. Frameworks build absolute URLs from it, and that includes password reset links:

const link = `https://${req.headers.host}/reset?token=${token}`;   // ✗

An attacker requests a reset for the victim's address with Host: attacker.example. The email is genuine, from you, and the link points at the attacker's server — which receives the token when the victim clicks. This is password reset poisoning, and it defeats the otherwise-correct flow from Chapter 8.4.1.

Two fixes, use both: build absolute URLs from configuration, never from the request, and allow-list valid Host values at the edge, rejecting anything else. The same header also enables cache poisoning, where a response generated with the attacker's host is stored and served to others.

3. HTTP request smuggling

Two servers parse the same bytes differently, and disagree about where one request ends and the next begins.

HTTP/1.1 has two ways to state a body's length: Content-Length and Transfer-Encoding: chunked. When both are present the specification says chunked wins, and implementations have historically disagreed.

CL.TE — the front end uses Content-Length, the back end uses Transfer-Encoding. The front end forwards what it thinks is one request; the back end sees the chunked body end earlier and treats the remainder as the start of the next request.

TE.CL — the reverse.

TE.TE — both support chunked, but one can be tricked into ignoring it with an obfuscated header (Transfer-Encoding: xchunked, or a space before the colon).

The impact is severe, because the smuggled prefix attaches to whichever request arrives next on that connection — someone else's.

• Queue poisoning: the next user's request is prefixed with the attacker's, so they receive the attacker's chosen response.
• Request capture: the victim's request, including cookies and body, is appended to something the attacker can read back.
• Bypassing front-end controls: authentication, WAF rules and path restrictions applied at the front end are skipped for the smuggled request.

Defences, in order:

Use HTTP/2 end to end. HTTP/2 has explicit framing with no length ambiguity (Chapter 5.6.1). Note the caveat: downgrading HTTP/2 to HTTP/1.1 at the back end reintroduces it, and that is a common deployment.

Reject ambiguous requests outright at the front end — any request with both headers, or a malformed Transfer-Encoding. Do not normalise and forward; reject.

Normalise on one parser. Have the front end rewrite the request rather than pass it through.

Disable connection reuse to the back end, which removes the shared queue at a real performance cost, as a last resort.

And the neighbouring bug: header injection. Unvalidated user input placed into a response header, with a CRLF in it, splits the response and injects headers or a whole second response. Reject \r and \n in any value that reaches a header — modern frameworks do this, and code that writes raw headers may not.

4. Rate limiting as a security control

Chapter 9.7.5 designs one and Chapter 10.15 covers the distributed view. The security-specific decisions:

Key on the right thing, in layers. Per account (targeted guessing), per IP (one source), per IP-range or ASN (cloud-hosted attacks), and per endpoint cost — a search endpoint that runs an expensive query needs a tighter limit than a static page.

Watch the distinct-accounts-per-source counter for credential stuffing (Chapter 8.4.1), which never triggers per-account limits.

Decide fail-open or fail-closed deliberately. If the rate-limit store is unreachable: fail open and an attacker who can degrade Redis removes all limits; fail closed and a Redis blip becomes a full outage. For login, fail closed; for a read endpoint, fail open — and write the decision down rather than leaving it to a catch.

Return 429 with Retry-After, and expose limit headers on every response so clients can behave, rather than only on rejection.

5. Bot defence

Automated traffic is scraping, credential stuffing, card testing, inventory hoarding and spam. The honest framing: you cannot distinguish a bot from a human with certainty, so this is about raising cost, not achieving detection.

The ladder, cheapest first:

Rate limits — stops the naive case and nothing sophisticated.

Proof of work / invisible challenges — the browser solves a small computational puzzle. Cloudflare Turnstile is the widely used implementation, and its appeal is that it is usually invisible and does not use image puzzles or track users across sites. This is the current default recommendation over traditional CAPTCHAs.

CAPTCHA — image or text puzzles. Increasingly weak (solving services are cheap and models beat many puzzles), and it has a real accessibility cost. Use it as an escalation for suspicious traffic, not as a gate on every user.

Device and browser fingerprinting — signals like canvas rendering, fonts and TLS handshake characteristics. It works, it is a privacy intrusion, and it may be regulated as personal data processing (Chapter 8.7). Be deliberate rather than accidental about adopting it.

Behavioural analysis — mouse movement, typing rhythm, navigation patterns. Effective and equally privacy-sensitive.

Managed bot protection — a CDN-level service with cross-customer intelligence, which is the practical answer at scale.

Two rules that keep this honest. Prefer challenging over blocking, because false positives block real customers and you rarely hear about it. And make the protected action idempotent and cheap to reverse — bot defence will fail sometimes, so a card test that gets through should be caught by the payment layer, not only by the bot layer.

6. File uploads

Uploads deserve a place here because every rule is an instance of something on this page.

Never trust the declared content type or the extension. Both are client-supplied. Check magic bytes, and re-derive the type server-side.

Do not use the client's filename. Generate your own identifier and store the original name as metadata only — that removes path traversal (Chapter 8.5.1) entirely.

Store outside the web root, ideally in object storage, and serve user content from a separate origin. A file served from your main origin runs in your origin's security context: an uploaded HTML or SVG file becomes stored XSS. A separate origin means it cannot touch your cookies or DOM.

Serve with Content-Disposition: attachment and nosniff for anything not deliberately rendered.

Enforce size limits at the proxy, not only in the application, so a large upload is rejected before it consumes memory.

Re-encode images rather than storing the original bytes. It strips embedded payloads and metadata, and it is the single most effective upload control.

Guard against decompression bombs — an archive or image that expands to gigabytes. Limit the decompressed size, not just the uploaded size.

Scan for malware if files are shared between users, and be honest that scanning is a partial control.

What the interviewer will push on

"An attacker is bypassing your rate limiter. What do you check?" X-Forwarded-For handling. The header is client-writable and append-only, so trusted values are counted from the right, one per proxy you operate — trust proxy: true trusts whatever the client injected. Then add that a CDN-specific header only helps if the origin is locked to the CDN's ranges.

"How can the Host header be dangerous?" Frameworks build absolute URLs from it, so an attacker sets Host: attacker.example on a password reset request and the genuine email from you contains a link to their server, which collects the token. Fix by building URLs from configuration and allow-listing hosts at the edge.

"What is HTTP request smuggling?" Two parsers disagreeing about where a request ends, exploited via Content-Length and Transfer-Encoding together. The impact is what matters — the smuggled prefix attaches to someone else's next request, enabling capture, poisoning and bypass of front-end controls. HTTP/2 end to end removes it, but downgrading at the back end brings it back.

"Which security headers would you set and why?" CSP as the one that limits the impact of injection, HSTS, nosniff, frame-ancestors, Referrer-Policy, Permissions-Policy, and remove X-XSS-Protection. Then volunteer Cache-Control: no-store on authenticated responses, because a CDN caching a personalised page is a real data leak that no header checklist flags.

"How would you stop bots?" Layered, and framed as raising cost rather than detecting: rate limits, then an invisible challenge like Turnstile, escalating to a harder challenge for suspicious traffic. Note that CAPTCHAs are increasingly weak and carry an accessibility cost, and that fingerprinting is effective and privacy-regulated. Challenge rather than block, because false positives are invisible to you.

"How do you handle file uploads securely?" Validate by magic bytes, generate your own filename, store outside the web root, serve from a separate origin, Content-Disposition: attachment with nosniff, size limits at the proxy, re-encode images, and guard against decompression bombs. The separate-origin point is the one that shows you understand why an uploaded SVG is stored XSS.

One thing to volunteer: point out that a CDN cache in front of an authenticated application is a data-leak risk that no security-header checklist covers — one missing Cache-Control: private, no-store and one user's account page is served to the next. It is a boring header with a severe failure mode, and it is caught in review far less often than CSP.

Recall

• Headers to set: CSP (limits injection impact), HSTS (includeSubDomains + preload is near-irreversible), nosniff, frame-ancestors, Referrer-Policy, Permissions-Policy. Remove X-XSS-Protection. And Cache-Control: private, no-store on authenticated responses — a CDN caching a personalised page is a real leak.
• X-Forwarded-For is client-writable and append-only: count trusted values from the right, one per proxy you operate. trust proxy: true is the vulnerability. A CDN header only helps if the origin is locked to the CDN's ranges. X-Forwarded-Proto decides whether Secure cookies get set.
• Host header poisoning: reset links built from req.headers.host send a genuine email pointing at the attacker. Build URLs from configuration and allow-list hosts.
• Request smuggling is two parsers disagreeing via Content-Length versus Transfer-Encoding, and the smuggled prefix attaches to someone else's request. HTTP/2 end to end fixes it; a downgrade at the back end reintroduces it. Reject ambiguous requests rather than normalising them.
• Rate limiting: layer per account, per IP, per range and per endpoint cost; watch distinct accounts per source for credential stuffing; decide fail-open or fail-closed per endpoint and write it down.
• Bot defence raises cost, it does not detect. Ladder: rate limits → invisible challenge (Turnstile) → harder challenge for suspicious traffic. CAPTCHAs are weakening and cost accessibility; fingerprinting is effective and privacy-regulated. Challenge, do not block.
• Uploads: magic bytes not content type, your filename not theirs, outside the web root, served from a separate origin, attachment + nosniff, size limits at the proxy, re-encode images, and guard against decompression bombs.

Self-test: Which end of X-Forwarded-For can you trust, and why? · How does a Host header turn a correct password reset into a takeover? · Why does a smuggled request affect a different user? · Which header prevents an uploaded text file becoming HTML? · What is the case for challenging rather than blocking bots? · Why serve user uploads from a different origin?

Next: 8.6.1 moves behind the application to the credentials it holds — where secrets live, how they rotate, and why the leaked key in a public repository is found in minutes.