Appearance
5.4.3 — Flow Control, Congestion Control & QUIC
October 1986. The link between Lawrence Berkeley Laboratory and the University of California at Berkeley — a few hundred metres apart — dropped from 32 kbit/s to 40 bit/s. A factor of a thousand.
Nothing was broken. No cable was cut, no router failed. Every machine was behaving exactly as designed: sending data, timing out when it was lost, and re-sending. The re-sends were the problem. Every retransmission added more traffic to a link that was already overloaded, which caused more loss, which caused more retransmissions. The network had entered a stable state where almost all its capacity carried copies of data that had already been sent.
This is congestion collapse, and Van Jacobson's fix — published in 1988 and still running in every operating system today — is the reason the internet works at all. This page is that mechanism, plus the one it is often confused with, and the protocol that finally replaced the whole arrangement.
1. Two different problems that both slow you down
They get conflated constantly, and separating them is the first thing to get right.
Flow control protects the receiver. A fast server sending to a phone with 200 KB of buffer will overflow that buffer, and the excess is simply dropped. The receiver knows how much room it has, so the receiver tells the sender.
Congestion control protects the network. A gigabit server sending across a link that can only carry 10 Mbit/s will fill the router's queue, and the excess is dropped. Nobody tells the sender anything — routers do not send "slow down" messages. The sender has to infer it.
That asymmetry is the whole reason congestion control is hard. Flow control is a conversation. Congestion control is a guess.
2. Flow control: the receive window
Every TCP segment carries a 16-bit receive window field: "I have this many bytes of free buffer space right now."
The sender must never have more unacknowledged data outstanding than that number. As the receiving application reads data out of the buffer, space frees up and the window grows again, which is advertised on the next acknowledgement.
The window can reach zero. If the application stops reading — it is busy, or blocked, or has stopped consuming — the buffer fills and the receiver advertises a window of 0. The sender stops completely.
Now a deadlock threatens. The sender is waiting for a window update; the window update travels in an acknowledgement; but the sender is not sending anything to acknowledge. If the update is lost, both sides wait forever.
The fix is the zero window probe: the sender periodically transmits a single byte purely to force a response. The response carries a fresh window advertisement, and either the deadlock is broken or the sender learns the window is still zero and waits longer.
This entire mechanism is backpressure, and it is worth naming as such because Chapter 3.8.4's Node streams and Chapter 10.9's reliability patterns are the same idea one and three layers up. A slow consumer must be able to slow a fast producer, or something overflows. TCP does it with a number in a header; a Node stream does it with a write() that returns false; a message queue does it by refusing to accept more work. When you understand the receive window, you understand backpressure everywhere.
The 16-bit field ran out of headroom. 65,535 bytes was generous in 1981 and is not now. The amount of data that should be in flight is the bandwidth-delay product:
\text{bytes in flight} = \text{bandwidth} \times \text{round-trip time}
For a 1 Gbit/s link with a 100 ms round trip, that is 125 \times 10^6 \times 0.1 = 12.5 MB. The window field can advertise 65 KB. Without a fix, that connection would run at about 5 Mbit/s on a gigabit link — using half a percent of the available capacity, purely because of a header field's width.
The fix is the window scale option, negotiated during the handshake: a shift factor applied to the window field, allowing windows up to 1 GB. It is negotiated in the SYN, so if a middlebox strips the option, the connection silently falls back to 64 KB and runs a hundred times too slowly with no error anywhere. That is a real and famously hard-to-diagnose failure, and it is one of the ossification examples from Chapter 5.1.
3. Congestion control: guessing how fast the network can go
The sender maintains a second limit, the congestion window (cwnd), which is its own estimate of what the network can carry. The actual amount in flight is
\min(\text{receive window},\ \text{congestion window})
so whichever constraint is tighter wins — the receiver's capacity or the network's.
Nobody tells the sender what cwnd should be. It has to be discovered, and the only signal available is loss. The core assumption of classical TCP is:
A lost packet means a router queue overflowed, which means I am sending too fast.
Hold that sentence. Section 6 is about what happens when it is wrong.
The algorithm has four parts.
Slow start — find the ceiling fast. Begin with a small window, historically 1 segment and now typically 10 (about 14 KB). Every acknowledgement increases cwnd by one segment, which means it doubles every round trip: 10, 20, 40, 80, 160.
The name is a historical joke — it is the fastest-growing phase. It is "slow" only relative to the previous behaviour of blasting a full window immediately.
Exponential growth is the right choice here because the sender has no idea whether the capacity is 10 KB or 10 MB, and doubling finds any ceiling in a logarithmic number of round trips. The cost of overshooting is one loss event; the cost of growing linearly would be minutes of underuse on a fast link.
Congestion avoidance — creep upward. Once cwnd passes a threshold (ssthresh), doubling is too aggressive. Growth switches to one segment per round trip, linear instead of exponential. The sender is now probing gently for extra capacity rather than hunting for the ceiling.
Fast recovery — react to a mild signal mildly. On three duplicate acknowledgements (Chapter 5.4.2), the sender re-sends the missing segment and halves cwnd. It does not collapse, because duplicate acks prove data is still flowing — the path works, one packet was lost.
Timeout — react to a severe signal severely. If the retransmission timer expires, nothing is getting through at all. cwnd drops to 1 segment and slow start begins again. This is the expensive case, and it is why a timeout costs so much more than a fast retransmit.
The sawtooth is not a flaw — it is the design. TCP is deliberately probing for a capacity that nobody publishes and that changes second by second as other flows come and go. The only way to know the limit is to exceed it occasionally. Every TCP connection on Earth is continuously running a small experiment to find out how fast it is allowed to go.
Why halving rather than some other factor. Halving on loss and adding one per round trip is called AIMD (additive increase, multiplicative decrease), and Chiu and Jain proved in 1989 that this specific combination converges to a fair and stable share among competing flows. Additive increase alone would not converge fairly; multiplicative increase would oscillate wildly. AIMD is a mathematical result, not a heuristic, and being able to say that is a genuinely strong answer.
4. What actually runs today
Reno and NewReno are the classical algorithm above. Simple, fair, and poor on modern links.
CUBIC is the Linux and Windows default and has been for years. Its problem with Reno is that on a fast link with a long round trip, growing by one segment per RTT takes a very long time to recover after a loss — a 10 Gbit/s intercontinental link could take hours to reach full speed. CUBIC replaces linear growth with a cubic function of the time since the last loss: it climbs fast at first, flattens as it approaches the window size that previously caused loss (probing carefully around the known limit), then accelerates again if no loss occurs. It is also deliberately independent of round-trip time, which fixes Reno's unfairness where short-RTT flows grab far more than distant ones.
BBR (Google, 2016) rejects the founding assumption entirely, and it is the most interesting development in the field for twenty years. Instead of treating loss as the congestion signal, it measures two things continuously: the maximum delivery rate observed, and the minimum round-trip time observed. From those it computes the actual bottleneck bandwidth and the path's true propagation delay, and it paces packets to send exactly at that rate.
The problem BBR solves is bufferbloat. Router manufacturers made queues enormous, reasoning that a bigger buffer drops fewer packets. But a loss-based algorithm keeps increasing until it sees loss, so it fills whatever buffer exists before it gets any feedback. A 1-second queue means a full second of latency added to every packet, on a connection that is not dropping anything. This is why a large upload used to make everything else on the same network unusable — the algorithm was working exactly as designed and the design was wrong for the hardware.
BBR keeps the queue nearly empty because it stops at the measured bandwidth rather than at the point of loss. YouTube's deployment reported meaningful throughput gains and large latency reductions. It also has fairness concerns when sharing a bottleneck with CUBIC flows, which version 2 works on.
The lesson worth extracting: the whole algorithm rests on an inference — loss means congestion — and that inference was true in 1988, is false on a wireless link where loss means interference, and is misleading on a bloated buffer where congestion arrives as delay long before it arrives as loss. Recognising that a mechanism depends on an assumption about its environment, and that the environment changed, is the kind of reasoning that generalises far beyond networking.
5. Head-of-line blocking: the guarantee that becomes a cost
TCP promises an ordered byte stream. That promise has a consequence people usually meet as a mystery performance problem.
A page loads ten images over one HTTP/2 connection. The images are independent — image 7 has nothing to do with image 3. But they are multiplexed over one TCP connection, so they are one byte stream.
If a single packet belonging to image 3 is lost, TCP has already received the packets for images 4 through 10. It cannot deliver them. It promised byte order, and byte 5,000 cannot be handed to the application before byte 4,000. So all ten images stall for a full retransmission round trip because of a loss affecting one of them.
This is head-of-line blocking, and note precisely where it lives: the application does not care about the ordering, and has no way to say so through the layer boundary. The abstraction is leaking exactly as Chapter 5.1 predicted.
HTTP/1.1's workaround was six connections per origin, which is why browsers opened six. Independent connections mean independent loss recovery — but also six handshakes, six TLS negotiations, six congestion windows each starting small, and six times the server memory.
HTTP/2 fixed the wrong layer. It multiplexed many streams over one connection, which removed HTTP's own head-of-line blocking (where request 2 waited for request 1's response). But it did nothing about TCP's, and by collapsing six connections into one it arguably made the TCP case worse. On a lossy mobile network, HTTP/2 can be slower than HTTP/1.1 for exactly this reason.
Fixing it properly requires the transport to know that the streams are independent — which means changing TCP, which cannot be done.
6. QUIC: rebuilding the transport where it can be changed
QUIC is a transport protocol that provides everything TCP does — reliability, ordering, congestion control — implemented in user space, on top of UDP. It became RFC 9000 in 2021, and HTTP/3 is HTTP over QUIC.
Chapter 5.4.1 made the key point: UDP here is not "the unreliable choice". It is the only place a new transport can be deployed, because TCP lives in kernels and middleboxes that will never all be updated.
Five things QUIC changes.
Streams are genuinely independent, so head-of-line blocking is gone. QUIC knows about streams natively. A lost packet blocks only the stream it belonged to; the other nine images are delivered immediately. This is the problem HTTP/2 could not solve, solved at the layer where it actually lives.
The handshake is merged with encryption. TCP needs one round trip, then TLS needs one or two more (Chapter 5.7). QUIC combines them: one round trip for a new connection, and zero for a resumed one — a returning client can send application data in its very first packet, using keys cached from before.
The 0-RTT case has a genuine and unavoidable caveat: that first flight is vulnerable to replay, because an attacker who records it can send it again and the server cannot distinguish it. So 0-RTT data must be restricted to idempotent requests (Chapter 9.6.3 defines idempotency). A GET is fine; a POST that charges a card is not.
A connection survives a change of IP address. A TCP connection is identified by the four-tuple, so walking out of Wi-Fi onto mobile changes your IP and kills every connection. QUIC identifies a connection by a connection ID carried in the packet, independent of addresses. Your video call continues across the network change. This is the most user-visible improvement and it is impossible to retrofit onto TCP, because the four-tuple identity is baked into every kernel and every NAT box.
Almost everything is encrypted, including the transport header. Not primarily for privacy — for evolvability. Chapter 5.1 explained ossification: middleboxes that can read a header form dependencies on it, and those dependencies freeze the protocol. QUIC encrypts its header so middleboxes cannot form them, which keeps the protocol changeable in ten years' time. It is a deliberate design choice aimed at the future, and it is one of the more elegant ideas in modern protocol work.
It is implemented in user space, so it can be updated with an application deployment instead of an operating system upgrade. Google can ship a congestion-control change to Chrome in weeks; a TCP change takes a decade.
The honest costs. UDP is more expensive per packet than TCP on most systems, because TCP has decades of kernel and network-card offload optimisation that QUIC does not yet get — measured CPU cost has been roughly twice TCP's, though the gap is closing with offload support. Some corporate firewalls block or throttle UDP entirely, so clients must be able to fall back to TCP. And user-space implementation means the protocol logic is in every application rather than shared in one kernel.
Adoption is now substantial: HTTP/3 is supported by every major browser and used by a large share of traffic from Google, Cloudflare, Meta and Akamai.
What the interviewer will push on
"Flow control versus congestion control." Receiver protection versus network protection. The sharpest framing: flow control is a conversation — the receiver tells you the number. Congestion control is a guess — nobody tells you anything, so the sender must infer it from loss.
"Why does TCP halve the window on loss and add one per round trip?" AIMD, and it is a proven result rather than a heuristic — Chiu and Jain showed this specific combination converges to a fair, stable allocation among competing flows.
"What is bufferbloat and why did bigger buffers make things worse?" A loss-based algorithm increases until it observes loss, so it fills whatever buffer exists before receiving any feedback. A huge queue therefore adds a huge latency to every packet on a connection that never drops one. BBR fixes it by measuring bandwidth and minimum RTT rather than waiting for loss.
"Why can HTTP/2 be slower than HTTP/1.1 on a lossy network?" HTTP/2 multiplexes onto one TCP connection, so one lost packet blocks every stream. HTTP/1.1's six connections gave independent loss recovery by accident. HTTP/2 fixed HTTP's head-of-line blocking and not TCP's.
"Why is QUIC built on UDP?" Not for speed and not to avoid reliability — QUIC is fully reliable. UDP is the only place a new transport can be deployed, because TCP is frozen in kernels and middleboxes. Then name the four wins: independent streams, a merged 1-RTT (or 0-RTT) handshake, connection migration across IP changes, and an encrypted header specifically to prevent future ossification.
"What is the risk of 0-RTT?" Replay — an attacker can resend the recorded first flight and the server cannot tell. So 0-RTT carries only idempotent requests.
"A gigabit link with a 100 ms RTT is running at 5 Mbit/s. Why?" Bandwidth-delay product is 12.5 MB and the unscaled receive window caps at 64 KB. Window scaling is negotiated in the SYN, so a middlebox stripping the option silently produces exactly this — no error, just a hundredfold slowdown.
One thing to volunteer: say that TCP's entire congestion control rests on one inference — loss means a queue overflowed — which was true in 1988, is false on wireless where loss means interference, and is misleading with bloated buffers where congestion shows up as delay long before loss. Naming the assumption and the environments that broke it is the observation that shows you understand the mechanism rather than the vocabulary.
Recall
- Flow control protects the receiver (it advertises a receive window, and a zero window needs a zero window probe to avoid deadlock). Congestion control protects the network, and nobody advertises anything — the sender must infer it. This is backpressure, the same idea as Node streams and queue-based reliability.
- The bandwidth-delay product is how much should be in flight; the 16-bit window field caps at 64 KB, so window scaling is what makes fast long links work — and a middlebox stripping the SYN option causes a silent hundredfold slowdown.
- The congestion window grows exponentially in slow start (doubling per RTT), linearly in congestion avoidance, halves on three duplicate acks, and collapses to 1 on a timeout — the sawtooth. AIMD is a proven fair-convergence result, not a heuristic.
- CUBIC replaces linear growth with a cubic curve and is RTT-independent; BBR abandons loss as the signal and measures bandwidth and minimum RTT, which is what fixes bufferbloat.
- The founding assumption — loss means a queue overflowed — is false on wireless and misleading with large buffers.
- Head-of-line blocking: one lost packet stalls every multiplexed stream, because TCP promised byte order and the application cannot say the streams are independent. HTTP/2 fixed HTTP's version and not TCP's.
- QUIC is a full transport in user space over UDP: independent streams, a merged 1-RTT (or 0-RTT, replay-limited to idempotent requests) handshake, connection migration by connection ID across IP changes, and an encrypted header to prevent ossification.
Self-test: State the one-line difference between flow and congestion control · Why is slow start's exponential growth the right choice? · Why does a timeout cost far more than three duplicate acks? · Explain bufferbloat, and why BBR avoids it · Why can HTTP/2 lose to HTTP/1.1 on a lossy link? · Why is QUIC's encrypted header not primarily about privacy?
Next: 5.5 goes back to the beginning of the request. Before any of this could happen, something had to turn google.com into 142.250.187.238 — and that lookup is a globally distributed database with no single owner, queried billions of times a second.