5.4.2 — TCP: Connections, Reliability & Ordering

IP promises nothing. A packet may be lost, delayed for a minute, duplicated by a confused router, or arrive after a packet that was sent later.

TCP hands you a stream of bytes that arrives complete, exactly once, in order.

There is no magic in between. Every guarantee is built from three mechanisms — number every byte, acknowledge what arrived, re-send what did not — and the whole difficulty is in the edge cases those three create.

1. Why a connection has to be established at all

A connection is not a physical thing. Nothing is reserved anywhere. A TCP connection is simply agreed state held at both ends: each side remembers a sequence number, a window size, and a set of buffers.

So "opening a connection" means getting both ends to agree on that state, and the specific thing they must agree on is where the numbering starts.

The exchange is the three-way handshake:

Three packets to establish agreed state. The client's data cannot start until step 3, so the handshake costs one full round trip — 150 ms on an intercontinental link, before a single byte of content moves.

Step 1 — SYN. The client sends a segment with the SYN flag set and its initial sequence number (ISN), say 1000. It is saying: "I want to start a connection, and my bytes will be numbered from 1000."

Step 2 — SYN-ACK. The server replies with SYN set ("I also want to start, and my bytes number from 5000") and ACK set with ack = 1001 ("I received your SYN; the next byte I expect from you is 1001").

Step 3 — ACK. The client confirms with ack = 5001. Both sides now know each other's numbering and both know that the other knows.

Why three and not two? Because the connection is bidirectional, and each direction needs its own numbering agreed and confirmed. Two packets would establish the client's numbering and tell the client the server's, but the server would never learn whether its own SYN arrived. The middle step carries both the server's SYN and its acknowledgement of the client's — which is why it is three packets and not four.

Why is the initial sequence number not just zero? Two reasons, and the second is a real attack.

If a connection between the same four-tuple (Chapter 5.4.1) is closed and immediately reopened, a delayed packet from the old connection could arrive and be accepted as part of the new one. Starting at a different number every time makes that packet fall outside the valid range and be discarded.

More seriously: if the ISN were predictable, an attacker who cannot see your traffic could still inject data into your connection by guessing the sequence number and spoofing the source address. This was a real, widely exploited attack in the 1990s. Modern stacks generate the ISN from a cryptographic hash of the four-tuple plus a secret and a timer (RFC 6528), so it is unpredictable without breaking the hash.

The cost of the handshake is the number to carry with you. One round trip before any data. On a local network that is under a millisecond. London to Sydney it is about 150 ms, and HTTPS adds one or two more round trips for the TLS handshake (Chapter 5.7), so a fresh connection to a distant server costs 300–450 ms before the request is even sent. That single fact drives connection reuse, HTTP keep-alive, connection pools, TLS session resumption and, ultimately, QUIC's zero-round-trip design in Chapter 5.4.3.

The SYN flood, and why the fix is elegant

Notice what the server must do at step 2: it has received a SYN and must remember the connection's state while waiting for step 3. That memory is allocated before the client has proved it exists.

A SYN flood attack sends thousands of SYNs per second with spoofed source addresses. Each one makes the server allocate state for a step 3 that never comes, and the connection table fills until legitimate clients are refused.

The fix, SYN cookies, is genuinely clever: instead of storing state, the server encodes the state into the initial sequence number it sends back, as a cryptographic hash of the four-tuple plus a timestamp and a secret. It then forgets everything. When step 3 arrives, the acknowledgement number is that value plus one, so the server can recompute the hash, verify it, and reconstruct the connection state from scratch. A forged ACK cannot produce a valid cookie without the secret, so the attack costs the server nothing. Chapter 8.5 covers the family.

2. Sequence numbers: numbering every byte, not every packet

This is the detail people get wrong, and it explains several otherwise-confusing behaviours.

TCP numbers bytes, not segments. If the connection starts at sequence 1000 and you send 500 bytes, they occupy sequence numbers 1000 through 1499, and the next segment starts at 1500.

Numbering bytes rather than packets is what makes the stream abstraction work. A sender can split a 1,000-byte write into two 500-byte segments, or coalesce two 500-byte writes into one segment, and the numbering is unaffected because it describes the data, not the packaging. This is the same fact that produces the framing problem from Chapter 5.4.1: TCP genuinely does not know where your messages begin.

The acknowledgement number means "the next byte I expect", not "the last byte I got". If a receiver sends ack = 1500, it is saying "I have everything up to and including 1499; send me 1500 next."

Acknowledgements are cumulative. An ack of 1500 confirms every byte below 1500, not just one segment. This is robust: if an acknowledgement is itself lost, the next one supersedes it and nothing is stuck.

The sequence number is 32 bits, and it wraps. At 4.3 billion bytes it returns to zero. On a 10 Gbit/s link that takes about 3.4 seconds, which is short enough that a delayed packet from before the wrap could be mistaken for a current one. The fix is the timestamp option, which adds a monotonically increasing value used to reject segments that are old regardless of their sequence number. This mechanism is called PAWS — protection against wrapped sequence numbers — and it is one of the reasons TCP options exist.

3. Loss, and the two ways TCP notices

A sender keeps every byte it has sent but not yet had acknowledged, in a retransmission buffer. It cannot discard data until the other side confirms receipt. Detecting that something was lost happens two ways, and the difference matters.

Way one — the timer runs out. Every segment starts a retransmission timeout (RTO). If no acknowledgement arrives before it expires, the sender assumes the segment was lost and re-sends.

Setting that timer is harder than it looks. Too short and you re-send data that was merely slow, wasting bandwidth and making congestion worse. Too long and recovery from real loss is sluggish.

TCP computes it from the measured round-trip time, using an exponentially weighted moving average — a running estimate where each new measurement nudges the average rather than replacing it:

\text{SRTT} \leftarrow (1-\alpha)\,\text{SRTT} + \alpha \cdot \text{sample}, \qquad \alpha \approx 0.125

But an average is not enough, because a network with wildly variable delay needs more margin than a steady one. So TCP also tracks the variation and sets

\text{RTO} = \text{SRTT} + 4 \times \text{RTTVAR}

That factor of four is the interesting part. It means the timeout adapts to jitter: a stable link gets a tight timeout and fast recovery, while a variable link gets a generous one and avoids spurious retransmissions. Jacobson's 1988 paper introducing this is one of the reasons the internet survived its first congestion collapses.

A timeout is expensive, because the minimum RTO in most stacks is around 200 ms — an eternity when the round trip is 20 ms. Which is why the second mechanism exists.

Way two — duplicate acknowledgements. Suppose segments 1, 2, 4 and 5 arrive; 3 was lost. The receiver acknowledges cumulatively, so it can only say "I have everything up to 3". When 4 arrives it says it again. When 5 arrives it says it again.

The sender now sees three duplicate acknowledgements for the same byte. That is a strong signal: later data is clearly arriving, so the path is working, and one specific segment is missing. So the sender re-sends segment 3 immediately without waiting for any timer. This is fast retransmit.

Why three duplicates and not one? Because a single duplicate is ambiguous — packets can arrive out of order for perfectly innocent reasons, such as two packets taking different routes (Chapter 5.3.2). Reordering usually displaces a packet by one or two positions. Three duplicates is the threshold chosen to distinguish real loss from ordinary reordering, and it is a tuned constant rather than a derived one.

SACK (selective acknowledgement) improves on the cumulative scheme. With plain cumulative acks, if segments 3 and 7 are both lost, the sender only learns about 3 and must discover 7 in a later round. SACK lets the receiver say "I have everything up to 3, and also 4–6 and 8–10", so the sender re-sends exactly 3 and 7 in one go. It is negotiated during the handshake and is universally supported.

4. Closing a connection, and the state everyone asks about

Closing is four-way, not three-way, and the reason is a genuine design decision.

client → FIN            "I have no more data to send"
server → ACK            "understood"
        ... the server may still be sending data here ...
server → FIN            "I have no more data either"
client → ACK            "understood"

TCP allows a half-close. Each direction is shut down independently, so one side can stop sending while continuing to receive. This is genuinely used: a client can send a request, signal that the request is complete by closing its sending direction, and still read the whole response. The four-way close is what makes that possible, and it is why the server's ACK and its FIN are separate packets — it may have data to finish sending in between.

Now TIME_WAIT, which appears in every production incident about sockets.

After sending the final ACK, the side that closed first does not free the connection. It enters TIME_WAIT and stays there for 2 × MSL — twice the maximum segment lifetime, conventionally 60 seconds on Linux, sometimes 240 seconds elsewhere.

Two reasons, and both are real:

The final ACK might be lost. If it is, the other side will retransmit its FIN. Something has to be there to answer it, or that side sits waiting and eventually errors. TIME_WAIT is the state that answers.

Delayed duplicates from this connection must expire before the four-tuple is reused. If a new connection opened immediately on the same four-tuple, a segment wandering the network from the old one could arrive and be accepted as legitimate data. Waiting twice the maximum lifetime guarantees every stray packet is dead.

The operational consequence. A server that closes connections first accumulates tens of thousands of sockets in TIME_WAIT, each holding an ephemeral port (Chapter 5.4.1). Under load that exhausts the port range.

The temptations and their honest assessment:

• SO_REUSEADDR lets a listening socket bind while old connections linger. Safe and standard, and should generally be set on servers.
• net.ipv4.tcp_tw_reuse allows reusing a TIME_WAIT socket for a new outbound connection when timestamps prove it is safe. Reasonable on clients.
• Shortening the timeout, or tcp_tw_recycle — do not. tcp_tw_recycle was removed from Linux in 4.12 because it broke badly for clients behind NAT, where many machines share one address and their timestamps do not agree.
• The real fix is to close fewer connections. Keep-alive and connection pooling. TIME_WAIT pressure is nearly always a symptom of connection churn, and treating the symptom by tuning kernel parameters is a well-known way to trade a visible problem for an intermittent one.

Which side ends up in TIME_WAIT is a design lever. If clients close first, the state lands on thousands of clients where nobody notices. If the server closes first, it all lands on one machine. This is why HTTP servers generally let the client close, and it is worth knowing when you set an idle timeout.

5. The state machine, and the states you will actually see

TCP is formally a state machine, and netstat shows you which state each connection is in. The ones worth recognising:

State	Meaning	What it usually tells you
LISTEN	waiting for connections	the server is up
SYN_SENT	sent SYN, no reply yet	firewall dropping, or nothing listening
SYN_RECV	got SYN, awaiting the final ACK	many of these means a SYN flood
ESTABLISHED	open, data flowing	normal
FIN_WAIT_1/2	we closed, waiting on them	the peer is slow to close
CLOSE_WAIT	they closed, we have not	almost always an application bug
TIME_WAIT	we closed first, waiting out stray packets	connection churn

CLOSE_WAIT deserves the emphasis. It means the remote side sent a FIN and the local application has not called close(). This is not a network problem — it is your code leaking sockets, forgetting to close a connection in an error path or a finally block. A pile of CLOSE_WAIT sockets that never drains is one of the most reliable indicators of a specific class of bug, and knowing that saves hours.

SYN_SENT that never progresses means the SYN went out and nothing came back. Either nothing is listening on that port and no rejection was sent, or a firewall is silently dropping. Note the distinction: a closed port normally sends a TCP RST, which produces an immediate "connection refused". Silence rather than refusal means something is dropping packets on purpose, which is what a firewall configured to DROP rather than REJECT does.

6. Nagle and delayed ACK: the interaction that causes mystery latency

Two independent optimisations, each sensible, which combine badly. This is worth knowing because the symptom is bizarre and the cause is invisible.

Nagle's algorithm stops a sender from emitting many tiny segments. Sending one byte costs 40 bytes of headers — 4,000% overhead — and a terminal session typing character by character would flood the network. So Nagle says: if there is unacknowledged data outstanding, buffer small writes until the acknowledgement arrives, then send everything accumulated as one segment.

Delayed acknowledgement stops a receiver from emitting bare acknowledgements. If the receiver waits a moment, it may have data of its own to send and can piggyback the acknowledgement on it. So it waits up to 200 ms (40 ms on Linux) before acknowledging on its own.

Now put them together, with an application that writes a request in two calls — say a header, then a body:

1. Sender writes the header. Nothing outstanding, so it goes immediately.
2. Sender writes the body. Nagle holds it, because the header is unacknowledged.
3. Receiver has the header, cannot act without the body, and has nothing to send — so delayed ACK holds the acknowledgement.
4. Nothing happens for up to 200 milliseconds, until the receiver's delayed-ack timer fires.

A request that should take 1 ms takes 200 ms, on a healthy network, with no packet loss. The classic symptom is "our API is fast except for a suspiciously round 40 ms or 200 ms on some calls".

The fixes, in order of preference:

Write the whole message in one call. If the header and body go out together, Nagle never holds anything. This is the correct fix and it is free.

TCP_NODELAY disables Nagle for that socket. Every HTTP server, every database driver and every RPC library sets it, and Node.js sets it by default on TCP sockets. It is the right setting for request/response protocols where each write is a complete message.

Never disable delayed ACK globally. It is a system-wide setting and it increases acknowledgement traffic for everyone.

What the interviewer will push on

"Why three packets in the handshake and not two?" The connection is bidirectional, so both directions' sequence numbers must be agreed and confirmed. Two packets would leave the server unsure its own SYN arrived. The middle packet carries both roles, which is why it is three and not four.

"Why is the initial sequence number random?" To reject delayed packets from a previous connection on the same four-tuple, and — the security answer — so an off-path attacker cannot guess it and inject data into your stream. This was a real, exploited attack.

"What is a SYN flood and how do you defend against it?" State allocated at step 2 for a step 3 that never comes. SYN cookies encode the state into the sequence number so the server stores nothing and can reconstruct it from a valid ACK. Explaining why a forged ACK cannot produce a valid cookie is the strong version.

"What is TIME_WAIT for, and why do you have 30,000 of them?" To answer a retransmitted FIN if the final ACK was lost, and to let stray packets die before the four-tuple is reused. Thirty thousand means connection churn on the side that closes first. The fix is keep-alive and pooling, not kernel tuning — and specifically not tcp_tw_recycle, which was removed for breaking clients behind NAT.

"You see hundreds of sockets in CLOSE_WAIT. What is wrong?" The application is not calling close() after the peer closed. It is a code bug, not a network one, and it is usually a missing close in an error path.

"An API call intermittently takes exactly 40 ms with no load. Explain." Nagle plus delayed ACK. Say the four-step interaction, then give the fixes in order: one write instead of two, then TCP_NODELAY.

"How does TCP decide something was lost?" Two ways with different costs: the retransmission timeout, computed from a smoothed round-trip estimate plus four times its variation so it adapts to jitter; and fast retransmit on three duplicate acknowledgements, which is far quicker. Three duplicates because one or two are indistinguishable from ordinary reordering.

One thing to volunteer: name the handshake's round-trip cost as a design constraint, not a fact. One RTT for TCP plus one or two for TLS means a fresh intercontinental connection costs 300–450 ms before the request is sent — which is the reason keep-alive, connection pools, session resumption and QUIC's 0-RTT all exist. It reframes the mechanism as the thing that shapes real architecture.

Recall

• A TCP connection is agreed state at both ends, and the three-way handshake exists because both directions' sequence numbers must be agreed and confirmed. It costs one round trip before any data, plus one or two more for TLS.
• The initial sequence number is randomised to reject stale packets from a previous connection on the same four-tuple, and to stop an off-path attacker injecting data.
• SYN cookies defeat a SYN flood by encoding the connection state into the sequence number instead of storing it, so a forged ACK cannot produce a valid cookie.
• TCP numbers bytes, not packets; an acknowledgement number means the next byte I expect, and acknowledgements are cumulative, so a lost ack is harmlessly superseded.
• Loss is detected by the retransmission timeout — smoothed round-trip time plus four times its variation, so it adapts to jitter — or far faster by fast retransmit on three duplicate acks, three being the threshold that separates real loss from ordinary reordering. SACK lets the receiver name the gaps.
• Closing is four-way because TCP supports a half-close. TIME_WAIT lasts 2×MSL to answer a retransmitted FIN and to let stray packets die; thousands of them mean connection churn, and the fix is pooling, not kernel tuning.
• CLOSE_WAIT piling up is an application bug — a socket the code never closed.
• Nagle plus delayed ACK produces a 40–200 ms stall when a message is written in two calls. Fix by writing once, or by setting TCP_NODELAY.

Self-test: Why not a two-way handshake? · What does ack = 1500 actually assert? · Why three duplicate acks rather than one? · What are the two independent reasons for TIME_WAIT? · What does a growing pile of CLOSE_WAIT tell you, and where do you look? · Walk the four steps of the Nagle / delayed-ACK stall.

Next: 5.4.3 answers the question this page left open — TCP now re-sends what was lost, but how fast should it send in the first place? That is congestion control, the mechanism that stopped the internet collapsing in 1986 and still decides how fast your downloads go.