10.20 — Networking for System Design

The network is the foundation of distributed systems. In a single-machine system, components communicate via memory. In a distributed system, they communicate over unreliable, slow, and constrained wires. Understanding networking at a systems level means knowing how bytes move from A to B, the protocols that govern this movement, and how to place boundaries between components.

The Client-Server Model

At its core, the internet operates on the client-server model. A client requests resources or actions, and a server fulfills those requests. In system design, these roles are relative. A backend API is a server to a mobile app, but a client to a database.

One request, one response. Every failure question in system design starts here: what if ① never arrives, and what if ② is lost after the server already did the work?

This request-response paradigm dictates how we handle failures (retries, timeouts) and scalability (load balancing requests across multiple servers).

IP and CIDR

An IP address uniquely identifies a device on a network. IPv4 provides roughly 4.3 billion addresses (e.g., 192.168.1.1), which we exhausted years ago. IPv6 expands this to practically infinite addresses.

Networks are grouped into subnets using CIDR (Classless Inter-Domain Routing). CIDR notation looks like 10.0.0.0/24. The /24 means the first 24 bits are fixed (the network prefix), leaving 8 bits (32 - 24 = 8) for host addresses. This subnet contains 2^8 = 256 addresses. Fewer than 256 are usable for machines: the first address names the network itself and the last is the broadcast address, and cloud providers reserve a few more (AWS keeps five per subnet for its router, DNS and future use), so a /24 on AWS gives you 251. That small detail matters when you size a subnet for an autoscaling group and it runs out of addresses at peak. The full treatment of addresses, masks and CIDR arithmetic is 5.3.1.

In cloud networking (like AWS VPC), you divide your address space into subnets:

• Public Subnet: Instances have public IPs and a route to the internet gateway. Use this for load balancers.
• Private Subnet: Instances have no public IPs. They can only access the internet outwards via a NAT Gateway. Use this for application servers and databases.

DNS: The Internet's Phonebook

DNS (Domain Name System) translates human-readable domain names (api.example.com) into machine-readable IP addresses (203.0.113.5).

When a client queries a domain:

1. It checks the local browser/OS cache.
2. It queries a Recursive Resolver (often provided by the ISP).
3. The resolver queries the Root Nameserver (finds the .com TLD server).
4. It queries the TLD Nameserver (finds the authoritative server for example.com).
5. It queries the Authoritative Nameserver (returns the exact IP for api.example.com).

For system design, DNS is your first point of traffic routing. You can configure DNS to return different IPs based on the client's location (Geo-DNS) or round-robin multiple IPs to distribute load. However, DNS caching by clients means DNS updates take time to propagate (governed by the TTL - Time To Live), making DNS unsuitable for real-time failover.

TCP, UDP, and QUIC

Transport layer protocols govern how packets move between IPs.

TCP (Transmission Control Protocol) is reliable, ordered, and error-checked. It establishes a connection via a 3-way handshake (SYN -> SYN-ACK -> ACK). If a packet drops, TCP retransmits it. This guarantees delivery but introduces latency, especially over lossy networks (Head-of-Line blocking: one dropped packet delays all subsequent packets). Use TCP for HTTP, databases, and file transfers.

UDP (User Datagram Protocol) is fire-and-forget. It offers no handshakes, no order guarantees, and no retransmissions. It is fast and lightweight. Use UDP where a late packet is worse than a lost one: video and voice calls, real-time gaming, and metrics aggregation, where a dropped frame or data point is acceptable but waiting for a retransmission is not. Note that on-demand video (a film on a streaming service) is not in this list. It is delivered as small files over HTTP, which means TCP or QUIC, because a buffered player can afford to wait for a lost chunk. The transport details live in 5.4.1 and 5.4.2.

QUIC is a newer transport protocol built on top of UDP. It implements its own reliability mechanisms but eliminates TCP's Head-of-Line blocking by multiplexing independent streams. If one stream loses a packet, other streams continue processing. QUIC also combines the transport and cryptographic (TLS) handshakes, reducing connection latency.

HTTP Versions

HTTP is the dominant application-layer protocol. It has evolved to solve performance bottlenecks.

HTTP/1.1: Text-based. Supports keep-alive (reusing a TCP connection for multiple requests). However, it suffers from HTTP Head-of-Line blocking: you can only send one request at a time per connection. Browsers circumvent this by opening up to 6 concurrent TCP connections per domain.

HTTP/2: Binary protocol. Multiplexes multiple requests concurrently over a single TCP connection using streams. This solves HTTP Head-of-Line blocking. It also compresses headers (HPACK). HTTP/2 originally included server push, where the server sends files the browser has not asked for yet. In practice it rarely helped and often wasted bandwidth, and Chrome removed support for it in 2022, so do not propose it in a design. The modern replacement is the 103 Early Hints response, which tells the browser what to start fetching. Versions are covered in full in 5.6.1.

HTTP/3: Runs on QUIC instead of TCP. Solves TCP Head-of-Line blocking. Faster handshakes (0-RTT for resumed connections) and better performance on lossy networks (like mobile connections transitioning between cell towers).

TLS Termination

TLS (Transport Layer Security) encrypts traffic. Handshaking and decrypting TLS is CPU-intensive.

In a distributed system, you rarely handle TLS on the application servers. Instead, you perform TLS Termination at the Load Balancer or API Gateway.

1. The client establishes a secure TLS connection with the Load Balancer.
2. The Load Balancer decrypts the request.
3. The Load Balancer forwards the plain HTTP request over the private internal network to the application server.

This offloads CPU work from app servers, simplifies certificate management (only the load balancer needs the cert), and allows the load balancer to read HTTP headers to make routing decisions.

The honest caveat. Step 3 sends unencrypted traffic across your own network. Anyone who gets a foothold inside that network, or a misconfigured mirror port, can read passwords and card numbers in transit. That was considered acceptable when the inside of a data centre was trusted, and it no longer is. Regulated systems (payments, health data) and zero-trust designs re-encrypt the inner hop: the proxy terminates the client's TLS, reads what it needs for routing, and opens a new TLS connection to the server, often mutual TLS where both sides present certificates (8.3). You pay a little CPU twice, and you keep the routing benefits.

Proxies and Reverse Proxies

A Forward Proxy sits in front of a group of clients and intercepts their outbound requests to the internet. Used for content filtering (corporate networks) or anonymization (VPNs).

A Reverse Proxy sits in front of a group of servers and intercepts incoming requests from the internet. The client thinks it is talking directly to the server. Nginx and HAProxy are common reverse proxies.

A reverse proxy in front of three servers. The client sees one address and one certificate (①). The proxy picks a server for each request and forwards it inside the private network (②). Whether that inner hop is re-encrypted is a decision, covered under TLS termination above.

Reverse proxies provide:

• Load Balancing: Distributing traffic across servers.
• TLS Termination: Handling HTTPS.
• Caching: Serving static assets without hitting app servers.
• Rate Limiting: Protecting downstream servers from abuse.

CDN (Content Delivery Network)

A CDN caches static (and sometimes dynamic) content on edge servers globally distributed close to users. When a user in Tokyo requests an image hosted in Virginia, downloading it directly incurs significant latency (distance). A CDN caches the image on a node in Tokyo, serving it instantly.

CDNs use DNS resolution to map the user to the closest edge node. They dramatically reduce origin server load and improve client latency. Modern CDNs (like Cloudflare) also act as a global reverse proxy, offering DDoS protection and edge computing functions.

Anycast

How does Google's 8.8.8.8 DNS resolver respond instantly worldwide? It uses Anycast.

In standard IP routing (Unicast), one IP points to exactly one server. In Anycast, multiple servers distributed globally announce the same IP address to the internet via BGP (Border Gateway Protocol).

When a client sends a packet to an Anycast IP, routers along the path direct the packet to the topologically "closest" server announcing that IP. If an Anycast node goes down, its BGP routes are withdrawn, and traffic automatically flows to the next closest node. Anycast is heavily used by CDNs and root DNS servers for global scale and resilience.

What the interviewer will push on

1. "Why not use DNS for fast failover?" DNS relies on TTLs. Even if you set TTL=60s, many clients and ISPs ignore it and cache longer. When a server dies, changing the DNS record won't instantly stop traffic to the dead IP. Use an active load balancer for fast failover instead.
2. "Explain the difference between TCP and HTTP Head-of-Line blocking." HTTP/1.1 HoL blocking happens when a large response delays subsequent requests on the same connection. HTTP/2 solves this with streams. TCP HoL blocking happens when a dropped packet stops the OS from delivering subsequent packets to the application, even for independent HTTP/2 streams. HTTP/3 (QUIC) solves this.
3. "How do you design secure communication between microservices?" They will look for mTLS (Mutual TLS) within a service mesh, where both the client and server present certificates to authenticate each other, ensuring traffic is encrypted even within the private network (Zero Trust).

Recall

• CIDR: 10.0.0.0/24 means the first 24 bits define the network, leaving 8 bits for 256 host IPs.
• UDP vs TCP: UDP is fast and lossy. TCP is reliable but susceptible to head-of-line blocking.
• QUIC / HTTP/3: Eliminates TCP head-of-line blocking by moving reliability to the stream level over UDP.
• TLS Termination: Load balancers decrypt traffic to save CPU on app servers and read HTTP headers for routing.
• Anycast: Multiple globally distributed servers share one IP address. Routers send traffic to the closest one.

Quiz

1. Why is HTTP/2 over TCP still susceptible to performance issues on lossy mobile networks?
2. A client requests an image. Describe how a CDN reduces latency compared to fetching it from the origin server.
3. Why do we put databases in private subnets, and how can they still download software updates?

Answers

1. Because HTTP/2 multiplexes streams over a single TCP connection. If one TCP packet drops, the OS halts delivery of all subsequent packets (TCP Head-of-Line blocking), stalling all HTTP/2 streams.
2. The CDN caches the image on edge nodes located physically close to the user (e.g., in their city). The request is routed to the nearest node, drastically reducing the physical distance and network hops required to retrieve the data.
3. Private subnets have no inbound internet access, preventing direct attacks on the database. They download updates via a NAT (Network Address Translation) Gateway located in a public subnet, which allows outbound connections but blocks uninitiated inbound traffic.

Next: 10.21 moves up from packets to contracts: REST, GraphQL and gRPC, idempotency keys, pagination and the API gateway.