13.6.12 — Kubernetes Networking, End to End

A team scales a gRPC service from 3 pods to 30 because one pod is at 95% CPU.

After the scale-up, one pod is still at 95% CPU and twenty-nine are idle.

Nothing is broken. The Service is correct, the endpoints list all thirty pods, kube-proxy has the right rules on every node. The problem is that Kubernetes balances connections, and the client opened one connection. This page builds the network model from the bottom so that this failure — and the five others like it — are obvious rather than mysterious.

1. The model, in four rules

Kubernetes does not implement networking. It states four requirements and lets a plugin satisfy them however it likes.

1. Every pod gets its own IP address.
2. Every pod can reach every other pod directly, on any node, with no address translation between them.
3. Every node can reach every pod.
4. The IP a pod sees itself as having is the IP others use to reach it.

Rule 2 is the important one, and it is the biggest simplification over plain Docker. No port mapping, no NAT, no "which host port did this land on". A pod on node A talks to a pod on node C using its real address, and both see the same address.

Rule 4 sounds obvious and is not. In Docker's bridge networking a container sees 172.17.0.3 while the outside world reaches it through the host's IP and a published port. That mismatch breaks any protocol that carries addresses inside its messages, and clustering software does this constantly. Kubernetes removes the problem by requiring the two to be the same.

Two words before the table, because the rest of the page leans on them.

A CIDR is a block of IP addresses written as an address and a length — 10.244.0.0/16. The number after the slash says how many bits at the front are fixed, so everything after them varies. A /16 fixes 16 bits and leaves 16 to vary, which is 65,536 addresses. A /24 leaves 8 bits, so 256 addresses, of which about 254 are usable. Bigger slash number, smaller block. That single arithmetic is behind every address-exhaustion problem on this page.

NAT — network address translation — is a router rewriting the addresses inside a packet as it passes through. Your home router does it: every device on your network shares one public address, and the router rewrites the source address on the way out and puts it back on the way in. It works, and it costs you the truth — the destination no longer sees who really sent the packet, and the sender cannot be reached from outside unless the router was told to forward a port. Rule 2 above exists to keep NAT out from between pods.

There are three separate address ranges and confusing them causes real outages:

Range	Who has an address	Where it exists
Node network	The machines	Your real network or VNet
Pod CIDR	Every pod	Routed or encapsulated by the CNI plugin
Service CIDR	Every Service	Nowhere — virtual, never on any interface

A Service IP belongs to no machine. Nothing answers it. It exists only as a rule in the kernel's packet handling, which section 3 unpacks.

And plan these ranges before you build. A pod CIDR overlapping your office network, your VPN pool, or a partner's network produces failures where half the destinations work and half do not — genuinely one of the hardest problems to diagnose because it looks intermittent (Chapter 13.2).

2. How a plugin delivers rule 2

Two families, and the choice has real consequences.

Overlay (encapsulation). Pod traffic is wrapped inside a packet addressed node-to-node — VXLAN or Geneve — and unwrapped at the other end. The physical network never learns pod addresses at all.

• Works on any network, including one you do not control.
• Costs a header (50 bytes for VXLAN), which reduces the usable packet size. The MTU — maximum transmission unit — is the largest packet a network link will carry, normally 1,500 bytes on Ethernet. Wrapping steals 50 of those, so the inside packet must be 50 bytes smaller. If the MTU is not adjusted, large packets are fragmented or silently dropped, and the symptom is bizarre: small requests work, large ones hang. MTU mismatch is the classic overlay bug and it is worth suspecting whenever "it works until the response is big".
• Costs a little CPU for wrapping and unwrapping.

Routed (native). The network itself knows how to reach pod addresses, either through routes the plugin writes into the machines' routing tables, or through BGP — the protocol routers use to tell each other "send anything for this address block to me". The plugin speaks it on each node, so the physical network learns where every pod lives.

• No encapsulation, no MTU problem, full speed, and pod IPs are visible to the wider network, which firewalls and monitoring like.
• Requires cooperation from the network — routes, or a router willing to speak BGP.

Plugin	Approach	Notable for
Flannel	VXLAN overlay	Simplest, no policy support
Calico	Routed (BGP) or overlay	Mature network policy, very widely used
Cilium	eBPF	Policy at layer 7, can replace kube-proxy, strong observability
Azure CNI	Pods get real VNet IPs	Native Azure integration (Chapter 13.6.15)
AWS VPC CNI	Pods get real VPC IPs	Native AWS integration, ENI limits per instance

The "pods get real network IPs" designs are excellent until you run out of addresses. A pod with a VNet IP consumes an address from your subnet, so a cluster of 50 nodes at 30 pods each needs 1,500 addresses plus growth. A /24 subnet holds 254. This is why Azure and AWS both added overlay modes, and why address planning is a design decision made before the first node.

eBPF deserves a note because it is where the industry is going. Rather than writing packet-filter rules, an eBPF plugin loads small programs into the kernel that handle packets directly. This removes the linear-list scaling problem entirely, allows policy based on HTTP paths and methods rather than only IPs and ports, and can replace kube-proxy altogether.

3. Services: a stable address in front of moving pods

Pods come and go and their IPs change. A Service is a name and an address that do not.

apiVersion: v1
kind: Service
metadata: { name: checkout-api, namespace: shop }
spec:
  selector:
    app: checkout-api          # (1)
  ports:
    - name: http
      port: 80                 # (2)
      targetPort: 3000         # (3)

(1) The Service does not know about pods. It knows a label selector. Whatever matches it and is ready receives traffic. A selector that does not match the pod labels is the single most common cause of "the pod is running but gets no traffic", and kubectl get endpoints checkout-api returning nothing confirms it in one command.

(2) The port the Service listens on.

(3) The port on the pod. Use the pod's named port here — targetPort: http — so a port change is one edit.

What actually happens to a packet

Three objects and one kernel rule:

1. The EndpointSlice controller watches for ready pods matching the selector and maintains a list of their IPs.
2. kube-proxy on every node watches that list and writes forwarding rules.
3. A packet to the Service IP is rewritten by the kernel to a chosen pod IP and routed there directly.

There is no proxy hop. The connection goes straight from the client pod to the destination pod. The Service IP existed only for the instant the destination was chosen.

Which brings us to the opening problem. The choice is made once, when the connection is established. For HTTP/1.1 with short-lived connections, that is per request and balancing works fine. For HTTP/2, gRPC, or any client with a connection pool, one connection carries thousands of requests and they all land on the pod chosen at the start.

Three fixes:

Client-side balancing against a headless Service. The client resolves all pod IPs and balances itself. Every gRPC library supports this.

A service mesh. A proxy in each pod understands HTTP/2 and balances per request (section 7).

Periodic reconnection. Configure a maximum connection age so clients redistribute. Crude, and it works.

The Service types

ClusterIP — internal only. The default, and what you want for anything other services call.

NodePort — opens the same high port (30000–32767) on every node, forwarding to the Service. Mostly a building block that LoadBalancer uses underneath, occasionally useful for bare metal.

LoadBalancer — asks the cloud controller manager for a real load balancer with a real IP, pointing at the NodePort on each node. One per Service, and each one costs money and often a public IP. Twenty services, twenty load balancers. This cost is why Ingress exists.

ExternalName — a DNS alias to something outside the cluster. No proxying, just a CNAME. Useful for pointing at a managed database with a name that stays stable when you migrate.

Headless (clusterIP: None) — no virtual IP at all. DNS returns every pod's address directly. This is the one that matters for two cases: clients that balance themselves, and StatefulSets, where each pod also gets its own name (Chapter 13.6.11).

$ nslookup checkout-api.shop.svc.cluster.local     # normal Service
Address: 10.0.42.17                                 # one virtual IP

$ nslookup db-headless.shop.svc.cluster.local      # headless
Address: 10.244.1.5                                 # every pod
Address: 10.244.2.9
Address: 10.244.3.2

externalTrafficPolicy — the source IP question

By default, traffic arriving at a node's NodePort may be forwarded to a pod on a different node, and that hop rewrites the source address. Your application logs the node's IP instead of the client's, and IP-based rate limiting or allow-lists silently break.

spec:
  externalTrafficPolicy: Local     # only forward to pods on this node

Local preserves the client IP and removes the extra hop. The trade is that a node with no pod of that Service drops the traffic, so the load balancer's health checks must be set up to only send traffic to nodes that have one — which cloud providers handle — and that balancing becomes uneven if pods are unevenly spread across nodes.

4. DNS, and the surprise inside it

CoreDNS runs as a Deployment in kube-system, and every pod is configured to use it.

The naming scheme:

<service>.<namespace>.svc.cluster.local          # a Service
<pod-name>.<service>.<namespace>.svc.cluster.local   # a StatefulSet pod

Within a namespace the short name works: checkout-api. Across namespaces, checkout-api.shop.

Now the surprise, because it costs real latency and shows up in profiles as unexplained delay. Every pod's /etc/resolv.conf looks like this:

nameserver 10.0.0.10
search shop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means: if a name has fewer than 5 dots, try each search domain first before trying the name as given.

So a lookup of api.external-vendor.com — three dots — is attempted as:

api.external-vendor.com.shop.svc.cluster.local   → NXDOMAIN
api.external-vendor.com.svc.cluster.local        → NXDOMAIN
api.external-vendor.com.cluster.local            → NXDOMAIN
api.external-vendor.com                          → finally resolves

Four lookups, each of which is A and AAAA, so eight queries for one external name. Multiply by every outbound request in a service that does not reuse connections and it is a measurable share of your latency and a large share of CoreDNS's load.

Two fixes:

Use a fully qualified name with a trailing dot — api.external-vendor.com. — which skips the search list entirely.

Or lower ndots for the pod:

spec:
  dnsConfig:
    options:
      - { name: ndots, value: "2" }

Also cache. NodeLocal DNSCache runs a resolver on each node so most lookups never leave the machine, and it is one of the highest-value additions to a busy cluster.

5. Getting traffic in: Ingress and the Gateway API

A LoadBalancer per Service does not scale in cost. Ingress consolidates: one load balancer, routing by hostname and path.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"     # (1)
spec:
  ingressClassName: nginx                                   # (2)
  tls:
    - hosts: [shop.example.com]
      secretName: shop-tls                                  # (3)
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service: { name: checkout-api, port: { number: 80 } }
          - path: /
            pathType: Prefix
            backend:
              service: { name: web, port: { number: 80 } }

(1) And here is Ingress's real weakness. The specification covers hosts, paths and TLS and nothing else. Timeouts, body size, rewrites, rate limits, authentication, canary weights — all of it is vendor-specific annotations. Your Ingress is not portable between controllers, and this is exactly why the Gateway API exists.

(2) Which controller handles this. Several can coexist — an internal one and an external one, for instance.

(3) A Secret holding the certificate. cert-manager automates this: it watches Ingress resources, requests certificates from Let's Encrypt or another issuer, proves ownership, writes the Secret, and renews before expiry. It is the standard answer to certificates in Kubernetes and it removes an entire category of 3 a.m. incident.

Controllers you will meet: ingress-nginx (most common), Traefik, HAProxy, Envoy-based ones, and cloud-native controllers like Azure Application Gateway Ingress Controller (Chapter 13.6.15).

The Gateway API

The successor, and the direction of travel. It fixes Ingress's two structural problems: everything real is an annotation, and one resource mixes concerns owned by different teams.

Three resources, three owners:

• GatewayClass — the infrastructure provider defines it.
• Gateway — the platform team owns it: listeners, ports, certificates.
• HTTPRoute — the application team owns it: hostnames, paths, header matching, traffic splitting, request mirroring.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: checkout }
spec:
  parentRefs: [{ name: shop-gateway }]
  hostnames: ["shop.example.com"]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /api } }]
      backendRefs:
        - { name: checkout-api-v1, port: 80, weight: 90 }   # (1)
        - { name: checkout-api-v2, port: 80, weight: 10 }

(1) Weighted traffic splitting as a first-class field — a canary with no annotations and no mesh. This one example is the clearest illustration of what the Gateway API adds.

Ingress is not deprecated and will work for years. New designs should prefer the Gateway API.

6. Network policies

Without any policy, every pod can reach every other pod in the cluster. A compromised front-end can connect directly to your database pods, bypassing every application-level control.

A NetworkPolicy is a pod-level firewall, expressed in labels rather than IPs.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: shop }
spec:
  podSelector: {}                    # (1)
  policyTypes: [Ingress, Egress]     # (2)
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: db-from-api, namespace: shop }
spec:
  podSelector:
    matchLabels: { app: db }         # (3)
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: checkout-api }   # (4)
      ports:
        - { protocol: TCP, port: 5432 }

(1) An empty selector matches every pod in the namespace.

(2) With no rules listed, this denies everything in both directions. This is the default-deny baseline, and it is where you start.

(3) This policy applies to database pods.

(4) Only pods labelled app: checkout-api may connect, and only on 5432.

Four things to get right:

Policies are additive and there is no deny rule. Traffic is allowed if any policy allows it. You restrict by having a default-deny in place and then adding narrow allowances.

A pod with no policy selecting it is completely open. Absence of a policy is not a default-deny.

Egress rules must allow DNS, or every name lookup in the namespace fails and the outage looks nothing like a network policy problem:

egress:
  - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]
    ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]

And the CNI plugin must enforce them. NetworkPolicy objects are accepted by the API whether or not anything implements them. On a plugin without policy support they are silently ignored, and you have a policy that exists, validates, and does nothing. Flannel alone is the common case.

7. Service mesh, briefly and honestly

A service mesh puts a proxy in the path of every request between services, giving you things applications otherwise implement individually.

What it genuinely provides:

Automatic mutual TLS between every pod, with certificate rotation. This is the strongest single reason teams adopt one.

Per-request load balancing, which solves the gRPC problem from the opening properly.

Retries, timeouts, circuit breaking and traffic splitting, configured rather than coded.

Uniform metrics and traces for every call, with no application changes.

Two architectures:

Sidecar (Istio's classic mode, Linkerd) — a proxy container injected into every pod. Costs a proxy's memory and CPU per pod and adds two hops of latency per request.

Ambient / node-level (Istio's newer mode, Cilium) — shared proxies per node instead of per pod. Much lower overhead, and the direction the ecosystem is moving.

The honest assessment: a mesh is a significant operational commitment. It adds a control plane to run and upgrade, a proxy in every request path, and a new layer to debug when something is slow. Adopt one when you have a specific problem it solves — mutual TLS everywhere, or per-request balancing for gRPC, or traffic-shifted deploys — and not because it is on the landscape diagram.

8. Debugging network problems

kubectl get endpointslices -l kubernetes.io/service-name=checkout-api   # (1)
kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash              # (2)
kubectl exec -it <pod> -- nslookup checkout-api                          # (3)
kubectl port-forward svc/checkout-api 8080:80                            # (4)
kubectl get networkpolicies -n shop                                      # (5)
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller            # (6)

(1) Start here for "no traffic reaching the pod". Empty means the selector does not match the labels, or no pod is passing readiness. It is a one-command answer.

(2) A pod with every network tool in it — dig, curl, tcpdump, traceroute. This is how you test from inside the cluster's network rather than guessing.

(3) Resolve from a pod that is actually experiencing the problem.

(4) Bypass the Service and Ingress entirely and talk straight to the pod. If this works and the Service does not, the problem is between them.

(5) Check what is being enforced before assuming the application is broken.

(6) Ingress controller logs show the routing decision, which settles most "404 from the ingress" questions.

The failure table:

Symptom	Usual cause
Service has no endpoints	Selector does not match labels, or readiness failing
Intermittent failures to one pod	One pod unhealthy but still in endpoints — readiness too lenient
DNS slow, high CoreDNS load	ndots:5 search-domain expansion
Works small, hangs on large responses	MTU mismatch on an overlay network
gRPC traffic all on one pod	Per-connection balancing
Client IP is a node's IP	externalTrafficPolicy: Cluster
Policy exists, has no effect	CNI plugin does not enforce policies
Everything breaks after adding egress policy	DNS not allowed
Pods stuck ContainerCreating, IP allocation error	Pod address range exhausted

What the interviewer will push on

"Why does all my gRPC traffic hit one pod?" Service balancing is per connection, and gRPC holds one long-lived HTTP/2 connection, so every request rides it to the pod chosen at connection time. Fixes: client-side balancing against a headless Service, a service mesh for per-request balancing, or forcing periodic reconnection. This is the question that separates people who have run Kubernetes from people who have read about it.

"A pod is running but gets no traffic." Check kubectl get endpoints first. Empty means the Service selector does not match the pod labels, or readiness is failing. Going to endpoints rather than to logs is the tell, because it distinguishes a routing problem from an application problem in one command.

"What is a Service IP?" A virtual address that exists on no interface. kube-proxy programmes rules so that packets to it are rewritten to a pod IP, and there is no proxy hop — the connection goes straight to the pod. Saying "it is not a real address anywhere" is the answer that shows the model is understood.

"Ingress or Gateway API?" Ingress covers hosts, paths and TLS, and everything else — timeouts, body size, rewrites, canaries — is vendor-specific annotations, so it is not portable. The Gateway API separates GatewayClass, Gateway and HTTPRoute by owner and makes traffic splitting a first-class field. New designs should prefer it; Ingress is not deprecated.

"How do you stop the front-end reaching the database directly?" A default-deny NetworkPolicy per namespace, then narrow allowances by label. Two follow-ups to volunteer: egress rules must permit DNS or everything breaks in a way that looks unrelated, and the CNI plugin must actually implement policies — otherwise the objects are accepted and ignored.

"What is ndots:5 and why does it matter?" Any name with fewer than five dots is tried against three search domains first, so an external hostname takes four lookups, doubled for IPv6. It is a real latency and CoreDNS load problem, fixed with a trailing dot on external names, a lower ndots, or NodeLocal DNSCache.

One thing to volunteer: mention that pod address ranges must be planned before the cluster exists, because a plugin that gives pods real network addresses can exhaust a subnet, and a pod CIDR that overlaps the office network or a partner's produces failures where half the destinations work. It is a five-minute decision at design time and effectively unfixable afterwards without rebuilding the cluster.

Recall

• Four rules: every pod has an IP · every pod reaches every other pod with no NAT · nodes reach pods · the IP a pod sees is the IP others use. Three ranges: node, pod CIDR, and Service CIDR, which exists on no interface.
• Overlay (VXLAN) works anywhere but costs a header — MTU mismatch makes small requests work and large ones hang. Routed (BGP) is faster and needs network cooperation. Real-VNet-IP plugins exhaust subnets, which is why overlay modes exist.
• A Service selects on labels, not pods. kubectl get endpoints empty = selector mismatch or readiness failing — the first command for "running but no traffic".
• Balancing is per connection, not per request, so gRPC and HTTP/2 pin to one pod. Fix with a headless Service and client-side balancing, or a mesh.
• Types: ClusterIP · NodePort (building block) · LoadBalancer, one cloud load balancer each — the cost that Ingress exists to avoid · ExternalName · headless for per-pod DNS and self-balancing clients.
• externalTrafficPolicy: Local preserves the client IP and removes a hop; Cluster rewrites the source to a node IP and breaks IP allow-lists.
• ndots:5 turns one external lookup into four (doubled for IPv6). Use a trailing dot, lower ndots, or NodeLocal DNSCache.
• Ingress = hosts, paths, TLS; everything else is vendor annotations. cert-manager automates certificates. Gateway API splits GatewayClass / Gateway / HTTPRoute by owner and makes weighted splitting first-class.
• NetworkPolicies are additive with no deny rule, a pod with no policy is fully open, egress rules must allow DNS, and a plugin without policy support ignores them silently.
• A mesh gives automatic mTLS, per-request balancing, retries and uniform telemetry — at the cost of a control plane and a proxy in every request path. Adopt for a named problem.

Self-test: Why does scaling a gRPC service change nothing? · Which command answers "running but no traffic", and what do the two possible causes mean? · Why does an overlay network break large responses? · What does externalTrafficPolicy: Local buy and cost? · What breaks first when you add an egress policy? · Why can a NetworkPolicy exist and do nothing?

Next: 13.6.13 is the configuration that decides whether your cluster is stable or mysteriously slow — requests and limits, why CPU and memory fail in completely different ways, storage that follows a pod, and every layer of autoscaling.