13.6.10 — The Node: kubelet, Runtime, kube-proxy and the Plug-in Points

A node reports Ready. Every pod on it is Running. Every dashboard is green.

And every request to the service times out.

The node ran out of disk. The kubelet noticed, set a condition called DiskPressure, and began evicting pods to reclaim space — starting with the ones that had declared no resource requests, which happened to be the entire application tier. The pods it evicted were recreated on the same node, because the scheduler was still told the node was Ready, and they were evicted again. The cluster was in a loop, and the summary column said everything was fine.

The control plane decides what should exist. The node decides what actually happens, and it has its own judgement, its own failure modes and its own eviction policy. This page is the machine.

1. What is on a node

Figure 2 — One node. The kubelet is the only thing here that talks to the API server about pods. Everything below it is reached through a plug-in interface — CRI for containers, CNI for networking, CSI for storage — which is why the runtime, the network and the storage layer can each be swapped without changing Kubernetes.

Three programs and three plug-in interfaces:

• kubelet — the node's agent. Talks to the API server, makes pods real, reports status.
• Container runtime (containerd) — reached through the CRI.
• kube-proxy — programmes Service routing rules.
• CNI plugin — gives pods their network.
• CSI driver — attaches and mounts storage.
• Device plugins — expose GPUs and other hardware.

2. The kubelet

One process per machine, and it is the only thing on the node that reads pod specifications from the API server.

Its loop:

1. Get the pods assigned to this node — from a watch, plus files on disk, plus (historically) an HTTP endpoint.
2. Compare with what is running, learned from the runtime.
3. Act on the differences — create missing containers, kill ones that should not exist, restart ones that failed.
4. Report status back to the API server.

It is the same reconcile loop as every controller, applied to one machine.

Creating a pod, step by step

① The sandbox comes first. Before any of your containers start, the kubelet asks the runtime to create a pod sandbox: a container running a tiny program that does nothing but sleep. This is the pause container, and it exists to hold the namespaces open.

Why it must exist: the pod's network namespace has to survive your application container crashing and restarting, or the pod would get a new IP every time it restarted. The pause container never crashes, so the namespace persists and the IP is stable for the pod's whole life. It is a few hundred kilobytes and it does nothing else. This is exactly the --network container:<name> mode from Chapter 13.6.6, made structural.

② The network is attached. The kubelet calls the CNI plugin, which creates the veth pair, allocates an IP from the node's range, and writes routes. The IP belongs to the sandbox, so every container in the pod shares it.

③ Volumes are set up. Storage is attached to the node and mounted where the pod expects it, and ConfigMap and Secret volumes are materialised as files.

④ Init containers run to completion, in order. Each must exit 0 before the next starts. Used for migrations, waiting on a dependency, or fetching configuration.

⑤ Application containers start, sharing the sandbox's namespaces.

⑥ Probes begin, and status flows back to the API server.

Probes: three questions, three consequences

This is the highest-value section on this page, because confusing two of these causes outages.

Readiness — "should I receive traffic?" Failing removes the pod from the Service's endpoint list. It restarts nothing. This is the probe that should check dependencies: if the database is unreachable, this pod cannot serve, so stop sending it requests.

Liveness — "am I broken beyond recovery?" Failing restarts the container. It should check only that the process itself is functioning — not deadlocked, not wedged, not out of file descriptors.

Startup — "have I finished starting?" While it is failing, the other two are suspended. For applications with slow boots. Use this rather than a large initialDelaySeconds on liveness, because a long initial delay stays in force for the pod's entire life and delays real failure detection forever.

The rule that prevents the classic outage: a liveness probe must never check a dependency.

Here is why, concretely. Your liveness endpoint queries the database. The database gets slow for thirty seconds. Every replica fails liveness at the same time. Every replica restarts at the same time, dropping every in-flight request, and comes back with cold caches and no connection pools — which makes the database slower. A brief degradation becomes a total outage, caused entirely by the health check.

readinessProbe:
  httpGet: { path: /ready, port: 3000 }      # (1)
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /healthz, port: 3000 }    # (2)
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet: { path: /healthz, port: 3000 }    # (3)
  failureThreshold: 30
  periodSeconds: 5

(1) /ready checks the database connection and any dependency needed to serve. Three failures and traffic stops arriving.

(2) /healthz returns 200 if the process is running. Nothing else. No database, no downstream call, no cache.

(3) The same cheap endpoint, but with 30 attempts at 5 seconds — 150 seconds to start, after which liveness takes over at its strict setting.

Probe types: httpGet (2xx or 3xx is a pass), tcpSocket (can it be connected to), exec (a command exiting 0 — expensive, since it starts a process every period), and grpc (the standard gRPC health protocol).

What else the kubelet does

Reports node status every few seconds: capacity, allocatable resources, conditions, images present, and a heartbeat via a Lease object. Miss the heartbeats and the node controller marks the node NotReady (Chapter 13.6.9).

Collects resource metrics through built-in cAdvisor, which is what kubectl top and the autoscaler read via the metrics server.

Cleans up images. When disk usage crosses a high threshold (85% by default), it deletes unused images until it reaches the low one (80%). This is why an image "already on the node" sometimes has to be pulled again.

Runs static pods. Any manifest placed in /etc/kubernetes/manifests is run by the kubelet directly, with no API server involved. This is the bootstrap trick that solves the chicken-and-egg problem — on a self-managed cluster the API server itself is a static pod, so the kubelet can start the control plane that it will then report to. A mirror pod appears in the API for visibility, but deleting it does nothing: the file is the truth.

Eviction: when the node is under pressure

This is the opening scene of the page. The kubelet monitors its own resources and, when they run short, starts removing pods to protect the machine.

Hard eviction thresholds — act immediately. Defaults are roughly: memory available below 100 Mi, disk free below 10%, inodes below 5%.

Soft thresholds — act after a grace period.

The order pods are evicted in is decided by their QoS class, and this is why declaring resources matters:

1. BestEffort — no requests or limits set. Evicted first, always.
2. Burstable — requests set, limits higher or absent. Evicted by how far over their requests they are.
3. Guaranteed — requests equal limits for every container. Evicted last.

A production service with no resource requests is the first thing thrown off a struggling node. It is a one-line omission with a severe consequence, and it is extremely common.

The node also taints itself under pressure — node.kubernetes.io/disk-pressure, memory-pressure, pid-pressure — which stops new pods being scheduled onto it. In the opening story, this taint is what should have broken the loop, and the reason to check kubectl describe node rather than the Ready column: conditions and taints live there, and the summary hides them.

3. CRI: how the kubelet talks to the runtime

A gRPC interface with two services: RuntimeService (sandboxes, containers, exec, logs) and ImageService (pull, list, remove).

Since Kubernetes 1.24 there is no Docker on a node. The kubelet speaks CRI to containerd or CRI-O directly. Chapter 13.6.3 covered what that did and did not change — the short version is that images are an OCI standard, so nothing about how you build changed.

The tool to know is crictl. When a pod will not start and kubectl is telling you nothing useful, this is how you ask the runtime directly, on the node:

crictl pods                            # (1)
crictl ps -a                           # (2)
crictl logs <container-id>             # (3)
crictl images                          # (4)
crictl inspect <container-id>          # (5)

(1) Sandboxes on this node. (2) Containers including exited ones. (3) Logs straight from the runtime. (4) What is actually cached here. (5) The full runtime view — mounts, the OCI config, the exit reason.

Use it when the kubelet and the API disagree, when an image pull is failing in a way the events do not explain, or when a container is exiting before the kubelet can report anything useful.

Runtime classes let different pods use different runtimes on the same cluster:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata: { name: gvisor }
handler: runsc
---
spec:
  runtimeClassName: gvisor      # this pod runs sandboxed

This is how you run untrusted workloads with a stronger boundary (Chapter 13.6.3) without changing anything else about the cluster.

4. kube-proxy: making Service addresses work

A Service has a virtual IP that belongs to no machine and no interface. Nothing answers it. kube-proxy is what makes packets sent to it arrive at a pod.

It watches Services and EndpointSlices and programmes the node's packet-handling rules. It is not in the data path — it writes rules, the kernel forwards the packets.

Three modes:

iptables (the long-time default). Writes DNAT rules per Service, choosing a backend with probability rules. Rule evaluation is a linear list, so a cluster with thousands of Services spends real time in rule updates and matching. This is the known scaling limit.

IPVS. Uses the kernel's load balancer, which is a hash table rather than a list. Constant-time lookup regardless of Service count, plus real balancing algorithms (round robin, least connections). The right choice for large clusters.

nftables (newer) and eBPF (Cilium), both replacing the linear-list problem with better data structures. eBPF lets you load a small, safety-checked program into the running kernel — the kernel verifies it cannot crash or loop forever, then runs it on every packet. So instead of writing rules for the kernel's existing packet filter to walk, you supply the code that does the work. eBPF-based networking can remove kube-proxy entirely, and it is where the ecosystem is heading.

The behaviour that surprises everyone, and it is not kube-proxy's fault:

Balancing is per connection, not per request. The rule is applied when a connection is established. A client that opens one long-lived HTTP/2 or gRPC connection sends every request over it, so every request lands on one pod. Add nine more pods and the traffic does not move. Chapter 13.6.12 covers the fixes.

5. CNI: how a pod gets an IP

A specification so small it fits on a page: a binary that, given a network namespace, sets up an interface and returns an IP.

The kubelet calls the plugin at sandbox creation, and the plugin creates the veth pair, allocates an address, and writes routes. Kubernetes itself contains no networking code for this, which is why there are a dozen plugins with genuinely different designs — Calico, Cilium, Flannel, Azure CNI, AWS VPC CNI. Chapter 13.6.12 compares them.

The rule every plugin must satisfy: every pod gets its own IP, and every pod can reach every other pod directly, with no NAT between them. How that is achieved is the plugin's business.

When the plugin is broken, pods stay in ContainerCreating with an event mentioning a network plugin error, and this is one of the few problems where the node's kubelet log is the fastest route to the answer.

6. CSI: how a pod gets storage

A standard interface so storage vendors write one driver that works on every orchestrator. It splits the work in two:

A controller component — one per cluster — creates and deletes volumes and asks the cloud to attach a disk to a node.

A node component — a DaemonSet — mounts the attached device into the pod's directory.

The sequence for a cloud disk: provision the disk → attach it to the node (a cloud API call taking tens of seconds) → format if new → mount into the pod.

Two things that bite:

Attach is slow and has limits. Every cloud caps how many disks may attach to one VM, by instance size. Hit the cap and pods stay Pending with a volume attachment error even though CPU and memory are free.

Zone binding. A cloud block disk exists in one availability zone and can only attach to a node in that zone. A pod using it can only be scheduled in that zone, and if that zone has no capacity the pod waits indefinitely. This produces the confusing case where a cluster is clearly half empty and a pod will not schedule.

7. Device plugins

How hardware other than CPU and memory is offered to pods. A DaemonSet advertises a resource name, and the scheduler treats it like any other:

resources:
  limits:
    nvidia.com/gpu: 1       # this pod needs one GPU

GPUs are the main use — the plugin discovers them, advertises them as node capacity, and the scheduler places only onto nodes with a free one. The same mechanism handles specialised network cards and other accelerators.

8. Node lifecycle and maintenance

kubectl get nodes                            # (1)
kubectl describe node aks-np1-vmss000002     # (2)
kubectl cordon <node>                        # (3)
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data   # (4)
kubectl uncordon <node>                      # (5)
kubectl top nodes                            # (6)

(1) Status and roles. Ready alone is not enough — the opening of this page is a Ready node in trouble.

(2) The command that actually answers node questions. Conditions (including the pressure ones), taints, allocatable versus capacity, the pods running here with their requests, and recent events.

(3) Mark unschedulable. Nothing new arrives; existing pods stay.

(4) Evict everything gracefully. DaemonSet pods are skipped because they belong to every node. This respects PodDisruptionBudgets, so it will refuse to take a service below its declared minimum — which is the mechanism that makes upgrades safe (Chapter 13.6.13).

(5) Allow scheduling again.

(6) Live CPU and memory. Note that it shows usage, while the scheduler decides on requests — a node can be 20% used and completely unschedulable because its requests are fully committed. That single mismatch explains a large share of "why won't this schedule, the node is empty" confusion.

Capacity versus allocatable, since it appears in describe: capacity is the machine's total; allocatable is what is left for pods after reserving for the kubelet, the runtime and the OS, plus the eviction threshold. On a small node this reservation is a significant fraction — a 2-core, 8 GB node may offer only about 1.6 cores and 5.5 GB to pods, and people are routinely surprised by where their memory went.

What the interviewer will push on

"Explain liveness versus readiness." Readiness controls traffic and should check dependencies; liveness restarts the container and must not. Then tell the incident: a liveness probe that checks the database makes every replica restart together during a database slowdown, dropping in-flight work and returning with cold caches — turning a degradation into an outage. Add that a startup probe is the correct fix for a slow boot, because a long initialDelaySeconds on liveness weakens failure detection permanently.

"What is the pause container for?" It holds the pod's namespaces open so the network namespace and the IP survive an application container crashing and restarting. Without it the pod would get a new IP on every restart. It is the same idea as docker run --network container:<name>, made structural.

"A pod is stuck in ContainerCreating. What do you check?" describe first for events: image pull failure, a missing ConfigMap or Secret, a volume that cannot attach, or a CNI error. Then on the node, crictl ps -a and the kubelet log. Naming the volume-attach limit and zone binding as causes is what marks experience.

"Why does gRPC traffic all hit one pod?" kube-proxy balances per connection, not per request, and gRPC holds one long-lived HTTP/2 connection. This question separates people who have run Kubernetes from people who have read about it.

"How does a node upgrade not take a service down?" Cordon, then drain, which evicts gracefully and respects PodDisruptionBudgets so a service never drops below its declared minimum. Without a budget, a drain can take every replica at once, and that looks like a platform failure rather than a missing two-line object.

"Which pods get evicted first when a node runs out of memory?" BestEffort — pods with no requests or limits at all — then Burstable by how far over their requests they are, then Guaranteed last. So a production service that declares no resources is the first thing thrown off a struggling node.

One thing to volunteer: point out that kubectl top shows usage while the scheduler decides on requests, so a node can look 20% busy and still be completely unschedulable. It is the fastest resolution to the most common "the cluster is empty but nothing will schedule" argument, and it leads naturally into the requests-versus-limits discussion that Chapter 13.6.13 covers.

Recall

• kubelet is the only thing on the node reading pod specs. Same reconcile loop, one machine. It reports status, runs probes, collects metrics, cleans up images at 85% disk, and runs static pods from /etc/kubernetes/manifests — which is how a self-managed control plane bootstraps.
• The pause container holds the pod's namespaces open, so the IP survives an application container restart.
• Pod startup order: sandbox → CNI attaches the network → volumes → init containers in order to completion → application containers → probes.
• Readiness controls traffic and should check dependencies. Liveness restarts and must not. A dependency check in liveness turns a slow database into a fleet-wide restart. Use a startup probe, not a long initialDelaySeconds.
• Eviction order is by QoS: BestEffort first, then Burstable by overage, Guaranteed last. A service with no requests is thrown off first. The node taints itself under memory, disk or PID pressure — visible in describe node, not in the Ready column.
• CRI = gRPC to containerd or CRI-O; no Docker on a node since 1.24. crictl is the tool when kubectl is not enough. RuntimeClass selects gVisor or Kata per pod.
• kube-proxy programmes rules and is not in the data path. iptables (linear, scales poorly) · IPVS (hash table, use for large clusters) · nftables and eBPF next. Balancing is per connection, so gRPC pins to one pod.
• CNI gives each pod an IP with no NAT between pods. CSI attaches and mounts storage — attach is slow, per-node disk counts are capped, and cloud disks are zone-bound, which strands pods in Pending.
• Maintenance: cordon then drain --ignore-daemonsets, which respects PodDisruptionBudgets. kubectl top shows usage; the scheduler uses requests. Allocatable is well below capacity on small nodes.

Self-test: Why does a pod keep its IP across a container restart? · Which probe may check the database, and what happens if the other one does? · Which pods are evicted first, and what one-line omission causes it? · What does crictl see that kubectl cannot? · Why does adding pods not spread gRPC traffic? · Why can a node at 20% CPU refuse to schedule anything?

Next: 13.6.11 is what you actually write — the pod specification field by field, and the six workload objects that create pods for you, including which one to reach for and why StatefulSets are slower on purpose.