13.6.9 — The Kubernetes Control Plane

kubectl apply -f deployment.yaml returns in 200 milliseconds:

deployment.apps/checkout-api configured

Nothing has been deployed. No container has started, no machine has been chosen, no image has been pulled. All that happened is that a row in a database changed.

Everything else — choosing machines, pulling images, starting containers, wiring networking, watching health — happens afterwards, asynchronously, performed by five components that never speak to each other. They only read and write that one database, through one door.

This page is those five components, that one door, and the exact path your YAML takes.

1. Everything is a resource, and there is one door

Kubernetes is a database with an opinionated API and a set of programs watching it. That sentence is more accurate than any diagram, and it explains most behaviour that seems strange.

Every object you work with — a pod, a service, a secret, a node — has the same four-part shape:

apiVersion: apps/v1          # (1)
kind: Deployment             # (2)
metadata:                    # (3)
  name: checkout-api
  namespace: shop
  labels: { app: checkout-api }
spec:                        # (4)
  replicas: 3
status:                      # (5)
  readyReplicas: 3

(1) Which API group and version. apps/v1 for Deployments, v1 (the core group, with no name) for Pods and Services, networking.k8s.io/v1 for Ingress. Versioning is per group, which is how the project adds features without breaking existing manifests.

(2) The type.

(3) Identity and labels. A name is unique within a namespace and a kind. Labels are how everything finds everything else — a Service does not know about pods, it knows about a label selector, and whatever matches it today is what receives traffic.

(4) spec is what you want. You write this.

(5) status is what is true. Controllers write this. You never write status, and the gap between spec and status is the work the cluster has to do. That split is the entire architecture in two fields.

And there is exactly one way in. The API server is the only component that talks to the datastore. The scheduler does not read etcd. The kubelet does not read etcd. Nothing else has the credentials or the code path.

Two enormous benefits follow from that single choke point. Every change passes through one place, so authentication, authorisation, admission policy, validation and auditing are complete by construction — there is no back door to secure separately. And every component sees a consistent view, because they all read the same store through the same server.

2. The API server: five checkpoints before anything is written

Figure 1 — One request, five checkpoints, then a fan-out. The write to etcd is the only thing that happens synchronously. Everything after it is independent programs reacting to a change they were watching for — which is why kubectl apply returns before anything has been deployed.

① Authentication — who are you? Client certificates, bearer tokens, service account tokens, or a plug-in provider (for AKS this is Microsoft Entra ID — Chapter 13.6.15). Kubernetes has no user database. There is no User object. It verifies a credential and extracts a username and group list, and identity management belongs to whatever issued the credential. That surprises people the first time.

② Authorisation — may you? Almost always RBAC: is there a role, bound to you, permitting this verb on this resource in this namespace? Deny by default.

③ Mutating admission — rewrite the object. Controllers that change what you submitted before it is stored. Built-in ones fill in defaults. Webhook-based ones do things like inject a sidecar container (this is how a service mesh appears in every pod without anyone editing a manifest), add labels, or set a default storage class.

④ Schema validation. Fields exist, types are right, required values present.

⑤ Validating admission — accept or deny. Policy engines run here: no :latest tags, images only from approved registries, resource limits required, signatures verified. Deny at this checkpoint and the object never enters the cluster at all — as opposed to being created and then flagged, which is what a scanning tool does. This is the difference between policy and reporting, and Chapter 13.6.14 builds it out.

Then the write to etcd, and the request returns. Everything you actually wanted to happen has not happened yet.

The other thing the API server does is serve watches. A client opens a long-lived connection and says "tell me about changes to pods". Every subsequent change is streamed to it. No component polls. This is what makes a cluster with thousands of objects cheap to run, and it is the mechanism the entire ecosystem — controllers, operators, GitOps tools, dashboards — is built on.

3. etcd: the only source of truth

A distributed key-value store using the Raft consensus algorithm (Chapter 10.7.2). Everything is in here: every pod, every secret, every config map, the whole cluster.

How it stores things: keys look like paths — /registry/pods/shop/checkout-api-7d9f-x2k — and values are serialised objects. etcdctl get /registry --prefix --keys-only lists the cluster.

Four properties that shape how Kubernetes behaves:

Strong consistency through Raft. One member is the leader; writes go through it and are committed once a majority has them. A majority is required, so cluster sizes are odd — three or five. Three members tolerate one failure; five tolerate two. An even number gains nothing, since four members still need three to agree.

Every write increments a global revision number, and objects carry it as resourceVersion. This gives you two things you use constantly without noticing. Optimistic concurrency: an update includes the version you read, and if the object changed since, your write is rejected with a conflict — this is why controllers retry, and why kubectl apply occasionally reports a conflict on a busy object. And resumable watches: a client that disconnects reconnects saying "from revision 12345" and receives everything it missed.

Watch is a first-class feature of etcd, which is why the API server can offer it efficiently.

A history is kept and must be compacted. Old revisions accumulate; compaction discards them and defragmentation reclaims the disk. Managed clusters do this for you and self-managed ones must.

The operational facts, because these are what actually hurt:

etcd is extremely sensitive to disk latency. Raft commits a write only after it is written to disk on a majority of members. Slow disks do not slow etcd a little — they make the whole cluster unresponsive, because every API call waits on a write. SSDs are a requirement, not an optimisation, and the health signal to watch is the write-to-disk latency percentile.

Back it up. etcdctl snapshot save produces a file that is the entire cluster definition. Lose etcd with no snapshot and you have lost the cluster — running pods keep running for a while, but nothing can be changed, healed or rescheduled, and rebuilding means recreating every object from whatever manifests you still have. This is the strongest argument for GitOps: if every object is in Git, etcd stops being irreplaceable.

There are size limits. The default database limit is 2 GB, and individual values are capped around 1.5 MB. The way people meet these limits is almost always the same: thousands of ConfigMaps or Secrets, or something writing objects in a loop.

Secrets are stored here as base64, which is encoding, not encryption. Anyone who can read etcd — or restore a backup of it — reads every secret. Encryption at rest is a configuration option that must be turned on, and managed services do it for you (Chapter 13.6.15).

4. The scheduler: choosing a machine

The scheduler watches for pods with no node assigned, and its output is one field. It does not start anything. It writes spec.nodeName and its job is finished.

Two phases, and the vocabulary is worth having exactly:

Filtering — which nodes could run this? Each node is checked against hard requirements:

• Are there enough free CPU and memory to satisfy the pod's requests? (Requests, not usage — section 5 of Chapter 13.6.13 explains why that distinction matters.)
• Does the node match the pod's node selector and required affinity?
• Does the pod tolerate the node's taints?
• Are the ports free, is the required volume attachable in this zone, is the node ready?

A node failing any check is out. If every node is filtered out, the pod stays Pending — and this is the single most common reason for a pending pod.

Scoring — which of the survivors is best? Each remaining node is scored by several plugins and the totals are compared:

• Spread across nodes and zones for pods with the same labels.
• Least allocated — prefer emptier nodes (the default), or most allocated if you configure bin-packing to save money.
• Image locality — a node that already has the image starts faster.
• Affinity preferences you expressed as "preferred" rather than "required".
• Taint toleration preferences.

Highest score wins, ties broken randomly. Then binding: the scheduler writes the node name back through the API server, and a kubelet on that node sees a pod assigned to it and takes over.

Preemption handles the case where nothing fits and the pod is important. If a pending pod has a higher priorityClass than pods already running, the scheduler can evict lower-priority pods to make room. This is how you guarantee that a critical service always gets capacity while batch work absorbs the shortage — and it is one of the ideas that came straight from Borg.

One structural fact worth carrying: the scheduler's decision is made once, at placement time, from the state at that moment. It does not move a pod later because the cluster became unbalanced. A node that is now overloaded stays overloaded unless something evicts pods — which is what the separate descheduler project exists to do.

5. The controller manager: dozens of loops in one binary

One process running roughly thirty independent control loops, each responsible for one kind of gap. Some of them:

Controller	Watches for	Acts by
Deployment	Deployment changes	Creating and scaling ReplicaSets
ReplicaSet	Pod count ≠ replicas	Creating or deleting pods
Node	Nodes going silent	Marking NotReady, tainting, evicting after a timeout
Job	Job objects	Running pods to completion
CronJob	The clock	Creating Jobs on schedule
Endpoints / EndpointSlice	Ready pods matching a Service selector	Maintaining the address list traffic goes to
Namespace	Namespace deletion	Deleting everything inside, then the namespace
ServiceAccount / token	New namespaces and accounts	Creating and rotating credentials
PersistentVolume	Claims	Binding claims to volumes

Every one is the same loop from Chapter 13.6.8: observe desired, observe actual, close the gap, repeat forever.

Three properties of that design are worth naming, because they explain most controller behaviour:

They are level-triggered, not edge-triggered. A controller acts on the current state, not on the event that woke it. This is why a missed event is not a disaster — the next resync sees the same gap and does the same thing. An edge-triggered system that misses an event has lost information forever; this one cannot.

They read from a local cache, not from etcd. Each controller runs an informer: a watch that maintains an in-memory copy of the objects it cares about, kept current by the event stream. Reads are local and free. Without this the API server would be crushed by controllers polling.

Work is queued and retried with backoff. A failed reconcile goes back on the queue with an increasing delay. This is why a broken thing retries forever without hammering anything, and why a fix sometimes takes a couple of minutes to be picked up.

One controller deserves a special note. The node controller is what makes the 03:14 outage from Chapter 13.6.8 self-repairing. Kubelets report in regularly; if one stops, the node is marked NotReady after about 40 seconds, and after a further grace period (5 minutes by default) its pods are marked for deletion and their controllers create replacements elsewhere. That total delay is deliberate — evicting instantly on a brief network blip would be far more damaging than waiting.

6. The cloud controller manager

The component that turns Kubernetes objects into cloud resources, split out so the core project holds no provider-specific code.

• Service of type LoadBalancer → an Azure Load Balancer, an AWS NLB, a Google forwarding rule, with the pods registered behind it.
• PersistentVolumeClaim → a managed disk, attached to the right node.
• Node lifecycle → labelling nodes with their region and zone, and removing Node objects for machines the provider says are gone.
• Routes → provider routing entries so pod traffic reaches other nodes, where the network model needs it.

On a managed service this runs for you and is one of the things you are paying for, together with the fact that a LoadBalancer Service in a manifest quietly becomes a real load balancer with a real public IP (Chapter 13.6.15).

7. Following one kubectl apply all the way

This is the trace to know cold. Every step names the component responsible.

1. kubectl reads your YAML, converts to JSON, finds the API endpoint from your kubeconfig, and sends PATCH /apis/apps/v1/namespaces/shop/deployments/checkout-api.
2. API server: authenticate, authorise, mutate, validate, admit. Any checkpoint can reject, and then nothing was written.
3. API server writes to etcd. Raft commits it on a majority. The command returns here.
4. Deployment controller sees the change through its watch. The spec says 3 replicas of image v2; the current ReplicaSet has 3 of v1. It creates a new ReplicaSet for v2 with 0 replicas and begins stepping: scale new up, wait for ready, scale old down, respecting maxSurge and maxUnavailable.
5. ReplicaSet controller sees a ReplicaSet wanting 1 pod and having 0. It creates a Pod object. The pod exists in the database with no node — status Pending.
6. Scheduler sees a pod with empty spec.nodeName. Filters, scores, binds. Writes the node name.
7. Kubelet on that node sees a pod assigned to it. Pulls the image if needed, asks containerd to create the sandbox and the containers (Chapter 13.6.10), and starts running probes.
8. Kubelet updates status through the API server: ContainerCreating, then Running, then Ready once the readiness probe passes.
9. EndpointSlice controller sees a now-ready pod whose labels match a Service selector, and adds its IP to the endpoint list.
10. kube-proxy on every node sees the endpoint change and updates the local forwarding rules, so Service traffic can now reach this pod.
11. Deployment controller sees the new pod ready, and takes the next step of the rollout.

Eleven steps, no direct calls between components. Every one of them read a change through a watch and wrote a change back. That is the whole architecture, and it is why you can lose any single component temporarily and the cluster degrades rather than collapses.

8. What breaks when a component is down

This table is the reason knowing the components pays off.

Down	What still works	What stops
API server	Every running pod keeps serving	Every change, all kubectl, every controller
etcd (no quorum)	Running pods keep serving	All writes; the API becomes read-only at best
Scheduler	Everything already placed	New pods stay Pending forever
Controller manager	Existing pods	Scaling, rollouts, node failure recovery, endpoint updates
CCM	Everything else	New load balancers and volume provisioning
kubelet on a node	Its containers keep running briefly	That node's updates; it goes NotReady and its pods are rescheduled

Read the first column again: running workloads survive a full control plane outage. The data plane keeps serving traffic. You lose the ability to change anything, and you lose self-healing — so the cluster is frozen rather than dead. This is a deliberate design property and it is why control plane upgrades are routine rather than terrifying.

9. Making the control plane highly available

Three or five control plane nodes, and the components behave differently from each other:

API servers are stateless and all active, behind a load balancer. Add as many as you like.

etcd needs an odd number for quorum. Three tolerates one failure, five tolerates two. Beyond five, write latency gets worse because every write waits for a larger majority.

Scheduler and controller manager use leader election. All replicas run; one holds a lease in the API and does the work; the others wait. Two schedulers both placing pods would double-book nodes, so this is a correctness requirement rather than an efficiency one.

Spread control plane nodes across availability zones, or a single data centre failure takes the quorum with it.

And on a managed service you do none of this. Azure, AWS and Google run the control plane, patch it, back up etcd and keep quorum. Running etcd correctly is a specialised job, and the managed control plane is the single best value in the whole platform — often free or a few tens of dollars a month, against an engineer's continuous attention.

What the interviewer will push on

"What happens when you run kubectl apply?" The trace in section 7. The detail that separates people is knowing the command returns after the etcd write and before anything is deployed, and that no component calls another — everything reacts to watches. If you can name the five admission checkpoints in order, that is the strongest version of this answer.

"Why is the API server the only thing that talks to etcd?" Because one door means authentication, authorisation, admission policy, validation and auditing are complete by construction, with no second path to secure. It also gives every component a consistent view. The good follow-up to volunteer is that this is also why a mutating webhook can inject a sidecar into every pod without anyone editing a manifest.

"Why is etcd always three or five members?" Raft needs a majority to commit, so an even number buys nothing — four members still need three. Three tolerates one failure, five tolerates two, and more than five makes writes slower. Then the operational point: etcd is disk-latency sensitive, so a slow disk makes the whole API unresponsive rather than slightly slower.

"A pod is Pending. Walk me through it." It has not been bound to a node, so this is a scheduler question, not an application one. Filtering removed every node: insufficient CPU or memory to meet its requests, an untolerated taint, a node selector matching nothing, or a volume that can only attach in another zone. kubectl describe pod and read the Events, because a pending pod has no logs at all.

"What is a controller, exactly?" A loop that reads desired state, observes actual state, acts to close the gap, forever. Level-triggered, so a missed event is harmless; backed by an informer cache so it does not poll the API server; with a work queue and exponential backoff on failure. Naming level-triggered is the tell of someone who has written one.

"What breaks if the control plane goes down?" Running pods keep serving traffic. You lose all changes, all scaling, all self-healing and all new scheduling. The cluster is frozen, not dead — which is why control plane upgrades are routine.

One thing to volunteer: point out that Secrets are stored in etcd base64-encoded rather than encrypted, so anyone who can read etcd or restore a snapshot of it reads every secret in the cluster — which makes encryption at rest and tight backup handling a requirement, not a hardening step. It reframes an etcd backup as a credential, which is the correct way to treat it.

Recall

• Every object is apiVersion, kind, metadata, spec, status. You write spec, controllers write status, and the gap between them is the work. Labels and selectors — not names — are how objects find each other.
• The API server is the only component that talks to etcd, which makes authn, authz, admission, validation and audit complete by construction. It also serves watches, so nothing polls.
• Five checkpoints in order: authenticate → authorise (RBAC) → mutating admission (injects sidecars, sets defaults) → schema validation → validating admission (policy denies, and the object is never stored).
• etcd: Raft, majority commit, odd sizes 3 or 5. Every write bumps a revision, giving optimistic concurrency (resourceVersion conflicts) and resumable watches. Very sensitive to disk latency. Back it up — it is the cluster. Secrets are base64, not encrypted.
• Scheduler = filter then score, then write spec.nodeName. Filtering uses requests, not usage; failing every filter means Pending forever. Preemption evicts lower-priority pods. Placement is decided once and never rebalanced.
• Controller manager runs ~30 loops. Level-triggered (a missed event is harmless), reading from informer caches, with queues and backoff. The node controller marks NotReady at ~40 s and evicts after ~5 minutes.
• Cloud controller manager turns LoadBalancer Services into real load balancers and claims into real disks.
• The eleven-step trace: kubectl → admission → etcd → Deployment controller → ReplicaSet controller → Pod Pending → scheduler binds → kubelet starts → status → EndpointSlice → kube-proxy.
• Control plane down = frozen, not dead. Running pods serve; changes, scaling and self-healing stop. HA: stateless API servers behind a load balancer, odd-numbered etcd, leader election for scheduler and controller manager.

Self-test: What has happened when kubectl apply returns? · Name the five admission checkpoints in order and say which one a policy engine uses. · Why is an even-numbered etcd cluster pointless? · What does the scheduler actually write? · What does level-triggered mean and why does it make controllers robust? · What still works when the entire control plane is down?

Next: 13.6.10 moves to the machine where the work happens — what the kubelet does every ten seconds, how a pod becomes containers through the runtime interface, and the four plug-in points that make a node work at all.