14.3.2 — Your First Cluster

Chapter 14.3.1 explained why one Docker engine on one machine is not enough, and the chapters after this one take Kubernetes apart piece by piece: the control plane, the node, the workload objects, the network. Those chapters land much better once you have watched a cluster do its job. So this page is a half-hour session at your own keyboard.

You will create a three-machine Kubernetes cluster on your laptop, build a small TypeScript service, and deploy three copies of it. Then you will break it on purpose in four ways and watch it repair itself. Every name you meet here, such as pod, ReplicaSet, Service, kubelet and etcd, is explained properly in the chapters that follow, and this page points to the right one each time.

1. What you need

Docker, from any of the setups in Chapters 14.2.1 and 14.2.6. Then two small command-line tools:

• kubectl (say "cube control") is the client for any Kubernetes cluster. It sends your requests to the cluster's API server, in the same way the docker command sends requests to dockerd (Chapter 14.2.1). You will use it for the rest of your Kubernetes life, on every cluster.
• kind stands for Kubernetes in Docker. It creates a cluster where each Kubernetes "machine" is actually a Docker container, with a whole node's software running inside it. It exists for learning and for automated checks, and it is what the Kubernetes project itself uses to test Kubernetes. It is not for production.

Install both with your system's package manager:

brew install kind kubectl                                  # macOS
winget install -e --id Kubernetes.kind                     # Windows
winget install -e --id Kubernetes.kubectl

On Linux, follow the download instructions on the kind and kubectl project pages, which give a single curl command for each. Check that both work with kind version and kubectl version --client.

Other tools that do the same job, which you will hear about: minikube (the oldest, runs a cluster in a VM or container), k3d (runs a small Kubernetes called k3s in Docker), and the Kubernetes switch in Docker Desktop's settings. Everything on this page works on any of them except the two kind commands for creating the cluster and loading images.

2. Create a cluster with three machines

A real cluster has a control plane, the machines that store the cluster's state and make decisions, and worker nodes, the machines that run your containers. We will make one of the first and two of the second, so that moving work between machines is something you can actually see.

# kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

kind create cluster --name shop --config kind.yaml
kubectl get nodes
docker ps --format '{{.Names}}\t{{.Image}}'

NAME                 STATUS   ROLES           AGE   VERSION
shop-control-plane   Ready    control-plane   62s   v1.35.0
shop-worker          Ready    <none>          41s   v1.35.0
shop-worker2         Ready    <none>          41s   v1.35.0

shop-control-plane   kindest/node:v1.35.0
shop-worker          kindest/node:v1.35.0
shop-worker2         kindest/node:v1.35.0

Three nodes, and each one is a Docker container. The VERSION column is the Kubernetes version your copy of kind picked, and yours may differ. kindest/node is an image containing everything a real node runs: the kubelet (the agent on every node that starts and watches containers, Chapter 14.3.4), containerd (the container runtime from Chapter 14.2.1), and on the control-plane node, the control plane itself. Inside each of these containers there is another container runtime running your containers. It is containers inside containers, which is why kind starts in a minute and is gone in seconds.

Where kubectl keeps its connection details

kind create cluster also wrote the cluster's address and a credential into a file called the kubeconfig, by default ~/.kube/config. It has three lists:

• clusters: where each API server is and which certificate proves it is genuine,
• users: the credentials you present,
• contexts: named pairings of one cluster with one user.

One context is current, and every kubectl command goes to it.

kubectl config get-contexts
kubectl config current-context

CURRENT   NAME        CLUSTER     AUTHINFO    NAMESPACE
*         kind-shop   kind-shop   kind-shop

kind-shop

This is the most important safety habit in Kubernetes: always know your current context. kubectl delete does not ask which cluster you meant. People who have production and a laptop cluster in the same kubeconfig have deleted production by running a cleanup command in the wrong terminal. Most teams show the current context in their shell prompt for exactly this reason.

3. Look around before deploying anything

kubectl get pods -A

NAMESPACE            NAME                                         READY   STATUS
kube-system          coredns-7c65d6cfc9-4xk2p                     1/1     Running
kube-system          coredns-7c65d6cfc9-9hzfm                     1/1     Running
kube-system          etcd-shop-control-plane                      1/1     Running
kube-system          kindnet-6rj8d                                1/1     Running
kube-system          kindnet-b7wq2                                1/1     Running
kube-system          kindnet-zt5mc                                1/1     Running
kube-system          kube-apiserver-shop-control-plane            1/1     Running
kube-system          kube-controller-manager-shop-control-plane   1/1     Running
kube-system          kube-proxy-5vt8n                             1/1     Running
kube-system          kube-proxy-hc2q7                             1/1     Running
kube-system          kube-proxy-x9rlw                             1/1     Running
kube-system          kube-scheduler-shop-control-plane            1/1     Running
local-path-storage   local-path-provisioner-57c5987fd4-ktb8r      1/1     Running

A pod is Kubernetes' smallest unit: one or more containers that always run together on the same machine. For now, read "pod" as "a container that Kubernetes is looking after". -A means all namespaces. A Kubernetes namespace is a named folder for objects, used to keep teams and environments apart. This is a different thing from the Linux namespaces of Chapter 14.1.3, despite the shared word.

Notice what is already running: Kubernetes runs its own control plane as pods. Read the list as a map of the next four chapters:

Pod	What it is	Chapter
kube-apiserver	The one front door	14.3.3
etcd	The cluster's database	14.3.3
kube-scheduler	Picks a node for each pod	14.3.3
kube-controller-manager	The repair loops	14.3.3
kube-proxy (one per node)	Makes Services work	14.3.6
kindnet (one per node)	Pod networking	14.3.6
coredns (two copies)	Names inside the cluster	14.3.6
local-path-provisioner	Disks for pods	14.3.7

The four control-plane pods all have the node's name on the end, because they are pinned to that machine. The three that appear once per node are pinned one to each machine.

4. Something to deploy: a small TypeScript service

We need a service that does three things a real one does: answers requests, reports whether it is alive and whether it is ready, and shuts down cleanly when asked. It also reports which pod answered, so you can see load balancing happen.

// server.ts
import { createServer } from "node:http";
import { hostname } from "node:os";

const VERSION = process.env.APP_VERSION ?? "dev";                  // (1)
let ready = false;
let shuttingDown = false;

const server = createServer((req, res) => {
  if (req.url === "/healthz") {                                    // (2)
    res.writeHead(200).end("ok\n");
    return;
  }
  if (req.url === "/readyz") {                                     // (3)
    res.writeHead(ready && !shuttingDown ? 200 : 503).end();
    return;
  }
  res.writeHead(200, { "content-type": "application/json" });
  res.end(JSON.stringify({ service: "checkout-api", version: VERSION, pod: hostname() }) + "\n"); // (4)
});

server.listen(8080, () => {
  console.log(`checkout-api ${VERSION} listening on 8080`);
  setTimeout(() => {                                               // (5)
    ready = true;
    console.log("ready");
  }, 2000);
});

process.on("SIGTERM", () => {                                      // (6)
  shuttingDown = true;
  console.log("SIGTERM received, draining");
  server.close(() => {
    console.log("closed, exiting");
    process.exit(0);
  });
});

(1) The version comes from an environment variable, which the image will set. That lets us build two versions of the same code and tell their answers apart during an update.

(2) The liveness endpoint answers "is this process still working at all?" It always says ok while the program runs. If it stops answering, the process is stuck, and Kubernetes will restart the container (Chapter 14.3.5).

(3) The readiness endpoint answers "should traffic be sent here right now?" It returns 503 Service Unavailable until the service has finished starting, and again once shutdown has begun. These two questions are different, and mixing them up is one of the most common Kubernetes mistakes: a service that is alive but still warming up should get no traffic, but it should not be killed either.

(4) The normal reply names the service, the version and hostname(). Inside Kubernetes, a container's hostname is its pod's name (the UTS namespace from Chapter 14.1.3 at work), so every reply says which copy answered.

(5) A pretend two-second warm-up, standing in for a real service connecting to its database or filling a cache. Run on your laptop with node server.ts (Node 22.18 or later runs TypeScript directly), it behaves like this:

checkout-api dev listening on 8080
                                         ← curl /readyz now: 503
ready
                                         ← curl /readyz now: 200

(6) Clean shutdown. Kubernetes sends SIGTERM before stopping a container. The handler flips readiness to "no" so traffic stops arriving, lets in-flight requests finish, and then exits. Without it, as Chapter 14.2.3 explained, a process running as PID 1 ignores SIGTERM, and gets killed after the grace period with requests still in flight. The output when stopped is SIGTERM received, draining then closed, exiting.

Package it, with a Dockerfile in the same folder:

FROM node:24-slim
ARG VERSION=dev
ENV APP_VERSION=$VERSION
WORKDIR /app
COPY server.ts .
USER node
EXPOSE 8080
CMD ["node", "server.ts"]

The ARG and ENV pair bakes the version into the image at build time. USER node runs the process as the unprivileged user the Node image provides, rather than as root. The exec form of CMD makes node PID 1 directly, so it receives SIGTERM (Chapter 14.2.3).

docker build --build-arg VERSION=1.0 -t checkout-api:1.0 .
kind load docker-image checkout-api:1.0 --name shop          # (1)

(1) Copies the image into every node. This step surprises everyone the first time. The nodes have their own container runtime, separate from your Docker's image store, so an image you just built is invisible to them. A real cluster would pull from a registry (Chapter 14.2.2); kind load skips the registry by copying the image straight in. If the command fails on your Docker setup, the fallback always works: docker save checkout-api:1.0 -o api.tar, then kind load image-archive api.tar --name shop.

5. Describe what you want: your first manifest

In Kubernetes you do not say "start three containers". You write down "there should be three containers like this", hand that to the cluster, and the cluster makes it true and keeps it true. That written-down description is a manifest, a YAML file. Here is the whole of ours: two objects in one file, separated by ---.

# app.yaml
apiVersion: apps/v1                       # (1)
kind: Deployment
metadata:
  name: checkout-api                      # (2)
  labels:
    app: checkout-api
spec:
  replicas: 3                             # (3)
  selector:
    matchLabels:
      app: checkout-api                   # (4)
  template:                               # (5)
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: api
          image: checkout-api:1.0         # (6)
          ports:
            - containerPort: 8080
          readinessProbe:                 # (7)
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 2
          livenessProbe:                  # (8)
            httpGet:
              path: /healthz
              port: 8080
          resources:                      # (9)
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 128Mi
---
apiVersion: v1
kind: Service                             # (10)
metadata:
  name: checkout-api
spec:
  selector:
    app: checkout-api                     # (11)
  ports:
    - port: 80                            # (12)
      targetPort: 8080

Every Kubernetes object has the same four top-level parts: apiVersion and kind say what type of object this is, metadata names it, and spec is what you want. The cluster adds a fifth part, status, which is what it actually sees. The whole of Kubernetes is controllers working to make status match spec (Chapter 14.3.1's one idea).

(1) The type. A Deployment is the object for "keep N identical copies of this running, and replace them gradually when I change it". apps/v1 says which part of the API it belongs to, and which version of it.

(2) Its name, unique within its namespace. We did not give a namespace, so it goes into the one called default.

(3) The desired count. Three copies.

(4) How the Deployment finds its pods: by label. A label is a key and value stuck on an object, here app: checkout-api. The selector says "the pods I am responsible for are the ones with this label". Kubernetes never connects objects by remembering a list of names; it always matches labels. This one idea explains a great deal of Kubernetes, and line 11 uses it again.

(5) The template is the pattern for each pod. Every copy is made from it, and it must carry the label the selector looks for, or the Deployment would create pods it cannot find and keep creating more.

(6) The image. Because the tag is not latest, Kubernetes uses a copy already on the node if there is one, which is what kind load gave it. With the tag latest, Kubernetes always tries to pull from a registry, fails to find our local image, and the pod shows ErrImagePull.

(7) The readiness probe. The kubelet calls /readyz every two seconds. Until it gets a 200, the pod is not given any traffic.

(8) The liveness probe. If /healthz fails several times in a row (three by default), the kubelet restarts the container.

(9) Resources. requests is what the scheduler reserves on a node for this pod: one twentieth of a processor (50m means 50 thousandths) and 64 MB of memory. limits.memory is the cgroup ceiling from Chapter 14.1.4; go over 128 MB and the container is killed. Chapter 14.3.7 is entirely about getting these right.

(10) A Service is one stable name and address in front of a changing set of pods. Pods come and go and get new addresses every time. The Service does not.

(11) The Service finds its pods by the same label. Any pod with app: checkout-api that is passing its readiness probe receives traffic. Nothing connects the Service to the Deployment directly; they simply look for the same label.

(12) Callers use port 80 on the Service, and it forwards to port 8080 on the pods.

6. Hand it to the cluster and watch

kubectl apply -f app.yaml
kubectl get pods -o wide --watch

deployment.apps/checkout-api created
service/checkout-api created

NAME                            READY   STATUS              NODE
checkout-api-6d4b9f7c5d-2kq8x   0/1     Pending             <none>
checkout-api-6d4b9f7c5d-7wmnz   0/1     Pending             <none>
checkout-api-6d4b9f7c5d-p9v4r   0/1     Pending             <none>
checkout-api-6d4b9f7c5d-2kq8x   0/1     ContainerCreating   shop-worker
checkout-api-6d4b9f7c5d-7wmnz   0/1     ContainerCreating   shop-worker2
checkout-api-6d4b9f7c5d-p9v4r   0/1     ContainerCreating   shop-worker
checkout-api-6d4b9f7c5d-2kq8x   0/1     Running             shop-worker
checkout-api-6d4b9f7c5d-7wmnz   0/1     Running             shop-worker2
checkout-api-6d4b9f7c5d-p9v4r   0/1     Running             shop-worker
checkout-api-6d4b9f7c5d-2kq8x   1/1     Running             shop-worker
checkout-api-6d4b9f7c5d-7wmnz   1/1     Running             shop-worker2
checkout-api-6d4b9f7c5d-p9v4r   1/1     Running             shop-worker

Press Ctrl+C to stop watching. Read those lines as a story, because each status is a different component finishing its step:

• Pending with no node: the object exists in the database, and the scheduler has not yet chosen a machine for it.
• ContainerCreating on a node: the scheduler has chosen, and that node's kubelet is setting up the pod: its network, then its container.
• Running but 0/1 ready: the container has started, and the readiness probe is still getting 503 during our two-second warm-up.
• 1/1: the probe passed, and only now does the Service send it traffic.

The pod names tell you who created whom. checkout-api is the Deployment. 6d4b9f7c5d identifies a ReplicaSet, the object the Deployment created to hold exactly this version of the template. The last five characters make each pod unique. Chapter 14.3.5 explains why a Deployment works through ReplicaSets: it is how rolling updates and rollbacks work.

When something does not reach 1/1, these three commands find out why, and they are the start of every investigation in Chapter 14.3.9:

kubectl describe pod checkout-api-6d4b9f7c5d-2kq8x       # (1)
kubectl logs checkout-api-6d4b9f7c5d-2kq8x               # (2)
kubectl get events --sort-by=.lastTimestamp | tail      # (3)

(1) Everything the cluster knows about one pod, ending in an Events section that lists, in order, each step the cluster took and each problem it hit: Scheduled, Pulled, Created, Started, or Failed with a reason.

(2) The container's own output. Here you will see checkout-api 1.0 listening on 8080 and then ready.

(3) The same kind of events for the whole namespace, newest last.

7. Talk to it

From your laptop, the quickest way in is a port forward:

kubectl port-forward svc/checkout-api 8080:80

In a second terminal:

for i in 1 2 3; do curl -s localhost:8080; done

{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-7wmnz"}
{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-7wmnz"}
{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-7wmnz"}

The same pod three times, and that is not a bug. kubectl port-forward to a Service picks one pod behind it and builds a tunnel to that pod through the API server. It is a debugging tool, and it never load-balances. To see what other services in the cluster see, call the Service from inside the cluster:

kubectl run tmp --rm -it --restart=Never --image=curlimages/curl -- \
  sh -c 'for i in 1 2 3 4 5 6; do curl -s http://checkout-api; done'

{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-p9v4r"}
{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-2kq8x"}
{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-2kq8x"}
{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-7wmnz"}
{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-p9v4r"}
{"service":"checkout-api","version":"1.0","pod":"checkout-api-6d4b9f7c5d-7wmnz"}

kubectl run starts a one-off pod from the small curlimages/curl image, --rm -it attaches your terminal and deletes the pod afterwards, and --restart=Never makes it a plain pod rather than something Kubernetes would restart.

Three things happened in that command, each covered in Chapter 14.3.6. The name checkout-api was turned into the Service's address by CoreDNS, the cluster's own name server. Each curl opened a new connection to that address, and the rules kube-proxy wrote on the node picked one of the ready pods at random for each connection. All three pods answered, in no fixed order, which is load balancing per connection.

8. Break it, four ways

This is the part worth the half hour. For each one, predict what will happen before you run it.

Break 1: delete a pod

kubectl delete pod checkout-api-6d4b9f7c5d-2kq8x
kubectl get pods

pod "checkout-api-6d4b9f7c5d-2kq8x" deleted

NAME                            READY   STATUS    RESTARTS   AGE
checkout-api-6d4b9f7c5d-7wmnz   1/1     Running   0          6m
checkout-api-6d4b9f7c5d-p9v4r   1/1     Running   0          6m
checkout-api-6d4b9f7c5d-x4rtd   1/1     Running   0          8s

A new pod, with a new name, eight seconds old. You deleted a pod, but you did not change what you asked for: the ReplicaSet still says three. Its controller noticed it had two, and created one more. Deleting a pod in Kubernetes is not "stop this". It is "replace this", which is why kubectl delete pod is how people deliberately force a fresh copy, and why it never works as a way to turn something off. To turn it off you change the desired state, for example to zero replicas.

Break 2: lose a whole machine

kind's machines are containers, so you can pull the plug on one with Docker:

docker stop shop-worker2
kubectl get nodes --watch

NAME                 STATUS     ROLES           AGE
shop-control-plane   Ready      control-plane   12m
shop-worker          Ready      <none>          12m
shop-worker2         Ready      <none>          12m
shop-worker2         NotReady   <none>          12m

About forty seconds pass between the plug being pulled and NotReady. The node's kubelet stopped reporting in, and the node controller waited a grace period before deciding the silence was real, rather than a brief network blip (Chapter 14.3.3).

Then, for about five more minutes, apparently nothing happens. The pod that was on shop-worker2 still shows in the list. This is deliberate: by default, every pod carries a rule saying "stay bound to an unreachable node for 300 seconds before being evicted". Evicting instantly on every short network problem would cause far more harm than waiting. After those five minutes, the pod is marked for deletion, the ReplicaSet sees it is short, and a replacement appears on shop-worker. Meanwhile the Service had already stopped sending traffic to the lost pod, because a pod on a node that is not ready stops counting as ready. Callers saw two pods instead of three, not errors.

Bring the machine back with docker start shop-worker2. It reports in, returns to Ready, and is available for new pods. Kubernetes does not move the pods back, because nothing is wrong now: three pods are running, as asked.

Break 3: ask for more

kubectl scale deployment checkout-api --replicas=5
kubectl get pods

Two new pods appear in seconds. Scaling is just changing the desired count, and the same controller that replaced the deleted pod creates the extra ones. Scale back with --replicas=3, and two pods receive SIGTERM, their readiness flips to "no" while they drain, and they exit.

Break 4: ship a broken version

First, a good update. Build version 1.1 and roll it out:

docker build --build-arg VERSION=1.1 -t checkout-api:1.1 .
kind load docker-image checkout-api:1.1 --name shop
kubectl set image deployment/checkout-api api=checkout-api:1.1      # (1)
kubectl rollout status deployment/checkout-api                      # (2)

deployment.apps/checkout-api image updated
Waiting for deployment "checkout-api" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "checkout-api" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "checkout-api" rollout to finish: 1 old replicas are pending termination...
deployment "checkout-api" successfully rolled out

(1) Change the image of the container named api. That changes the pod template, and any change to the template starts a rollout.

(2) Waits and reports progress. By default the Deployment adds one new pod, waits until it is ready, removes one old pod, and repeats. That way the service never has fewer than three ready copies. Run the in-cluster curl loop from section 7 during a rollout and you will see 1.0 and 1.1 answers mixed together, then only 1.1.

Now a broken one. Version 9.9 does not exist anywhere:

kubectl set image deployment/checkout-api api=checkout-api:9.9
kubectl get pods

NAME                            READY   STATUS             RESTARTS   AGE
checkout-api-5f8c7d9b6-hxk2m    0/1     ImagePullBackOff   0          40s
checkout-api-7b9d4c8f5c-4nq7z   1/1     Running            0          3m
checkout-api-7b9d4c8f5c-8vlkp   1/1     Running            0          3m
checkout-api-7b9d4c8f5c-tc2wd   1/1     Running            0          3m

One new pod, stuck, and all three old pods still serving. The rollout created one new pod first, and that pod never became ready, because its image cannot be found. ImagePullBackOff means "pulling failed, and I am waiting longer and longer between retries" (Chapter 14.3.9). Since the new pod never became ready, the Deployment never removed an old one. A broken release with sensible defaults is a stalled rollout, not an outage. Left alone, kubectl rollout status gives up after ten minutes with exceeded its progress deadline.

Undo it:

kubectl rollout history deployment/checkout-api
kubectl rollout undo deployment/checkout-api

rollout undo switches the Deployment back to its previous template. The ReplicaSet for 1.1 still exists, scaled to three, so going back is instant and needs no new image pull. That is the whole reason old ReplicaSets are kept around.

9. The habit that matters most: change the file, not the cluster

Look at what we just did. We changed the image twice with kubectl set image. The cluster now runs 1.1, and app.yaml on your disk still says 1.0. If someone runs kubectl apply -f app.yaml tomorrow, they will roll back to 1.0 without meaning to.

kubectl diff -f app.yaml

-        image: checkout-api:1.1
+        image: checkout-api:1.0

kubectl diff shows what apply would change, and here it shows the cluster and the file disagreeing. Commands that change the live cluster directly, such as scale, set image and edit, are called imperative. They are fine on a laptop and for emergencies. Changing the file and applying it is declarative, and it is how real teams work, because then the file in Git is always the truth. Chapter 14.3.8 takes this to its end point, GitOps, where a program in the cluster applies whatever is in Git and nobody runs apply by hand at all.

Put the file right and apply it: change line (6) to checkout-api:1.1, then kubectl apply -f app.yaml, and kubectl diff now shows nothing.

10. Clean up

kind delete cluster --name shop

The three node containers are deleted, and the cluster's entry is removed from your kubeconfig. Nothing else on your machine was changed.

The commands from this page

Command	Does
kubectl get <kind>	List objects
-o wide / --watch	More columns / live updates
kubectl describe <kind> <name>	Details and events
kubectl logs <pod>	Container output
kubectl apply -f <file>	Make the cluster match the file
kubectl diff -f <file>	Preview an apply
kubectl delete <kind> <name>	Remove an object
kubectl port-forward svc/<name> L:R	Tunnel to one pod
kubectl run tmp --rm -it …	One-off debugging pod
kubectl scale deploy/<name> --replicas=N	Change the count
kubectl rollout status / history / undo	Watch, list, revert a rollout
kubectl config current-context	Which cluster am I on?

What the interviewer will push on

"What happens when you delete a pod that belongs to a Deployment?" It is replaced. The ReplicaSet's controller sees fewer pods than desired and creates a new one with a new name, usually on whichever node the scheduler picks. What the interviewer is checking is whether you think in desired state: the question behind the question is "how would you actually stop it?", and the answer is to change the replica count or delete the Deployment.

"What is the difference between a liveness probe and a readiness probe?" Liveness failing restarts the container; readiness failing removes the pod from the Service's traffic without restarting it. The tell is the example: a service that is warming up its cache should fail readiness and pass liveness. The classic wrong answer is pointing both probes at the same endpoint that checks the database, which turns a database outage into every pod restarting at once.

"How does a Service know which pods to send traffic to?" By label selector, and only to pods that are passing readiness. Nothing links the Service to the Deployment directly. Mention that a typo in a label is the most common reason a Service has no endpoints.

"A rollout pushed a bad image. What happens, and what do you do?" With default settings, one new pod is created, fails to pull, never becomes ready, and the old pods keep serving, so the rollout stalls rather than causing an outage. kubectl rollout undo goes back instantly because the previous ReplicaSet still exists. Volunteering why the old pods were never removed, that a new one must be ready first, is what shows you understand the mechanism.

"Why did kubectl port-forward always hit the same pod?" Because it tunnels to one pod that it picks once. It is a debugging tool, not a load balancer.

One thing to volunteer: mention the gap that imperative commands open between the live cluster and the files in Git, and that kubectl diff is how you see it. Teams that understand this move to GitOps; teams that do not eventually roll something back by accident.

Recall

• kubectl is the client for every cluster; kind runs each node as a Docker container (for learning and testing, not production). minikube, k3d and Docker Desktop's Kubernetes do the same job.
• kubeconfig = clusters + users + contexts; every command goes to the current context. Know it before you run delete.
• The control plane runs as pods in kube-system: apiserver, etcd, scheduler, controller-manager; plus kube-proxy and the network plugin on every node, and CoreDNS.
• kind load docker-image copies an image into the nodes, because their runtime is separate from your Docker. Tag latest makes Kubernetes always pull, and so fail on local images.
• Every object: apiVersion, kind, metadata, spec (what you want) and status (what is). Controllers make status match spec.
• Deployment → ReplicaSet → pods, all found by label selectors. A Service is a stable name in front of the pods its selector matches that are ready.
• Readiness gates traffic; liveness restarts. Statuses in order: Pending (unscheduled) → ContainerCreating → Running 0/1 (not ready) → 1/1.
• port-forward hits one pod; in-cluster calls are spread per connection by kube-proxy.
• Delete a pod and it comes back. A lost node goes NotReady after about 40 s, and its pods are evicted after a further 300 s by default.
• A bad image stalls a rollout (ImagePullBackOff, old pods keep serving); rollout undo is instant because the old ReplicaSet is kept.
• Imperative commands drift the cluster from Git; kubectl diff shows it; declarative apply from files is the habit.

Self-test: Why does a new pod appear when you delete one? · What would happen if the template's labels did not match the selector? · Why did port-forward show one pod but the in-cluster loop show three? · Why did the broken rollout not take the service down? · What does kubectl diff show after a kubectl set image?

Next: 14.3.3 opens the control plane you just used. When kubectl apply returned in a fraction of a second, nothing had been deployed yet. What had actually happened, and who did the rest?