14.3.9 — Troubleshooting Field Guide

It is 10:40 on a Tuesday, a release went out at 10:30, and someone posts in the team channel: "checkout is down". You run one command:

kubectl get pods -n shop

NAME                            READY   STATUS             RESTARTS      AGE
checkout-api-7b9d4c8f5c-4nq7z   0/1     CrashLoopBackOff   6 (82s ago)   9m
checkout-api-7b9d4c8f5c-8vlkp   0/1     CrashLoopBackOff   6 (75s ago)   9m
checkout-api-7b9d4c8f5c-tc2wd   0/1     CrashLoopBackOff   6 (88s ago)   9m

That one word in the STATUS column has already told you which part of the system is failing, and therefore which three commands to run next. This page is about reading that column, and the handful of other places Kubernetes writes down what went wrong, so that "checkout is down" becomes a cause in two minutes instead of an hour.

Chapter 14.3.8 gave the short version as a table. This page works through every common failure the same way:

• what you see, word for word,
• what it means, in terms of which component gave up and why,
• how to confirm it,
• the usual causes, most likely first,
• the fix.

Most of them come with a broken manifest you can apply to the kind cluster from Chapter 14.3.2 to watch the failure happen on purpose. Seeing each one once, safely, is the fastest way to recognise it at 10:40 on a Tuesday.

1. The method: find which step stopped

A pod's life is a sequence of steps, each done by a different component (Chapters 14.3.3 and 14.3.4). The scheduler places it. The kubelet on that node sets it up: pulls the image, prepares its configuration, creates the container. The container starts and has to stay running. The readiness probe has to pass. Only then does the Service send it traffic. Every failure on this page is one of those steps not finishing, and the status tells you which one.

The figure below is the map for the rest of the page. Start at the top with the status you see, and follow it down to the section that covers it.

Figure 1 — Which step stopped? Each status in the middle row means the step above it did not finish. Name the step, and you know which component to question: the scheduler, the kubelet, or your own program.

The same five commands answer nearly everything, and the order matters:

kubectl get pods -o wide                         # (1)
kubectl describe pod <name>                      # (2)
kubectl logs <name> --previous                   # (3)
kubectl get events --sort-by=.lastTimestamp      # (4)
kubectl get pod <name> -o yaml                   # (5)

(1) Status, restarts, and which node. The node column matters more than it looks: if every failing pod is on the same node, the problem is the node, not your application.

(2) The Events list at the bottom is the answer more often than anything else. It is the cluster's diary for this pod: scheduled where, pulled what, failed why. Read it before logs.

(3) The logs of the container that died, not the one that just restarted. Without --previous, you see a fresh start with no error in it.

(4) Everything that happened recently in this namespace, newest last. Events are kept for about an hour by default, so look early.

(5) The full object, including its status. This is where exact exit codes, last termination reasons and probe results live.

One more habit before the cases: count the pods. If no pod exists at all for a Deployment, the pod was never created, so no pod-level command will help. Look one level up, at the ReplicaSet: kubectl describe rs -l app=<name>. The two usual reasons are a namespace quota (section 2) and an admission policy refusing the pod (Chapter 14.3.8).

To follow along, with the kind cluster and checkout-api:1.0 image from Chapter 14.3.2 still in place, save the broken manifests shown in each section into one file, broken.yaml, and apply it with kubectl apply -f broken.yaml. Delete everything afterwards with kubectl delete -f broken.yaml.

2. Pending: no machine will take it

What you see. The pod is Pending and the NODE column says <none>. In describe:

Events:
  Type     Reason            From               Message
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated
  taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient cpu. preemption: 0/3 nodes are
  available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

What it means. The scheduler (Chapter 14.3.3) tried every node, and each one was ruled out for some reason. The message is a tally of those reasons, and learning to read it is the whole skill here. Read the example: three nodes; one was ruled out because of a taint, a "keep off" marker that only pods with a matching toleration may ignore, and that is the control-plane node; the other two were ruled out because they did not have enough processor left. The part after preemption: says the scheduler also checked whether evicting lower-priority pods would make room, and it would not.

Reproduce it:

apiVersion: v1
kind: Pod
metadata:
  name: too-big
spec:
  containers:
    - name: api
      image: checkout-api:1.0
      resources:
        requests:
          cpu: "64"

A request for 64 processors on a laptop cluster. Note that the scheduler only looks at requests, never at actual use (Chapter 14.3.7). A node with idle processors but already fully requested is full as far as the scheduler is concerned.

The usual causes, most likely first:

Message says	Cause	Fix
Insufficient cpu / memory	Requests too big, or cluster full	Lower requests, or add nodes
untolerated taint	Node reserved for others	Add toleration, or use other nodes
didn't match Pod's node affinity/selector	Label rule matches no node	Fix the selector or the node labels
unbound immediate PersistentVolumeClaims	Disk not provisioned	Check the PVC (Chapter 14.3.7)
Too many pods	Node's pod count limit	Add nodes

Two subtleties worth knowing. First, a cluster autoscaler (Chapter 14.3.7) reacts to exactly this state: a pod Pending for lack of resources is its signal to add a node, so in a cloud cluster, Pending for a minute or two may be normal and resolve itself. Second, a namespace quota does not produce a Pending pod at all. It stops the pod being created, so you see fewer pods than replicas with no error on any pod. The error is on the ReplicaSet: kubectl describe rs shows exceeded quota.

3. ErrImagePull and ImagePullBackOff: the image cannot be fetched

What you see. ErrImagePull on the first failure, then ImagePullBackOff, which means "it failed, and I am now waiting longer and longer between attempts", up to five minutes apart. In describe:

Warning  Failed  kubelet  Failed to pull image "checkout-api:does-not-exist": failed to pull and
unpack image "docker.io/library/checkout-api:does-not-exist": failed to resolve reference
"docker.io/library/checkout-api:does-not-exist": pull access denied, repository does not exist
or may require authorization

What it means. The node's container runtime asked a registry for the image and got a refusal. The important detail is in the error text, so read it to the end. Notice how the short name checkout-api was expanded to docker.io/library/checkout-api: an image name with no registry in it means Docker Hub. Also notice that Docker Hub answers "access denied" for a repository that does not exist at all, so that it does not reveal which private names exist. "Access denied" can mean "you typed it wrong".

Reproduce it: a pod with image: checkout-api:does-not-exist.

The causes, by what the message ends with:

Message ends with	Cause	Fix
not found / manifest unknown	Wrong tag	Check the tag exists
access denied / unauthorized	Typo, or private and no credentials	Fix name; add pull credentials
toomanyrequests	Registry rate limit	Authenticate; use a pull-through cache
no match for platform in manifest	No variant for this CPU	Build multi-platform (Chapter 14.2.6)
i/o timeout / no such host	Node cannot reach registry	Check outbound network and DNS

The fix for credentials is an imagePullSecrets entry on the pod, or on its service account, that points at a Secret of type kubernetes.io/dockerconfigjson. On AKS with Azure Container Registry, the cleaner answer is to let the cluster's own identity pull, with no secret at all (Chapter 14.4.1). The rate-limit case is the one that surprises teams, because it appears suddenly in the middle of a large deploy (Chapter 14.2.2 gives the numbers).

Check a tag exists without pulling it: docker manifest inspect registry.example.com/shop/api:1.4 prints the manifest if it exists and errors if not.

4. CreateContainerConfigError: something the pod refers to is missing

What you see.

NAME             READY   STATUS                       RESTARTS   AGE
missing-secret   0/1     CreateContainerConfigError   0          20s

Warning  Failed  kubelet  Error: secret "db-credentials" not found

What it means. The image is fine, but before creating the container, the kubelet has to assemble its configuration: environment variables, mounted files, the user to run as. Something that configuration refers to does not exist. The container is never created, so there are no logs to read. The answer is only in the events.

Reproduce it:

apiVersion: v1
kind: Pod
metadata:
  name: missing-secret
spec:
  containers:
    - name: api
      image: checkout-api:1.0
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

The usual causes:

• A Secret or ConfigMap that does not exist in this namespace. Objects are looked up in the pod's own namespace only. The Secret existing in default does not help a pod in shop.
• The object exists, but the key does not: couldn't find key password in Secret shop/db-credentials. Often a typo, or a key named DB_PASSWORD in one file and password in another.
• runAsNonRoot: true with an image whose default user is root: container has runAsNonRoot and image will run as root. Set runAsUser to a numeric non-root ID, or fix the image's USER line (Chapter 14.2.3).

The fix is to create the missing object, or correct the name. The kubelet retries on its own and the pod starts within seconds, with no restart needed. If a missing value is genuinely optional, mark the reference optional: true and the container starts without it.

5. CrashLoopBackOff: it starts, then dies, again and again

What you see. RESTARTS climbing, and the status alternating between Running, Error and CrashLoopBackOff. Like image pulls, restarts back off: about 10 seconds, then 20, 40 and so on, up to 5 minutes between attempts. A container that then runs for 10 minutes without crashing has its backoff reset.

What it means. Kubernetes did its part. It scheduled the pod, pulled the image, created the container and started it. The program then exited. CrashLoopBackOff is not an error of its own; it is the kubelet saying "I keep restarting this, and it keeps exiting". The cause is almost always in the program's own output, from the run that died.

Reproduce it:

apiVersion: v1
kind: Pod
metadata:
  name: crasher
spec:
  containers:
    - name: api
      image: checkout-api:1.0
      command: ["node", "-e", "console.error('DATABASE_URL is not set'); process.exit(1)"]

Confirm it in two commands:

kubectl logs crasher --previous
kubectl get pod crasher -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

DATABASE_URL is not set

{"exitCode":1,"reason":"Error","startedAt":"2026-10-18T10:41:02Z","finishedAt":"2026-10-18T10:41:02Z"}

The exit code is a strong clue on its own, and this table is worth knowing by heart:

Exit code	Meaning	Usual cause
1	The program chose to fail	Read its log
126	Found, not executable	Missing chmod +x
127	Command not found	Wrong command, or a CRLF script (Chapter 14.2.6)
137	Killed by SIGKILL	Memory limit (section 6) or liveness kill
139	Crashed reading bad memory	A native library bug
143	Stopped by SIGTERM	A normal shutdown

Exit codes above 128 are "killed by a signal", and the signal number is the code minus 128: 137 is 128 + 9, which is SIGKILL, and 143 is 128 + 15, which is SIGTERM.

The usual causes, most likely first:

1. Missing or wrong configuration. An environment variable, a config file, a connection string. The log says so plainly, as in the example.
2. A dependency is not reachable at startup, and the program exits instead of retrying. The database is still starting, or its Service name is wrong. Programs that will run in Kubernetes should retry their connections at startup rather than exiting, because start order is never guaranteed.
3. The liveness probe is killing a slow starter. The program needs 40 seconds to warm up, the liveness probe starts checking after a few seconds and restarts it before it is ever ready. describe shows Liveness probe failed followed by Killing. The fix is a startupProbe, which holds off the liveness probe until the program has started once (Chapter 14.3.5).
4. The wrong program for this CPU. The log has a single line: exec format error. The image was built for ARM and the node is x86, or the other way round (Chapter 14.2.6).
5. Permissions. The image runs as a non-root user and the program tries to write somewhere it may not, such as a read-only root filesystem. The log says EACCES or permission denied.

One trap: a container that keeps exiting with code 0. A program that finishes its work and exits successfully, inside a Deployment, gets restarted forever, because a Deployment's job is to keep it running. One-off work belongs in a Job (Chapter 14.3.5).

6. OOMKilled: over the memory limit

What you see. Often it looks like CrashLoopBackOff at first glance. describe gives the real reason:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

What it means. The container's cgroup reached its memory.max (Chapter 14.1.4), and the kernel killed the process at once. There is no warning and no graceful shutdown, because the kernel cannot politely refuse an allocation that is already happening. In-flight requests are lost.

Reproduce it:

apiVersion: v1
kind: Pod
metadata:
  name: memory-hog
spec:
  containers:
    - name: api
      image: checkout-api:1.0
      command: ["node", "-e", "const a = []; setInterval(() => a.push(Buffer.alloc(10 * 1024 * 1024, 1)), 100)"]
      resources:
        limits:
          memory: 64Mi

The program allocates 10 MB every tenth of a second and never frees it. Node itself starts at about 40 MB, so the limit is hit in well under a second.

Three different "out of memory" deaths, which need different fixes:

• The container hit its own limit. OOMKilled, exit 137, as above. Either the limit is too low for the real workload, or the program leaks.
• The whole node ran out of memory, and the node's kernel picked a victim. The pod may be killed even though it was under its own limit. This happens when pods without limits, or with limits far above their requests, crowd a node. kubectl describe node shows a SystemOOM event.
• The kubelet evicted the pod before the kernel had to act. The status is Evicted, not OOMKilled, and section 11 covers it.

The fix depends on which. For the first: look at the memory graph before the kill. A steady climb is a leak; a flat line just under the limit is a limit set too low. Also set the language runtime's own heap ceiling below the container's limit, with --max-old-space-size for Node or -XX:MaxRAMPercentage for Java, so the runtime collects garbage before the kernel kills it.

7. Running but 0/1: the readiness probe is failing

What you see. The container runs, restarts stay at zero, and READY says 0/1. Callers get no response from this pod, because the Service is not sending it any traffic.

Warning  Unhealthy  kubelet  Readiness probe failed: Get "http://10.244.1.5:8080/": dial tcp
10.244.1.5:8080: connect: connection refused

What it means. Kubernetes is doing exactly what you asked: holding back traffic until the probe passes. The probe's error message tells you which of three things is wrong:

Probe says	Meaning
connection refused	Nothing listening on that address and port
context deadline exceeded / timeout	Listening, but answering too slowly
statuscode: 503 (or any non-2xx)	Answering, and saying "not ready"

The connection refused case has one cause that catches almost everyone once. Reproduce it:

apiVersion: v1
kind: Pod
metadata:
  name: localhost-only
  labels:
    app: localhost-only
spec:
  containers:
    - name: api
      image: checkout-api:1.0
      command: ["node", "-e", "require('node:http').createServer((q, s) => s.end('ok')).listen(8080, '127.0.0.1')"]
      readinessProbe:
        httpGet:
          path: /
          port: 8080

The program listens on 127.0.0.1, which means "only accept connections from inside this pod". The kubelet's probe comes from the node, to the pod's own address, 10.244.1.5, and nothing is listening on that address. The program works perfectly when you test it with curl localhost:8080 from inside the pod, which is what makes this so confusing. The fix is to listen on 0.0.0.0, meaning every address the pod has. Many frameworks default to 127.0.0.1 in development mode for safety, so check the framework's host setting.

The other causes:

• Wrong port in the probe. The probe checks 8080 and the program listens on 3000.
• The readiness check calls a dependency that is down. Then every pod fails readiness together and the whole service drops out, even though each pod could have served some requests. Readiness should usually check only this pod's own state.
• Timeouts too tight. The default probe timeout is 1 second. A probe that does real work under load can exceed that.

8. The Service has no endpoints

What you see. Pods are Running and 1/1, and calls to the Service still fail: connection refused, or a timeout, or 503 from an ingress or gateway.

kubectl describe service orders

Name:              orders
Selector:          app=order-api
Type:              ClusterIP
IP:                10.96.141.27
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         <none>

What it means. Endpoints: <none> says the Service matched no ready pods. A Service finds pods only by label (Chapter 14.3.2), so either no pod has the labels the selector asks for, or the ones that do are not ready.

Reproduce it: apply the orders Service below. It selects app: order-api, and no pod in the cluster has that label.

apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector:
    app: order-api
  ports:
    - port: 80
      targetPort: 8080

Confirm it by putting the two side by side:

kubectl get service orders -o jsonpath='{.spec.selector}'; echo
kubectl get pods --show-labels

{"app":"order-api"}

NAME                            READY   STATUS    LABELS
order-service-5c9f8b7d6-2xkqp   1/1     Running   app=order-service,pod-template-hash=5c9f8b7d6

order-api against order-service. One word, and the Service has nothing behind it.

The usual causes:

1. Label and selector do not match: a typo, a rename on one side only, or the Service in a different namespace from the pods.
2. The pods match but none is ready, which is section 7.
3. Endpoints exist but targetPort is wrong. Then describe does list addresses, and connections to them are refused. Check that targetPort is the port the program actually listens on.

9. DNS: the name does not resolve

What you see. In the application's log, getaddrinfo ENOTFOUND orders, could not resolve host, or Name or service not known.

What it means. The pod asked the cluster's name server, CoreDNS (Chapter 14.3.6), for an address, and got none. Test from a pod in the same namespace, because that is the only test that matches what the application sees:

kubectl run dnstest -n shop --rm -it --restart=Never --image=busybox:1.37 -- nslookup orders

Server:    10.96.0.10
Address:   10.96.0.10:53

Name:      orders.shop.svc.cluster.local
Address:   10.96.141.27

Every Service gets the name <service>.<namespace>.svc.cluster.local. A pod can use just orders only because its own namespace is added automatically. The pod's /etc/resolv.conf lists search suffixes such as shop.svc.cluster.local, and the resolver tries each one in turn. So the usual causes are:

• The Service is in another namespace. From shop, the name orders finds orders in shop only. To reach orders in billing, use orders.billing.
• The Service does not exist, or has a different name. kubectl get svc -A | grep orders.
• A network policy blocks DNS. This is the classic one after a team applies a "deny all outbound traffic" policy for security: DNS is outbound traffic too, to port 53 on CoreDNS, and every name lookup in the namespace starts failing. The fix is a policy that allows outbound traffic to CoreDNS on port 53 over both UDP and TCP (Chapter 14.3.6).
• CoreDNS itself is unhealthy. kubectl get pods -n kube-system -l k8s-app=kube-dns. Rare, and it affects the whole cluster at once, so you would know.

A performance note that looks like a DNS problem. The pod's resolver setting called ndots:5 means any name with fewer than five dots is first tried with every search suffix. So a lookup of api.payments.example.com first tries api.payments.example.com.shop.svc.cluster.local and two more invented names before trying the real one. Under heavy load, that multiplies DNS traffic several times over. Writing external names with a trailing dot, api.payments.example.com., skips the search list.

10. Node NotReady

What you see. In kubectl get nodes, one node says NotReady, and the pods on it stop being ready.

What it means. The node's kubelet has stopped reporting in, or is reporting that something essential is broken. After about 40 seconds of silence the node controller marks it NotReady, and by default its pods are evicted 300 seconds after that (Chapters 14.3.2 and 14.3.3).

kubectl describe node shop-worker2

Read the Conditions table first:

Condition	Healthy value	When it is wrong
Ready	True	False or Unknown
MemoryPressure	False	Kubelet is evicting for memory
DiskPressure	False	Disk or image space low
PIDPressure	False	Too many processes

Unknown and False mean different things. Unknown, with the message Kubelet stopped posting node status, means the control plane has heard nothing: the machine is off, the kubelet has crashed, or the network between them is down. False means the kubelet is talking and reporting a problem. The most common one on a fresh node is container runtime network not ready ... cni plugin not initialized, meaning the pod-network plugin (Chapter 14.3.4) has not started on that node.

What to do. In a cloud, check the VM itself in the provider's console. If you can reach the machine, systemctl status kubelet and journalctl -u kubelet show why the kubelet stopped. On a managed cluster, the usual right answer is to replace the node rather than repair it, since node pools exist so that individual machines do not matter (Chapter 14.4.1).

11. Evicted: the node pushed the pod out

What you see. Pods with status Evicted, and in describe:

Status:   Failed
Reason:   Evicted
Message:  The node was low on resource: memory. Threshold quantity: 100Mi, available: 87232Ki.

What it means. The kubelet watches the node's free memory and disk. When either drops below a threshold, it removes pods to protect the node, before the kernel is forced to start killing processes at random. The Deployment then creates replacements, usually on other nodes.

Who goes first is decided by the pod's quality-of-service class (Chapter 14.3.7):

1. BestEffort: pods with no requests or limits at all. Evicted first.
2. Burstable: pods using more than they requested, worst offenders first.
3. Guaranteed: requests equal to limits. Evicted last.

So the real fix is nearly always "set requests that match real use", which both stops the node being overfilled in the first place and protects your pod when something else misbehaves.

Disk pressure has its own common causes: container logs, files written to a pod's emptyDir scratch volume, and old images. A pod writing gigabytes of log output can get itself evicted for using too much of the node's disk.

Evicted pods stay listed as records of what happened, and do not run. Clean them up with kubectl delete pods --field-selector=status.phase=Failed -n shop.

12. Terminating forever

What you see. You deleted a pod, or a namespace, and minutes later it still says Terminating.

What it means. Deletion in Kubernetes has two halves. First the object is marked for deletion. Then everything that needs to finish its cleanup does so, and only when the last one is done is the object removed. Two things hold it in between:

• The node the pod was on is gone or unreachable. The kubelet is the one that confirms a pod's containers have stopped, and an unreachable kubelet cannot confirm anything, so the API server waits.
• A finalizer: a name listed in the object's metadata.finalizers, which means "some controller still has cleanup to do before this goes". A storage controller detaching a disk, or a cloud controller deleting a load balancer, use finalizers. If that controller is broken or was uninstalled, nothing ever removes the name.

kubectl get pod <name> -o jsonpath='{.metadata.finalizers}'

The fix, carefully. For a pod on a node that is truly gone, kubectl delete pod <name> --grace-period=0 --force removes the record. Read this before you do it: force-deleting does not stop anything that may still be running. It only removes Kubernetes' record of it. For a StatefulSet pod (Chapter 14.3.5), a replacement then starts elsewhere, and if the old one was in fact still running on a node that had only lost its network, two copies of a database are now writing to the same data. Confirm the node is really off first.

A namespace stuck in Terminating almost always contains an object whose finalizer's controller no longer exists. kubectl get all -n <ns> misses many kinds of object, so use kubectl api-resources --verbs=list --namespaced -o name | xargs -n1 kubectl get -n <ns> --ignore-not-found to find the leftover.

13. A rollout or a drain that will not finish

A rollout stuck halfway. kubectl rollout status waits, and eventually reports exceeded its progress deadline (10 minutes by default). kubectl get pods shows new pods failing in one of the ways above, while the old pods keep serving (Chapter 14.3.2 broke one on purpose). Find the failure on the new pods, then kubectl rollout undo to put the old version back while you fix it.

A node drain that will not finish. During an upgrade, kubectl drain keeps printing:

error when evicting pods/"checkout-api-7b9d4c8f5c-4nq7z" -n "shop" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.

This is a PodDisruptionBudget doing its job (Chapter 14.3.7): it refuses to let the service drop below its declared minimum. It becomes a deadlock when the budget can never be satisfied, for example a budget of minAvailable: 2 on a Deployment with only 2 replicas. Evicting any pod would leave 1, so no pod can ever be evicted. The fix is to scale up first, or set the budget to allow at least one pod down. Removing the budget to force the drain through is the move that turns an upgrade into an outage.

14. Forbidden: you, or a pod, may not do that

What you see.

Error from server (Forbidden): pods is forbidden: User "priya@example.com" cannot list resource
"pods" in API group "" in the namespace "shop"

What it means. Your identity was recognised, and RBAC (Chapter 14.3.8) has no rule allowing that action. The message names everything you need: who, which verb (list), which resource (pods), which API group ("" is the core group) and which namespace.

Test permissions without guessing:

kubectl auth can-i list pods -n shop
kubectl auth can-i list pods -n shop --as=system:serviceaccount:shop:checkout-api

The second form checks what a pod's service account may do, which is how you debug an application that calls the Kubernetes API and gets a 403. Grant the narrowest Role that covers the verb and resource in the message, in that namespace. Granting cluster-admin to make the error go away is how clusters get compromised.

15. When none of that fits: look from inside

kubectl exec -it <pod> -- sh gives you a shell in a running container, if the image has one. Many production images do not, on purpose (Chapter 14.2.3).

kubectl debug solves that. It adds a temporary extra container, with all the tools you want, into the running pod:

kubectl debug -it checkout-api-7b9d4c8f5c-4nq7z --image=busybox:1.37 --target=api

--target=api puts the debug container into the same process namespace as the api container, so ps shows the application's processes, and its files can be reached under /proc/1/root. The application is not restarted and its image is not changed.

kubectl debug node/<node> -it --image=busybox:1.37 does the same for a whole node: it starts a pod on that node with the node's own filesystem mounted at /host, which is how you read node logs on a managed cluster where you have no SSH access.

What the interviewer will push on

"A pod is in CrashLoopBackOff. Walk me through what you do." kubectl describe pod for events and the last termination reason and exit code, then kubectl logs --previous for the output of the run that died. The tell is --previous, because without it you read a fresh start that has no error in it. Then name the usual causes in order: configuration, a dependency unreachable at startup, the liveness probe killing a slow starter, the wrong CPU architecture. The weak answer is "restart it".

"What is the difference between OOMKilled and Evicted?" OOMKilled is the kernel killing a container that reached its own cgroup limit, exit 137, instantly. Evicted is the kubelet removing a pod because the node is low on memory or disk, chosen by quality-of-service class. Interviewers ask this to check whether you know that requests and limits do different jobs: limits cause the first, missing requests make you the first victim of the second.

"Pods are running and ready, but the Service returns nothing. Where do you look?" kubectl describe service and its endpoints, then compare the selector with kubectl get pods --show-labels. Mention targetPort as the second cause when endpoints do exist.

"A pod is Pending. How do you read the scheduler's message?" As a tally: how many nodes, and how many were excluded for each reason. Add that the scheduler uses requests, not real usage, and that quotas stop pods being created at all, so a quota problem shows on the ReplicaSet, not on a pod.

"Would you force-delete a pod stuck in Terminating?" Only after confirming its node is really gone, and never casually for a StatefulSet, because force-deleting removes the record and not the process. This is the answer that shows judgement rather than commands.

One thing to volunteer: say that you start every investigation by asking which step of the pod's life stopped, scheduling, image, configuration, running or readiness, because the status names the step and the step names the component. It turns a list of error strings into a method, and it is how experienced operators find causes in minutes.

Recall

• Method: which step stopped? ① scheduled ② image ③ config ④ stays up ⑤ ready. Five commands: get -o wide · describe (read Events first) · logs --previous · get events --sort-by · get -o yaml. No pod at all → look at the ReplicaSet (quota, admission).
• Pending: read FailedScheduling as a tally. Causes: requests too big (scheduler uses requests, not use), taints, affinity, unbound PVC. Autoscaler may fix it.
• ImagePullBackOff: read the message's end: not found · access denied (often a typo) · toomanyrequests · no platform match · network. Short names mean Docker Hub.
• CreateContainerConfigError: missing Secret/ConfigMap/key in this namespace, or runAsNonRoot vs a root image. No logs exist; events only.
• CrashLoopBackOff: backoff 10 s doubling to 5 min. logs --previous + exit code. 137 = 128+9 SIGKILL, 143 = 128+15 SIGTERM, 127 not found, 126 not executable. Causes: config, dependency at startup, liveness on a slow starter (startupProbe), exec format error, permissions. Exit 0 in a Deployment = should be a Job.
• OOMKilled: container limit, exit 137, no graceful shutdown. Differs from node OOM and from eviction. Set the runtime's heap below the limit.
• 0/1 ready: probe says refused (nothing listening — 127.0.0.1 instead of 0.0.0.0), timeout, or non-2xx.
• No endpoints: selector vs --show-labels, pods not ready, or wrong targetPort.
• DNS: <svc>.<ns>.svc.cluster.local; cross-namespace needs svc.ns; deny-all egress blocks port 53; ndots:5 multiplies lookups.
• Node NotReady: Unknown = silence, False = reported problem (often CNI). Replace, don't repair. Evicted: BestEffort → Burstable → Guaranteed.
• Terminating forever: unreachable node or a finalizer. Force-delete removes the record, not the process.
• Drain blocked by a PDB that can never be met: scale up or loosen it. Forbidden: kubectl auth can-i … --as=. kubectl debug for images with no shell.

Self-test: Why is kubectl logs without --previous useless for a crash loop? · What does exit code 137 tell you, and what are its two causes? · Why does a program listening on 127.0.0.1 fail its readiness probe? · Why does a quota problem not show up on any pod? · What can go wrong when you force-delete a StatefulSet pod?

Next: 14.4.1 moves to a real managed cluster. What Azure actually runs for you in AKS, every component it adds on top of upstream Kubernetes, and which configuration choices you can never change after creation.