13.6.11 — Pods and the Workload Objects

Somebody deletes a pod to "restart it". Ninety seconds later there is a new pod with a new name, doing the same thing.

Somebody else scales a Deployment to zero, then back to three, and gets three new pods with three new names and three new IPs — and the database pods, which look identical in every dashboard, come back as db-0, db-1 and db-2, each reattached to the exact disk it had before.

Both behaviours are correct and they come from different objects. This page is the pod specification field by field, then the six objects that create pods for you, and the rules for choosing between them.

1. The pod

A pod is one or more containers that share a network namespace and can share volumes. Same IP, same port space, localhost between them.

Why not just a container? Because some things genuinely belong on one machine sharing one network stack: a log shipper reading a file the application writes, a proxy handling outbound connections, a process refreshing a credential file, a helper syncing configuration from somewhere. Anything that would otherwise need a shared local socket, a shared directory, or a loopback connection.

The pod is also the unit of scheduling. Containers in a pod are always on one node, always started together, always stopped together.

And a pod is disposable and never repaired. It gets a new IP each time, it is replaced rather than fixed, and nothing should ever address a pod directly — which is what Services are for (Chapter 13.6.12).

The specification, annotated

apiVersion: v1
kind: Pod
metadata:
  name: checkout-api
  labels: { app: checkout-api, tier: backend }     # (1)
  annotations:
    prometheus.io/scrape: "true"                    # (2)
spec:
  serviceAccountName: checkout                      # (3)
  automountServiceAccountToken: false               # (4)
  securityContext:                                  # (5)
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 2000                                   # (6)
    seccompProfile: { type: RuntimeDefault }
  terminationGracePeriodSeconds: 45                 # (7)
  initContainers:
    - name: migrate                                 # (8)
      image: registry.example.com/checkout@sha256:1a2b...
      command: ["node", "dist/migrate.js"]
  containers:
    - name: api
      image: registry.example.com/checkout@sha256:1a2b...   # (9)
      ports:
        - containerPort: 3000
          name: http                                # (10)
      env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef: { name: checkout-db, key: url }   # (11)
        - name: POD_IP
          valueFrom:
            fieldRef: { fieldPath: status.podIP }   # (12)
      envFrom:
        - configMapRef: { name: checkout-config }   # (13)
      resources:
        requests: { cpu: "100m", memory: "256Mi" }  # (14)
        limits:   { memory: "256Mi" }               # (15)
      volumeMounts:
        - { name: config, mountPath: /etc/app, readOnly: true }
        - { name: tmp, mountPath: /tmp }
      securityContext:
        allowPrivilegeEscalation: false             # (16)
        readOnlyRootFilesystem: true                # (17)
        capabilities: { drop: ["ALL"] }
      readinessProbe:
        httpGet: { path: /ready, port: http }
      livenessProbe:
        httpGet: { path: /healthz, port: http }
      lifecycle:
        preStop:
          exec: { command: ["sleep", "5"] }         # (18)
  volumes:
    - name: config
      configMap: { name: checkout-config }
    - name: tmp
      emptyDir: {}                                  # (19)

(1) Labels are the whole coordination mechanism. A Service selects on them, a Deployment owns pods through them, a network policy matches on them. Names are for humans; labels are for machines.

(2) Annotations are unstructured metadata for tools, not for selection. Ingress controllers, certificate managers and monitoring agents all read their configuration from here.

(3) The identity this pod presents to the API server and, with workload identity, to the cloud (Chapter 13.6.15).

(4) Do not mount the API token unless the pod calls the API. Mounted by default, and a compromised container with a mounted token holds a cluster credential.

(5) Pod-level security settings, inherited by every container.

(6) fsGroup sets group ownership on mounted volumes, which is how a non-root process gets write access to a cloud disk. Without it, a runAsNonRoot pod with a persistent volume fails with permission denied — a very common first-time failure.

(7) How long between SIGTERM and SIGKILL. Default 30 seconds. Raise it if your shutdown genuinely takes longer, and see section 7 for the sequence.

(8) Init containers run to completion, in order, before any application container starts. Migrations, waiting for a dependency, fetching configuration. If one fails, the pod restarts it according to the restart policy and the application never starts.

(9) Digest-pinned (Chapter 13.6.4), so the running thing is unambiguous.

(10) Name the port. Probes and Services can then refer to http instead of 3000, so changing the port is one edit.

(11) A value from a Secret.

(12) The downward API — pod metadata as environment variables. Node name, pod name, namespace, IP, and resource limits are all available, which is how applications tag their logs and metrics with where they are running.

(13) Every key in a ConfigMap as environment variables at once.

(14) and (15) The most consequential lines in the file, and Chapter 13.6.13 is devoted to them.

(16) Blocks gaining privileges through a setuid binary.

(17) Read-only root filesystem, with emptyDir volumes wherever the process genuinely writes.

(18) A preStop hook — see section 7 for why this sleep is not a hack.

(19) emptyDir is scratch space that lives and dies with the pod, shared by all its containers. With medium: Memory it is a tmpfs.

The three multi-container patterns

Containers in a pod share a network namespace and can share volumes, and those are exactly the two things these patterns exploit. All three have been in use since Google described them, and knowing the names matters because people use them as shorthand in design discussions.

① Sidecar — a helper that adds a capability to the main container.

The main container does its job; the sidecar does something alongside it. A log shipper reading the file the application writes. A metrics exporter. A process that refreshes a credential file every hour. A helper that pulls configuration from somewhere and drops it on the shared volume.

They communicate through a shared volume or through localhost, which is only possible because they are in one pod.

Service meshes work by injecting a sidecar into every pod through a mutating webhook (Chapter 13.6.9), which is why your pods suddenly have two containers after a mesh is installed and nobody edited a manifest.

② Ambassador — a helper that sits between your container and the outside world.

Your application connects to localhost and the ambassador handles what happens next. It is a proxy that your container treats as the real thing.

The clearest example is a database connection. Your application is written to connect to localhost:5432. The ambassador container listens there and forwards to the real database — handling connection pooling, retries, failover between replicas, or TLS to the actual server.

containers:
  - name: api
    env:
      - name: DATABASE_URL
        value: postgres://app@localhost:5432/checkout   # (1)
  - name: db-proxy                                       # (2)
    image: cloud-sql-proxy:2
    args: ["--port=5432", "shop-pg.postgres.database.azure.com"]

(1) The application believes the database is local. It has no connection pooling logic, no failover logic, no TLS configuration.

(2) The ambassador makes that true. It holds the pool, the retries and the encrypted connection to the real server.

Why this is worth a name: it lets you change how the outside world is reached without changing the application at all. The application's view stays localhost:5432 forever.

③ Adapter — a helper that changes your container's output into the shape something else expects.

The reverse direction of the ambassador. Your container produces something in its own format; the adapter translates it into the format the rest of the system requires.

A legacy application writes health information as plain text on a file. Your monitoring system expects Prometheus-format metrics on an HTTP endpoint. An adapter container reads the file from the shared volume and serves the translated version — and the legacy application is never modified.

The rule that connects all three: put a container in the pod when it must share the network stack or a filesystem with the main container and live and die with it. If it does not need either of those, it is a separate Deployment, and putting it in the pod just makes both harder to scale.

Init containers and native sidecars

Init containers run to completion, in order, before any application container starts. Migrations, waiting for a dependency to be reachable, fetching configuration. If one fails, the pod's restartPolicy decides what happens next and the application containers never start.

And the fix for a long-standing problem: native sidecars. Historically a sidecar was just another container in containers:, which broke Jobs — the pod could never complete, because the log shipper never exits. Since Kubernetes 1.29, an init container with restartPolicy: Always is a native sidecar: it starts before the application containers, keeps running alongside them, and is not counted when deciding whether the pod has finished.

initContainers:
  - name: log-shipper
    image: fluent-bit:3
    restartPolicy: Always      # makes this a sidecar, not an init container

This also fixes startup ordering, since a mesh proxy is up before the application starts making calls through it.

Ephemeral containers are the fourth kind, and they are added to a running pod rather than declared. You cannot put one in a manifest. kubectl debug injects one into a pod that is already running, sharing its namespaces, so you get a full toolbox next to a container that has no shell of its own (Chapter 13.6.14). They have no probes, no resources and no restart — they exist to be looked through and thrown away.

Pods that borrow the host's namespaces

Normally a pod gets its own namespaces (Chapter 13.6.2). Three settings hand it the host's instead, and you should be able to recognise them because they are powerful, occasionally necessary, and blocked by every serious security policy.

spec:
  hostNetwork: true      # (1)
  hostPID: true          # (2)
  hostIPC: true          # (3)

(1) The pod uses the node's network stack directly. Its localhost is the node's, and any port it binds is bound on the node with no Service involved. hostPort is the narrower version — publish one container port on the node without taking the whole stack.

(2) The pod sees every process on the node, not just its own.

(3) The pod shares the node's shared memory.

Who legitimately needs these: node-level agents in a DaemonSet. A monitoring agent needs hostPID to see the node's processes. A CNI plugin needs hostNetwork because it is configuring the node's networking. A log collector needs the node's filesystem.

Why everything else is refused them: hostNetwork removes network isolation and lets the pod bind node ports and reach anything the node can reach. hostPID lets it inspect and signal every process on the machine, including other tenants' containers. This is what Chapter 13.6.14's baseline Pod Security Standard means when it says it blocks host namespaces, and it is why a request for them in an application manifest deserves a conversation rather than an approval.

Pod phases, and what is not one

Every pod has a status.phase, and there are exactly five. This is the pod's own life, and it is smaller than the list of things kubectl get pods prints.

Phase	Meaning
Pending	Accepted, but not yet running — waiting to be scheduled, or pulling images, or waiting on init containers
Running	Bound to a node, all containers created, at least one running or starting
Succeeded	All containers exited 0 and will not be restarted — Jobs end here
Failed	All containers terminated and at least one failed
Unknown	The node cannot be reached, so the phase cannot be reported

Now the part that confuses everyone, and it is worth being precise about: CrashLoopBackOff is not a phase.

Neither is ImagePullBackOff, ContainerCreating, Terminating, Error, Completed, Init:0/2 or OOMKilled. kubectl get pods prints a helpful summary that mixes the pod's phase with the reason a container is waiting or terminated — and those are separate fields, status.containerStatuses[].state.

Why this matters in practice rather than as trivia: a pod showing CrashLoopBackOff is in phase Running. So a script or an alert that checks status.phase will report it as healthy. Anything watching pod health must read container states and the Ready condition, not the phase. This catches people who automate against the API for the first time.

CrashLoopBackOff itself is worth naming exactly: the container keeps exiting, so the kubelet is waiting before restarting it again, with the delay doubling each time — 10 s, 20 s, 40 s, up to five minutes. The status is the kubelet's patience, not the error. The error is in kubectl logs --previous.

restartPolicy, and why it is a pod-level setting

Three values, set on the pod, applying to every container in it.

Value	Behaviour	Used by
Always	Restart a container whenever it exits, success or failure	Deployments, StatefulSets, DaemonSets — and it is the default
OnFailure	Restart only on a non-zero exit	Jobs and CronJobs
Never	Never restart; the pod goes to Succeeded or Failed	Jobs, when you want each attempt to be a fresh pod

The constraint that surprises people: a Deployment's pods may only use Always. The API rejects anything else. That is not arbitrary — a Deployment's promise is that a given number of pods exist and keep running, and a pod that is allowed to finish contradicts it. If your workload is meant to end, it is a Job, not a Deployment. A Deployment running a script that exits 0 produces an endless restart loop where nothing is actually wrong, which is a genuinely common early mistake.

And note the restart happens in place, on the same node, keeping the same pod and the same IP (Chapter 13.6.10's pause container is why). The restart count goes up; the pod is not replaced. A pod is only replaced when something deletes it.

Bare pods, and why nothing looks after them

A bare pod is one you created directly, with no controller above it.

kubectl run tmp --image=nicolaka/netshoot -- sleep 3600    # a bare pod

Nothing recreates it. No ReplicaSet, no owner, no reconcile loop (Chapter 13.6.9). If its node dies, the pod is gone permanently. If somebody drains the node, it is evicted and never comes back.

And it quietly blocks cluster maintenance, which is the reason it appears here rather than as a footnote. The cluster autoscaler will not remove a node holding a bare pod, because it has no way to recreate it elsewhere — so one debugging pod somebody forgot about can keep an expensive node alive for weeks (Chapter 13.6.13). Always pass --rm on a debugging pod, or clean up afterwards.

The rule: bare pods are for experiments that you watch and delete. Everything else gets a controller.

2. Deployment — the default for stateless services

Manages a ReplicaSet, which manages pods. Two levels, and the second one exists so that a rollout can hold two versions at once.

apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout-api, namespace: shop }
spec:
  replicas: 3
  revisionHistoryLimit: 5                    # (1)
  progressDeadlineSeconds: 600               # (2)
  selector:
    matchLabels: { app: checkout-api }       # (3)
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                            # (4)
      maxUnavailable: 0                      # (5)
  template:                                  # (6)
    metadata:
      labels: { app: checkout-api }
    spec:
      containers: [ ... ]

(1) How many old ReplicaSets to keep for rollback. Each is an empty object, not running pods — cheap history.

(2) If the rollout makes no progress for this long, it is marked failed. Without it a broken rollout hangs silently forever, and nothing tells you.

(3) The selector is immutable after creation. Change it and you must delete and recreate the Deployment, which is a small but genuine outage. Choose labels carefully once.

(4) How many pods above the desired count may exist during a rollout.

(5) How many may be unavailable. Zero means capacity never drops — a new pod must be ready before an old one goes. This is the setting you want for a service that matters, and it costs one extra pod's worth of resources during a deploy.

(6) The pod template. Changing anything in here triggers a new rollout; changing replicas does not.

How a rolling update actually proceeds

With maxSurge: 1, maxUnavailable: 0 and three replicas:

start        v1 v1 v1
add one      v1 v1 v1 v2(starting)
ready        v1 v1 v1 v2 ← readiness probe passed, now receiving traffic
remove one   v1 v1 v2                       ← old pod gets SIGTERM
... repeat until:
finish       v2 v2 v2

The readiness probe is the gate. Without a meaningful one, Kubernetes considers a pod ready the moment its container starts, so it replaces the old version with new pods that are not yet serving. A rolling update with a bad readiness probe is a rolling outage, and it looks like a mysterious application error rather than a configuration one.

kubectl rollout status deployment/checkout-api        # (1)
kubectl rollout history deployment/checkout-api       # (2)
kubectl rollout undo deployment/checkout-api          # (3)
kubectl rollout undo deployment/checkout-api --to-revision=4
kubectl rollout restart deployment/checkout-api       # (4)
kubectl rollout pause deployment/checkout-api         # (5)

(1) Blocks until done or failed — this is what a CI pipeline waits on. (2) The revisions available to roll back to. (3) Back to the previous ReplicaSet, which is why it is instant: the old ReplicaSet still exists and is simply scaled back up. (4) A graceful restart of every pod with no change to the spec. This is the correct way to "restart the service", and it works by stamping an annotation on the template so the rollout machinery does its normal thing. (5) Stop mid-rollout — useful for a manual canary: pause after one new pod, look at metrics, then resume or undo.

strategy: Recreate kills everything, then starts the new version. Downtime, on purpose. The honest use is a schema migration that two versions cannot both survive, or a workload holding a lock that only one instance may hold.

Blue/green and canary are not built in. They are done with two Deployments and a Service selector switch, or with a controller such as Argo Rollouts or Flagger that automates the traffic shifting and the metric checks (Chapter 13.7).

3. StatefulSet — when identity matters

For workloads where each instance is not interchangeable: databases, brokers, anything doing peer discovery or leader election.

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: db }
spec:
  serviceName: db-headless              # (1)
  replicas: 3
  podManagementPolicy: OrderedReady     # (2)
  updateStrategy:
    rollingUpdate: { partition: 0 }     # (3)
  selector:
    matchLabels: { app: db }
  template: { ... }
  volumeClaimTemplates:                 # (4)
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: managed-csi-premium
        resources: { requests: { storage: 100Gi } }

What you get that a Deployment does not give you:

Stable names. db-0, db-1, db-2 — not random suffixes. A pod that is replaced comes back with the same name.

Stable network identity. With a headless Service, each pod gets its own DNS name: db-0.db-headless.shop.svc.cluster.local. Peers can address each other individually, which is what clustered software requires.

(1) The headless Service that provides those per-pod names. It must exist.

(2) Ordered start and stop. db-0 fully ready before db-1 begins; shutdown in reverse. This is the feature and the reason a StatefulSet rolls slowly, and it exists because a database cluster generally must not lose two members at once.

(3) partition is a genuinely useful and underused feature. Set it to 2 and only pods with an ordinal at or above 2 update — so db-2 takes the new version and db-0 and db-1 do not. Verify, then lower the partition. A canary for stateful workloads, built in.

(4) volumeClaimTemplates gives each pod its own PersistentVolumeClaim, created automatically and named after the pod. db-1 always reattaches to data-db-1.

Two behaviours that catch people:

Deleting the StatefulSet does not delete the volume claims. Deliberate — it is what stops an accidental delete destroying your data — and it means recreating the set reattaches the existing data. It also means the claims accumulate silently if you scale down and forget.

Scaling down does not delete claims either, so scaling back up rejoins the original data.

And the honest advice about databases in Kubernetes: it works, good operators handle failover, backup and restore well, and a managed database is still the right default unless you have a specific reason and someone who owns it. Chapter 13.6.16 comes back to this.

4. DaemonSet — one pod per node

Runs one copy on every node (or every node matching a selector). New node, new pod, automatically.

The uses are all infrastructure: log collectors, metrics agents, network plugins, storage drivers, security agents.

spec:
  template:
    spec:
      tolerations:
        - operator: Exists           # (1)
      nodeSelector:
        kubernetes.io/os: linux      # (2)

(1) Tolerate every taint, so the agent runs even on nodes reserved for particular workloads. A log collector that skips your GPU nodes is a log collector with a hole in it.

(2) Restrict to matching nodes — necessary on mixed-OS clusters.

DaemonSet pods are skipped by kubectl drain, because they belong to the node rather than to a workload.

5. Job and CronJob

A Job runs pods until a number of them succeed, then stops.

apiVersion: batch/v1
kind: Job
metadata: { name: nightly-reconcile }
spec:
  completions: 10            # (1)
  parallelism: 3             # (2)
  completionMode: Indexed    # (3)
  backoffLimit: 4            # (4)
  activeDeadlineSeconds: 3600  # (5)
  ttlSecondsAfterFinished: 86400  # (6)
  template:
    spec:
      restartPolicy: OnFailure    # (7)
      containers: [ ... ]

(1) Ten successful runs are needed. (2) Three at a time. (3) Each pod gets an index in an environment variable, so pod 0 handles shard 0 and so on. This is how you split a batch across pods without a queue. (4) Retries before the Job is marked failed. Without a limit, a permanently broken job retries forever with exponential backoff. (5) A wall-clock ceiling. Without it, a job that hangs holds its resources indefinitely. (6) Delete the Job this long after it finishes. Without this, finished Jobs and their pods accumulate until listing the namespace is painful and etcd is carrying thousands of dead objects. (7) OnFailure restarts the container in place; Never creates a new pod per attempt. Never leaves the failed pods behind, which is better for debugging and worse for clutter.

A CronJob creates Jobs on a schedule:

apiVersion: batch/v1
kind: CronJob
spec:
  schedule: "*/15 * * * *"
  timeZone: "Asia/Kolkata"          # (1)
  concurrencyPolicy: Forbid         # (2)
  startingDeadlineSeconds: 300      # (3)
  successfulJobsHistoryLimit: 3     # (4)
  failedJobsHistoryLimit: 3
  jobTemplate: { ... }

(1) Without this, schedules are in the control plane's timezone, usually UTC. A "midnight" job that runs at 05:30 local time is this field being absent.

(2) Forbid skips a run if the previous one is still going. The default, Allow, will happily pile up overlapping runs of a job that has become slow, and that is how a scheduled task takes down a database. Replace kills the old one and starts fresh.

(3) If the control plane was unavailable at the scheduled time, still start if it is within this window — otherwise skip.

(4) Keep three of each. The defaults keep more than most people want and the objects are easy to forget about.

And the guarantee to be honest about: a CronJob is at least once, not exactly once. Under some failure conditions a scheduled run can fire twice. Jobs that must not double-run need to be idempotent or take a lock. This is not a defect to work around; it is the standard property of any distributed scheduler.

6. Ownership, and why deleting works

Every object created by a controller carries an ownerReference pointing at its parent. A Deployment owns ReplicaSets, which own Pods.

Delete the Deployment and garbage collection deletes everything underneath, because those references form a tree. This is why the whole thing disappears cleanly, and it is why deleting a pod does nothing lasting: the pod's owner still exists and still declares that a pod should be there.

--cascade=orphan breaks the links instead, leaving the children running. Occasionally the right tool when you are replacing a controller under a live workload.

Finalizers are the other half. A finalizer is a string on an object that blocks deletion until whatever put it there removes it — used so that a cloud load balancer or disk is cleaned up before the object disappears. A stuck finalizer is why an object sits in Terminating forever, and the correct fix is to find out which controller owes the cleanup, not to strip the finalizer by hand — that leaks the cloud resource it was protecting.

7. Termination, second by second

Knowing this sequence explains almost every deploy-time error, and it contains one genuine race condition.

When a pod is deleted:

1. The pod is marked terminating and removed from Service endpoints. This happens in parallel with step 2, and that is the race.
2. The preStop hook runs, if there is one.
3. SIGTERM is sent to PID 1 of each container.
4. The grace period counts down (terminationGracePeriodSeconds, default 30).
5. SIGKILL for anything still alive.

The race is worth understanding properly. Removing the pod from the endpoint list requires the EndpointSlice controller to notice, then kube-proxy on every node to update its rules — that takes a moment. Meanwhile SIGTERM has already been delivered. So there is a window where the application has begun shutting down but traffic is still arriving, and the symptom is a handful of connection-refused errors on every single deploy.

The preStop sleep is the standard fix and it is not a hack:

lifecycle:
  preStop:
    exec: { command: ["sleep", "5"] }

SIGTERM is not sent until preStop finishes, so those five seconds are spent still serving normally while the endpoint removal propagates. Then shutdown begins with no traffic left in flight. Five seconds is the usual value.

And the application half is still required (Chapter 13.6.5): handle SIGTERM, stop accepting new connections, finish in-flight requests, close pools, exit. terminationGracePeriodSeconds must be longer than your slowest legitimate request, or SIGKILL cuts it off.

8. Namespaces and labels

A namespace is a scope for names, not a security boundary on its own. Two Deployments called api can coexist in different namespaces. What makes a namespace an actual boundary is what you attach to it: RBAC roles, resource quotas, limit ranges, network policies and Pod Security Standards (Chapters 13.6.13 and 13.6.14).

Not everything is namespaced. Nodes, PersistentVolumes, StorageClasses, ClusterRoles and CustomResourceDefinitions are cluster-wide. kubectl api-resources --namespaced=false lists them.

Labels are the coordination mechanism, and a consistent scheme pays off constantly. The standard set is worth adopting because tools understand it:

labels:
  app.kubernetes.io/name: checkout-api
  app.kubernetes.io/instance: checkout-api-prod
  app.kubernetes.io/version: "1.4.2"
  app.kubernetes.io/component: api
  app.kubernetes.io/part-of: checkout

kubectl get pods -l app=checkout-api,tier=backend      # (1)
kubectl get pods -l 'environment in (staging,prod)'    # (2)
kubectl get pods -l '!canary'                          # (3)

(1) Equality, ANDed. (2) Set membership. (3) Objects without the label at all.

What the interviewer will push on

"Deployment or StatefulSet?" Deployment for anything stateless and interchangeable. StatefulSet when instances need stable names, stable per-pod DNS, and their own persistent volume that follows them — databases, brokers, anything with peer discovery. The detail that lands is partition in the update strategy, which gives you a built-in canary for stateful workloads, and that deleting a StatefulSet deliberately leaves its volume claims behind.

"Walk me through a rolling update." New ReplicaSet created, scaled up by maxSurge, each new pod gated by its readiness probe, old pods removed within maxUnavailable, repeat. maxUnavailable: 0 means capacity never drops. The point to make: without a meaningful readiness probe, a rolling update is a rolling outage, because pods are considered ready as soon as the container starts.

"How do you restart a service?" kubectl rollout restart, which does a normal graceful rolling update. Not kubectl delete pod, which is uncontrolled — and explaining why deleting a pod does nothing lasting, because its owner still declares it should exist, is the answer underneath the answer.

"Why do we get connection errors during every deploy?" Endpoint removal and SIGTERM happen in parallel, and endpoint removal has to propagate to kube-proxy on every node, so traffic arrives at a pod that has started shutting down. A preStop sleep of about five seconds holds SIGTERM back while the removal propagates. This question separates people who have shipped in Kubernetes from people who have read the docs.

"How do you stop a CronJob overlapping itself?" concurrencyPolicy: Forbid. The default is Allow, which piles up runs when a job becomes slow. Then volunteer that CronJob is at-least-once, not exactly-once, so anything that must not double-run needs to be idempotent or take a lock.

"What are init containers for, and what changed recently?" Setup that must complete before the application starts — migrations, waiting on dependencies. The change is native sidecars: an init container with restartPolicy: Always keeps running alongside the application and is not counted for pod completion, which finally makes sidecars work inside Jobs and fixes startup ordering with a mesh proxy.

"What are the pod phases?" Pending, Running, Succeeded, Failed, Unknown — five, and that is all. The follow-up that separates people is that CrashLoopBackOff is not one of them: it is a container waiting reason, the pod is in phase Running, and therefore any alert written against status.phase reports a crash-looping pod as healthy. Health has to be read from container states and the Ready condition.

"Name the multi-container patterns." Sidecar adds a capability alongside the main container; ambassador sits between your container and the outside world so the application only ever talks to localhost; adapter translates your container's output into the format something else expects. Then the rule that decides when to use any of them: it belongs in the pod only if it must share the network stack or a volume and live and die with the main container — otherwise it is a separate Deployment.

"Why can't a Deployment's pods use restartPolicy: Never?" Because a Deployment promises that a number of pods exist and keep running, and a pod that is allowed to finish contradicts that, so the API rejects it. A workload meant to end is a Job. The tell that someone has hit this is knowing the symptom: a Deployment running a script that exits 0 restarts forever with nothing actually wrong.

One thing to volunteer: point out that a Deployment's selector is immutable, so an ill-chosen label scheme can only be corrected by deleting and recreating the Deployment — a real outage for a decision made in five seconds on day one. Adopting the app.kubernetes.io/* labels from the start costs nothing and avoids it.

Recall

• A pod is containers sharing a network namespace and volumes — one IP, one port space, localhost between them. Disposable, never repaired, never addressed directly.
• Labels coordinate everything; annotations configure tools. fsGroup is what lets a non-root pod write to a mounted volume. The downward API exposes pod metadata as environment variables. Turn off automountServiceAccountToken unless the pod calls the API.
• Three multi-container patterns: sidecar (adds a capability alongside) · ambassador (your app talks to localhost, it handles the outside world) · adapter (translates your app's output into what something else expects). If it needs neither the shared network nor a shared volume, it is a separate Deployment.
• Init containers run to completion in order. An init container with restartPolicy: Always is a native sidecar (1.29+) — runs alongside, not counted for completion, fixes Jobs and startup ordering. Ephemeral containers are injected into a running pod by kubectl debug and cannot be declared.
• hostNetwork/hostPID/hostIPC borrow the node's namespaces. Legitimate for DaemonSet agents, blocked by the baseline policy for everything else, because they remove network isolation and expose every process on the machine.
• Five phases only: Pending, Running, Succeeded, Failed, Unknown. CrashLoopBackOff is not a phase — it is a container waiting reason, and such a pod is in phase Running, so anything alerting on status.phase calls it healthy. The BackOff delay doubles to five minutes; the error is in logs --previous.
• restartPolicy is pod-level: Always (the default, and the only value a Deployment accepts) · OnFailure · Never. A workload meant to finish is a Job — a Deployment running a script that exits 0 restarts forever. Restarts happen in place, same pod, same IP.
• A bare pod has no controller, so nothing recreates it and the cluster autoscaler will not remove the node holding it — one forgotten debug pod keeps an expensive node alive. Use --rm.
• Deployment → ReplicaSet → Pods. maxUnavailable: 0 keeps capacity flat. The readiness probe is the gate — without one, a rolling update is a rolling outage. progressDeadlineSeconds stops a silent hang. The selector is immutable.
• kubectl rollout restart is the right way to restart; undo is instant because the old ReplicaSet still exists; pause gives a manual canary.
• StatefulSet = stable names, per-pod DNS via a headless Service, volumeClaimTemplates so each pod keeps its disk, ordered start/stop, and partition for a staged rollout. Deleting it keeps the claims.
• DaemonSet = one per node, tolerate everything, skipped by drain. Job: set backoffLimit, activeDeadlineSeconds, ttlSecondsAfterFinished; Indexed mode shards work. CronJob: set timeZone, concurrencyPolicy: Forbid, history limits — and it is at-least-once.
• ownerReferences make cascading delete work, which is why deleting a pod achieves nothing. A stuck Terminating object is a finalizer — fix the controller, do not strip it.
• Termination: endpoint removal ‖ preStop → SIGTERM → grace period → SIGKILL. The parallel step is a real race — a preStop sleep of ~5 s is the standard fix.
• A namespace is a name scope, not a boundary until you attach RBAC, quotas, policies and network policies.

Self-test: Why does deleting a pod not restart a service in any meaningful sense? · What exactly does maxUnavailable: 0 cost you? · Which StatefulSet field gives you a canary, and what happens to volumes when you delete the set? · Why do deploys produce a few connection errors, and what is the fix? · What makes an init container a sidecar? · Which two CronJob settings prevent the two most common failures?

Next: 13.6.12 is how anything reaches those pods — the flat network model, what a Service address really is, why one gRPC client pins to one pod, and how traffic from outside gets in.