13.6.14 — Packaging, Policy and Operating a Cluster

Twelve services. Each needs a Deployment, a Service, an Ingress, a ConfigMap, a ServiceAccount, an HPA and a PodDisruptionBudget. Three environments.

That is 252 YAML files, of which about 240 differ from another one by a replica count, an image tag and a hostname.

Copy them and every fix must be made twelve times, or thirty-six. Miss one and it drifts. This page is how the industry handles that — and then everything else you need to actually run a cluster: who is allowed to do what, what policy stops a bad manifest reaching the cluster at all, and the order to run commands in when something is broken.

1. Helm

Helm packages a set of manifests as a chart, with a values file, versioned releases and rollback.

checkout-chart/
├── Chart.yaml            # name, version, dependencies
├── values.yaml           # defaults
├── values-prod.yaml      # per-environment overrides
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    └── _helpers.tpl      # reusable snippets

# templates/deployment.yaml
spec:
  replicas: {{ .Values.replicaCount }}                          # (1)
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}   # (2)
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}@{{ .Values.image.digest }}"   # (3)
          resources:
            {{- toYaml .Values.resources | nindent 12 }}        # (4)

(1) A value substituted from values.yaml or the command line.

(2) The checksum annotation from Chapter 13.6.13, generated automatically — a config change alters the pod template and triggers a rolling update.

(3) Digest-pinned image, assembled from values.

(4) nindent 12 is where most Helm frustration lives. The template engine is text substitution, so you are responsible for producing correct YAML indentation. Get it wrong and the error message describes a YAML parse failure at a line number in generated output you cannot see.

helm install checkout ./chart -f values-prod.yaml     # (1)
helm upgrade checkout ./chart --set image.digest=sha256:9f2b... --atomic --wait   # (2)
helm rollback checkout 3                               # (3)
helm history checkout                                  # (4)
helm template ./chart -f values-prod.yaml              # (5)
helm diff upgrade checkout ./chart                     # (6)

(1) First install. (2) --wait blocks until resources are ready; --atomic rolls back automatically if they are not. These two flags are what make a Helm upgrade in a pipeline safe rather than optimistic. (3) Back to revision 3. Helm stores each release as a Secret in the namespace, so history is in the cluster. (4) What you can roll back to. (5) Render locally without installing. The first thing to run when output is not what you expected. (6) Show what would change — a plugin, and worth installing on day one.

What Helm genuinely provides: one artefact per application, per-environment values, release history with rollback, dependencies between charts, and a distribution format for third-party software.

The honest criticism, because it shapes when to use it:

It is text templating over YAML. The template is not valid YAML while it is being written, so editors cannot help, and whitespace errors produce incomprehensible messages.

Conditionals become unreadable fast. A chart supporting many options ends up with logic that is genuinely hard to follow.

Third-party charts are often deeply configurable in ways that are hard to audit. You are installing several thousand lines of generated manifests. helm template before installing anything into a cluster you care about is a good habit, particularly for cluster-wide resources and RBAC.

2. Kustomize

Patches plain YAML with overlays. No templating language.

base/
├── kustomization.yaml
├── deployment.yaml       # valid YAML, applies as-is
└── service.yaml
overlays/
├── staging/kustomization.yaml
└── prod/
    ├── kustomization.yaml
    └── replicas-patch.yaml

# overlays/prod/kustomization.yaml
resources: [../../base]
namespace: shop-prod
images:
  - name: checkout-api
    digest: sha256:9f2b...        # (1)
replicas:
  - { name: checkout-api, count: 6 }
patches:
  - path: resources-patch.yaml    # (2)
configMapGenerator:
  - name: checkout-config
    literals: [LOG_LEVEL=info]    # (3)

(1) Image substitution as a first-class operation, not a template variable. (2) A strategic merge patch — write only the fields you are changing. (3) Generated ConfigMaps get a hash suffix in their name, so changing a value creates a new name, which changes the pod template, which triggers a rollout. Kustomize's answer to the config-change problem, and it is more elegant than the checksum annotation.

Built into kubectl — kubectl apply -k overlays/prod. No extra tool.

The trade: the base files are always valid YAML and always readable, which is a genuine advantage. But Kustomize is weaker for real packaging — no versioning, no dependencies, no distribution, no release history.

The split most teams land on: Kustomize for your own applications, Helm for third-party software. Publishing something for others to install is where Helm is clearly right.

3. Operators and custom resources

You can add your own object types to the Kubernetes API, and they behave exactly like built-in ones — kubectl get, RBAC, watches, everything.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster                    # not a built-in type
metadata: { name: checkout-db }
spec:
  instances: 3
  storage: { size: 100Gi }
  backup:
    barmanObjectStore:
      destinationPath: "https://acct.blob.core.windows.net/backups"

A CustomResourceDefinition registers the type. An operator is a controller that watches those objects and does the work — in this case provisioning a three-node PostgreSQL cluster with replication, running failover when the primary dies, taking backups, and handling minor-version upgrades.

This is the operator pattern: encoding what an experienced administrator would do into a control loop. Same reconcile loop as everything else (Chapter 13.6.9), applied to domain knowledge.

Operators you will meet: cert-manager (certificates), Prometheus Operator (monitoring), CloudNativePG or Zalando (PostgreSQL), Strimzi (Kafka), External Secrets, Argo CD.

Writing one is a real project — reconciliation logic, status reporting, upgrade paths, failure handling. Using a good one is usually excellent. The distinction to hold: an operator is only as good as the operational knowledge encoded in it, so a well-maintained one from the software's own community is a very different proposition from an abandoned one from a blog post.

4. GitOps

Instead of a pipeline holding cluster credentials and pushing changes, a controller runs inside the cluster, watches a Git repository, and reconciles.

git push → Git repo → Argo CD or Flux (in-cluster) → applies → cluster matches Git

Why this is more than a workflow preference:

Git is the desired state, so the cluster is auditable. What should be running is a commit history with reviews and authorship.

Drift is corrected automatically. Somebody edits a Deployment by hand and the controller puts it back — visibly, with a message.

No cluster credentials in CI. The pipeline pushes an image and updates a tag in a repository. It never has access to the cluster, which removes a large piece of blast radius.

Rollback is git revert.

And it makes the etcd risk from Chapter 13.6.9 much smaller. If every object is in Git, losing etcd means rebuilding a cluster and pointing the controller at the repository, rather than reconstructing state from memory.

Argo CD has a strong user interface showing sync status and diffs per application. Flux is a set of controllers with no interface, composed as you like. Both are mature; the choice is mostly about whether you want the dashboard.

The one thing to decide deliberately: what happens to things GitOps does not manage. Auto-pruning removes anything not in Git, which is powerful and occasionally deletes something a person created deliberately.

5. RBAC

Four objects, and the model is small enough to learn in one sitting.

• Role — permissions within one namespace.
• ClusterRole — permissions cluster-wide, or on non-namespaced things like nodes.
• RoleBinding — grants a Role (or a ClusterRole, scoped to one namespace) to subjects.
• ClusterRoleBinding — grants a ClusterRole everywhere.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: deployer, namespace: shop }
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "patch", "update"]     # (1)
  - apiGroups: [""]
    resources: ["pods", "pods/log"]                         # (2)
    verbs: ["get", "list"]
---
kind: RoleBinding
metadata: { name: team-deployers, namespace: shop }
subjects:
  - { kind: Group, name: "shop-engineers", apiGroup: rbac.authorization.k8s.io }   # (3)
roleRef: { kind: Role, name: deployer, apiGroup: rbac.authorization.k8s.io }

(1) No delete and no create — this role can update existing deployments but not remove them.

(2) Subresources are separate permissions. pods/log for logs, pods/exec for shells. Granting pods/exec is effectively granting the ability to run arbitrary code as that workload, and it should be treated as a much bigger permission than "read pods".

(3) A group, from the identity provider — Entra ID on AKS (Chapter 13.6.15). Kubernetes has no user objects, so this string comes from whatever authenticated the request.

Purely additive: there are no deny rules. Permissions accumulate from every binding that applies to you.

Service accounts. Every pod has one, and the default is mounted automatically. Turn that off where it is not needed (automountServiceAccountToken: false) — otherwise a compromised container holds a live cluster credential.

Two commands that make RBAC tractable:

kubectl auth can-i delete pods --namespace shop                    # (1)
kubectl auth can-i list secrets --as system:serviceaccount:shop:checkout  # (2)

(1) What can I do? (2) What can that service account do? This is how you verify a workload's permissions before an auditor does.

The permissions to be genuinely careful with: secrets (read is a credential dump), pods/exec (code execution), escalate and bind (grant yourself more), impersonate (become somebody else), and any write access to admission webhook configurations (rewrite everything entering the cluster).

6. Pod Security Standards and admission policy

Three levels, applied by labelling a namespace:

Level	Meaning
privileged	No restrictions
baseline	Blocks the obviously dangerous — host namespaces, privileged containers, most host paths
restricted	The target — non-root, no privilege escalation, all capabilities dropped, seccomp profile, read-only root filesystem

apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    pod-security.kubernetes.io/enforce: restricted     # (1)
    pod-security.kubernetes.io/warn: restricted        # (2)

(1) Reject anything that does not comply. (2) Warn as well, which is how you roll this out — set warn first, watch what would break, then enforce.

For anything beyond that, a policy engine. Pod Security Standards cover pod security settings only; they cannot express "images must come from our registry" or "every Deployment must have a PodDisruptionBudget".

Kyverno writes policies as Kubernetes resources, in YAML:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: require-digest }
spec:
  validationFailureAction: Enforce
  rules:
    - name: no-tags
      match: { any: [{ resources: { kinds: [Pod] } }] }
      validate:
        message: "Images must be referenced by digest."
        pattern:
          spec:
            containers:
              - image: "*@sha256:*"

OPA Gatekeeper uses a policy language called Rego — more powerful, steeper to learn. Validating Admission Policy is now built into Kubernetes itself, using CEL expressions, with no webhook to run.

Policies worth having on day one: no :latest, images only from approved registries, resource requests required, runAsNonRoot, no hostPath, no privileged, and image signatures verified.

And the point that makes this different from scanning: a validating policy denies the object at admission, so it never enters the cluster (Chapter 13.6.9). A scanner tells you afterwards. One is a control, the other is a report.

Verify signatures at admission, so a compromised registry is not a compromised production — the enforcement half of the signing from Chapter 13.6.4.

7. Observability, in one section

Kubernetes ships with none of this. All four layers are things you add.

Metrics. Prometheus scrapes pods and components; Grafana displays them. The metrics that matter most on a cluster, in order: CPU throttling (Chapter 13.6.13), memory working set against limit, pod restart counts, pending pod counts, and node allocatable versus requested.

Logs. Something collects standard output from every pod and ships it — Fluent Bit, Vector, or a cloud agent. The application's only job is to log to stdout as structured JSON with the trace ID in it.

Traces. OpenTelemetry instrumentation with a collector. This is what turns "the checkout is slow" into "the third call in this chain takes 3 seconds", and it is the thing you miss most once you have more than a handful of services.

Events. The cluster's own account of what it did, and they expire — the default retention is one hour. Ship them somewhere, because after an incident the events explaining it are usually already gone.

8. Debugging, in order

This sequence answers most problems, and the order is deliberate.

kubectl get pods -o wide                        # (1)
kubectl describe pod <name>                     # (2)
kubectl logs <name> --previous                  # (3)
kubectl get events --sort-by=.lastTimestamp     # (4)
kubectl exec -it <name> -- sh                   # (5)
kubectl debug -it <name> --image=nicolaka/netshoot --target=api    # (6)
kubectl get endpoints <service>                 # (7)

(1) Status, restart count and which node. A high restart count with Running status is a crash loop that has recovered — the most easily missed signal here.

(2) The Events section at the bottom is the answer more often than anything else. Read it before you read logs.

(3) --previous is the one people forget. After a crash, the current container's logs show a fresh start; the previous container's logs contain the error that caused it.

(4) Everything the cluster did recently, in order.

(5) A shell — if the image has one.

(6) An ephemeral debug container sharing the target's namespaces. This is the answer for distroless images (Chapter 13.6.5): you get a full toolbox next to your container without adding a shell to production.

(7) For anything traffic-related, start here (Chapter 13.6.12).

The status table:

Status	Cause
Pending	No node fits — requests too large, a taint, an unsatisfiable affinity, or an unbound volume
ContainerCreating (stuck)	Image pull, missing ConfigMap or Secret, volume attach, or a CNI failure
ImagePullBackOff	Wrong name or tag, missing credentials, or a registry rate limit
CrashLoopBackOff	Exits repeatedly — read logs --previous
OOMKilled (137)	Memory limit exceeded
Error (1)	The process exited non-zero — application logs
Running, no traffic	Readiness failing, or the Service selector does not match the labels
Periodic restarts	Liveness too strict, or checking a dependency
Terminating forever	A finalizer, or a process ignoring SIGTERM
Evicted	Node pressure — the pod had no requests
Init:0/2	An init container has not completed

Two diagnostics worth having as reflexes. kubectl get events --field-selector type=Warning -A shows everything unhealthy in the whole cluster in one line of output. And kubectl describe node <node> is where conditions and taints live — a node can report Ready while under disk pressure and quietly evicting pods (Chapter 13.6.10).

9. Upgrades

Kubernetes releases three times a year and each version is supported for about a year. Falling behind is a real risk: the upgrade path is one minor version at a time, so being four versions behind means four sequential upgrades.

The order is fixed: control plane first, then node pools, one at a time.

What breaks upgrades, and it is nearly always the same thing: removed API versions. A resource that moved from v1beta1 to v1 stops being served eventually, and manifests referencing the old version fail to apply. kubectl warns about deprecated APIs, and tools like pluto and kubent scan your manifests and cluster for them. Read the release notes' removal section before each upgrade — it is short and it is the part that matters.

Node upgrades are a cordon and drain (Chapter 13.6.10), which is why PodDisruptionBudgets are what make an upgrade a non-event.

What the interviewer will push on

"Helm or Kustomize?" Helm for packaging and third-party software, because it gives versioned releases, dependencies, and rollback. Kustomize for your own manifests, because it patches real YAML rather than templating text, and it is built into kubectl. Naming Helm's actual weakness — text templating over YAML, so whitespace errors produce incomprehensible messages — is what makes the answer credible rather than a preference.

"What is an operator?" A custom resource type plus a controller that reconciles it, encoding what an experienced administrator would do — provisioning, failover, backup, upgrades. Same control loop as everything else, applied to domain knowledge. The judgement to add: writing one is a real project, using a well-maintained one is usually excellent.

"Why GitOps rather than a deploy pipeline?" Git becomes the desired state, so the cluster is auditable and drift is corrected automatically; CI never holds cluster credentials; rollback is a revert. And it reduces the etcd risk, because rebuilding means pointing a controller at the repository.

"How do you stop someone deploying :latest?" A validating admission policy that denies it, so the object never enters the cluster. Contrast with a scanner, which reports after the fact. Then the set worth having on day one: approved registries only, requests required, runAsNonRoot, no privileged, signatures verified.

"Which RBAC permissions would you scrutinise?" secrets (reading them is a credential dump), pods/exec (arbitrary code execution as that workload), escalate and bind, impersonate, and write access to webhook configurations. Then volunteer automountServiceAccountToken: false, because the default mount means a compromised container holds a cluster credential it never needed.

"A pod is CrashLoopBackOff. Walk me through it." describe and read Events first, then logs --previous — because the current container's logs are from a fresh start and the error is in the previous one. Then check the usual causes: a missing ConfigMap or Secret, a failing dependency at boot, an OOM kill, or a liveness probe that is too strict.

One thing to volunteer: mention that Kubernetes events expire after about an hour by default, so the explanation for an incident is usually gone by the time anyone investigates. Shipping events to your logging system is a small change that makes post-incident analysis possible at all, and almost nobody does it until the second time it bites.

Recall

• Helm = charts, values per environment, versioned releases, rollback, dependencies. --wait --atomic makes a pipeline upgrade safe; helm template and helm diff before installing. Weakness: text templating over YAML, so whitespace errors are unreadable.
• Kustomize patches valid YAML with overlays, is built into kubectl, and hashes generated ConfigMap names so a config change triggers a rollout. Weaker at packaging. Common split: Kustomize for your apps, Helm for third-party.
• CRD + controller = an operator, encoding administrator knowledge as a reconcile loop. Using a well-maintained one is usually excellent; writing one is a project.
• GitOps: a controller in the cluster reconciles from Git. Auditable, self-correcting, no cluster credentials in CI, git revert to roll back, and it shrinks the etcd loss risk.
• RBAC = Role/ClusterRole + RoleBinding/ClusterRoleBinding, additive, no deny. Subresources are separate — pods/exec is code execution. Scrutinise secrets, escalate, bind, impersonate, webhook writes. kubectl auth can-i --as verifies a service account. Turn off automountServiceAccountToken.
• Pod Security Standards — privileged / baseline / restricted is the target; set warn before enforce. Policy engines (Kyverno, Gatekeeper, built-in CEL policies) deny at admission, so a bad object never enters the cluster — unlike a scanner, which reports afterwards.
• Observability is all added: metrics (watch CPU throttling first), stdout logs shipped, OpenTelemetry traces, and events, which expire in about an hour.
• Debug order: get pods -o wide → describe and read Events → logs --previous → events → exec → kubectl debug for distroless → get endpoints for traffic problems.
• Upgrades: three releases a year, one minor version at a time, control plane then nodes. Removed API versions are what breaks them — scan with pluto/kubent. PodDisruptionBudgets make node upgrades a non-event.

Self-test: What are Helm's two flags that make a pipeline upgrade safe? · How does Kustomize trigger a rollout on a config change? · Why is pods/exec a bigger permission than it looks? · What is the difference between a validating policy and a scanner? · Which log flag matters after a crash loop, and why? · What breaks a Kubernetes version upgrade most often?

Next: 13.6.15 moves to a real managed cluster. What Azure actually runs for you in AKS, every component it adds on top of upstream Kubernetes, and which of the many configuration choices you can never change after creation.