13.6.13 — Resources, Storage, Scheduling and Autoscaling

A service's p99 latency is 4 seconds. It should be 80 milliseconds.

CPU usage sits at 34%. Memory is fine. There are no errors, no restarts, no failing probes. The database is fast. The traces show the time is spent inside the application, doing nothing in particular.

The container has a CPU limit of 500 millicores, and it is being throttled for 60% of every scheduling period. The CPU graph reads 34% because that is the average across the period, including the portion where the kernel refused to schedule it at all. Nothing in the default dashboard shows this. The metric that shows it is called container_cpu_cfs_throttled_seconds_total and almost nobody looks at it until they have been burned once.

This page is the configuration that decides whether a cluster is stable or quietly broken.

1. Requests and limits

The most consequential and most misunderstood two lines in Kubernetes.

resources:
  requests: { cpu: "100m", memory: "256Mi" }
  limits:   { memory: "256Mi" }

A request is what the scheduler reserves. It is used to decide which node has room, and it is subtracted from that node's allocatable capacity for as long as the pod exists. It is not enforced at run time — a pod may use more if the node has spare.

A limit is a hard ceiling, enforced by cgroups (Chapter 13.6.2).

Units, because both are easy to get wrong:

CPU is in cores. 1000m = 1 core. 100m is a tenth of a core. This is time, not a processor — a container with 500m may run on eight cores for a twentieth of each period.

Memory is in bytes, and the suffixes matter: Mi is 1,048,576 bytes; M is 1,000,000. Mixing them produces a 5% discrepancy that shows up as unexpected OOM kills at the edge.

The difference that causes outages

CPU is compressible. Memory is not. These two words explain every resource incident you will ever see.

Exceed a CPU limit and you are throttled, not killed. The kernel enforces the limit over a 100-millisecond period: use your quota and you are simply not scheduled until the next period begins.

Work through the arithmetic once, because it makes the failure obvious. A limit of 500m means 50 ms of CPU time per 100 ms period. A request handler needs 80 ms of CPU. It runs for 50 ms, is stopped for 50 ms, runs for its remaining 30 ms. A request that should take 80 ms takes 130 ms, and the CPU graph reports 65% usage. Under concurrency it is far worse, and the shape of the problem is a service that is slow with no errors and healthy-looking graphs.

Exceed a memory limit and the container is killed immediately. OOMKilled, exit code 137, no graceful shutdown, no chance to finish in-flight work. You cannot politely slow down an allocation, so there is nothing else the kernel can do.

What to actually set

The current mainstream advice, and the reasoning behind each line:

Always set memory requests and limits, equal to each other. Predictable, prevents one pod starving a node, and puts the pod in the best eviction class.

Always set CPU requests, because they are what guarantees you a share when the node is contended, and because CPU-based autoscaling cannot compute a percentage without them.

Usually omit CPU limits. This surprises people, so here is the argument. A CPU limit throttles you even when the node is completely idle — you are stopped from using capacity that nobody else wants. Requests already guarantee your share under contention, which is the actual protection you need. Set a CPU limit only where you must protect neighbours from a genuinely runaway workload, or where a platform policy requires one.

QoS classes fall out of this

The class is derived, not declared, and it decides who dies first when a node runs out of memory:

Class	Condition	Evicted
Guaranteed	Requests = limits for every container, both resources	Last
Burstable	Some requests set	Middle, by overage
BestEffort	Nothing set at all	First

A production service with no resource declarations is BestEffort and is the first thing thrown off a struggling node. One omitted stanza, severe consequence, extremely common.

Governing a namespace

apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: shop }
spec:
  limits:
    - type: Container
      default:        { memory: "512Mi" }        # (1)
      defaultRequest: { cpu: "100m", memory: "256Mi" }
      max:            { memory: "4Gi" }          # (2)
---
apiVersion: v1
kind: ResourceQuota
metadata: { name: shop-quota, namespace: shop }
spec:
  hard:
    requests.cpu: "20"                            # (3)
    requests.memory: 40Gi
    persistentvolumeclaims: "20"
    count/deployments.apps: "50"

(1) Applied to any container that declares nothing — this alone eliminates BestEffort pods in the namespace.

(2) A ceiling on what any one container may ask for.

(3) A namespace-wide budget. Note the important behaviour: once a ResourceQuota on CPU or memory exists, every pod in the namespace must declare requests or it is rejected. That is a feature — it forces the declaration — but it surprises teams the day it is introduced, because previously-working manifests start failing admission.

The trap that survives all of this

A process inside the container still sees the host's CPU count and the host's memory through the usual interfaces (Chapter 13.6.2). A runtime or thread pool sizing itself from "the machine" sizes for a 64-core, 256 GB node while living in a 500m, 512Mi box. Modern JVMs and .NET read cgroup limits; many libraries do not.

Pass the real numbers in explicitly:

env:
  - name: MEMORY_LIMIT
    valueFrom:
      resourceFieldRef: { resource: limits.memory }     # the downward API
  - name: CPU_LIMIT
    valueFrom:
      resourceFieldRef: { resource: limits.cpu }

2. Configuration and secrets

ConfigMap for non-sensitive configuration, Secret for sensitive. Structurally almost the same object, with different handling.

apiVersion: v1
kind: ConfigMap
metadata: { name: checkout-config, namespace: shop }
data:
  LOG_LEVEL: info
  app.yaml: |
    retries: 3
    timeoutMs: 2000

Two ways to consume it:

envFrom:
  - configMapRef: { name: checkout-config }    # (1)
volumeMounts:
  - { name: config, mountPath: /etc/app }      # (2)

(1) Every key as an environment variable. Simple, and fixed at container start.

(2) Every key as a file. Preferred, for three reasons: files can update without a restart, environment variables are inherited by every child process and appear in crash dumps, and a file can hold structured content.

Three behaviours that catch people:

Changing a ConfigMap restarts nothing. A pod started with the old values keeps them; a mounted file updates within a minute or so, but only if the application re-reads it. Most applications read configuration once at startup, so nothing happens.

The standard fix is a checksum annotation on the pod template:

template:
  metadata:
    annotations:
      checksum/config: "{{ sha256sum (toYaml .Values.config) }}"

Changing the config changes the annotation, which changes the pod template, which triggers a normal rolling update. Helm generates this; it is the mechanism, not a Helm feature.

subPath mounts do not update. Ever. Mounting a single file with subPath — which people do to avoid replacing a whole directory — creates a copy that never refreshes. If you need live updates, mount the directory.

Mark stable ConfigMaps and Secrets immutable: true. The kubelet then stops watching them, which measurably reduces API server load in a large cluster, and it protects against an accidental edit taking effect fleet-wide.

Secrets

A Secret is base64-encoded, not encrypted. Base64 is an encoding for moving binary through text — anyone with read access decodes it in one command. Four things to do about it:

Turn on encryption at rest for etcd, so a backup or a disk is not a credential dump. Managed services do this (Chapter 13.6.15).

Restrict access with RBAC. get secrets in a namespace is a powerful permission and should be treated as one.

Prefer an external secret store — Azure Key Vault, HashiCorp Vault, AWS Secrets Manager — reached either through the Secrets Store CSI driver, which mounts values directly as files with no Kubernetes Secret involved, or through the External Secrets Operator, which syncs them into Secrets. The CSI driver is the stronger option because the value never becomes a cluster object at all.

Best of all, avoid the secret. Workload identity gives a pod a cloud identity with no stored credential anywhere (Chapter 13.6.15). A secret that does not exist cannot leak.

3. Storage

Three objects and a driver:

• PersistentVolumeClaim (PVC) — "I need 100 GB of fast storage that one node can write to." Written by you.
• StorageClass — how claims are satisfied: which driver, which disk type, which parameters.
• PersistentVolume (PV) — the actual volume, usually created automatically in response to a claim.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: fast }
provisioner: disk.csi.azure.com
parameters: { skuName: Premium_LRS }
reclaimPolicy: Retain                    # (1)
allowVolumeExpansion: true               # (2)
volumeBindingMode: WaitForFirstConsumer  # (3)
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: checkout-data }
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast
  resources: { requests: { storage: 100Gi } }

(1) Retain keeps the disk when the claim is deleted. The default is usually Delete, which means removing a claim destroys the data immediately and permanently. Set Retain on anything you would miss.

(2) Allows growing the volume later by editing the claim. Shrinking is never possible.

(3) The single most valuable line here. With the default Immediate binding, the disk is created as soon as the claim exists — in some zone — and the pod can then only be scheduled in that zone, which may have no capacity. WaitForFirstConsumer delays creating the disk until the scheduler has chosen a node, and then creates it in the right zone. Without it you get pods stuck Pending with a volume node-affinity conflict, on a cluster that is visibly half empty.

Access modes, and what they really mean:

Mode	Meaning	Reality
ReadWriteOnce	One node may mount it read-write	What every cloud block disk gives you
ReadOnlyMany	Many nodes, read-only	Reference data
ReadWriteMany	Many nodes, read-write	Needs a network filesystem — Azure Files, NFS, EFS
ReadWriteOncePod	Exactly one pod	The strict version, for anything that must never have two writers

ReadWriteOnce is per node, not per pod — a detail that matters, because two pods on the same node can both mount it, which is a data-corruption risk for software that assumes a single writer. ReadWriteOncePod is the fix.

Other volume types:

emptyDir — scratch space living and dying with the pod, shared by its containers. medium: Memory makes it a tmpfs, and that memory counts against the pod's limit, so a large in-memory scratch directory can OOM-kill you in a way that looks unrelated.

hostPath — a path on the node. Almost always wrong: it ties the pod to one machine, and a writable hostPath is a straightforward route to compromising the node. Legitimate only for node-level agents in a DaemonSet, and restricted policies block it.

Volume snapshots — VolumeSnapshot objects create point-in-time copies through the CSI driver, and a new claim can be restored from one. This is the correct backup primitive for stateful workloads in Kubernetes, and tools like Velero build cluster-wide backup on top of it.

4. Controlling where pods land

nodeSelector — the simple one:

nodeSelector: { agentpool: gpu }

Node affinity — the expressive one:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:      # (1)
      nodeSelectorTerms:
        - matchExpressions:
            - { key: kubernetes.io/os, operator: In, values: [linux] }
    preferredDuringSchedulingIgnoredDuringExecution:     # (2)
      - weight: 100
        preference:
          matchExpressions:
            - { key: node.kubernetes.io/instance-type, operator: In, values: [Standard_D8s_v5] }

(1) Hard: no matching node, no scheduling.(2) Soft: prefer, but schedule anyway.

IgnoredDuringExecution in both names means what it says: if the node's labels change later, a running pod is not moved.

Pod anti-affinity — keeping replicas apart:

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels: { app: checkout-api }
      topologyKey: kubernetes.io/hostname     # (1)

(1) topologyKey is the axis of separation — one pod per hostname here, or use topology.kubernetes.io/zone for one per zone.

Without this, three replicas can land on one node, and one machine failure takes the whole service down. That is the Chapter 13.1 availability-zone mistake repeated one layer lower.

Topology spread constraints — the modern, better version:

topologySpreadConstraints:
  - maxSkew: 1                                  # (1)
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway           # (2)
    labelSelector:
      matchLabels: { app: checkout-api }

(1) The counts across zones may differ by at most one.(2) ScheduleAnyway prefers the spread but does not block scheduling if it cannot be achieved. DoNotSchedule makes it a hard rule — and makes pods stay Pending when a zone is out of capacity, which is usually worse than an uneven spread.

Anti-affinity is expensive to evaluate at large scale because it compares against every other pod. Topology spread is cheaper and expresses the real intent — even distribution — better.

Taints and tolerations — the inverse:

kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

tolerations:
  - { key: workload, operator: Equal, value: gpu, effect: NoSchedule }

A taint repels every pod that does not tolerate it. This is how you reserve nodes: expensive GPU nodes for the workloads that need them, or a node pool for one team.

Three effects: NoSchedule (no new pods), PreferNoSchedule (avoid if possible), NoExecute (also evicts pods already running that do not tolerate it).

Selectors and taints are complementary, and you usually want both. A toleration lets a pod onto a tainted node; it does not send it there. Without a node selector as well, the pod may land anywhere.

Priority and preemption:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: critical }
value: 1000000
globalDefault: false

A pending high-priority pod can evict lower-priority ones to make room. This is how you guarantee that a payment service always gets capacity while batch work absorbs any shortage — an idea taken straight from Borg (Chapter 13.6.8).

PodDisruptionBudget — the object that makes maintenance safe:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: checkout-api }
spec:
  minAvailable: 2                     # (1)
  selector:
    matchLabels: { app: checkout-api }

(1) At least two must remain available during voluntary disruptions — a node drain, a cluster upgrade, an autoscaler removing a node.

Without one, a routine node upgrade can drain every replica of a service at once, and the outage looks like a platform failure rather than a missing four-line object. It is the clearest example in Kubernetes of the system doing exactly what you declared, which included nothing.

Two warnings. minAvailable equal to the replica count blocks all maintenance permanently — drains hang forever waiting for permission that will never come. And a budget only constrains voluntary disruptions; a node losing power ignores it entirely.

5. Autoscaling: four layers

Horizontal Pod Autoscaler — more pods

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: checkout-api }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout-api }
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }   # (1)
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300                            # (2)
      policies: [{ type: Percent, value: 50, periodSeconds: 60 }]
    scaleUp:
      stabilizationWindowSeconds: 0                              # (3)
      policies: [{ type: Percent, value: 100, periodSeconds: 30 }]

(1) 70% of the CPU request, not of the node. This is why CPU requests are mandatory for CPU-based scaling — with no request there is no denominator and the autoscaler simply does not work.

(2) Wait five minutes of sustained lower load before scaling down, so a brief dip does not remove capacity you are about to need.

(3) Scale up immediately, and by up to 100% every 30 seconds. Asymmetric behaviour is correct: being slow to add capacity costs you an outage, being slow to remove it costs a little money.

The algorithm itself is one line:

desired = ceil(current × (currentMetric / targetMetric))

with a tolerance band (10% by default) so it does not react to noise.

Vertical Pod Autoscaler — better requests

Adjusts requests based on observed usage. In Off mode it only recommends, and that is how most teams should use it: run it for a fortnight, read the recommendations, set requests deliberately.

Automatic mode conflicts with the HPA on the same resource — one changes replicas based on usage against requests while the other changes the requests — so do not point both at CPU.

Cluster Autoscaler — more machines

Watches for pods that cannot be scheduled and adds nodes; removes nodes that have been underused for a while.

The lag is the sum of everything: notice (up to a minute) + provision the VM (1–3 minutes) + join the cluster + pull the image + start + pass readiness. Realistically three to six minutes. This is why the HPA must scale up before the node pressure arrives, and why a minReplicas with genuine headroom is not waste.

Scale-down is blocked by more than people expect: pods without a controller, pods with local storage, pods whose PodDisruptionBudget will not permit eviction, and pods with restrictive affinity. A node that will not go away is usually one pod holding it, and the autoscaler's log names which.

KEDA — scale on anything, including to zero

The Kubernetes Event-Driven Autoscaler scales on external signals: queue length, stream lag, database rows, a schedule, a cloud metric.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef: { name: order-worker }
  minReplicaCount: 0                    # (1)
  maxReplicaCount: 50
  triggers:
    - type: azure-servicebus
      metadata: { queueName: orders, messageCount: "20" }   # (2)

(1) Scale to zero when the queue is empty — impossible with the plain HPA, which has a minimum of one.

(2) One replica per 20 queued messages.

Queue depth is a far better signal than CPU for worker workloads, because it measures the backlog directly rather than a symptom of it. A worker waiting on slow I/O has low CPU and a growing queue, and CPU-based scaling does exactly the wrong thing.

And the newest layer: node autoprovisioning. Rather than scaling fixed node pools, a provisioner looks at the pending pods and creates a machine of the right size and type for them, including spot instances. Karpenter is the widely known implementation, and AKS ships it as Node Autoprovisioning (Chapter 13.6.15). It typically both improves packing and cuts the scale-up delay.

What the interviewer will push on

"Requests versus limits?" Requests are for scheduling and are not enforced; limits are enforced by cgroups. Then the difference that matters: CPU is compressible so exceeding it throttles — a slow service with no errors — and memory is not, so exceeding it kills with OOMKilled and exit code 137. Then the recommendation: memory request equal to limit, CPU request set, CPU limit usually omitted because it throttles you even on an idle node while requests already guarantee your share.

"A service is slow, CPU looks fine, no errors. Where do you look?" CPU throttling. The average usage graph hides it because throttled time is not busy time. Check the throttling metric, then either raise or remove the CPU limit. This is the single most valuable diagnostic on this page.

"What happens to a pod with no resource declarations?" It is BestEffort and is evicted first when a node runs short. Fix with a LimitRange giving namespace defaults, and note that a ResourceQuota on CPU or memory forces every pod to declare requests.

"Pods are Pending and the cluster is half empty. Why?" Requests do not fit anywhere (the scheduler uses requests, not usage), an untolerated taint, an unsatisfiable affinity rule, or a volume bound in a zone with no capacity — which volumeBindingMode: WaitForFirstConsumer prevents. Naming the volume case is what marks experience.

"What does a PodDisruptionBudget protect against?" Voluntary disruptions — node drains, upgrades, autoscaler scale-down — taking too many replicas at once. Then the two warnings: minAvailable equal to the replica count blocks maintenance forever, and it does nothing about a node losing power.

"How would you scale a queue worker?" KEDA on queue depth, scaling to zero when idle. CPU is the wrong signal because a worker blocked on I/O has low CPU and a growing backlog. Then mention the total scale-up lag — HPA reaction plus node provisioning plus image pull is minutes, so headroom in minReplicas is not waste.

One thing to volunteer: point out that CPU-based autoscaling silently does nothing when CPU requests are unset, because the target is a percentage of the request and there is no denominator. The HPA reports unknown metrics and the service simply never scales — a failure that is invisible until the traffic that needed it arrives.

Recall

• Requests schedule (and are not enforced); limits are enforced by cgroups. CPU compressible → throttled: slow, no errors, and invisible on a usage graph. Memory not compressible → OOMKilled, exit 137, no graceful shutdown.
• Memory request = limit. Set CPU requests. Usually omit CPU limits — a limit throttles you on an idle node while requests already guarantee your share. Mi ≠ M.
• QoS is derived: requests = limits everywhere → Guaranteed (evicted last); some set → Burstable; nothing set → BestEffort, evicted first. LimitRange gives defaults; ResourceQuota forces every pod to declare requests.
• Mount config as files, not environment variables. A ConfigMap change restarts nothing — use a checksum annotation on the pod template. subPath mounts never update. Mark stable ones immutable: true.
• Secrets are base64, not encrypted. Encryption at rest, tight RBAC, and prefer the Secrets Store CSI driver (value never becomes a cluster object) or workload identity, so there is no secret at all.
• volumeBindingMode: WaitForFirstConsumer stops a disk being created in a zone the pod cannot use. reclaimPolicy: Retain for anything valuable. ReadWriteOnce is per node, not per pod — ReadWriteOncePod is the strict one. hostPath is almost always wrong.
• Anti-affinity or topology spread, or three replicas land on one node. Taints repel; tolerations permit but do not attract — pair with a selector. PriorityClass enables preemption.
• A PodDisruptionBudget is what makes a drain safe — and minAvailable = replicas blocks maintenance forever, and it does nothing for an involuntary failure.
• HPA targets a percentage of the request, so no request means no scaling at all. Asymmetric behavior: scale up fast, down slowly. VPA in recommendation mode. Cluster Autoscaler lag is 3–6 minutes. KEDA scales on queue depth and to zero — the right signal for workers.

Self-test: What is the observable difference between exceeding a CPU limit and a memory limit? · Why is a CPU limit usually a bad idea on an idle node? · What one line prevents zone-stranded volumes? · What does a toleration not do? · Why does an HPA silently do nothing? · Which pods are evicted first and what one omission causes it?

Next: 13.6.14 covers everything around the manifests — Helm and its honest weaknesses, operators and custom resources, GitOps, RBAC and admission policy, and the debugging order that finds a broken pod in three commands.