13.6.16 — AKS and the Rest of Azure: One System, End to End

A customer taps Pay. Between that tap and the confirmation screen, fourteen Azure services and about nine Kubernetes objects do something.

This page follows that request through all of them, then follows a code change from a pull request to those same pods, then prices it, then finishes with the question worth asking before any of it: should this have been a cluster at all?

1. The system

Figure 1 — One checkout system. The cluster is in the middle and is the least interesting part. What makes it production-grade is what surrounds it: identity with no passwords, private links to data services, a registry the nodes can pull from without credentials, and a pipeline that never holds a cluster credential.

2. Following the tap

① Front Door and the WAF. The request lands at Azure's edge, closest to the customer. TLS terminates here, static assets are served from cache (Chapter 13.4), and the web application firewall drops obvious attacks before they cost you a single cycle of compute.

② Application Gateway. Inside your virtual network. A second WAF layer for anything Front Door passed, and the entry point into the private network where the cluster lives.

③ The ingress controller. A pod in the cluster. It reads the hostname and path and picks a Service (Chapter 13.6.12).

④ The Service. A virtual IP that exists on no interface. kube-proxy's rules rewrite the destination to one of the six ready api pods, and the connection goes straight there.

⑤ The pod. Spread across three availability zones by topology spread constraints, kept off one node by the same, scaled by an HPA between 6 and 30 (Chapter 13.6.13).

⑥ The database. Azure Database for PostgreSQL Flexible Server, reached over a private endpoint — an address inside your VNet, so traffic never touches the public internet. And no password: the pod uses workload identity to obtain an Entra token and authenticates with it (Chapter 13.6.15).

⑦ Secrets. What genuinely must be a secret — a payment provider's key — is in Key Vault, mounted as a file by the CSI driver, so it never becomes a Kubernetes Secret and never sits in etcd.

⑧ The queue. The API writes an order event to Azure Service Bus and returns. The customer's request finishes here. Everything after this is asynchronous.

⑨ The worker. KEDA sees the queue depth and scales the worker deployment from zero. Workers generate the invoice, write the PDF to Blob Storage, and send the confirmation email.

⑩ Telemetry. Every step emitted structured logs to stdout, collected into Log Analytics; metrics went to managed Prometheus; the trace spans the whole path through OpenTelemetry.

Notice what is not in that list: not one password, connection string, or certificate was stored in the cluster. Every hop used either a private network path or a token obtained from an identity. That is the difference between a demonstration and a production system, and it is achievable because AKS integrates with Entra ID properly.

3. The manifests, with the Azure pieces marked

apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout
  namespace: shop
  annotations:
    azure.workload.identity/client-id: "8b1e...-...-a92f"      # (1)
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata: { name: checkout-kv, namespace: shop }
spec:
  provider: azure
  parameters:
    clientID: "8b1e...-...-a92f"                                # (2)
    keyvaultName: shop-kv
    objects: |
      array:
        - |
          objectName: payment-provider-key
          objectType: secret
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout-api, namespace: shop }
spec:
  replicas: 6
  selector: { matchLabels: { app: checkout-api } }
  strategy:
    rollingUpdate: { maxSurge: 2, maxUnavailable: 0 }
  template:
    metadata:
      labels:
        app: checkout-api
        azure.workload.identity/use: "true"                     # (3)
    spec:
      serviceAccountName: checkout
      topologySpreadConstraints:                                # (4)
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: checkout-api } }
      nodeSelector: { workload: general }
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: api
          image: shopacr.azurecr.io/checkout-api@sha256:9f2b... # (5)
          env:
            - name: PGHOST
              value: shop-pg.postgres.database.azure.com
            - name: SERVICEBUS_NAMESPACE
              value: shop-bus.servicebus.windows.net            # (6)
          volumeMounts:
            - { name: kv, mountPath: /mnt/secrets, readOnly: true }
            - { name: tmp, mountPath: /tmp }
          resources:
            requests: { cpu: "200m", memory: "512Mi" }
            limits:   { memory: "512Mi" }                       # (7)
          readinessProbe:
            httpGet: { path: /ready, port: 3000 }
          livenessProbe:
            httpGet: { path: /healthz, port: 3000 }             # (8)
          lifecycle:
            preStop: { exec: { command: ["sleep", "5"] } }      # (9)
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
      volumes:
        - name: kv
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes: { secretProviderClass: checkout-kv }
        - name: tmp
          emptyDir: {}
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: checkout-api, namespace: shop }
spec:
  minAvailable: 4                                               # (10)
  selector: { matchLabels: { app: checkout-api } }

(1) Links this service account to an Azure managed identity. The federated credential in Azure trusts system:serviceaccount:shop:checkout (Chapter 13.6.15).

(2) The same identity, used by the CSI driver to read Key Vault. No credential is stored to read the secret store.

(3) The label the webhook looks for. It injects the projected token and the environment variables the Azure SDKs read, which is what makes DefaultAzureCredential() work with nothing configured.

(4) Spread evenly across the three zones, ScheduleAnyway so a zone shortage does not leave pods Pending (Chapter 13.6.13).

(5) Digest, not a tag (Chapter 13.6.4), from a registry the nodes' kubelet identity may pull from with no image pull secret.

(6) A namespace, not a connection string. The SDK obtains a token from the workload identity.

(7) Memory request = limit; CPU request set, no CPU limit — the reasoning is in Chapter 13.6.13.

(8) /healthz checks the process only. /ready checks the database. Getting this backwards is the outage in Chapter 13.6.10.

(9) Five seconds so endpoint removal propagates before SIGTERM (Chapter 13.6.11).

(10) Four of six must stay up during any drain, which is what makes an AKS node upgrade a non-event.

4. The path from a commit to those pods

# .github/workflows/deploy.yml
- uses: azure/login@v2
  with:
    client-id: ${{ secrets.AZURE_CLIENT_ID }}     # (1)
    tenant-id: ${{ secrets.AZURE_TENANT_ID }}
    subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

- run: |
    az acr login -n shopacr
    docker buildx build --platform linux/amd64 \
      -t shopacr.azurecr.io/checkout-api:${{ github.sha }} --push .   # (2)

- run: |
    DIGEST=$(az acr manifest show-metadata \
      -r shopacr -n checkout-api:${{ github.sha }} --query digest -o tsv)
    cosign sign --key azurekms://... "shopacr.azurecr.io/checkout-api@$DIGEST"  # (3)
    yq -i ".images[0].digest = \"$DIGEST\"" overlays/prod/kustomization.yaml
    git commit -am "checkout-api ${{ github.sha }}" && git push          # (4)

(1) Federated credentials again, this time for the pipeline. GitHub's OIDC token is exchanged for an Azure token, so there is no secret in GitHub — the secrets.* values here are identifiers, not passwords.

(2) Build and push to Azure Container Registry.

(3) Sign the digest, so admission policy can require a valid signature before anything runs (Chapter 13.6.14).

(4) The pipeline's last act is a Git commit. It never touches the cluster and holds no cluster credential.

Then Flux, running inside the cluster, notices the commit, applies the manifests, and the Deployment controller runs the rolling update (Chapter 13.6.9). The blast radius of a compromised CI system is now "can push a commit that a human reviews", not "can do anything to production".

5. Which Azure service is doing what

Service	Its job here	Why not do it in the cluster
Front Door	Global edge, TLS, caching, WAF	Blocks attacks before they cost compute
Application Gateway	Regional WAF, VNet entry	Inspection outside the cluster
Entra ID	Authentication for people and workloads	The identity system already exists
Key Vault	Secrets and certificates, with rotation and audit	Kubernetes Secrets are base64, not encrypted
ACR	Images, scanning, geo-replication, cached upstreams	Also solves public registry rate limits
PostgreSQL Flexible Server	The database, with backups, HA and patching	Running a database well is a specialist job
Service Bus	Durable queue with dead-lettering	Message loss is unacceptable and durability is hard
Blob Storage	Invoices and static assets	Object storage is not a cluster's job
Azure Monitor	Logs, metrics, alerts, dashboards	Kubernetes ships none of this
Defender for Containers	Image scanning, runtime detection, posture	Security posture across everything, not one cluster
Azure Policy	Gatekeeper policies applied fleet-wide	Central enforcement, one place to see compliance

The pattern in that right-hand column is worth naming: the cluster runs your code, and everything with a durability, security or global-scale requirement is a managed service. That is not a compromise, it is the design.

And the strongest single reason for it: the state. A cluster you can delete and rebuild in twenty minutes is a cluster you can upgrade fearlessly, replace when a permanent decision turns out wrong (Chapter 13.6.15), and reason about simply. Every stateful thing you put inside it takes that property away.

6. What it costs

A realistic monthly figure for this system, moderate traffic, one region:

Item	Rough monthly
AKS Standard control plane	$73
3 system nodes (D4s_v5)	$420
3–8 user nodes (D8s_v5), averaging 5	$1,400
PostgreSQL Flexible Server, HA	$500
Application Gateway WAF v2	$250
Front Door Standard	$40
Log Analytics ingestion (~50 GB/day)	$350
ACR Premium	$50
Service Bus Standard, Blob, disks, IPs, egress	$150
Total	≈ $3,200

Two lines deserve comment. Log ingestion at $350 is more than the control plane and the registry combined, and it grows with your log level, not with your traffic — dropping debug logging in production is often the single largest saving available. And the user nodes are the biggest line, which is why right-sizing resource requests returns more than any other optimisation (Chapter 13.6.13).

The same workload on Azure Container Apps, which is the managed container service, would run closer to $400–700 — because there are no nodes, no system pool, and no cluster.

That gap is the whole question of the next section.

7. Was a cluster the right answer?

Be honest about it, because for a large share of teams it is not.

Azure Container Apps runs the same container image. It gives you rolling deploys, revisions with traffic splitting, autoscaling including to zero, HTTPS with certificates, Dapr, KEDA-based scaling, managed identity, and private networking. There is no cluster: no node pools, no upgrades, no CNI decision, no etcd, no kubectl.

Choose Container Apps when: you run a handful of services, have no platform team, want scale-to-zero, and have no unusual scheduling needs. For a team of eight running six services, this is the correct answer, and picking it is not a compromise — it removes an entire operational domain.

Choose AKS when at least two of these are true: you run many services across several teams · you need advanced scheduling (GPUs, specific hardware, batch alongside serving) · you need per-team isolation with quotas and policy · you want one operational model across clouds and on-premises · you depend on ecosystem software packaged as operators · you have people whose job includes the platform.

And the fact that makes deferring easy, repeated because it is the most useful sentence in this Part: the container is the same artefact. The image running on Container Apps is the image a pod runs. Moving later means writing manifests, not rewriting software. Adopting Kubernetes early costs a year of operating instead of building; adopting it late costs a few weeks of YAML.

8. If you do run it, the checklist

Cluster: Standard tier · three availability zones · private cluster or authorised IP ranges · Azure CNI Overlay with a policy engine · address ranges planned against every network you will ever peer with · a tainted system pool separate from user pools · auto-upgrade and node OS channels with a maintenance window.

Identity: Entra ID authentication · Azure RBAC or tight Kubernetes RBAC · workload identity everywhere, so no application credential exists · automountServiceAccountToken: false where the API is not called.

Workloads: requests and limits on everything (memory request = limit, no CPU limit) · readiness and liveness that mean different things · PodDisruptionBudgets · topology spread across zones · preStop sleep · digest-pinned images · non-root, read-only root filesystem, all capabilities dropped.

Platform: GitOps as the only path to the cluster · admission policy denying :latest, unapproved registries, missing requests and unsigned images · a default-deny NetworkPolicy per namespace · secrets from Key Vault through the CSI driver · Container Insights with sensible filtering, managed Prometheus, and events shipped somewhere before they expire.

Data: managed services for the database, the queue and object storage · Retain reclaim policy for anything valuable · volume snapshots for what stays in the cluster.

What the interviewer will push on

"Design a production system on AKS." Walk the request path — edge and WAF, gateway, ingress, Service, pods across zones, private endpoints to managed data services — and then name the thing that makes it production-grade: no stored credentials anywhere, because workload identity and private endpoints replace them. Finish with the pipeline never holding a cluster credential because GitOps reverses the direction.

"Where do your secrets live?" Key Vault, mounted by the Secrets Store CSI driver so they never become Kubernetes Secrets in etcd, with the driver itself authenticating by workload identity. Then the better answer underneath: most "secrets" disappear entirely — the database, the queue and storage all accept Entra tokens, so there is no connection string to protect.

"Why is the database not in the cluster?" Because a cluster you can delete and rebuild in twenty minutes is worth far more than the savings, and every stateful thing inside removes that property. Managed PostgreSQL brings backups, HA, failover and patching that an operator would make you own. It works in-cluster with a good operator, and it is still not the default choice.

"How does a deploy reach the cluster?" CI builds, pushes and signs an image, then commits a digest to a repository. A GitOps controller inside the cluster reconciles. CI never holds cluster credentials, so a compromised pipeline can propose a change a human reviews, not act on production.

"AKS or Container Apps?" Container Apps for a handful of services with no platform team — same image, rolling deploys, scale-to-zero, no cluster. AKS when you have many services, several teams, unusual scheduling, or operator-packaged software. And the point that ends the argument: the container is the same artefact, so migrating later is manifests rather than a rewrite.

"What is the first thing you would fix on an inherited cluster?" Resource requests, because they decide both stability and cost — no requests means BestEffort and first to be evicted, oversized requests reserve capacity nobody uses on every node. Then PodDisruptionBudgets and the readiness/liveness distinction, because those two together cause most self-inflicted outages.

One thing to volunteer: point out that the cluster is the least interesting part of this architecture. What makes it production-grade is the identity model, the private network paths and the deployment direction — and every one of those is a decision made outside Kubernetes. Teams that treat AKS as the whole design end up with a well-run cluster full of stored passwords.

Recall

• The request path: Front Door + WAF → Application Gateway → ingress controller → Service → pods across three zones → private endpoints to PostgreSQL, Service Bus and Blob. No password anywhere — workload identity supplies tokens.
• Everything durable is a managed service. The cluster runs code; the database, queue, object storage, secrets, registry and monitoring are Azure services. The payoff is a cluster you can delete and rebuild in twenty minutes.
• Key manifest lines: service account annotated with a client ID · azure.workload.identity/use: "true" label · Key Vault via the Secrets Store CSI driver, so no Kubernetes Secret exists · digest-pinned image from ACR pulled by the kubelet identity · topology spread · memory request = limit · /healthz process-only, /ready dependency-aware · preStop sleep · PodDisruptionBudget.
• The pipeline holds no cluster credential. OIDC federation for Azure login, build, push, sign the digest, commit the digest to Git, and a GitOps controller inside the cluster reconciles. A compromised CI can propose, not deploy.
• Cost shape: nodes dominate, and log ingestion is often larger than the control plane and registry combined and scales with log level, not traffic. Right-sizing requests returns the most.
• The same system on Azure Container Apps costs roughly a fifth and has no cluster to operate. Choose it for a handful of services with no platform team. Choose AKS for many services, several teams, unusual scheduling, or operator-packaged software.
• The container is the same artefact, so migrating to Kubernetes later is manifests, not a rewrite. Early adoption costs a year; late adoption costs weeks.
• The inherited-cluster fixes, in order: resource requests · PodDisruptionBudgets · the readiness/liveness distinction · default-deny network policy · admission policy · events shipped before they expire.

Self-test: Which credentials exist in this system, and why is the answer almost none? · Why is the database outside the cluster? · Which direction does the deploy travel, and what does that buy? · Which cost line grows with configuration rather than traffic? · What makes deferring Kubernetes cheap? · Name the first three things to fix on a cluster you have just inherited.

Next: 13.7 covers the pipelines that build and ship all of this — build versus release, artefacts, GitHub Actions, Jenkins and Azure DevOps, and a complete enterprise-grade CI/CD lab you can run for free on your own machine.