13.6.15 — AKS: What Azure Adds on Top of Kubernetes

Two commands and you have a cluster:

az group create -n shop-rg -l centralindia
az aks create -g shop-rg -n shop-aks --node-count 3 --generate-ssh-keys
az aks get-credentials -g shop-rg -n shop-aks
kubectl get nodes

Four minutes. No etcd to install, no certificates to generate, no API server to configure, no control plane to keep alive at 3 a.m.

And a cluster created with those defaults is not one you should run a business on. Half the decisions that matter were made for you, and several of them cannot be changed afterwards without building a new cluster. This page is what Azure runs on your behalf, every component it adds on top of upstream Kubernetes, and — the section to read twice — which choices are permanent.

1. What "managed" actually means

Azure runs the control plane in its own subscription, invisible to you. The API server, etcd, the scheduler, the controller manager and the cloud controller manager (Chapter 13.6.9) all run on Microsoft-operated infrastructure. You never see them, cannot SSH to them, and are not billed for the virtual machines underneath them.

What Azure does for you: runs the control plane across availability zones, patches and upgrades it, backs up and maintains etcd, keeps quorum, rotates the certificates, scales the API server with the cluster, and provides the endpoint.

What is still yours: the nodes, everything running on them, networking design, storage, identity, security policy, cost, upgrades of the node pools, and every application concern.

The tiers, and the difference is a real one:

Tier	What you get
Free	A service-level objective of 99.5% on the API server. No financial guarantee.
Standard	A financially backed SLA — 99.95% with availability zones, 99.9% without — plus support for larger clusters
Premium	Standard, plus long-term support: about two years on a Kubernetes version instead of about one

Use Standard for anything in production. The cost is roughly a tenth of a dollar per cluster-hour — about seventy dollars a month — and the Free tier's control plane is explicitly not guaranteed. Premium is for organisations that genuinely cannot upgrade three times a year.

And note what the SLA covers: API server availability. Your workloads keep running through a control plane outage (Chapter 13.6.9); what you lose is the ability to change anything.

2. The two resource groups

This confuses everyone exactly once. Creating a cluster creates two resource groups:

Your group — holds the Microsoft.ContainerService/managedClusters object. One resource.

The node resource group, named MC_<your-group>_<cluster>_<region> — holds everything Azure creates on the cluster's behalf: the virtual machine scale sets for the nodes, the load balancers, the public IPs, the managed disks, the network security groups, the route tables.

Rules for it: do not rename it, do not delete anything in it by hand, do not apply a deny-assignment lock to it. Azure manages the contents, and manual edits are reconciled away or break the cluster. You may set its name at creation and apply tags to it, and that is about it.

Deleting the cluster deletes this group and everything in it — including any disk holding data whose storage class used the default Delete reclaim policy (Chapter 13.6.13).

3. Node pools

A node pool is a virtual machine scale set of identically configured nodes. A cluster has one or more.

System pools versus user pools:

System pools host the cluster's own workloads — CoreDNS, the metrics server, the CSI drivers. Every cluster needs at least one, it must have at least one node, and it must run Linux.

User pools run your applications and may scale to zero.

The pattern worth adopting on any cluster that matters: a small system pool tainted CriticalAddonsOnly=true:NoSchedule, and separate user pools for applications. This stops an application pod that misbehaves from starving CoreDNS, and it means scaling application capacity never touches the cluster's own services.

az aks nodepool add -g shop-rg --cluster-name shop-aks -n apps \
  --node-vm-size Standard_D8ds_v5 \
  --node-count 3 --zones 1 2 3 \                     # (1)
  --enable-cluster-autoscaler --min-count 3 --max-count 20 \
  --os-sku AzureLinux \                              # (2)
  --node-osdisk-type Ephemeral \                     # (3)
  --max-pods 110 \                                   # (4)
  --labels workload=general --node-taints ""

(1) Spread across three availability zones. Nodes are distributed and, with topology spread constraints on your workloads (Chapter 13.6.13), your replicas land in different zones. Zones cannot be added to an existing pool — you create a new pool.

(2) Azure Linux is Microsoft's own container-host distribution: smaller, fewer packages, therefore a smaller attack surface and faster to patch. Ubuntu remains the alternative and the more familiar one.

(3) An ephemeral OS disk lives on the VM's local storage rather than a network disk. Faster, free, and it disappears when the node does — which is exactly right, because a node is disposable. Use it whenever the VM size has enough local cache.

(4) Maximum pods per node. The default depends on the network plugin, and it is a permanent choice for that pool. Set too low and you waste capacity; too high and you exhaust IP addresses.

Other pool options worth knowing:

Spot pools run on Azure's spare capacity at a large discount and can be evicted with 30 seconds' notice. Correct for batch, CI runners and anything interruptible; never for a stateful service. They arrive tainted so nothing lands there without a toleration.

Windows pools for .NET Framework applications that cannot run on Linux. They cannot be system pools, and Windows images are large, so pulls are slow.

GPU pools for machine learning, with the device plugin exposing GPUs as a schedulable resource (Chapter 13.6.10).

4. Networking — the decisions that are hardest to reverse

This is the section that determines whether you are rebuilding the cluster in a year.

Option	Where pod IPs come from	Use when
Azure CNI Overlay	A private range outside the VNet	The default choice now
Azure CNI (node subnet)	Real VNet addresses	Something outside must reach pods directly
Azure CNI Powered by Cilium	Overlay, with eBPF data plane	You want eBPF policy and performance
kubenet	Overlay with route tables	Legacy — do not choose for anything new

Why overlay became the default is an address-arithmetic argument. With traditional Azure CNI every pod consumes a VNet address. A 50-node cluster at 30 pods per node needs 1,500 addresses plus headroom for upgrades — and a /24 subnet holds 254. Teams ran out of addresses and could not grow, and a VNet address plan is not something you casually redo.

Overlay gives pods addresses from a private range that exists only inside the cluster. Nodes hold VNet addresses; pod traffic between nodes is encapsulated. You can run tens of thousands of pods on a small node subnet. The cost is that pods are not directly addressable from the VNet — which is fine, because traffic should arrive through a Service or an Ingress anyway.

Choose the node-subnet mode only when something outside genuinely must reach a pod IP directly, which is rare and usually a legacy integration.

Network policy: AKS offers Azure NPM, Calico, or Cilium. Chapter 13.6.12 applies unchanged. The decision that matters is making one, because with no policy engine the NetworkPolicy objects you write are accepted and ignored.

Outbound traffic has three shapes:

A load balancer (default) — outbound goes through the cluster's public load balancer, which rewrites the source address of every outgoing packet to its own. That rewrite is SNAT, source network address translation (Chapter 13.6.12 defines NAT), and because many pods now share one address, each outbound connection has to be told apart by a port number — and there is a finite supply of them per destination. The failure mode to know is SNAT port exhaustion: enough simultaneous connections to one destination use up those ports, and new connections fail intermittently while existing ones keep working. It looks exactly like a flaky partner API, which is why it costs a day the first time.

A NAT gateway — far more outbound ports, and the right choice for anything making many outbound calls.

User-defined routing — you send everything through your own firewall. Standard in regulated environments, and it requires allowing the endpoints AKS itself needs.

Locking down the API server, in increasing order of strictness:

Authorised IP ranges — the public endpoint only answers listed addresses. Simple, effective, and remember to include your CI runners' egress addresses.

API Server VNet Integration — the API server gets an address in your VNet, so cluster traffic to it never leaves the network.

Private cluster — no public endpoint at all, reached through a private endpoint and private DNS. The operational consequence is real: kubectl only works from inside the network, so you need a VPN, a bastion, a self-hosted CI runner, or the az aks command invoke escape hatch.

5. Ingress and the mesh

Four ways in, and they are genuinely different products:

The application routing add-on — a managed NGINX ingress controller, with optional automatic DNS records and certificates from Key Vault. The simplest sensible answer, and it is the managed version of the controller most people would install anyway.

Application Gateway Ingress Controller (AGIC) — an Azure Application Gateway acts as the ingress. The reason to choose it is the web application firewall, which inspects traffic outside the cluster before it arrives. The trade is that it is one gateway per cluster and configuration updates take longer than an in-cluster controller's.

Application Gateway for Containers — the newer service, built around the Gateway API (Chapter 13.6.12), with much faster configuration updates and first-class traffic splitting.

The Istio-based service mesh add-on — a managed Istio, with Azure handling upgrades. Chapter 13.6.12's assessment stands: adopt it for a specific problem, usually mutual TLS everywhere or per-request balancing for gRPC.

And a small detail that saves an afternoon: a LoadBalancer Service is public by default. For an internal one:

metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

6. Identity — the part AKS does best

Four different identity mechanisms, and mixing them up is the most common confusion in AKS.

① The cluster identity. A managed identity the cluster itself uses to create load balancers, disks and network resources in the node resource group. Managed for you.

② The kubelet identity. What nodes use to pull images from a container registry. --attach-acr grants it the AcrPull role on a registry, which is why the pull "just works" with no image pull secret anywhere.

③ Entra ID for authentication — who you are.

az aks create ... --enable-aad --aad-admin-group-object-ids <group-id>

Now kubectl authenticates with your corporate account, including conditional access and multi-factor authentication. This is the answer to Chapter 13.6.9's observation that Kubernetes has no user database — the identity provider is Entra ID, and RBAC bindings reference Entra group IDs:

subjects:
  - kind: Group
    name: "6f8a2c14-...-9d3b"        # an Entra group object ID

④ Azure RBAC for Kubernetes authorisation — what you may do. Optionally, instead of managing Kubernetes RBAC objects, grant Azure roles such as Azure Kubernetes Service RBAC Reader or Writer at a subscription, resource group, cluster or namespace scope. Permissions then live in the same place as the rest of your Azure access, which is a genuine operational simplification for a large organisation.

Workload identity — the one to actually learn

The problem: a pod needs to read from Azure Blob Storage, or a database, or Key Vault. The old answer was a connection string in a Secret — which must be created, distributed, rotated, and never leaked.

Workload identity removes the credential entirely.

az aks update -g shop-rg -n shop-aks \
  --enable-oidc-issuer --enable-workload-identity        # (1)

az identity create -g shop-rg -n checkout-identity        # (2)

az identity federated-credential create \                 # (3)
  --name checkout-fed --identity-name checkout-identity -g shop-rg \
  --issuer "$(az aks show -g shop-rg -n shop-aks --query oidcIssuerProfile.issuerUrl -o tsv)" \
  --subject "system:serviceaccount:shop:checkout"

az role assignment create \                               # (4)
  --assignee <identity-client-id> \
  --role "Key Vault Secrets User" --scope <key-vault-id>

(1) The cluster becomes an OpenID Connect identity provider, publishing signing keys at a public URL.

(2) A managed identity in Azure — a principal that can be granted roles.

(3) The link, and this is the whole mechanism. It says: a token issued by this cluster for the service account checkout in the namespace shop may be exchanged for a token for this Azure identity. The trust is on the service account name, so no secret is shared.

(4) Grant that identity real permissions in Azure.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout
  namespace: shop
  annotations:
    azure.workload.identity/client-id: "<identity-client-id>"
---
# pod template
spec:
  serviceAccountName: checkout
  labels: { azure.workload.identity/use: "true" }      # (5)

(5) A webhook sees this label and injects the token file path and the environment variables the Azure SDKs look for. Your code calls DefaultAzureCredential() and it works — with no secret in the cluster, no rotation, and nothing to leak.

This is the single most valuable AKS feature to adopt, and it replaces the older AAD Pod Identity, which is deprecated.

7. Storage

Four CSI drivers, installed and managed for you:

Driver	What it is	Access mode
Azure Disk	Block storage attached to one node	ReadWriteOnce
Azure Files	SMB or NFS shares	ReadWriteMany
Azure Blob	Object storage mounted as a filesystem	ReadWriteMany, large sequential data
Azure NetApp Files	High-performance NFS	Demanding workloads

Default storage classes exist — managed-csi (standard SSD) and managed-csi-premium. Both use reclaimPolicy: Delete, so deleting a claim destroys the disk and its data. Create your own class with Retain for anything that matters (Chapter 13.6.13).

Disks are zone-bound. A premium disk in zone 1 attaches only to a node in zone 1, so a pod using it is confined to that zone. volumeBindingMode: WaitForFirstConsumer is what stops this stranding pods, and it is set on the built-in classes.

Every VM size caps how many disks may attach to it. Hit the cap and pods stay Pending with a volume error while CPU and memory sit idle.

8. Add-ons and extensions

This is the "on top of Kubernetes" list, and each item replaces something you would otherwise install and maintain yourself.

Azure Monitor Container Insights — collects logs from every pod and container into a Log Analytics workspace, queried with KQL. The cost model is per gigabyte ingested, so a chatty application at debug level is expensive, and the first tuning most teams do is filtering namespaces and reducing log levels.

Azure Monitor managed Prometheus + Azure Managed Grafana — Prometheus metrics scraped, stored and displayed, without running the Prometheus stack yourself. Standard Kubernetes dashboards come pre-built.

Microsoft Defender for Containers — vulnerability scanning of images in the registry, runtime threat detection on nodes, and posture checks against the cluster configuration.

Azure Policy add-on — Gatekeeper (Chapter 13.6.14), driven by Azure Policy definitions. This is how a central team enforces "no privileged containers" and "images only from approved registries" across every cluster in a subscription, with compliance visible next to every other Azure policy result.

Key Vault Secrets Provider — the Secrets Store CSI driver, mounting Key Vault secrets and certificates directly as files. Combined with workload identity, no secret ever becomes a Kubernetes object.

KEDA add-on — managed event-driven autoscaling (Chapter 13.6.13), scaling on Service Bus queue depth, Event Hubs lag or Storage queue length, including to zero.

GitOps (Flux) extension — Flux installed and managed as a cluster extension, configurable across many clusters at once from Azure. This is how a platform team applies a baseline to fifty clusters.

Dapr extension — a runtime offering service invocation, state stores, pub/sub and secret access through a uniform API, as a sidecar.

Image Cleaner — removes unused and vulnerable images from nodes automatically, which otherwise slowly fills node disks (Chapter 13.6.10).

Virtual nodes — schedule pods onto Azure Container Instances instead of a VM, through a virtual kubelet. Pods start in seconds with no node provisioning, which is genuinely useful for bursty batch work. The constraints are real: no DaemonSets, limited networking, and no host access.

9. Scaling and upgrades

Cluster autoscaler per node pool, with a minimum and maximum count (Chapter 13.6.13).

Node Autoprovisioning — the Karpenter-based option that creates right-sized nodes for pending pods instead of scaling fixed pools. It usually improves packing and reduces the scale-up delay, because it picks a VM size that fits what is waiting rather than adding another of whatever the pool has.

Upgrades, and the settings that make them uneventful:

az aks update -g shop-rg -n shop-aks \
  --auto-upgrade-channel stable \                  # (1)
  --node-os-upgrade-channel NodeImage              # (2)

az aks nodepool update -g shop-rg --cluster-name shop-aks -n apps \
  --max-surge 33%                                  # (3)

(1) Automatic Kubernetes upgrades, one minor version behind the newest — rapid, stable, patch and none are the choices.

(2) Node OS image updates, which carry the security patches. Separate from the Kubernetes version and equally important.

(3) How many extra nodes may be added during an upgrade. The default of one node at a time is safe and slow; 33% is a reasonable production value. Nodes are added, workloads drained onto them, old nodes removed — so PodDisruptionBudgets are what keep your service up (Chapter 13.6.13).

Planned maintenance windows confine both to chosen hours, which is how you avoid an automatic upgrade during a sale.

10. Cost

What you pay for:

• Nodes — virtual machines, billed as virtual machines. The dominant cost.
• The control plane — free tier, or about $0.10/hour for Standard.
• Disks, load balancers, public IPs, egress bandwidth.
• Log ingestion, which surprises people more than any other line.

The four biggest savings, in order of how much they usually return:

Right-size requests. Requests reserve capacity whether or not it is used. A fleet with requests set at three times actual usage is paying for three times the nodes. Run the VPA in recommendation mode and act on it (Chapter 13.6.13).

Spot node pools for anything interruptible — 60–90% off.

Reserved instances or savings plans for the steady baseline — around 40% off for a one-year commitment.

Scale to zero — user pools with a minimum of zero, plus KEDA for queue workers, so nothing runs overnight.

11. The decisions you cannot easily change

Read this list before creating a cluster, because each of these is either impossible to change or requires a new cluster or a new node pool.

Decision	Reality
Pod CIDR, Service CIDR, DNS service IP	Fixed at creation
Network plugin and data plane	Effectively a creation-time choice
Private cluster or public	Fixed at creation
Availability zones for a node pool	Fixed — create a new pool
Node VM size for a pool	Fixed — create a new pool and migrate
max-pods per node for a pool	Fixed per pool
The node resource group name	Set at creation only
OS SKU of an existing pool	Create a new pool

Two of these deserve the most thought. Address ranges, because an overlap with your corporate network or a subnet too small to grow into is discovered months later and is close to unfixable (Chapter 13.2). And private-versus-public, because it changes how every engineer and every pipeline reaches the cluster.

The mitigation that makes all of this survivable: keep everything in Git and treat clusters as replaceable. With infrastructure as code (Chapter 13.8) and GitOps (Chapter 13.6.14), building a correct new cluster and moving to it is a planned afternoon rather than a rebuild of institutional memory.

What the interviewer will push on

"What does AKS actually manage?" The control plane — API server, etcd, scheduler, controllers — running in Microsoft's subscription, patched, backed up, kept in quorum, and free of VM charges. Everything from the nodes upward is yours. The tell of someone who has run it is knowing the Free tier has an objective rather than an SLA, and that the SLA covers API server availability while workloads keep serving through a control plane outage.

"Why did Azure move to CNI Overlay?" Because giving every pod a VNet address exhausts subnets — 50 nodes at 30 pods needs 1,500 addresses and a /24 holds 254 — and a VNet address plan cannot be casually redone. Overlay puts pod addresses in a private range so the node subnet stays small. The trade is that pods are not directly reachable from the VNet, which is fine because traffic should arrive via a Service.

"How does a pod get access to Key Vault without a secret?" Workload identity: the cluster is an OIDC issuer, a federated credential trusts a specific service account in a specific namespace, and a webhook injects the token so the Azure SDK exchanges it for an Azure token. No secret exists to rotate or leak. Add that it replaced AAD Pod Identity, which is deprecated.

"What is the node resource group and what are the rules?" The MC_* group Azure creates to hold nodes, load balancers, disks and IPs. Do not rename it, do not edit its contents by hand, do not lock it — Azure reconciles it. And deleting the cluster deletes everything in it, including disks whose storage class used the default Delete policy.

"Which AKS decisions are permanent?" Address ranges, network plugin, private-versus-public, and per-pool zones, VM size and max-pods. Then the mitigation: keep everything in Git and treat clusters as replaceable, which turns a permanent decision into a planned migration.

"How do you authenticate and authorise people?" Entra ID integration for authentication, so kubectl uses corporate accounts with conditional access, and RBAC bindings reference Entra group object IDs. For authorisation, either Kubernetes RBAC or Azure RBAC for Kubernetes, which keeps permissions alongside the rest of your Azure access.

One thing to volunteer: point out that log ingestion is the AKS cost line that surprises teams most — Container Insights bills per gigabyte, so an application logging at debug level across fifty pods can cost more than the nodes running it. Naming it shows you have owned a bill, not just a cluster.

Recall

• Azure runs the control plane in its own subscription — API server, etcd, scheduler, controllers — patched, backed up, and not billed as VMs. Free tier = an objective; Standard = a 99.95% SLA with zones; Premium = long-term support. The SLA covers the API server; workloads keep serving through a control plane outage.
• Two resource groups. The MC_* node resource group holds scale sets, load balancers, IPs and disks. Do not rename, edit or lock it. Deleting the cluster deletes it and its disks.
• System pool tainted CriticalAddonsOnly, separate user pools. Zones, VM size and max-pods are fixed per pool. Spot for interruptible work, ephemeral OS disks by default, Azure Linux for a smaller surface.
• Azure CNI Overlay is the default choice — pods get private addresses so the VNet subnet is not exhausted. Node-subnet CNI only when something outside must reach pods. kubenet is legacy. Pick a network policy engine or your policies are ignored.
• Outbound: load balancer (watch SNAT port exhaustion), NAT gateway for heavy egress, or user-defined routing through a firewall. Lock the API server with authorised IP ranges, VNet integration, or a private cluster — which means kubectl only works from inside the network.
• Four identities: cluster identity · kubelet identity (--attach-acr for image pulls) · Entra ID for authentication, with RBAC bound to Entra group IDs · Azure RBAC for authorisation.
• Workload identity is the one to learn: OIDC issuer + federated credential trusting system:serviceaccount:<ns>:<name> + a webhook injecting the token. No secret exists. It replaced AAD Pod Identity.
• Storage: Disk (ReadWriteOnce, zone-bound, per-VM attach limits) · Files and Blob (ReadWriteMany) · NetApp. Default classes are Delete — make a Retain class for real data.
• Add-ons: Container Insights (billed per GB ingested) · managed Prometheus + Grafana · Defender · Azure Policy (Gatekeeper) · Key Vault CSI · KEDA · Flux GitOps extension · Dapr · Image Cleaner · virtual nodes on ACI.
• Upgrades: auto-upgrade channel + separate node OS image channel + --max-surge + maintenance windows, made safe by PodDisruptionBudgets. Cost: right-size requests first, then spot, reservations and scale-to-zero.

Self-test: What exactly does the AKS SLA cover, and what keeps working without it? · Why does giving pods VNet addresses fail at scale? · What are the four AKS identities, and which one removes secrets entirely? · What does the federated credential's subject string trust? · Name four decisions you cannot change after cluster creation. · Which cost line surprises teams most?

Next: 13.6.16 puts it together — one real system on AKS with the registry, secrets, ingress, database, monitoring and pipeline all wired up, what each Azure service is doing in it, and an honest account of when a cluster was the wrong answer.