14.1.2 — Virtual Machines & Hypervisors

You open the Azure portal, pick a size called Standard_D4s_v5, and two minutes later you have a Linux server with 4 processors and 16 GB of memory. You log in, run nproc, and it says 4. You run free -h, and it says 16 GB.

That machine does not exist. Somewhere in a data centre there is a physical server with, say, 96 processor cores and 768 GB of memory, and your "server" is a slice of it, running side by side with the slices of a dozen other customers who will never know you are there. Your Linux kernel believes it owns real hardware. It is wrong, and it never finds out.

The program that pulls this off is called a hypervisor, and this page explains exactly how it works: how one operating system can be fooled into thinking it owns a computer, how the fooling is done safely so one customer cannot touch another, and how it is done so fast that you lose only a few percent of speed. You need this for two reasons. First, every Kubernetes node you will ever run in a cloud is a virtual machine, so containers in production almost always run inside VMs. Second, you cannot really understand what a container saves you until you know what a VM costs.

Chapter 2.9 gave the overview (2.9). This page is the mechanism underneath it. It leans on one idea from Chapter 2.1, so here it is again in two sentences: the processor runs code in either a privileged mode, where it can do anything, or an unprivileged mode, where some instructions are forbidden. The operating system kernel runs privileged, your programs run unprivileged, and when a program tries something forbidden the processor stops it and hands control to the kernel (2.1).

1. The whole trick in one paragraph

Here is the idea, and everything else on this page is detail on top of it.

Let the guest's code run directly on the real processor, at full speed, and step in only at the moments it tries to touch something that is shared. Adding two numbers, looping over an array, calling a function: none of that affects anybody else, so the hypervisor lets it run untouched. Changing the memory map, disabling interrupts, talking to a disk controller: those would affect the real machine, so the processor is arranged to stop the guest at exactly those instructions and hand control to the hypervisor. The hypervisor then does a pretend version of what the guest asked for, updating its private record of that guest's machine instead of the real one, and lets the guest carry on as if it had worked.

This is called trap-and-emulate. Trap means the processor stops the guest and jumps into the hypervisor. Emulate means the hypervisor imitates the effect the guest wanted. The guest never sees the gap, because by the time it runs its next instruction, everything looks exactly as it would on a real machine.

Why not just simulate the whole processor in software instead? You can, and that is what an emulator does: it reads each guest instruction and acts it out in code. It works for any processor on any other processor, which is how you can run an old games console on a laptop. But every single instruction now costs tens or hundreds of real instructions, so it runs ten to a hundred times slower. A hypervisor is fast precisely because it does not do this. It runs almost everything natively and only intervenes on the rare instructions that need it. The rest of this page is the story of how hard it turned out to be to catch exactly those rare instructions and nothing else.

2. Where it began: one mainframe, many people

Virtualisation is from the 1960s, not the cloud era, and the original reason is worth knowing because it is the same reason as today.

In the mid-1960s a company owned one computer, an IBM mainframe the size of a room, and many people needed to use it. The common way to share it was batch processing: you handed in your program on punched cards, waited hours, and got a printout back. Researchers at IBM's Cambridge Scientific Center in Massachusetts wanted something better. They wanted every person at a terminal to feel as if the mainframe were theirs alone.

Their answer, CP-40 in 1967 and then CP-67 on the System/360 Model 67, was the first real hypervisor. "CP" stood for Control Program. It ran directly on the hardware, and its only job was to create virtual machines. Inside each one, a user ran a small single-user operating system called CMS. If you crashed your CMS, you crashed your virtual machine and nobody else's. IBM turned this into a product, VM/370, in 1972, and it is still sold today under the name z/VM. Virtual machines on mainframes have been in continuous use for more than fifty years.

The rule that says when a processor can be virtualised

In 1974 two researchers, Gerald Popek and Robert Goldberg, wrote down exactly what a processor needs for trap-and-emulate to work. Their rule is short, and it explains the next thirty years of history, so it is worth getting precisely.

They split instructions into two groups, defined by what the instruction does:

• A sensitive instruction is one that reads or changes the state of the real machine: the memory map, the interrupt setting, the processor's mode, a device. These are the ones a hypervisor has to intercept, because letting a guest run them for real would let it affect the whole machine.
• A privileged instruction is one the processor refuses to run in unprivileged mode. Try it, and the processor traps to whoever runs in privileged mode.

The rule: a processor can be virtualised by trap-and-emulate if every sensitive instruction is also a privileged instruction. Read it slowly. If it holds, the hypervisor simply runs the guest kernel in unprivileged mode. Every sensitive thing the guest tries is, by the rule, privileged, so it traps, so the hypervisor gets control at exactly the right moment. Nothing slips through. IBM's mainframes satisfied the rule, which is why CP-67 worked.

3. Why the PC broke the rule

For about twenty-five years after that, virtualisation was a mainframe thing. The reason it did not come to ordinary servers is that the x86 processor in every PC and server breaks the Popek and Goldberg rule.

x86 has four privilege levels called rings, numbered 0 to 3. Ring 0 is the most powerful and is where the kernel runs. Ring 3 is where programs run. Rings 1 and 2 exist but almost nothing ever used them.

The obvious plan for a hypervisor is: put the hypervisor in ring 0, push the guest kernel down into ring 1, and let every sensitive instruction the guest tries trap upward. That plan fails, because x86 has instructions that are sensitive but do not trap. In 2000, John Robin and Cynthia Irvine counted them and found 17.

The clearest one is POPF, and it is worth walking through, because it shows why "just run the guest in a lower ring" cannot work. POPF loads a set of processor flags from memory. One of those flags turns hardware interrupts on or off, and the kernel uses this constantly: switch interrupts off, do something delicate, switch them back on.

• Run in ring 0, POPF sets the interrupt flag exactly as asked.
• Run in ring 1, POPF silently ignores the interrupt flag and sets the others. There is no error and no trap. The instruction just quietly does less.

So the guest kernel in ring 1 runs POPF to switch interrupts back on. The processor ignores that part. The hypervisor is never told, because nothing trapped. The guest kernel now believes interrupts are on. The hypervisor's record of this guest says they are off, so it never delivers the timer interrupt the guest is waiting for. The guest hangs, waiting forever for a tick that will never come.

Here is that exact failure as a small program. It is a toy, not a real hypervisor, but it models the one decision a hypervisor makes on every guest instruction, and you can run it with node --experimental-transform-types toy-vmm.ts:

type Instr =
  | { op: "add"; n: number }            // ordinary work: runs straight on the CPU
  | { op: "cli" }                        // "disable interrupts": privileged
  | { op: "popf"; interruptsOn: boolean } // restore flags: sensitive, but on old x86 does NOT trap

interface VirtualCpu { name: string; acc: number; interruptsOn: boolean }  // (1)

let exits = 0;

function runGuest(vcpu: VirtualCpu, program: Instr[], hardwareAssist: boolean): void {
  for (const ins of program) {
    switch (ins.op) {
      case "add":
        vcpu.acc += ins.n;                                    // (2)
        break;
      case "cli":
        exits++;                                              // (3)
        vcpu.interruptsOn = false;
        break;
      case "popf":
        if (hardwareAssist) {
          exits++;                                            // (4)
          vcpu.interruptsOn = ins.interruptsOn;
        }
        // without hardware help: the CPU silently ignores the bit,
        // nothing traps, and the hypervisor never finds out  (5)
        break;
    }
  }
}

const program: Instr[] = [
  { op: "add", n: 5 },
  { op: "cli" },
  { op: "add", n: 10 },
  { op: "popf", interruptsOn: true },   // guest kernel turns interrupts back on
];

for (const assist of [false, true]) {
  exits = 0;
  const vcpu: VirtualCpu = { name: "guest-1", acc: 0, interruptsOn: true };
  runGuest(vcpu, program, assist);
  console.log(
    `${assist ? "VT-x   " : "old x86"}: acc=${vcpu.acc} interruptsOn=${vcpu.interruptsOn} exits=${exits}`,
  );
}

old x86: acc=15 interruptsOn=false exits=1
VT-x   : acc=15 interruptsOn=true exits=2

(1) The hypervisor's private record of one guest processor. A real one holds every register and flag. This toy keeps one ordinary register, acc, and the one flag we care about. The guest never sees this object. It only sees the effects.

(2) Ordinary work runs directly. No trap, no hypervisor involvement, full speed. In a real VM this is the overwhelming majority of instructions, which is why VMs are fast.

(3) CLI (clear interrupts) is privileged on x86, so it traps even on old processors. The hypervisor gets control, counts an exit, and updates its record: this guest now has interrupts off.

(4) With hardware assistance, POPF traps too, so the hypervisor sees the guest switching interrupts back on and updates its record to match.

(5) Without it, nothing happens at all. No exit is counted, the record still says interrupts are off, and the guest's belief and the hypervisor's record have split apart. That is the bug the output shows: the first line ends with interruptsOn=false even though the guest's last instruction asked for true. The second line, with one extra exit, gets it right.

This is the whole reason x86 virtualisation was hard. The problem was never speed. It was that the processor did not reliably tell the hypervisor when it needed to step in.

4. Three ways around the broken rule

Engineers found three ways around it, in this order, and all three are still in use somewhere.

Way 1: rewrite the guest's code before it runs (VMware, 1999)

VMware, founded in 1998, shipped the first x86 hypervisor in 1999, and its trick was binary translation. Before letting a chunk of guest kernel code run, the hypervisor reads it, finds any of those 17 troublesome instructions, and replaces each one with a short sequence that calls the hypervisor instead. The rewritten chunk is cached, so the translation is paid once and the fast version is reused every time that code runs again. Guest program code in ring 3 needs no rewriting, because none of the troublesome instructions matter there, so it runs completely natively.

It is worth seeing why this is better than the pure emulator from section 1. An emulator acts out every instruction. Binary translation runs almost every instruction natively and patches only the few that would misbehave, so the cost is a small fraction of the emulator's. It let unmodified Windows and Linux run side by side on one ordinary server for the first time, and it is the technology that built the server-virtualisation industry in the 2000s. It was also enormously complicated to write and maintain, which is why only a few companies ever did it.

Way 2: change the guest so it asks politely (Xen, 2003)

A team at the University of Cambridge took the opposite approach: if the guest kernel misbehaves, edit the guest kernel. Their hypervisor, Xen, published in 2003, required a modified Linux kernel that never used the troublesome instructions. Wherever the kernel would have done something sensitive, it instead made a hypercall, a deliberate call into the hypervisor, the same way a program makes a system call into the kernel. This is called paravirtualisation: the guest knows it is virtualised and cooperates.

It was very fast, and it had one fatal limit: you had to be able to change the guest's source code. That was fine for Linux and impossible for Windows. Xen became the hypervisor underneath Amazon's cloud when it launched in 2006.

Paravirtualisation did not die. It moved to devices. Section 6 shows that almost every VM today uses paravirtual drivers for its disk and network, because cooperating is still the fastest way to move data.

Way 3: fix the processor (Intel VT-x 2005, AMD-V 2006)

The real fix came from the chip makers. Intel added VT-x in late 2005 and AMD added AMD-V in 2006. Instead of squeezing a hypervisor and a guest kernel into the same four rings, they added a whole new dimension: two modes of operation, each with all four rings inside it.

• Root mode is where the hypervisor runs.
• Non-root mode is where every guest runs. Inside it, the guest kernel really does run in ring 0, and the guest's programs really do run in ring 3. The guest is not pushed anywhere, so the POPF problem disappears: in ring 0, POPF behaves correctly.

The hypervisor then tells the processor which events should stop the guest. When one of them happens, such as the guest switching interrupts, touching a device, or changing the memory map, the processor saves the guest's state, switches to root mode and jumps into the hypervisor. That switch is called a VM exit. When the hypervisor has dealt with it, it tells the processor to resume the guest, which is a VM entry.

The guest's saved state lives in a block of memory called the VMCS (virtual machine control structure), one per virtual processor. It holds the guest's registers, the reason for the last exit, and the list of events that should cause the next one. The processor reads and writes it automatically on every exit and entry.

To see this as a loop, the figure below follows one virtual processor through one exit and back. Read it top to bottom.

Figure 1 — One exit and back. The guest runs natively until an event the hypervisor asked to see. The processor saves the guest into its VMCS and switches to the hypervisor, which handles the event and resumes the guest. A healthy VM does this thousands of times a second and spends almost all its time in ①.

Follow the numbers. At ① the guest runs untouched. At ② it does something on the hypervisor's list, and the processor saves it into the VMCS and switches modes. At ③ and ④ the hypervisor reads why, and does the pretend version. At ⑤ the processor loads the guest back from the VMCS and the guest carries on, unaware. The speed of a VM is decided almost entirely by how often ② happens and how long each trip takes.

A surprising fact from history: the first hardware support was often slower than VMware's software. A 2006 paper by two VMware engineers, Keith Adams and Ole Agesen, measured it and found that each VM exit on the first VT-x chips cost thousands of processor cycles, so a workload that exited often ran faster with clever binary translation. Every chip generation since has made exits cheaper and added ways to avoid them. By around 2010, hardware assistance was the clear winner, and today every mainstream hypervisor depends on it.

You can check whether your own processor has it. On Linux, grep -cE 'vmx|svm' /proc/cpuinfo prints how many processor threads advertise Intel's vmx or AMD's svm flag. Zero means either no support, or it is switched off in the firmware settings, which is a common reason a laptop refuses to run Docker Desktop or WSL 2.

5. Memory: an address translated twice

The processor problem is solved. Memory is the second, harder problem, and it decides whether memory-heavy software like a database runs well in a VM.

Recall how memory works without virtualisation (Chapter 2.5, 2.5). Every program uses virtual addresses. The kernel keeps a page table for each program, which maps each virtual page to a real page of physical memory. The processor walks that table on every memory access it does not already have cached, and it caches recent answers in a small, very fast store called the TLB (translation lookaside buffer).

Now add a hypervisor. The guest kernel still builds page tables, mapping its programs' addresses to what it believes are physical addresses. But those are not real physical addresses. They are the guest's private numbering, and the hypervisor has to map them again onto the real memory chips. So there are three kinds of address:

Name	Who uses it	Mapped by
Guest virtual	Programs in the guest	Guest's page tables
Guest physical	The guest kernel	Hypervisor's tables
Host physical	The real memory chips	Nobody, it is real

The old way: shadow page tables

Before hardware help, the hypervisor faked this with shadow page tables. It kept a hidden third table that mapped guest virtual addresses straight to host physical addresses, and pointed the real processor at that one. The problem is keeping it correct. Every time the guest kernel changed its own page table, which happens on every new process, every memory allocation and every page fault, the hypervisor had to notice and update the shadow to match. To notice, it marked the guest's page tables read-only so any write would trap. The result was a storm of VM exits for memory-heavy work, and this is the main reason databases had a reputation for running badly in VMs in the 2000s.

The new way: nested page tables

AMD in 2007 and Intel in 2008 taught the processor to do both translations itself. AMD calls it NPT or RVI, Intel calls it EPT (extended page tables). The guest kernel manages its own page tables freely with no exits at all. When the processor needs an address, it walks the guest's table to get a guest physical address, then walks the hypervisor's table to turn that into a host physical address, all in hardware.

This has a hidden cost, and the arithmetic is worth doing once, because it explains why "huge pages" matter so much in VMs. On a 64-bit machine, a page table has 4 levels, so a TLB miss costs 4 memory reads to walk it. With nested paging, every one of the guest's 4 steps produces a guest physical address that must itself be translated through the hypervisor's 4 levels, and so must the final answer.

Count it step by step. The guest walk has 4 levels, and before reading each level's entry the processor must translate where that entry lives, which costs a 4-read walk of the host table plus the 1 read of the entry itself, so 5 reads per level, and 4 \times 5 = 20. Then the final guest physical address, the one the program actually wanted, needs one more 4-read host walk. That gives

4 \times (4 + 1) + 4 = 24

which you read as "four guest levels, each costing a four-step host walk plus one read, plus one last four-step host walk, equals twenty-four". The general form, for a guest table with n levels and a host table with m levels, is (n+1)(m+1) - 1, and with n = m = 4 that is 5 \times 5 - 1 = 24.

So a TLB miss in a VM can cost 24 memory reads instead of 4, six times as many. That is why the TLB hit rate matters even more inside a VM, and why the standard advice for a database VM is to use huge pages: 2 MB or 1 GB pages instead of 4 KB ones. A bigger page means one TLB entry covers far more memory, so there are far fewer misses, and the page tables have fewer levels to walk.

One last memory fix: tagging the TLB. When the processor switches from one VM to another, the cached translations in the TLB belong to the wrong guest. Early on, the only safe move was to throw the whole TLB away on every switch, which made the next few thousand memory accesses slow. Intel's VPID and AMD's ASID tag every TLB entry with the owner's ID, so entries from different guests can sit side by side and nothing needs to be thrown away.

6. Devices: three ways to give a VM a network card

The processor and memory are now handled by hardware. Devices, meaning disks and network cards, are where most of the remaining cost lives, and there are three approaches, from slowest to fastest.

Emulation. The hypervisor pretends to be a real, well-known piece of hardware, such as a specific Intel network card from the 2000s, down to its registers. The guest uses its normal driver for that card, unmodified. Every time the driver pokes a register, that is a VM exit, and the hypervisor acts out what the card would have done. It works with any guest and it is slow, because sending one network packet can take several exits. The program most often doing this pretending is QEMU, which you will meet again on Mac and Windows in Chapter 14.2.6.

Paravirtual devices. The guest uses a driver that knows it is in a VM, and instead of poking fake registers one at a time, it puts whole batches of requests into a ring of shared memory and makes one call to say "there is work". The standard for this is called virtio, and Linux has shipped virtio drivers for years. This is Xen's idea from section 4, surviving where it pays most. Almost every cloud VM uses paravirtual disks and network cards.

Passthrough. The VM is given direct access to a real device, and the hypervisor gets out of the way entirely. To make this work for many VMs at once, network cards can support SR-IOV (single root I/O virtualisation), which lets one physical card present itself as dozens of separate small cards, each handed to a different VM. Azure calls its version of this Accelerated Networking, and it is on by default for most VM sizes.

The cloud providers then went one step further and moved the hypervisor's device work onto separate hardware. Amazon's Nitro system, introduced from 2017, puts networking, storage and security onto dedicated cards in each server, so almost the whole of the main processor is left for customers. Azure does similar work with its own custom cards. The direction of travel for twenty years has been the same: move every job the hypervisor does in software into hardware, until the hypervisor itself is a small piece of code that mostly just sets things up.

7. Type 1 and Type 2, and why the line is blurry

You will hear hypervisors sorted into two types. The distinction is real but less clean than it sounds, and the blurry part is exactly the part that matters for Docker on Windows.

A Type 1 hypervisor runs directly on the hardware with no operating system underneath. VMware ESXi and Xen are the classic examples. This is what data centres use, because nothing sits between the hypervisor and the metal.

A Type 2 hypervisor is a normal program running on a normal operating system. VirtualBox and VMware Workstation on a laptop are the classic examples. It is convenient, because you keep your own operating system and its device drivers, and it is slower, because every disk or network request from a guest has to go through the host operating system too.

The figure below puts the two side by side, and adds the two real systems that do not fit either box. Look at where the full operating system sits in each column.

Figure 2 — Four real stacks. The textbook types are the first two columns. KVM and Hyper-V are what most of the world actually runs, and neither fits the textbook cleanly.

KVM (kernel-based virtual machine), merged into Linux in 2007, is a module that turns the Linux kernel itself into a hypervisor. Each VM is an ordinary Linux process, usually QEMU, and the module uses VT-x or AMD-V to run that process's guest code in non-root mode. So it runs on a full operating system, like Type 2, but the hypervisor is the kernel, like Type 1. KVM runs Google Cloud and, through Nitro, most of Amazon's cloud. People argue about which type it is. The argument does not matter; what matters is that the guest gets hardware-assisted speed.

Hyper-V, Microsoft's hypervisor, is the case to remember for Docker. When you switch on Hyper-V, Windows restarts and loads the hypervisor first, and then your Windows desktop starts up as a guest on top of it. Microsoft calls that privileged guest the root partition. It is a guest with special rights to manage the others, but it is a guest. This is how WSL 2 works: your Windows and a small Linux VM both run side by side on Hyper-V, which is why the Linux VM is so fast to start and why Docker Desktop can use it. Azure itself runs on a version of Hyper-V. Chapter 14.2.6 builds on exactly this picture.

8. What a VM still costs, even with all the help

With hardware help for the processor and memory, and paravirtual drivers for devices, a VM's raw processing speed is within a few percent of bare metal. But some costs come from the design itself, from the fact that every VM runs a whole separate operating system, and no hardware feature can remove them. These are the costs containers were invented to avoid, so it is worth being exact about each.

A kernel's memory, paid once per VM. An idle Linux system uses a few hundred megabytes for its kernel, its caches and its background services before your application starts. Ten VMs on one server means ten of those. Two hypervisor features fight this, and both are worth knowing because both cause surprises:

• Kernel same-page merging (KSM) scans memory for pages that are byte-for-byte identical across VMs, such as the same kernel code loaded ten times, and keeps one shared, read-only copy. If a VM writes to it, the hypervisor quietly makes that VM a private copy first. It saves real memory, at the cost of processor time spent scanning. It has also been used in research attacks that infer what another VM is doing by timing those private copies, so many clouds keep it off between customers.
• Ballooning lets the hypervisor take memory back from a guest that does not need it. A special driver inside the guest, the balloon, asks the guest kernel for memory as if it were a normal program, and hands those pages back to the hypervisor. The guest gives up memory politely instead of having it stolen. The surprise is that a guest under balloon pressure starts swapping, and the cause is invisible from inside.

Caching the same file twice. The guest kernel keeps recently read file data in its memory, so the next read is instant. The guest's disk is really a file on the host, so the host may cache the same data again. Two copies of the same bytes, in the same physical memory. Cloud hypervisors mostly avoid this by bypassing the host cache for guest disks, but on a laptop running a Type 2 hypervisor it is real.

Boot time. Starting a VM means running firmware, a bootloader, a kernel that probes its devices, and an init system that starts dozens of services. On a modern cloud this takes tens of seconds, sometimes a minute or two including the time to set up the disk and network. That is fast for a new server and very slow for a request that just arrived.

Mixing up the disk traffic. Each guest sends its disk a tidy stream of requests, often for neighbouring blocks. The hypervisor receives those tidy streams from many guests at once and has to send them all to one physical storage system, so what arrives there is an interleaved jumble. This is sometimes called the I/O blender. On spinning disks, where reading a far-away block means physically moving a head, it could cut throughput dramatically. On modern SSDs and networked cloud storage the effect is much smaller, but it is still why storage-heavy systems often get their own disks.

A virtual processor that is not running. Your VM's 4 processors are really 4 threads that the hypervisor schedules onto real cores, alongside other VMs' threads. Sometimes your virtual processor is ready to run and simply is not being run, because the real core is busy with someone else. Linux measures this as steal time, and you can see it as the st column in top. If st is consistently above a few percent, your VM is short of real processor and no tuning inside the guest will fix it. This is the "noisy neighbour" problem, and it is the first thing to check when a cloud VM is mysteriously slow.

The lock holder that got paused. A kernel protects shared data with spinlocks: a processor that wants the lock loops, checking again and again, until whoever holds it lets go. On real hardware the holder lets go within a microsecond. In a VM, the hypervisor can pause the virtual processor holding the lock to give its real core to another VM. Now the other virtual processors spin uselessly, burning real time, waiting for a holder that is not even running. This is called lock-holder preemption. Modern guests avoid it with paravirtual spinlocks: after spinning a while, the guest gives up and makes a hypercall that says "wake me when this lock is free", and modern processors can detect long spinning and exit to the hypervisor on their own.

9. The line between VM and container is moving

By now the cost of a VM sounds like it comes from being a VM. It actually comes from carrying a whole general-purpose machine: firmware, emulated devices, a full init system. Strip those out and a VM can be very small and very fast.

Firecracker, which Amazon open-sourced in 2018, is a hypervisor built on KVM that offers only the handful of devices a server actually needs: a paravirtual disk, a paravirtual network card, a serial console, and almost nothing else. Its own documentation reports a guest starting in under 125 milliseconds with less than 5 MB of memory overhead per VM. These are called microVMs, and they are what AWS Lambda and AWS Fargate run your code in. Each customer's function gets its own kernel, which is a VM boundary, at close to container speed.

Two other projects come at the same gap from the container side:

• Kata Containers runs each Kubernetes pod inside its own small VM, so the pod looks like a container to Kubernetes but has its own kernel underneath.
• gVisor, from Google, puts a kernel written in Go between the container and the real kernel. The container's system calls go to gVisor first, which answers most of them itself. It is not a VM, but it gives the container a much smaller part of the real kernel to attack.

So the honest difference between a VM and a container is not speed or size. It is one question: does the workload get its own kernel, or share the host's? Everything else is how much machinery comes along with that choice. Chapter 14.1.3 now takes apart the sharing option, piece by piece.

10. Why your containers will run inside VMs anyway

This is where the two technologies meet in real life, and it is the point most beginners miss.

When you create a Kubernetes cluster on Azure (Chapter 14.4.1), each "node" is a VM. When you run a container on one of those nodes, it is a process on a Linux kernel that is itself a guest on Hyper-V. The two layers do different jobs. The VM boundary separates you from other customers, because the cloud provider does not trust its customers' kernels to be shared. The container boundary separates your own services from one another, cheaply, because you trust your own code enough to share a kernel between them.

So VMs did not lose to containers. They moved down a layer. The cloud is VMs, and containers are how you pack your own work onto them.

What the interviewer will push on

"What is the difference between a Type 1 and a Type 2 hypervisor?" Type 1 runs directly on the hardware; Type 2 runs as a program on a host operating system and pays for the host in the path of every device request. What they are checking is whether you know where the line blurs. The good answer adds that KVM makes the Linux kernel itself the hypervisor, and that turning on Hyper-V makes Windows a privileged guest of its own hypervisor. The memorised answer stops at "bare metal versus hosted" and cannot place KVM.

"Why was x86 hard to virtualise, and what did VT-x change?" Because about 17 x86 instructions were sensitive but did not trap when run outside ring 0, so a hypervisor could not rely on the processor to tell it when to step in. POPF silently dropping the interrupt flag is the example to give. VT-x added a separate root mode for the hypervisor, so the guest kernel runs in a real ring 0 and the hypervisor chooses which events cause a VM exit. The tell is naming the Popek and Goldberg rule, or at least its idea: every sensitive instruction must trap. The common wrong answer is "VMs were slow because emulation is slow", which confuses a hypervisor with an emulator.

"Why do memory-heavy workloads suffer more in a VM, and what do you do about it?" Nested page tables make a TLB miss walk two sets of tables, up to 24 memory reads instead of 4. The fix is huge pages, which cut the number of misses. Mention that before nested paging, shadow page tables made it far worse by turning page-table updates into VM exits.

"A cloud VM is slow and nothing inside it looks busy. Where do you look?" At steal time, the st column in top. The interviewer is checking whether you know a virtual processor can be ready and not running. High steal means the physical host is oversubscribed, and the fix is a different VM size or a dedicated host, not tuning the application.

"If containers are lighter, why does AWS Lambda use VMs?" Because different customers' code must not share a kernel, and Firecracker microVMs make a VM boundary cheap enough to use per function. This answer shows you understand that the real difference is the kernel, not the weight.

One thing to volunteer: say that the history of virtualisation is the history of moving the hypervisor's work into hardware, first the processor with VT-x, then memory with EPT, then devices with SR-IOV, then the whole device layer onto cards like Nitro. It frames every follow-up and shows you see the direction the industry is moving, not just its current state.

Recall

• Trap-and-emulate: the guest runs natively; only sensitive instructions trap to the hypervisor, which imitates their effect on a private record. Emulators act out every instruction and are 10–100× slower.
• Popek and Goldberg (1974): trap-and-emulate works if every sensitive instruction (touches real machine state) is also privileged (traps outside kernel mode). IBM CP-67 (1967–68) satisfied it.
• x86 broke the rule with 17 sensitive-but-silent instructions. POPF in ring 1 drops the interrupt flag without trapping, so guest and hypervisor disagree and the guest hangs.
• Three fixes: VMware binary translation (1999, rewrite guest kernel code) · Xen paravirtualisation (2003, modified guest makes hypercalls) · VT-x/AMD-V (2005/2006, root and non-root modes, VM exit/entry, state in the VMCS).
• Memory: shadow page tables caused exit storms; EPT/NPT does both translations in hardware, but a TLB miss can cost (4+1)(4+1)-1 = 24 reads instead of 4, so use huge pages. VPID/ASID tag the TLB per guest.
• Devices: emulation (slow, any guest) · virtio paravirtual drivers (standard) · SR-IOV passthrough (Azure Accelerated Networking). Nitro moves device work onto separate cards.
• Types: Type 1 on bare metal, Type 2 on a host OS. KVM makes Linux the hypervisor; Hyper-V makes Windows a privileged guest, which is how WSL 2 runs.
• Built-in costs: a kernel per VM (KSM, ballooning help) · double caching · tens-of-seconds boot · I/O blending · steal time (st in top) · lock-holder preemption.
• The real line is the kernel, not the weight: Firecracker microVMs boot in under 125 ms for Lambda and Fargate. In the cloud, containers run inside VMs, and each boundary does a different job.

Self-test: Why can't a hypervisor just run the guest kernel in ring 1? · What happens on a VM exit, and where is the guest's state kept? · Derive the 24 reads of a nested page walk. · What does a high st value tell you? · Why does turning on Hyper-V make your Windows desktop a guest?

Next: 14.1.3 takes the other road. Instead of giving each workload its own kernel, a container shares one kernel and asks it to keep separate books for each workload: what it can see, what it can use, and what files it runs on.