2.1 — Kernel, User Space & System Calls

Part 1 built a machine: a CPU that fetches and executes instructions (1.5), memory that feeds it (1.6), all of it switches and voltages at the bottom. But that machine, on its own, is almost unusable. It can run one program that owns everything — every byte of memory, every device, the whole CPU — and if that program crashes or misbehaves, the machine is simply dead until someone reboots it. There is no "open a file," no "run two apps at once," no "stop that program from reading another's passwords." Raw hardware offers raw power and zero safety.

The operating system (OS) is the software that turns that raw, dangerous machine into something civilised — something many programs can share safely, that survives any one of them crashing, and that offers clean services ("give me a file," "give me some memory," "send this over the network") instead of raw hardware. This chapter is about the OS's beating heart — the kernel — and the single most important boundary in all of systems software: the wall between ordinary programs and the kernel, and the guarded gate through it called the system call. Understand this one boundary and a hundred later mysteries — why Node's fs.readFile is "slow," why a container is isolated, why a segfault kills your process but not your laptop — all click into place.

1. The core problem: one machine, many untrusting programs

Picture what must be true for your laptop to work right now. A browser, a music player, a code editor, and dozens of background services are all "running at once" on a handful of CPU cores. None of them can read the others' memory (or your banking tab would leak to any random app). If the music player crashes, the browser doesn't even notice. Each thinks it has the whole machine to itself. Someone is enforcing all of that — refereeing access to the CPU, the memory, and the devices, and doing it by force, because you cannot simply trust every program to behave.

That referee is the kernel, and it faces a genuine dilemma. To do its job — to stop a misbehaving program from touching another's memory or halting the CPU — the referee itself must have powers no ordinary program is allowed to have: the power to touch any memory, command any device, halt or reschedule anything. But if those god-powers were available to every program, there'd be no referee at all, just anarchy. So there must be two levels of privilege: a trusted inner circle that can do anything, and everyone else, who can't. And crucially, "everyone else" must not be able to simply promote themselves. The separation has to be enforced by something no software can override — the hardware itself.

2. Two worlds: kernel space and user space

Every modern CPU provides, in silicon, at least two privilege levels (also called modes or, on x86, rings). They are a hardware feature — a couple of bits in a CPU register that change what instructions are legal:

• Kernel mode (x86 "ring 0," the supervisor mode): the privileged level. Code running here can execute any instruction, access any physical memory, and talk directly to devices. The kernel runs here, and only the kernel.
• User mode (x86 "ring 3"): the restricted level. Code here can do ordinary arithmetic and access only its own allotted memory. The moment it tries a privileged instruction — halt the CPU, remap memory, poke a device, read an address it wasn't given — the hardware refuses and traps into the kernel. Every program you run — your browser, your Node server, ls — runs here.

We call the world of privileged code kernel space and the world of ordinary programs user space. This isn't a software convention the kernel politely asks programs to honour; it is a wall the CPU enforces on every single instruction. A user-mode program physically cannot execute a ring-0 instruction — the transistors won't let it. This is the same "make illegal states unrepresentable" instinct from the finite state machine (1.2), now enforced by hardware: the dangerous powers are simply unavailable in the mode ordinary code runs in.

Figure 1 — The great divide. Programs live in user space with restricted powers. The kernel lives in kernel space with full hardware access. The only way up is through a controlled gate — the system call.

Now, a question that should be nagging you: if user-mode programs can't touch devices or arbitrary memory, how does your browser ever read a file or send a network packet? It obviously does those things constantly. The answer is the gate in that wall.

3. The system call: a guarded gate through the wall

A user program that needs something only the kernel can do — read a file, allocate memory, open a socket, create a new process — does not do it. It asks the kernel to do it on its behalf. That request is a system call (syscall): the controlled, deliberate mechanism by which user space crosses into kernel space, gets one specific privileged job done, and comes back. It is the entire public API of the kernel — everything a program can ask the operating system to do is some system call. On Linux there are around 300–400 of them (read, write, open, close, mmap, fork, execve, socket, and so on).

The word "call" makes it sound like a normal function call, but it is profoundly different, and the difference is the whole point. A normal function call jumps to another address in the same program, in the same mode. A system call must change the CPU's privilege level from ring 3 to ring 0 — and it must do so without letting the program jump to wherever it likes in the kernel (or a malicious program would leap straight into the middle of privileged code and skip every safety check). So the crossing is deliberately narrow and controlled. Here is the mechanism, step by step:

1. Set up the request. The program places a number identifying which syscall it wants (e.g. on x86-64 Linux, read is syscall #0) into a designated CPU register, and the arguments (which file, which buffer — a reserved block of the program's own memory set aside to receive the data — and how many bytes) into other registers — exactly the "calling convention" idea from the 1.5 trace, now used to talk to the kernel.
2. Execute the trap instruction. The program runs a single special instruction — syscall on x86-64 (older code used the software interrupt int 0x80). This is the only legal way up. It does something no ordinary instruction can: it atomically switches the CPU to kernel mode and jumps to one fixed, kernel-chosen entry point — never anywhere else. The program doesn't get to pick where in the kernel it lands; the hardware and a kernel-installed address decide. This is what makes the gate safe: one door, one doorman.
3. The kernel takes over. Now in ring 0, the kernel's syscall handler reads the syscall number, looks it up in a table, and — critically — validates everything. Are you allowed to read this file? Is that buffer pointer actually inside your memory and not someone else's? Are the arguments sane? This validation is why the boundary is a security boundary and not just a technicality: the kernel never trusts the values user space handed it. Then it performs the real work (talking to the disk driver, copying data), using its full privileges.
4. Return. The kernel puts the result (bytes read, or an error code) into a register, then executes a matching instruction (sysret) that drops the CPU back to user mode and resumes the program right after its syscall instruction — as if a normal function had returned. The program is none the wiser that it briefly visited another world.

Two terms you'll now meet constantly: file descriptor and socket

Step 1 said the program passes "which file." It does not pass a filename — it passes a file descriptor (fd), and you'll see the term in every remaining chapter, so here it is precisely.

When a program opens something, the kernel does the real work (finding the file, checking permissions) and keeps the messy details — where the data lives, how far you've read — in its own memory, where user space can't touch it. What it hands back to the program is simply a small integer — a ticket number — that indexes the kernel's private table of that process's open things. That integer is the file descriptor. From then on the program says "read from fd 5," and the kernel looks up entry 5 in that process's table to find what it actually refers to. It's a coat-check ticket: you hold a numbered stub, the kernel holds the coat. This design is why the kernel stays in control (you can only name things it already gave you a ticket for) and why fds are per-process (your fd 5 and my fd 5 are unrelated). Every process starts with three fds already open — 0, 1, and 2 — which you'll meet as standard input, output, and error in Chapter 2.8.

Because Unix treats nearly everything as a file (Chapter 2.6), a file descriptor might refer to a file on disk, a terminal, a pipe between programs — or a socket: the endpoint of a network connection. A socket is just "an open network conversation" given the same file-descriptor treatment, so a program can read() and write() a network connection with the exact same calls it uses for a file. That uniformity is why this chapter's read() discussion applies equally to disks and networks. (How sockets actually carry data across a network is Part 5; here you only need "a socket is a network connection you talk to like a file.")

This round trip — user → trap → kernel mode → work → return → user — is called a mode switch (or mode transition). Hold onto that phrase; in the expert lens we'll see it costs real time, and that cost quietly explains a huge amount of performance engineering.

Figure 2 — A system call up close. The program traps into a single kernel entry point, the kernel validates and does the privileged work, then returns to user mode. Every "open a file," "send a packet," "get more memory" is this dance.

You've been making syscalls all along

You almost never write a raw syscall instruction, and here's the chain that hides it — worth seeing, because it demystifies "high-level" code. When your JavaScript calls fs.readFileSync('x.txt'), Node calls a C++ function, which calls the C standard library's read() wrapper function, which is a tiny piece of assembly that sets up the registers and executes the syscall instruction. The C standard library (libc / glibc on Linux) exists largely to provide these friendly wrappers so ordinary code never touches raw traps. So the layers stack: your language → its runtime → libc wrapper → the trap → the kernel. Every print, every file read, every network call in every language bottoms out here. (You can watch it happen: the Linux tool strace ./myprogram prints every system call a program makes, live. Running strace ls and seeing the openat, read, write calls scroll by is one of the great "oh, that's what's underneath" moments — do it once and this chapter becomes concrete forever.)

4. What the kernel actually contains

"The kernel" isn't one blob; it's a collection of subsystems, and — pleasingly — the rest of Part 2 is essentially a guided tour of them. Every one runs in kernel space:

• The scheduler decides which program gets the CPU next and for how long, creating the illusion that dozens of programs run "simultaneously" on a few cores (Chapter 2.3).
• The memory manager hands out memory, and — through the magic of virtual memory — gives every program its own private illusion of the whole address space so none can see another's data (Chapter 2.5).
• File systems turn "read report.pdf" into the right blocks on a physical disk (Chapter 2.6).
• Device drivers are the kernel's translators for hardware — one per keyboard, disk, GPU, network card — speaking each device's private language (Chapter 2.7).
• The network stack implements TCP/IP so programs can send() and recv() without knowing anything about Ethernet frames (Part 5).

A design question worth knowing: should all of this live inside the one privileged kernel (a monolithic kernel, like Linux — fast, because subsystems call each other directly with no boundary crossings, but a bug in any driver can crash everything), or should most of it be pushed out into user-space services talking via messages, leaving only a tiny privileged core (a microkernel, like the one inside macOS/iOS and QNX — more robust and isolated, since a driver crash needn't take down the system, but slower, because those messages cross boundaries constantly)? This is the same tractability-vs-performance and isolation-vs-speed trade-off you keep meeting — it recurs almost identically for microservices in Part 10. Linux chose monolithic (with loadable modules) and won the server world on performance; the debate was one of the most famous in the field (the 1992 Torvalds–Tanenbaum flame war). There's no free lunch, only a chosen trade.

5. Boot: how the machine climbs from silicon to a running OS

We've described the kernel as the thing in charge — but when you press the power button, the kernel isn't even in memory yet; it's a file on a disk the CPU can't yet read a file system to find. So how does a dead machine pull itself up into a running OS? This is booting (from "bootstrapping" — pulling yourself up by your own bootstraps, an apt image for a machine loading the very software needed to load software). It's a chain of ever-more-capable stages, each just smart enough to find and launch the next:

1. Power-on → firmware. When power hits, the CPU begins executing instructions from a fixed address hard-wired to a chip on the motherboard: the firmware — the modern UEFI (or the legacy BIOS). This firmware is tiny and hardware-specific. It runs the power-on self-test (checking RAM and devices exist), then hunts for a bootloader on your storage devices.
2. Firmware → bootloader. The firmware loads a small program called the bootloader (e.g. GRUB on Linux, the Windows Boot Manager) from a known spot on the disk into memory and jumps to it. The bootloader's one job: find the operating-system kernel on disk, load it into memory, and hand over control. (It's a separate stage because the firmware is too dumb to understand your file systems or your choice of OS — the bootloader bridges that gap, and is why dual-boot menus exist.)
3. Bootloader → kernel. The bootloader loads the kernel image into memory and jumps to its entry point. The kernel now takes over for good: it initialises its subsystems (sets up memory management, starts the scheduler, detects hardware and loads drivers), mounts the root file system, and switches the CPU into its normal operating state.
4. Kernel → the first process. Finally, the kernel starts one user-space program — the init process (on modern Linux, systemd; it always gets process ID 1). This is the ancestor of every other process on the machine. init then starts everything else: system services, the network, the login screen. From here, user space comes alive, and the machine is "booted."

Notice the elegant pattern: each stage is a small, dumb thing whose only real skill is launching the next, slightly smarter thing. Firmware can't read your files; the bootloader can, just enough to find the kernel; the kernel can do everything but delegates the actual applications to init and its children. A chain of bootstraps, silicon to desktop, in a second or two.

6. The expert lens

A syscall is expensive — and that cost shapes real systems. A mode switch isn't free: the CPU must save the user program's state, switch privilege level, jump to the kernel, and later restore everything — hundreds to thousands of cycles, and (post-Spectre, section 5) more, because mitigations added work to every boundary crossing. This is the reason read()-ing a file one byte at a time is catastrophically slow (one syscall per byte) while reading in big buffered chunks is fast (one syscall per many kilobytes) — the syscall count dominates, not the bytes. It's why languages buffer I/O by default, why "batch your work across the boundary" is a perennial optimisation, and why modern interfaces like Linux's io_uring (Chapter 2.7) exist specifically to let a program submit thousands of I/O requests with almost no syscalls. The kernel even maps a few read-only "syscalls" (like get the current time) directly into user space via the vDSO so they need no mode switch at all. Every one of these tricks is the same insight: crossing the wall costs, so cross it less.

The boundary is the security boundary — and breaking it is the whole game. Because user space is walled off and can only enter the kernel through validated syscalls, the kernel is the enforcer of all isolation: process from process, user from user, container from container (Chapter 2.8/Chapter 2.9). This is why the most severe security vulnerabilities are privilege escalation bugs — a flaw that lets user-space code trick the kernel into running its code in ring 0, thereby becoming the referee and owning the machine. Every syscall's argument-validation is a potential crack; a single missed check on a pointer can be a total system compromise. When you later reason about container escapes, sandbox breakouts, or why running untrusted code is dangerous, you're reasoning about this wall and the integrity of its one gate.

"It's just a library call" is an abstraction with a floor. The reason a segfault kills only your process (not the machine) is this very design: your program runs in ring 3, so its wildest mistake — dereferencing a bad pointer — is caught by the hardware and reported to the kernel, which calmly terminates just your process and reclaims its memory, machine unharmed. Contrast a bug in the kernel (ring 0): there's no higher referee to catch it, so it takes down everything — the Windows "blue screen," the Linux "kernel panic." Same bug, catastrophically different blast radius, entirely because of which side of the wall it happened on.

Next chapter: we've said the kernel runs "programs" and the scheduler juggles them — but what is a running program, precisely? Chapter 2.2 defines the process: the OS's container for a running program, how one is created (the beautiful, strange fork), how it's torn down, and how the kernel keeps every process's world separate from every other's.

Recall

• The operating system turns raw, unsafe hardware into a shared, protected platform; its privileged core is the kernel, and the rest of Part 2 tours the kernel's subsystems.
• Hardware enforces two privilege levels: kernel mode (ring 0, unrestricted, only the kernel) and user mode (ring 3, restricted, every program). The wall between kernel space and user space is enforced per-instruction by the CPU — software cannot self-promote.
• A system call is the only gate through the wall: the program loads a syscall number + args into registers, executes a syscall trap that atomically switches to ring 0 at one fixed kernel entry point, the kernel validates everything and does the privileged work, then returns to user mode. This round trip is a mode switch.
• You reach syscalls through layers — language → runtime → libc wrapper → trap → kernel; strace shows them live. The kernel holds the scheduler, memory manager, file systems, drivers, and network stack (monolithic like Linux vs microkernel like macOS/QNX).
• Booting is a bootstrap chain: firmware (UEFI/BIOS) → bootloader (GRUB) → kernel → the first process (init/systemd, PID 1) → all other programs.

Self-test: Why must the privilege wall be enforced by hardware, not software? Walk through what happens when a program calls read(). Why is reading a file one byte at a time so much slower than in big chunks? Why does a user-program segfault spare the machine while a kernel bug panics it? Name the four boot stages in order.

Quiz Bank

FoundationalWhat is the difference between user space and kernel space, and why does it exist?

User space is where ordinary programs run in a restricted CPU privilege level (ring 3): they can do arithmetic and touch only their own memory, but cannot execute privileged instructions or access hardware directly. Kernel space is where the kernel runs in the privileged level (ring 0): it can execute any instruction, access any memory, and command devices. It exists to enforce isolation and safety — so a buggy or malicious program can't touch another's memory, hog the CPU, or crash the machine. Crucially the separation is enforced by hardware (the CPU's privilege bits), so no program can promote itself.

FoundationalWhat is a system call?

A system call is the controlled mechanism by which a user-space program requests a privileged service from the kernel — reading a file, allocating memory, opening a socket, creating a process. It's the kernel's entire public API. The program sets a syscall number and arguments in registers, executes a trap instruction (syscall) that switches the CPU to kernel mode at a single fixed entry point, the kernel validates and performs the work, then returns control and the result to user mode.

AppliedTrace what happens, step by step, when a program calls read().

(1) The program (via a libc wrapper) loads the syscall number for read and its arguments — file descriptor, buffer pointer, byte count — into CPU registers. (2) It executes the syscall trap instruction, which atomically switches to kernel mode (ring 0) and jumps to the kernel's one fixed syscall entry point — a mode switch. (3) The kernel reads the number, looks up the handler, and validates: are you allowed this file? is that buffer inside your own memory? (4) It performs the privileged work (asks the disk driver, copies data into your buffer). (5) It places the result in a register and executes sysret, dropping back to user mode and resuming the program right after the trap. The program sees it as an ordinary function return.

AppliedWhy is reading a file one byte at a time far slower than reading in large chunks, even though the total bytes are identical?

Because the cost is dominated by the number of system calls, not the number of bytes. Each read() is a mode switch — save state, switch to ring 0, validate, switch back — costing hundreds to thousands of CPU cycles. Reading one byte per call means one full mode switch per byte; reading 64 KB per call amortises one mode switch over 65,536 bytes. This is why runtimes buffer I/O, and it's the practical reason "batch across the syscall boundary" is a standard optimisation.

InterviewWhy does a user-space program crashing (segfault) not bring down the whole machine, but a kernel bug does?

A user program runs in ring 3, so any illegal action — dereferencing a bad pointer, executing a privileged instruction — is caught by the hardware and trapped into the kernel, which acts as a higher referee: it terminates just that process and reclaims its resources; every other process and the machine continue unharmed (that's a "segmentation fault"). A kernel bug runs in ring 0, where there is no higher referee to catch it — so a bad pointer or illegal state there corrupts the trusted core and halts everything: a kernel panic (Linux) or blue screen (Windows). The blast radius differs entirely because of which side of the privilege wall the fault occurred on.

InterviewWhat's the difference between a monolithic kernel and a microkernel, and what's the trade-off?

A monolithic kernel (e.g. Linux) puts all core subsystems — scheduler, memory manager, file systems, drivers, network stack — inside the single privileged kernel address space, so they call each other directly. It's fast (no boundary crossings between subsystems) but less isolated (a bug in any driver can crash the whole kernel). A microkernel (e.g. the Mach-derived core in macOS/iOS, QNX) keeps only the barest essentials in ring 0 and pushes drivers/file systems out into user-space services that communicate via message passing. It's more robust and isolated (a driver crash needn't kill the system) but slower (constant message-passing across boundaries). It's the classic performance-vs-isolation trade-off, echoed later by monolith-vs-microservices.

StaffA latency-sensitive service does millions of tiny network reads and its profile shows most time in the kernel. Using this chapter, list the levers to reduce syscall overhead.

The symptom is syscall-bound: too many mode switches. Levers, all variations of "cross the wall less": (1) Batch/buffer — read larger chunks per syscall, or coalesce many small logical reads into fewer large ones. (2) Use scalable I/O interfaces — epoll to handle many connections with few syscalls, or io_uring, which lets the program submit and complete thousands of I/O operations through shared memory ring buffers with near-zero syscalls per operation. (3) Reduce per-crossing cost — ensure hot "syscalls" that can use the vDSO (like time-of-day) aren't trapping needlessly; be aware Spectre-era mitigations add per-crossing cost, sometimes tunable. (4) Move work kernel-side — e.g. sendfile/zero-copy to avoid bouncing data across the boundary, or eBPF to run filtering logic in-kernel. (5) Amortise differently — larger socket buffers, batched syscalls like recvmmsg. The staff framing: identify that throughput is bounded by boundary crossings, then attack the crossing count and per-crossing cost, not the byte count.

Flashcards

FlashKernel mode vs user mode (ring 0 vs ring 3)

Kernel mode (ring 0): unrestricted, only the kernel. User mode (ring 3): restricted, all normal programs. Enforced by the CPU per instruction.

FlashWhat is a system call?

The only gate from user space into the kernel: a trap that switches to ring 0 at a fixed entry point to request a privileged service (read, write, mmap, fork, socket…).

FlashWhat is a mode switch, and why care?

The user↔kernel privilege transition during a syscall; it costs hundreds+ of cycles, so minimizing syscall count is a major optimization.

FlashRole of libc in system calls

The C standard library provides friendly wrapper functions that set up registers and execute the raw trap, so normal code never writes a syscall instruction directly.

FlashMonolithic vs microkernel

Monolithic (Linux): all subsystems in ring 0, fast, less isolated. Microkernel (QNX, macOS core): minimal ring 0 + user-space services, isolated, slower.

FlashThe boot chain (4 stages)

Firmware (UEFI/BIOS) → bootloader (GRUB) → kernel → init/systemd (PID 1) → all other processes.

FlashTool to see a program's system calls live

strace ./program on Linux — prints every syscall (openat, read, write, …) as it happens.

Scenario Drill

DrillYou run untrusted user-submitted code on your servers. Explain, using this chapter, why the CPU privilege model is your foundation of safety — and where it is NOT enough on its own.

The privilege model is the first line: untrusted code runs in user space (ring 3), so by hardware it cannot touch other processes' memory, command devices, or halt the machine — every dangerous action must go through a system call the kernel can inspect and refuse. That's why a random buggy or hostile program can crash itself (a segfault the kernel cleanly contains) without harming the host. But ring 3 alone is not enough, because a user-space process can still make legitimate syscalls to do harmful things: read files it can access, open network connections, spawn processes, or consume all CPU/memory — and it can attempt privilege-escalation attacks, hunting for a kernel bug in syscall validation to break into ring 0 and own the machine. So real sandboxing layers more kernel-enforced walls on top: restricting which syscalls are even allowed (seccomp), giving the code its own isolated view of processes, filesystem, and network (namespaces → Chapter 2.8), capping its resources (cgroups), and often running it inside a container or VM (Chapter 2.9) so a kernel-level escape still hits another wall.

The chapter's lesson: the privilege boundary is the enforcement mechanism, but safety comes from narrowing what the trusted side will do on the untrusted side's behalf — reducing the gate's attack surface, not just having a gate.