14.1.4 — Build a Container From Scratch

In Chapter 14.1.3 you made a rough container with one unshare command and a chroot, and you borrowed Docker to get the filesystem. That was enough to see the idea. This page does it properly, with no container software installed at all, in the same order a real container runtime does it:

1. get a root filesystem,
2. stack a writable layer on top of it,
3. put a process into fresh namespaces and swap its root,
4. wire up a network,
5. cap what it can use,
6. make root inside not root outside,
7. and finally put all of it into one short program that behaves like a tiny docker run.

Each step is small, and each one has a visible result you can check. By the end, nothing that Docker or Kubernetes does to start a container will be a mystery, because you will have done every part of it by hand.

0. What you need

A Linux machine where you can be root. Any of these works:

• a small cloud VM running Ubuntu or Debian,
• Ubuntu inside WSL 2 on Windows, which is a real Linux kernel in a lightweight VM (Chapter 14.1.2),
• a Linux laptop or a VM in VirtualBox.

A Mac does not work directly, because macOS has none of these kernel features. Use a Linux VM on it.

Everything below runs as root, so start a root shell once and stay in it:

sudo -i
uname -r                         # (1)
stat -fc %T /sys/fs/cgroup       # (2)

6.8.0-45-generic
cgroup2fs

(1) The kernel version. Anything from the last few years is fine. The Go program in section 7 needs at least 5.7, and every current distribution is well past that.

(2) Checks that the machine uses cgroup v2, the modern version of the resource-limit feature from Chapter 14.1.3. cgroup2fs means yes. If you see tmpfs, the machine is on the old v1 layout, and section 5 will not work as written. Any distribution from 2022 onward boots with v2.

All the files go under one directory, /var/lib/mini, so cleaning up at the end is one command.

1. A root filesystem, without Docker

A container needs a directory that looks like a whole Linux system: a /bin with programs in it, an /etc with configuration, a /lib with libraries. This is what an image provides. The smallest honest one is Alpine Linux's "mini root filesystem", a plain tar archive the Alpine project publishes for exactly this use.

mkdir -p /var/lib/mini/rootfs && cd /var/lib/mini
curl -fsSLO https://dl-cdn.alpinelinux.org/alpine/v3.24/releases/x86_64/alpine-minirootfs-3.24.2-x86_64.tar.gz   # (1)
tar -xzf alpine-minirootfs-3.24.2-x86_64.tar.gz -C rootfs                                                       # (2)
ls rootfs
du -sh rootfs

bin  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
8.6M	rootfs

(1) Downloads Alpine 3.24.2, the current stable release as of October 2026. Alpine releases twice a year, so if this exact file has gone, open https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/x86_64/ and pick the alpine-minirootfs-…-x86_64.tar.gz file listed there. On an ARM machine, replace x86_64 with aarch64.

(2) Unpacks it into rootfs. Nothing is running. This is just files.

About 9 MB for a complete Linux userland. It is that small because of two choices Alpine makes. Almost every command, from ls to ps to wget, is one program called BusyBox, with the other names being links to it. And the C library is musl, a small alternative to the larger glibc that Ubuntu uses. Look at ls -l rootfs/bin | head and you will see ls -> /bin/busybox, ps -> /bin/busybox, and so on.

This directory is what a container image is, before the layering. Docker would have downloaded the same kind of tar archive from a registry. Chapter 14.2.2 shows the exact format.

2. A writable layer on top: OverlayFS by hand

We could run a container straight on rootfs, but then every change it made would land in rootfs itself, and a second container would see the first one's mess. What we want is what Docker does: keep rootfs read-only and shared, and give each container its own thin layer for its changes. That is OverlayFS (Chapter 14.1.3 described its rules). Here you set it up by hand.

cd /var/lib/mini
mkdir -p c1/upper c1/work c1/merged                      # (1)
mount -t overlay overlay \
  -o lowerdir=rootfs,upperdir=c1/upper,workdir=c1/work \
  c1/merged                                              # (2)
ls c1/merged
ls c1/upper

bin  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

(1) Three directories for one container. upper will hold this container's changes. work is scratch space OverlayFS needs for itself, and it must be on the same disk as upper. merged is where the combined view will appear.

(2) The mount. lowerdir is the read-only bottom, upperdir is the writable top, and the merged view appears at c1/merged. The first ls shows a whole Linux system. The second shows that upper is empty, because nothing has changed yet.

Now make three kinds of change and watch where each one lands. These three experiments are the whole of OverlayFS.

echo "hello" > c1/merged/root/note.txt                   # (1) a new file
echo "10.200.0.1 host" >> c1/merged/etc/hosts           # (2) change an existing file
rm c1/merged/etc/motd                                    # (3) delete a file
find c1/upper -ls | awk '{print $3, $NF}'
ls rootfs/etc/motd && grep -c 10.200 rootfs/etc/hosts

drwxr-xr-x c1/upper
drwxr-xr-x c1/upper/root
-rw-r--r-- c1/upper/root/note.txt
drwxr-xr-x c1/upper/etc
-rw-r--r-- c1/upper/etc/hosts
c--------- c1/upper/etc/motd
rootfs/etc/motd
0

(1) A new file goes straight into upper. The lower layer never knew about it.

(2) Changing a file that exists below triggers copy-up: the kernel first copies the whole file into upper, then applies the change there. The lower layer's copy is untouched. With a 2-byte file this is instant. With a 2 GB database file it copies 2 GB, which is why you never keep a database's data inside a container's own layer.

(3) Deleting a file that exists below cannot actually delete it, because the lower layer is read-only. So OverlayFS leaves a marker in upper, and the line that starts with c--------- is that marker. It is a special "character device" file with device number 0, 0, which OverlayFS reads as "this name has been deleted, hide anything below it". This is called a whiteout. The last two lines prove the point: rootfs/etc/motd still exists, and the lower layer's hosts file does not contain the line we added (grep -c counts 0 matches). Only the merged view shows the changes.

That third experiment is the reason a Dockerfile line that deletes a file never makes the image smaller. The bytes are still in the layer below, and that layer is still downloaded on every pull. Chapter 14.2.3 builds on exactly this.

One more case: deleting a whole directory and making a new one with the same name.

rm -rf c1/merged/opt && mkdir c1/merged/opt
getfattr -n trusted.overlay.opaque c1/upper/opt

# file: c1/upper/opt
trusted.overlay.opaque="y"

The new opt in upper is marked opaque. That attribute tells OverlayFS "do not look below this directory at all", so nothing from the old /opt leaks through into the new one. If getfattr is missing, install the attr package.

Before moving on, unmount the view: run umount c1/merged. The upper directory keeps every change, and section 3 mounts it again from inside the container. Two live overlay mounts sharing one upper is not allowed, so the host's copy has to go first.

How this becomes a real image with many layers. lowerdir accepts a list, separated by colons, with the topmost layer first: lowerdir=layer3:layer2:layer1. Each Dockerfile instruction that adds files produces one such directory. Docker passes the whole stack to exactly this mount call. Two containers from the same image are two mounts with the same lowerdir list and different upperdirs, which is why a hundred containers from one image store the image only once.

3. Namespaces and a new root, by hand

Now the process. Open a terminal on the machine. This is terminal 1, and it will become the container.

cd /var/lib/mini
unshare --mount --uts --ipc --net --pid --fork --cgroup /bin/bash   # (1)

(1) Starts a new bash inside six fresh namespaces. Each flag creates one (Chapter 14.1.3 explains each):

• --mount gives it its own list of mounted filesystems,
• --uts its own hostname,
• --ipc its own shared memory,
• --net its own, completely empty, network,
• --pid its own process numbers,
• --cgroup its own view of the resource-limit tree.

--fork is needed with --pid, because a running process can never change its own PID. A brand-new child has to be born inside the new PID namespace to become its PID 1, so unshare forks and runs bash as that child.

One quiet but important default: unshare --mount marks every mount in the new namespace as private. That means a mount made inside does not appear on the host, and a mount made on the host later does not appear inside. Without that, the next step would also change the host.

Now swap the root. Still in terminal 1:

mount -t overlay overlay \
  -o lowerdir=rootfs,upperdir=c1/upper,workdir=c1/work c1/merged   # (1)
cd c1/merged
mkdir -p .oldroot
pivot_root . .oldroot                                                # (2)
cd /
mount -t proc proc /proc                                             # (3)
umount -l /.oldroot && rmdir /.oldroot                               # (4)
hostname box1
exec /bin/sh                                                         # (5)

(1) Mount the overlay again, this time inside the new mount namespace. The host does not see this mount. When this namespace ends, the mount disappears by itself.

(2) pivot_root makes merged the new / and moves the old root, the host's whole filesystem, to /.oldroot. It is a real swap of the root of this namespace, not just a change of where / points for one process.

(3) A fresh /proc for the new PID namespace. /proc is how tools like ps learn about processes. Without remounting it, ps would read the host's /proc and show the host's processes, even though our process can no longer signal them.

(4) Detach the host's filesystem completely. -l means "lazy": unhook it now and finish once nothing is using it. After this line there is no path from inside back to the host's files. This step is the difference between pivot_root and chroot. With chroot, the host's root is still mounted, just not pointed at, and a root process can climb back out with a well-known trick. The trick is to chroot into a subdirectory, which leaves the current directory outside the new root, and then cd .. repeatedly. Here there is nothing left to climb back to.

(5) Replace bash with Alpine's own shell. exec means the shell takes over this process instead of starting a new one, so it stays PID 1.

Now look around. Still inside:

/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh
    6 root      0:00 ps
/ # hostname
box1
/ # ip addr
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
/ # cat /etc/alpine-release
3.24.2
/ # ls /.oldroot
ls: /.oldroot: No such file or directory

Each line is one namespace doing its job. Two processes on a machine running hundreds: the PID namespace. A hostname that the host does not have: the UTS namespace. A single loopback interface, switched off: the network namespace, which is completely cut off for now. An Alpine release file on what may be an Ubuntu host: the mount namespace plus the overlay. And no way back to the host's files: pivot_root plus the unmount.

4. Give it a network

A new network namespace is a sealed room. It has a loopback interface and nothing else, so it cannot reach the host or the internet, and nothing can reach it. Docker builds the same four pieces you are about to build, every time it starts a container on its default network.

The figure shows what you are building. Read it from the container on the left to the internet on the right.

Figure 1 — The four pieces of container networking. ① a virtual cable with one end in each namespace, ② a virtual switch on the host joining every container's cable, ③ the host agreeing to forward packets, ④ the host rewriting outgoing packets so replies come back to it.

A veth pair is a virtual network cable: two interfaces, and whatever goes in one comes out of the other. A bridge is a virtual network switch: plug many interfaces into it and they can all talk to each other. NAT (network address translation) is the host rewriting the source address of outgoing packets to its own, so the internet sees the host and sends the replies to it, and the host passes them back to the container. Your home router does the same thing for every device in your house.

Leave terminal 1 running, and open terminal 2 on the host as root:

CPID=$(pgrep -P "$(pgrep -xn unshare)")                                  # (1)
ip link add br0 type bridge
ip addr add 10.200.0.1/24 dev br0 && ip link set br0 up                  # (2)
ip link add veth-host type veth peer name veth-ctr                       # (3)
ip link set veth-host master br0 up
ip link set veth-ctr netns "$CPID"                                       # (4)
sysctl -qw net.ipv4.ip_forward=1                                         # (5)
iptables -t nat -A POSTROUTING -s 10.200.0.0/24 ! -o br0 -j MASQUERADE   # (6)
iptables -A FORWARD -i br0 -j ACCEPT
iptables -A FORWARD -o br0 -j ACCEPT                                     # (7)

(1) Find the container's PID as the host sees it. pgrep -xn unshare finds the newest unshare process, and pgrep -P finds its child, which is our container's PID 1. Inside it is 1. Out here it is an ordinary number like 48211. One process, two numbers.

(2) Create the bridge and give it an address. That address, 10.200.0.1, will be the container's gateway, meaning the address it sends everything to when the destination is not on its own little network.

(3) Create the virtual cable. Both ends start on the host.

(4) Push one end into the container's network namespace. It disappears from the host's ip link list at this moment and appears inside the container. The other end is plugged into the bridge.

(5) Allow the host to pass packets between interfaces. By default Linux only handles packets addressed to itself.

(6) NAT for anything leaving the container network. MASQUERADE means "rewrite the source address to whatever address the outgoing interface has". ! -o br0 means "only for packets leaving by some interface other than the bridge", so container-to-container traffic is left alone.

(7) Allow forwarding to and from the bridge. Many machines block forwarding by default, and installing Docker itself sets that default to block, so these two rules make sure our packets are not dropped.

Back in terminal 1, inside the container, finish the job:

/ # ip link set lo up
/ # ip link set veth-ctr name eth0
/ # ip addr add 10.200.0.2/24 dev eth0 && ip link set eth0 up
/ # ip route add default via 10.200.0.1
/ # echo "nameserver 1.1.1.1" > /etc/resolv.conf
/ # ping -c1 1.1.1.1
64 bytes from 1.1.1.1: seq=0 ttl=57 time=4.1 ms
/ # apk add --no-cache busybox-extras
OK: 9 MiB in 18 packages

The container is on the internet. The resolv.conf line tells the container which server to ask when it needs to turn a name like dl-cdn.alpinelinux.org into an address, and the apk add line proves those lookups work. That new file lives in this container's upper layer. Alpine's package manager just downloaded and installed a package into the container's writable layer. The host's own packages were not touched.

Now publish a port. Start a tiny web server inside, then forward a host port to it. -f keeps it in the foreground, so Ctrl+C stops it later.

/ # mkdir -p /www && echo "hello from box1" > /www/index.html
/ # busybox-extras httpd -f -p 80 -h /www

In terminal 2:

curl -s http://10.200.0.2/                                                          # (1)
iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 10.200.0.2:80   # (2)

hello from box1

(1) The host is plugged into the same bridge, so it can reach the container's address directly.

(2) This is docker run -p 8080:80. DNAT means destination NAT: any packet arriving at the host on port 8080 has its destination rewritten to the container's address and port 80. Another machine can now open http://<this host's address>:8080/ and reach the container. The rule applies to packets arriving from outside. Requests the host makes to its own localhost:8080 take a different path through the kernel and skip it, which is why Docker also runs a small helper program, docker-proxy, to handle that case.

5. Cap what it can use

Our container can see only its own world, but it can still use as much of the machine as it likes. The fix is a cgroup, a group with limits (Chapter 14.1.3 introduced it). In terminal 2:

cat /sys/fs/cgroup/cgroup.subtree_control                 # (1)
mkdir /sys/fs/cgroup/box1                                 # (2)
echo 64M          > /sys/fs/cgroup/box1/memory.max
echo 0            > /sys/fs/cgroup/box1/memory.swap.max   # (3)
echo 50           > /sys/fs/cgroup/box1/pids.max
echo "50000 100000" > /sys/fs/cgroup/box1/cpu.max         # (4)
echo "$CPID"      > /sys/fs/cgroup/box1/cgroup.procs      # (5)

cpuset cpu io memory hugetlb pids rdma misc

(1) Which limit types are switched on for groups below the top. You need cpu, memory and pids in this list. On machines run by systemd they are on already.

(2) Making a directory makes a group. The kernel fills it with control files immediately.

(3) 64 MB of memory, and no swap. Without the second line, a process over the limit can push memory out to disk instead of being stopped, and the experiment below would just get slow.

(4) Half a processor: 50,000 microseconds of processor time in every 100,000 microsecond window.

(5) Move the container's PID 1 into the group. Every process it starts from now on is born into the same group.

Now break each limit on purpose, inside the container in terminal 1. First stop the web server with Ctrl+C.

/ # cat /dev/zero | head -c 200M | tail                    # (1)
Killed
/ # for i in $(seq 100); do sleep 300 & done               # (2)
/bin/sh: can't fork: Resource temporarily unavailable
/ # yes > /dev/null &                                      # (3)

(1) A memory hog in one line. tail has to keep the last ten lines of its input, and 200 MB of zero bytes is one enormous line with no line break, so tail tries to hold all of it in memory. At 64 MB the kernel kills it. The word Killed is the shell reporting that its child died from the kill signal.

(2) A runaway loop that starts processes. After roughly 48 of them the group reaches 50 processes, counting the shell, head and friends, and the next attempt to start one fails. This is the defence against a fork bomb, a program that does nothing except copy itself until the machine has no room left. Stop the sleepers with kill $(pgrep sleep).

(3) A processor hog. yes prints y forever as fast as it can.

Read the evidence from the host in terminal 2:

grep oom_kill /sys/fs/cgroup/box1/memory.events
cat /sys/fs/cgroup/box1/pids.events
grep -E 'nr_throttled|throttled_usec' /sys/fs/cgroup/box1/cpu.stat
top -b -n1 | grep -w yes

oom_kill 1
max 51
nr_throttled 412
throttled_usec 20563210
  48890 root      20   0    1608    640    576 R  50.0   0.0   0:41.33 yes

Each limit leaves a record. oom_kill 1 is one process killed for memory. max 51 counts the attempts to start a process that the limit refused. nr_throttled is the number of 100 ms windows in which the group used up its half-processor and was made to wait, and top shows yes held at exactly 50 percent. The memory limit kills and the processor limit only slows down, which is the difference that produces two completely different symptoms in production, and Chapter 14.3.7 comes back to it. Stop yes with kill %1.

6. Root inside, nobody outside: a user namespace

Everything so far ran as real root. If a process in our container escaped through a kernel bug, it would be root on the host. A user namespace fixes this by giving the container its own numbering of users, so that user 0 inside is some unimportant user outside.

You do not even need root for this. In a third terminal, as your normal user. On Ubuntu 24.04 and later, ordinary users are blocked from creating user namespaces by a default security setting, and the command below fails with Operation not permitted. For this experiment, lift that block with sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0, and set it back to 1 afterwards.

id -u
unshare --user --map-root-user --mount --pid --fork --mount-proc /bin/bash    # (1)
id
cat /proc/self/uid_map                                                        # (2)
touch /etc/should-not-work                                                    # (3)

1000
uid=0(root) gid=0(root) groups=0(root),65534(nogroup)
         0       1000          1
touch: cannot touch '/etc/should-not-work': Permission denied

(1) --map-root-user maps your own user to root inside the new user namespace. You started as user 1000, and inside, id says you are root.

(2) The mapping itself, in three columns: inside ID, outside ID, how many. 0 1000 1 reads "user 0 in here is user 1000 out there, for one user". A real runtime maps a whole block, such as 0 100000 65536, so a container can have many users of its own.

(3) The proof. "Root" cannot write to the host's /etc, because when the kernel checks permission on a real file, it translates the inside user back to the outside one. The file belongs to the host's root, and you are really user 1000. Inside its own namespaces, though, this "root" has real powers: it could mount filesystems and set up networking for itself, because those powers only apply to things this namespace owns.

This is the basis of rootless containers, where even the container runtime runs as an ordinary user. Podman works this way by default, and Docker has a rootless mode. Docker's normal mode does not use user namespaces, for compatibility reasons around file ownership on mounted folders, which is why Chapter 8.6.2 treats running containers as non-root as a separate hardening step. Type exit to leave.

7. The whole thing as one program

Everything in sections 2, 3 and 5 can be one program. This is about a hundred lines of Go and it does what a real low-level runtime like runc does at its core. Go is the natural language here because both Docker and runc are written in it, and its standard syscall package exposes every call we need. You do not need to know Go to follow it: every numbered line is walked through below.

It expects the Alpine files from section 1 at /var/lib/mini/rootfs. It does not set up the network, which keeps it short; section 4's commands still work against it.

// mini: a container runtime in one file. Linux only, run as root.
//
//	mini run <name> <command> [args...]
package main

import (
	"fmt"
	"os"
	"os/exec"
	"path/filepath"
	"runtime"
	"syscall"
)

const base = "/var/lib/mini" // (1)

func must(err error) {
	if err != nil {
		fmt.Fprintln(os.Stderr, "mini:", err)
		os.Exit(1)
	}
}

func main() {
	if len(os.Args) < 4 {
		fmt.Fprintln(os.Stderr, "usage: mini run <name> <command> [args...]")
		os.Exit(2)
	}
	switch os.Args[1] {
	case "run":
		run(os.Args[2], os.Args[3:])
	case "init":
		initContainer(os.Args[2], os.Args[3:])
	default:
		fmt.Fprintln(os.Stderr, "mini: unknown command", os.Args[1])
		os.Exit(2)
	}
}

// run happens on the host, outside the container.
func run(name string, argv []string) {
	cg := filepath.Join("/sys/fs/cgroup", "mini-"+name) // (2)
	must(os.MkdirAll(cg, 0o755))
	for file, value := range map[string]string{
		"memory.max":      "64M",
		"memory.swap.max": "0",
		"pids.max":        "50",
		"cpu.max":         "50000 100000",
	} {
		must(os.WriteFile(filepath.Join(cg, file), []byte(value), 0o644))
	}
	cgfd, err := syscall.Open(cg, syscall.O_DIRECTORY|syscall.O_RDONLY, 0)
	must(err)

	cmd := exec.Command("/proc/self/exe", append([]string{"init", name}, argv...)...) // (3)
	cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr
	cmd.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWPID | syscall.CLONE_NEWNS | syscall.CLONE_NEWUTS |
			syscall.CLONE_NEWIPC | syscall.CLONE_NEWNET, // (4)
		UseCgroupFD: true,
		CgroupFD:    cgfd, // (5)
	}
	err = cmd.Run()
	syscall.Close(cgfd)
	os.Remove(cg) // (6)
	if exitErr, ok := err.(*exec.ExitError); ok {
		os.Exit(exitErr.ExitCode())
	}
	must(err)
}

// initContainer runs as PID 1 inside the new namespaces, before the user's command.
func initContainer(name string, argv []string) {
	runtime.LockOSThread() // namespace calls act on one OS thread; stay on it
	dir := filepath.Join(base, "containers", name)
	lower := filepath.Join(base, "rootfs")
	upper := filepath.Join(dir, "upper")
	work := filepath.Join(dir, "work")
	merged := filepath.Join(dir, "merged")
	for _, d := range []string{upper, work, merged} {
		must(os.MkdirAll(d, 0o755))
	}

	must(syscall.Unshare(syscall.CLONE_NEWCGROUP)) // (7)
	must(syscall.Mount("", "/", "", syscall.MS_PRIVATE|syscall.MS_REC, ""))
	must(syscall.Mount("overlay", merged, "overlay", 0,
		"lowerdir="+lower+",upperdir="+upper+",workdir="+work)) // (8)
	must(syscall.Sethostname([]byte(name))) // (9)

	oldRoot := filepath.Join(merged, ".oldroot")
	must(os.MkdirAll(oldRoot, 0o700))
	must(syscall.PivotRoot(merged, oldRoot)) // (10)
	must(os.Chdir("/"))
	must(syscall.Mount("proc", "/proc", "proc",
		syscall.MS_NOSUID|syscall.MS_NODEV|syscall.MS_NOEXEC, "")) // (11)
	must(syscall.Unmount("/.oldroot", syscall.MNT_DETACH)) // (12)
	must(os.Remove("/.oldroot"))

	const prSetNoNewPrivs = 38
	if _, _, errno := syscall.RawSyscall(syscall.SYS_PRCTL, prSetNoNewPrivs, 1, 0); errno != 0 { // (13)
		must(errno)
	}

	os.Setenv("PATH", "/usr/sbin:/usr/bin:/sbin:/bin")
	env := []string{"PATH=" + os.Getenv("PATH"), "HOSTNAME=" + name, "TERM=xterm"}
	path, err := exec.LookPath(argv[0])
	must(err)
	must(syscall.Exec(path, argv, env)) // (14)
}

The program has two halves. run happens on the host and prepares things. initContainer happens inside the new namespaces and finishes the setup before handing over to your command. Walk the numbers in order:

(1) Where everything lives. The shared, read-only Alpine files are at /var/lib/mini/rootfs, and each container gets its own folder under /var/lib/mini/containers/<name> for its upper layer.

(2) The resource limits, created before the container exists. This is section 5 as code: one directory per container under /sys/fs/cgroup, with the same four limits written into its control files.

(3) The program starts a copy of itself. /proc/self/exe is a Linux shortcut that always points at the program currently running. So mini run box1 /bin/sh starts mini init box1 /bin/sh as a child. Why the round trip? Some setup has to happen after the process is inside the new namespaces and before your command starts, such as mounting the overlay inside the new mount namespace. Only code we control can run in that gap, so we run our own program there first. runc does exactly the same thing, which is why you will see a process called runc init if you catch it at the right moment.

(4) The namespaces, created at the moment the child is born. These flags go to the clone system call, which is how Linux creates processes. The child starts life with its own PID numbering, mount list, hostname, shared memory and empty network. This is section 3's unshare.

(5) The child is born directly inside the cgroup. UseCgroupFD tells Linux to place the new process into the group as part of creating it. The older way, writing the PID into cgroup.procs after starting, leaves a tiny moment where the process runs with no limits at all, and a process that forks fast can escape the limit in that moment. This needs Linux 5.7 or later.

(6) Clean up when the container exits. When PID 1 of a PID namespace dies, the kernel kills every other process in it, so by now the group is empty and can be removed. The overlay mount needs no cleanup at all: it lived inside the container's mount namespace, and that namespace was destroyed with its last process.

(7) The cgroup namespace, created only now. It makes the process's current group look like the root of the whole tree, so the container sees its own limits as / and learns nothing about the host's layout. It has to happen after step 5 has put the process in its group, or the "root" it sees would be the wrong group. The line after it makes all mounts private, as unshare did for us in section 3.

(8) The overlay mount from section 2, now done by the program, inside the new mount namespace, so the host never sees it.

(9) The hostname, inside the new UTS namespace, so the host's name is untouched.

(10) pivot_root, exactly as in section 3: merged becomes / and the host's root moves to /.oldroot.

(11) A fresh /proc, mounted with three safety options: no programs run from it, no device files work in it, and no setuid programs get extra rights from it.

(12) Detach the host's filesystem, so there is no path back.

(13) No new privileges. This switches on a kernel setting that says no program started from here on can gain more rights than it already has, even a setuid program like su or passwd. It closes a whole class of tricks for climbing from an ordinary user to root inside the container. Docker sets this when you pass --security-opt no-new-privileges. It goes through the low-level prctl call because Go's standard library has no friendly wrapper for it, and 38 is the number Linux gives this setting.

(14) Become the user's command. Exec replaces our Go program with your command in the same process, so your command becomes PID 1, as in a real container. Our Go code is gone from memory after this line. That is also why the PID 1 signal rules from Chapter 14.1.3 now apply to your command.

One line is not numbered but matters: runtime.LockOSThread() at the top of initContainer. Go runs code on several operating-system threads and moves work between them freely. Calls like Unshare change only the thread that makes them. Without this line, the cgroup namespace could be created on one thread and the final Exec happen on another, and the setting would be silently lost. Locking keeps every step of the setup on the same thread.

Build it and run it. You can build on any computer with Go installed, including Windows or a Mac, by telling Go to produce a Linux program. Then copy the file to the Linux machine.

GOOS=linux GOARCH=amd64 go build -o mini .        # on any machine
./mini run box2 /bin/sh                           # on the Linux machine, as root

/ # hostname
box2
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 /bin/sh
    2 root      0:00 ps
/ # cat /proc/self/cgroup
0::/
/ # cat /sys/fs/cgroup/memory.max 2>/dev/null || echo "cgroup files not mounted inside"
cgroup files not mounted inside

The first two results match what you built by hand. 0::/ is the cgroup namespace from step 7: the container sees its own group as the top of the tree. The last line shows one thing a real runtime adds that ours does not: it mounts the group's control files inside the container at /sys/fs/cgroup, so programs like the Java runtime can read their own limits (the trap described in Chapter 14.1.3).

8. What you built, next to what Docker does

You did	Real runtime does
Downloaded a tar	Pulls layers from a registry
One lowerdir	Many layers in lowerdir
unshare, clone flags	Same calls, via runc
pivot_root	pivot_root
bridge, veth, NAT	Same, done by Docker or a network plugin
cgroup.procs, limit files	Same files, often through systemd
--map-root-user	User namespaces, if enabled
no_new_privs	Plus dropped capabilities and a seccomp filter

Almost every row is the same mechanism you used. The real runtime adds the parts this page skipped: a list of allowed kernel powers that drops most of root's (capabilities), a filter that blocks dozens of dangerous system calls (seccomp), a read-only /sys and hidden parts of /proc, a proper /dev with only safe devices, a mounted cgroup directory, and the bookkeeping to start, stop and list many containers. The configuration for all of it is written into one JSON file, config.json, in a format called the OCI runtime specification. runc reads that file and makes the same calls you made. Chapter 14.2.1 shows who writes the file and who reads it.

9. Clean up

In terminal 1, type exit. The container's PID 1 ends, the kernel destroys its namespaces, and with the mount namespace go the overlay mount and /proc. Then, in terminal 2:

rmdir /sys/fs/cgroup/box1
ip link del br0                                                 # the veth end dies with its namespace
iptables -t nat -D POSTROUTING -s 10.200.0.0/24 ! -o br0 -j MASQUERADE
iptables -t nat -D PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 10.200.0.2:80
iptables -D FORWARD -i br0 -j ACCEPT && iptables -D FORWARD -o br0 -j ACCEPT
rm -rf /var/lib/mini

Look at what survived until the last line: the upper layers. c1/upper still held the note, the edited resolv.conf and the whiteout. This is exactly the state of a stopped Docker container. Its writable layer stays on disk until you run docker rm, which is why docker start on a stopped container finds the files it left behind, and why stopped containers slowly fill a disk.

What the interviewer will push on

"Walk me through what happens when a container starts, at the kernel level." The steps, in order:

1. Unpack the image layers, and stack them with OverlayFS.
2. Create the process with clone, with a flag for each namespace.
3. Put it in its cgroup.
4. Inside, make mounts private, pivot_root into the merged directory, mount /proc, and detach the old root.
5. Drop capabilities, set no-new-privileges, and install the seccomp filter.
6. exec the command, so it becomes PID 1.

What they are checking is order and reasons, for instance why /proc must be remounted, or why the runtime re-executes itself as runc init. The memorised answer lists namespaces and stops.

"Why pivot_root and not chroot?" chroot only changes where one process's / points; the old root is still mounted and a root process can climb out by chrooting into a subdirectory and walking ... pivot_root swaps the namespace's root mount, and unmounting the old root leaves nothing to climb back to. The tell is mentioning the unmount, because pivot_root without it is not much better.

"How does a container reach the internet, and how does -p 8080:80 work?" A veth pair connects the container's namespace to a bridge on the host; the host forwards packets and MASQUERADEs them on the way out; -p adds a DNAT rule that rewrites the destination of packets arriving on 8080. Add that local connections to localhost:8080 skip that rule, which is why Docker runs docker-proxy. The common wrong answer is "Docker opens the port", as if the container were listening on the host directly.

"Why does deleting a file in a Dockerfile not shrink the image?" OverlayFS cannot remove from a read-only lower layer, so the delete is a whiteout in a new layer, and the bytes stay in the old layer that is still pulled. Fix it by never adding the file, or by deleting it in the same instruction that created it, or with a multi-stage build (Chapter 14.2.3).

"What does a user namespace actually protect against?" A process that escapes the container escapes as its mapped outside user, which owns nothing. Root inside keeps powers only over objects its namespace owns. Mention that Docker's default does not enable it and that rootless Podman does.

One thing to volunteer: point out the gap a runtime must close between creating a process and limiting it. Writing a PID into cgroup.procs after the process has started leaves a window where it runs unlimited, which is why modern runtimes create the process directly inside its cgroup. It is a small detail, and it shows you have thought about how this is built rather than how it is described.

Recall

• Six steps, in order: root filesystem → OverlayFS layer → namespaces + pivot_root → network → cgroup limits → user namespace. A runtime does the same, configured by one OCI config.json.
• Alpine minirootfs is a ~9 MB Linux userland as a tar; small because of BusyBox (one program, many names) and musl.
• OverlayFS by hand: lowerdir (read-only, colon-separated list, top first), upperdir (this container's changes), workdir (scratch, same disk), merged view. New file → upper. Change → copy-up of the whole file. Delete → whiteout, a c device 0/0. Replace a directory → opaque attribute trusted.overlay.opaque="y".
• unshare --mount makes mounts private by default; --pid needs --fork because a process cannot change its own PID. Remount /proc or ps reads the host.
• pivot_root then unmount the old root leaves no way back; chroot can be escaped by chrooting deeper and walking ...
• Networking: veth pair (virtual cable) + bridge (virtual switch) + ip_forward + MASQUERADE out; -p is a DNAT rule in PREROUTING; localhost skips it, hence docker-proxy.
• cgroup v2 files: memory.max (+ memory.swap.max 0) kills → memory.events oom_kill; pids.max refuses forks → pids.events max; cpu.max "50000 100000" throttles → cpu.stat nr_throttled.
• User namespace: uid_map 0 1000 1 = inside 0 is outside 1000; root inside cannot touch the host's files; basis of rootless containers.
• Go runtime details: re-exec via /proc/self/exe (like runc init) · born into its cgroup with UseCgroupFD (no unlimited window) · cgroup namespace after joining the group · LockOSThread because namespace calls are per thread · no_new_privs · Exec so the command is PID 1.
• A stopped container is its upper layer still on disk, which is why docker rm exists.

Self-test: Where does each of the three kinds of change land in OverlayFS, and what does a whiteout look like? · Why must /proc be remounted? · Name the four pieces that connect a container to the internet. · Why is placing the process in its cgroup at birth better than writing its PID afterwards? · What does 0 100000 65536 in uid_map mean?

Next: 14.2.1 opens the toolbox. Typing docker run sets off a chain through four separate programs, the CLI, the daemon, containerd and runc, and you now know what the last of them does, because you just wrote a small version of it.