13.6.4 — Images, Layers and Registries

Two teams deploy at the same minute. One pushes a 90 MB image and the deploy completes in eleven seconds. The other pushes a 1.4 GB image and the deploy takes four minutes, times out the health check, and rolls back.

Both images changed by the same four kilobytes of JavaScript.

The difference is not the size of the images. It is where in the stack the change landed, and what a registry does when it is asked for bytes it has seen before. This page is what an image actually is — on disk, in a registry, and in the JSON that ties it together — because almost every slow build, mysterious cache miss and "which version is running?" argument is answered by these structures.

1. An image is not a file

An image is a small JSON document that points at other things by cryptographic hash. That is the whole design, and everything convenient about containers falls out of it.

Three kinds of object exist in a registry:

Layers (blobs). Compressed tar archives of filesystem changes. This is the actual content — the base OS files, your node_modules, your compiled output.

The config (also a blob). A JSON object describing how to run the image: the default command, environment variables, working directory, exposed ports, the user, the CPU architecture, and the ordered list of layer identifiers.

The manifest. A small JSON document listing the config and the layers, each by digest and size. The manifest is the image. Everything else is content it references.

Everything is addressed by its digest, which is the SHA-256 hash of the object's bytes, written as sha256: followed by 64 hexadecimal characters.

This is called content addressing, and it changes what identity means. A name like nginx:1.27 is a label somebody attached and can move tomorrow. A digest is derived from the bytes themselves — it cannot point at different content, because different content would produce a different digest. If two machines hold sha256:9f2b..., they hold identical bytes. No trust, no version negotiation, no ambiguity.

Look at a real manifest:

docker manifest inspect nginx:1.27

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:5ef79149e0ec84a7a9f9284c3f91aa3c20608f8391f5445eabe92ef07dbda03c",
    "size": 8449
  },
  "layers": [
    { "digest": "sha256:302e3ee...", "size": 29150592 },   // (1)
    { "digest": "sha256:e2d2c1e...", "size": 40906394 },   // (2)
    { "digest": "sha256:f9c9a0a...", "size": 627 }         // (3)
  ]
}

(1) The Debian base filesystem — 29 MB compressed. Every image built FROM debian shares this exact blob.

(2) The layer that installed nginx.

(3) 627 bytes. A layer that copied in a small configuration file. Layers can be tiny, and this is the fact the fast-deploying team in the opening was exploiting.

The manifest is 700 bytes. The image is 70 MB. Pulling starts by fetching the manifest, then asking for only the layer digests not already on the machine.

2. What a layer contains

A layer is a tar archive of the filesystem changes made by one build instruction, not a snapshot of the whole filesystem.

Build this:

FROM alpine                      # layer 1: the whole Alpine root filesystem, ~7 MB
RUN apk add --no-cache curl      # layer 2: only the files apk created
COPY server.js /app/server.js    # layer 3: one file, ~2 KB

Layer 2 contains the curl binary, the libraries it needed, and the modified package database. It does not contain /bin/sh, which came from layer 1 and did not change.

Three rules govern layers, and every image size problem is one of them:

A layer only ever adds. Once written, it is immutable and content-addressed. Nothing later can shrink it.

Deleting writes a whiteout marker. Chapter 13.6.2 showed the mechanism. Concretely:

RUN curl -o big.tar.gz https://example.com/big.tar.gz && tar -xf big.tar.gz   # +500 MB
RUN rm big.tar.gz                                                             # +0 MB saved

The final image is 500 MB larger and the file is not there. The bytes sit in the earlier layer, get pushed to the registry, and get pulled on every deploy, invisibly. The fix is one line:

RUN curl -o big.tar.gz https://example.com/big.tar.gz \
 && tar -xf big.tar.gz \
 && rm big.tar.gz          # same layer — the file never exists in a finished layer

Modifying a file copies it up. Changing one byte of a 300 MB file in a lower layer writes a fresh 300 MB copy into your layer. Chapter 13.6.5 shows where this catches people with chmod and chown.

docker history is the tool that finds the guilty layer:

docker history myapp:1.4 --no-trunc --format "{{.Size}}\t{{.CreatedBy}}"

487MB    RUN /bin/sh -c apt-get install -y build-essential
124MB    RUN /bin/sh -c npm ci
 29MB    /bin/sh -c #(nop) ADD file:... in /
  2KB    COPY dist/ /app/dist/

The answer is almost always one instruction, and here it is obvious: a compiler toolchain shipped into production. Section 4 of Chapter 13.6.5 removes it.

3. The config blob: everything that is not files

The config carries the runtime behaviour and the build history.

docker image inspect nginx:1.27 --format '{{json .Config}}' | jq

{
  "Env": ["PATH=/usr/local/sbin:...", "NGINX_VERSION=1.27.0"],
  "Cmd": ["nginx", "-g", "daemon off;"],
  "WorkingDir": "/",
  "ExposedPorts": { "80/tcp": {} },
  "User": "",
  "Labels": { "maintainer": "NGINX Docker Maintainers" }
}

Alongside it sits rootfs.diff_ids — the layer digests in order — and history, the record of every instruction that built the image.

Two things are worth knowing about this object.

The config's own digest is the image ID. The IMAGE ID shown by docker images is the SHA-256 of this config blob, not of the image as a whole. This is why the image ID differs from the digest you see in a registry, which is the hash of the manifest. Two different names for two different objects, and it confuses people permanently until they see it once.

The build history is public. Every instruction, including its full command line, is recorded here and readable by anyone who can pull the image. A secret passed as a build argument appears in this history in plain text. That is not a leak in the sense of a bug; it is the documented behaviour of a field designed to be read. Chapter 13.6.5 covers the correct alternative.

4. One tag, several architectures

An ARM laptop and an x86 server need different binaries. The registry solves this with one more level of indirection.

A manifest list (the OCI name is an image index) is a manifest of manifests. It lists one real manifest per platform:

{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    { "digest": "sha256:aaa...", "platform": { "os": "linux", "architecture": "amd64" } },
    { "digest": "sha256:bbb...", "platform": { "os": "linux", "architecture": "arm64" } }
  ]
}

The client picks. docker pull node:22 on an ARM Mac fetches the index, finds the arm64 entry, and pulls that manifest. On an x86 server the same command pulls a completely different image. One tag, two sets of bytes, chosen by the puller.

Build one with buildx:

docker buildx build --platform linux/amd64,linux/arm64 -t registry/api:1.4 --push .

The failure this prevents is a genuinely common half-day of confusion: an image built on an ARM laptop and pushed to a registry runs fine locally and fails on an x86 server with exec format error. The error means "these are ARM instructions and this is an x86 CPU". It looks like a corrupted binary; it is a missing --platform.

5. Tags, digests, and which one to deploy

A tag is a mutable pointer. myapp:1.4 is a name in a registry's database that points at a manifest digest today and can point at a different one tomorrow. Nothing stops anyone with push access from moving it.

A digest is immutable by construction.

docker pull nginx:1.27                                    # (1)
docker pull nginx@sha256:9f2b...                          # (2)

(1) "Whatever 1.27 means right now." (2) "Exactly these bytes, forever."

Why :latest is not a version. It is a tag like any other, applied by convention to the most recent push, mutable, and meaningless in a rollback. Three specific failures follow from deploying it:

• You cannot say what is running. Two pods started an hour apart may be running different code with the same tag.
• You cannot roll back, because the previous image has no name.
• imagePullPolicy changes behaviour silently. In Kubernetes, a tag of latest defaults the pull policy to Always while any other tag defaults to IfNotPresent, so the same manifest behaves differently based on a string.

The practice that works:

registry.example.com/api:1.4.2          # human-readable release
registry.example.com/api:a3f9c21        # the git commit — the one CI produces
registry.example.com/api@sha256:9f2b…   # the digest — the one you deploy

Tag by commit SHA so every image traces to exactly one commit; deploy by digest so the running thing is unambiguous. Kubernetes accepts a digest in the image field, and Chapter 13.6.11 uses one.

And apply the same reasoning to your base image. FROM node:22-slim is mutable too — it is rebuilt with new patches regularly, so a build that succeeded last month may produce different software today. Pinning the base by digest gives reproducibility; pairing that with an automated update tool gives you patches as well. Pinning without automation is how images quietly go two years without a security update.

6. What a registry actually is

A registry is an HTTPS API over a blob store, defined by the OCI Distribution Specification. There are only a handful of routes and knowing them makes the whole thing concrete.

GET  /v2/                                   # (1)
GET  /v2/<name>/manifests/<reference>       # (2)
HEAD /v2/<name>/blobs/<digest>              # (3)
GET  /v2/<name>/blobs/<digest>              # (4)
POST /v2/<name>/blobs/uploads/              # (5)
PUT  /v2/<name>/manifests/<reference>       # (6)

(1) Version check and where the authentication challenge comes from. (2) Fetch a manifest by tag or by digest. (3) Ask whether a blob exists without downloading it. This one line is the whole efficiency story. (4) Download a blob. (5) Start an upload. (6) Write the manifest, which is what makes the image visible under that tag.

A pull, step by step

1. GET /v2/ returns 401 with a WWW-Authenticate header naming a token service.
2. The client fetches a bearer token scoped to repository:library/nginx:pull.
3. GET .../manifests/1.27 returns the manifest (or the index, if multi-platform, and then the platform-specific manifest).
4. For each layer digest, the client checks its local content store. Anything already present is skipped entirely.
5. Missing layers are downloaded in parallel and verified — the digest is recomputed from the received bytes and compared. A mismatch fails the pull.
6. Layers are decompressed into the snapshotter's directories, ready for OverlayFS.

Step 4 is why the second image from the same base pulls almost instantly, and it is why the deploy in the opening paragraph took eleven seconds: only the top layer had changed, so only a few kilobytes moved.

A push, step by step

1. For each layer, HEAD /v2/<name>/blobs/<digest>. A 200 means "already here" and the client skips the upload entirely.
2. Missing layers are uploaded.
3. The config blob is uploaded.
4. The manifest is PUT, which publishes the tag.

This is why the ordering of Dockerfile instructions decides deploy speed and not just build speed. Change one line of source in a well-ordered Dockerfile and only the final small layer is new — everything below it gets a 200 on the HEAD and never travels. Change something near the bottom of the file and every layer above it is invalidated and must be rebuilt, re-uploaded, and re-downloaded by every node.

Cross-repository mounting is the last optimisation: pushing an image to a second repository in the same registry can reference blobs that already exist elsewhere in that registry instead of re-uploading them.

7. Running your own, and surviving public ones

Public registries rate-limit. Docker Hub limits anonymous pulls per IP address over a rolling window, and a CI farm or a Kubernetes cluster behind one NAT gateway is a single IP address making hundreds of pulls. The symptom is toomanyrequests in the middle of a deploy, which looks like a network fault and is not.

Two fixes, and you want both:

Authenticate, so pulls count against an account rather than an IP.

Run a pull-through cache, a registry configured to fetch from upstream on a miss and keep a copy. Every cloud registry offers this, and the plain registry:2 image does it in a few lines of configuration. The cluster pulls from you, you pull from upstream once.

Run your own registry in one command for a lab:

docker run -d -p 5000:5000 --restart=always --name registry registry:2
docker tag myapp:1.4 localhost:5000/myapp:1.4
docker push localhost:5000/myapp:1.4

Retention is not optional. Every CI build pushes a new image, and untagged manifests plus their blobs stay until something removes them. Registry storage bills grow quietly and steadily. Every managed registry has retention rules — keep the last N tags, delete untagged manifests older than a week — and setting them on day one avoids an unpleasant conversation in month eight.

Garbage collection is two-phase. Deleting a tag removes a pointer; the blobs stay until a garbage collection pass finds them unreferenced. This is why deleting images sometimes appears to free no space at all.

8. Trust: signing, provenance and what is inside

Content addressing guarantees the bytes did not change in transit. It does not tell you who produced them. An attacker with push access can create a perfectly valid image with a perfectly valid digest.

Signing closes that. Cosign — from the Sigstore project — signs an image's digest and stores the signature in the registry alongside it:

cosign sign --key cosign.key registry.example.com/api@sha256:9f2b...
cosign verify --key cosign.pub registry.example.com/api@sha256:9f2b...

Signing without verification is decoration. The value appears when the cluster refuses to run an unsigned image, enforced at admission (Chapter 13.6.14).

An SBOM — software bill of materials — is a list of every package inside the image, with versions, generated at build time and attached to the image in the registry. When a vulnerability is announced in some library, an SBOM turns "which of our 300 images contain it?" from a week of investigation into a query.

Provenance attestations record how the image was built: which source commit, which builder, which parameters. BuildKit generates these with --provenance=true. This is the defence against a build system being tampered with, and it is the practical part of what the industry calls supply-chain security (Chapter 8.6.2).

Scanning reads the SBOM against vulnerability databases. Two pieces of advice that save enormous time: run it in CI and fail the build on new critical findings, and triage by reachability rather than by count, because a vulnerability in a package your code never calls is not the same risk as one in your HTTP parser. The most effective scanner reduction is not a tool at all — it is a smaller base image, because a package that is not present cannot be vulnerable.

9. Housekeeping on a build machine

docker system df                     # (1)
docker system df -v                  # (2)
docker image prune                   # (3)
docker builder prune --keep-storage 20GB   # (4)
docker system prune -a --volumes     # (5)

(1) A summary of disk used by images, containers, volumes and build cache, with a reclaimable column. (2) The per-object breakdown, which tells you which image or volume is enormous. (3) Removes dangling images — layers left behind when a tag moved to a new build. (4) The build cache is usually the biggest single consumer on a CI machine, and it grows without limit unless capped. (5) Destructive. Removes everything not currently in use, including named volumes, which may contain data. Fine on a build agent, never run casually on a machine holding anything you care about.

A dangling image is one whose tag has moved on. Build myapp:latest ten times and you have one tagged image and nine untagged ones holding full layer sets. On a build machine this is tens of gigabytes within a week, and it is the usual answer to "the CI agent ran out of disk again".

What the interviewer will push on

"What is a container image?" A manifest listing a config blob and a set of layer blobs, all addressed by SHA-256 digest, where each layer is a tar of one build step's filesystem changes. The strong answer names content addressing and says why it matters: identical digest means identical bytes, which is what makes the artefact you tested the artefact you ran. The weak answer is "a snapshot of a filesystem", which cannot explain sharing or caching.

"Why did deleting a large file not shrink the image?" A layer is immutable, so the delete only writes a whiteout marker in a later layer that hides the name. The bytes are still stored and still pulled. Delete in the same instruction that created the file, or use a multi-stage build. Mentioning docker history as the way you found the layer is what turns a definition into experience.

"Tag or digest, and why?" Tags are mutable pointers, so they cannot answer "what is running" or support a reliable rollback. Tag by commit SHA for traceability, deploy by digest for certainty. Then extend it to FROM — base image tags are mutable too, so pin them by digest and automate the updates, because pinning alone means never patching.

"You changed one line of code and the deploy pushed 800 MB. Why?" The changed content is in an early layer, so everything above it was invalidated. Usually a COPY . . before the dependency install, so every source edit reinstalls and re-uploads all dependencies. Reordering makes only the top layer new, and the registry's HEAD check skips the rest.

"How does a multi-architecture image work?" One tag points at an index listing one manifest per platform, and the client selects by its own OS and architecture. The tell is knowing exec format error means an architecture mismatch, because that is the error that actually shows up.

"How do you know an image is what you think it is?" The digest proves integrity but not origin, so sign with cosign and verify at admission so unsigned images cannot run. Add an SBOM and provenance attestations so you can answer "which images contain this library?" as a query rather than an investigation.

One thing to volunteer: point out that public registry rate limits are counted per IP address, so an entire cluster or CI farm behind one NAT gateway shares one budget and hits the limit as a group. The failure appears mid-deploy as a network-looking error, and the fix is a pull-through cache plus authenticated pulls. It is the kind of outage that costs a day the first time and ten minutes every time after.

Recall

• An image is a manifest pointing at a config blob (command, env, ports, user, architecture) and layer blobs (tar of one step's changes), everything addressed by SHA-256 digest. Same digest = identical bytes, guaranteed.
• The IMAGE ID is the hash of the config, the registry digest is the hash of the manifest. Different objects, different values.
• Layers only add. Deleting writes a whiteout and saves nothing; modifying triggers a full copy-up. Delete in the same RUN, or multi-stage. docker history finds the guilty instruction.
• The build history, including build arguments, is readable by anyone who can pull the image.
• A manifest list / index maps one tag to one manifest per platform, chosen by the client. Wrong architecture gives exec format error.
• Tags are mutable, digests are not. Tag by commit SHA, deploy by digest, pin FROM by digest and automate updates. :latest also silently changes Kubernetes' pull policy.
• Registry efficiency is one API call: HEAD /v2/<name>/blobs/<digest> — present means skip. This is why instruction order decides push and pull size, not just build time.
• Public registries rate-limit per IP, so a whole cluster shares one budget — authenticate and run a pull-through cache. Set retention; garbage collection is two-phase, so deleting a tag frees nothing immediately.
• Sign with cosign and verify at admission, attach an SBOM and provenance, scan by reachability not count. docker system df -v finds the disk hog; the build cache is usually it.

Self-test: Why does the second image from the same base pull almost instantly? · What is the difference between an image ID and an image digest? · What exactly does a whiteout entry cost you at deploy time? · Why is pinning a base image by digest incomplete on its own? · Which single registry API call makes pushes cheap? · Why does deleting an image sometimes free no disk?

Next: 13.6.5 is the file that produces all of this — every Dockerfile instruction, what each one does to the layer stack, and the ordering that turns a four-minute build into fifteen seconds.