Containers

a container is not a small virtual machine. it is an ordinary linux process (tree) that the kernel has been told to lie to — about what processes exist, what the filesystem looks like, what the network is, who root is — plus an accountant capping what it may consume. the lying is namespaces, the accounting is cgroups, and everything else (images, registries, orchestrators) is packaging around those two syscall families (Tanenbaum, Andrew S., 2008).

what a container actually is#

namespaces: the six that matter#

a namespace wraps one global kernel resource so a process group gets a private view of it. spawn a process with clone() or unshare() flags and it lives in new namespaces:

(two more exist — cgroup and time — worth knowing but rarely the point.) the user namespace is the security-critical one: it lets a container’s uid 0 be an unprivileged uid on the host, which is what “rootless” means below.

cgroups v2: the accountant#

• control groups meter and cap resource use per process-tree: cpu.max (bandwidth quota), memory.max (hard cap — exceed it and the cgroup’s own oom killer fires), io.max, pids.max (fork-bomb fence).
• v2’s improvement is a single unified hierarchy: one tree under /sys/fs/cgroup, every controller attached to the same nodes, so “this pod gets 2 cpus and 4 gb” is one subtree — the primitive kubernetes resource limits compile down to.
• namespaces decide what you can see; cgroups decide what you can spend. isolation needs both — a container that can see nothing but eat all the ram is still a noisy neighbour.

not a vm#

the isolation row is the honest one: a kernel exploit escapes every container on the host, which is why multi-tenant clouds run containers inside micro-vms (firecracker, kata) — taking both columns. 𐃏

images and layers#

union filesystems and copy-on-write#

• an image is a stack of read-only layers, each a tarball of filesystem changes. at run time, overlayfs merges them: the layers become read-only lowerdir s, a fresh empty upperdir goes on top, and the container sees the union.
• writes are copy-on-write: modifying a file from a lower layer first copies it up to upperdir, then edits the copy. deletes write a whiteout entry that masks the lower file. the image is never touched — which is why 200 containers can share one image at the cost of one, and why anything written to the container layer dies with the container (volumes below are the escape hatch).

content-addressable everything#

layers and manifests are named by the sha256 of their bytes. the demo image built below really is just three digests:

$ docker image inspect layerdemo --format '{{range .RootFS.Layers}}{{println .}}{{end}}'
sha256:88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f
sha256:e48e9d6daab227918910cc87acc76d6ca91c822d44ba4633bbe0354ac5531381
sha256:6e1631f50f3a107cc640c8591071adc8f4df45704b5181756ac7ae78d9028b41

identical layers dedupe on disk, in transfer, and in the registry — the same content-addressing bet as git and the build-system caches.

dockerfile mechanics#

the cache rules#

each instruction produces one layer, and the builder replays instructions top-down against its cache: an instruction is a cache hit if the instruction text is unchanged and (for COPY and ADD) the checksums of the copied files are unchanged. the first miss invalidates everything after it — layers below depend on layers above, exactly the dirty-cone propagation from build systems.

the practical layering discipline follows immediately: order layers by change frequency — base image, then system packages, then dependency manifests (requirements.txt, package.json) + install, then your source code last, so the daily edit invalidates one cheap COPY instead of re-running pip install.

multi-stage builds, worked#

ship the artefact, not the toolchain: build in one stage, COPY --from the result into a minimal runtime stage. executed:

# ---- stage 1: "build" ----
FROM alpine:3.20 AS build
WORKDIR /src
RUN echo "banner built in stage 1" > banner.txt   # heavy toolchain lives here

# ---- stage 2: runtime (ships without the toolchain) ----
FROM alpine:3.20
COPY --from=build /src/banner.txt /banner.txt
COPY hello.sh /hello.sh
CMD ["sh", "/hello.sh"]

== build 2 (nothing changed):
#6 CACHED
#7 CACHED
#8 CACHED
== build 3 (hello.sh edited):
#6 [build 3/3] RUN echo "banner built in stage 1" > banner.txt
#6 CACHED
#8 [stage-1 2/3] COPY --from=build /src/banner.txt /banner.txt
#8 CACHED
#9 [stage-1 3/3] COPY hello.sh /hello.sh
== run it:
banner built in stage 1
hello from pid 1 on b8c22872ae9a
one more line

read the trace: the unchanged build stage and banner COPY stayed cached; editing hello.sh re-ran only its own COPY layer. and the run line quietly demonstrates two namespaces — the shell is pid 1 (pid namespace) on a random hostname (uts namespace). more directly:

$ docker run --rm alpine:3.20 sh -c 'ps; hostname'
PID   USER     TIME  COMMAND
    1 root      0:00 sh -c ps; hostname
    7 root      0:00 ps
d5493fe8cb30

a machine running hundreds of processes, and inside the namespace: two.

networking#

• bridge (default): each container gets a network namespace with a veth pair — one end inside as eth0, the other plugged into a software bridge on the host; outbound traffic is nat’d, inbound needs explicit publishing (-p 8080:80 programs a dnat rule). containers on the same bridge reach each other by ip (and by name, on user-defined bridges).
• host: no network namespace at all — the container shares the host stack, ports and all. fast, and exactly as isolated as it sounds.
• none: a loopback and silence; you build the rest yourself.
• container:x / pod networking: join another container’s network namespace — how kubernetes pods give all their containers one ip and a shared localhost. see internet networks for the stack underneath.

volumes#

• the writable layer is a scratchpad that dies with the container; anything worth keeping lives in a volume — a host-managed directory mounted into the container’s mnt namespace, bypassing overlayfs (and its copy-up cost) entirely.
• bind mounts map an explicit host path (great for development: source code mounted live); named volumes let the engine own the path (right for databases); tmpfs mounts keep secrets in ram only.
• rule of thumb: image = code, immutable; volume = state, precious; container layer = cache, disposable.

registries and digests#

• a registry is an http content store for image manifests and layer blobs; push and pull move only the layers the other side lacks (content addressing again).
• tags are mutable pointers: python:3.12 can point at different bytes tomorrow. digests are immutable: alpine@sha256:d9e853e87e5552... (the real digest of the image the demos above pulled) names exactly one manifest forever.
• production discipline: humans read tags, machines pin digests — ci resolves the tag once and deploys by digest, or the supply chain inherits a mutable dependency. the same hermeticity argument as reproducible builds, one level up.

security, one paragraph#

defence is subtraction. rootless: run the engine and containers in a user namespace so container-root is a nobody on the host — a breakout lands unprivileged. capabilities: root’s powers are roughly forty separate flags (CAP_NET_ADMIN, CAP_SYS_ADMIN, …); runtimes drop most by default and you should drop the rest (--cap-drop=ALL plus the two you need). seccomp: a syscall allowlist filter — the default profiles block a few dozen exotic syscalls, closing most historical kernel-escape routes. add read-only root filesystems and no-new-privileges, and the honest residual risk is the shared kernel itself — which is the vm row in the earlier table, and why the paranoid run micro-vms.

orchestration, one honest paragraph#

one host runs containers; a fleet needs a scheduler. kubernetes’ model in one breath: the unit is the pod (containers sharing network and volumes, scheduled together); you declare desired state (deployment: replicas: 3) and controllers run reconciliation loops that diff desired against actual and act — a crashed pod is rescheduled not because something noticed the crash, but because the loop noticed the count was 2. services give pods a stable virtual ip (the service discovery problem, solved by the platform). that is genuinely the whole idea; the other 900 pages are api surface.

the oci spec#

“docker” stopped being one thing in 2015: the open container initiative standardises the image format (layers + manifest + config), the runtime spec (what a bundle is and what create, start and kill mean), and the distribution spec (the registry http api). consequence: images build with docker or buildah, run on containerd or cri-o or podman, and any registry serves them — the formats are the interface, the tools compete. podman’s whole pitch is oci compatibility without a root daemon.

see also#

• microservices — what all these boxes are deployed to run
• build systems — layer caching is the build dag’s staleness problem in a trenchcoat
• linux — the kernel these processes all share
• concurrency — isolation by process when threads share too much

References#

Tanenbaum, Andrew S. (2008). Modern Operating Systems, Pearson.