Core skills · Lesson 7/8

Virtualization & Containers — Many Machines, One Box

43 min 1 quiz Graded labs: 3

What you'll learn

How a hypervisor runs whole virtual machines, from trap-and-emulate to hardware support (VT-x, AMD-V, nested paging)
Why a container is just an ordinary process with a restricted view, built from namespaces
How container images are stacked layers merged by OverlayFS, and what a whiteout is
How cgroups limit CPU, memory and process counts, hierarchically
What isolation each approach does and does not give you, and where gVisor and microVMs fit in

You will simulate PID namespaces in Python, merge image layers in Go and enforce hierarchical memory limits in JavaScript.

Two ways to share one machine 🖥️

A big server is too large for one program, so we split it. There are two very different ways to do that:

     virtual machines                        containers
+-----------+ +-----------+          +-----------+ +-----------+
|  app A    | |  app B    |          |  app A    | |  app B    |
|  libs     | |  libs     |          |  libs     | |  libs     |
|  guest OS | |  guest OS |          +-----------+ +-----------+
|  kernel   | |  kernel   |          |   container runtime     |
+-----------+ +-----------+          +-------------------------+
|        hypervisor       |          |   ONE shared host kernel |
+-------------------------+          +-------------------------+
|        hardware         |          |        hardware         |
+-------------------------+          +-------------------------+

A virtual machine pretends to be a whole computer, so each one boots its own kernel. A container is a group of normal processes on the host's kernel, each given a private view of the system. Everything else in this lesson follows from that one difference.

Hypervisors and virtual machines

A hypervisor (virtual machine monitor) must, as Popek and Goldberg put it in 1974, give a guest an environment equivalent to the real machine, keep control of the real resources, and run most instructions directly on the CPU for speed. The classic technique is trap and emulate: the guest kernel runs without full privileges, so every privileged instruction it tries (changing page tables, talking to devices, disabling interrupts) traps into the hypervisor, which emulates the effect on the guest's virtual hardware.

Early x86 broke that rule: some sensitive instructions (POPF is the famous one) silently behaved differently instead of trapping. VMware worked around it with binary translation and Xen with paravirtualization (a guest kernel modified to call the hypervisor directly). Since 2005-2006 CPUs have hardware virtualization: Intel VT-x and AMD-V add a guest mode that traps exactly what the hypervisor asks for. Nested paging followed (AMD NPT in 2007, Intel EPT in 2008) and adds a second level of address translation, guest-physical → host-physical, on top of the page tables from the memory lesson. Disks and network cards are usually paravirtual (virtio) because emulating real hardware register by register is slow.

Type 1 (bare metal) hypervisors run directly on the hardware: Xen, VMware ESXi, Microsoft Hyper-V. KVM turns the Linux kernel itself into one (QEMU supplies the virtual devices)
Type 2 (hosted) hypervisors run as an application on a normal OS: VirtualBox, VMware Workstation

Containers: processes with a restricted view 📦

There is no "container" object in the Linux kernel. A container is an ordinary process (and its children) placed in a set of namespaces, each of which gives it a private copy of one global resource. namespaces(7) lists eight kinds:

Mount: the list of mounted file systems (the container gets its own root file system)
UTS: the hostname and domain name
IPC: System V IPC objects and POSIX message queues
PID: process IDs (the first process inside becomes PID 1)
Network: network interfaces, IP addresses, routing tables, firewall rules, ports
User: user and group IDs, so UID 0 inside can be an unprivileged UID outside
Cgroup: which part of the cgroup tree the process sees as its root
Time: the boot-time and monotonic clocks (since Linux 5.6)

You can see a process's namespaces as links in /proc/<pid>/ns/ and list them all with lsns. As root, unshare --pid --fork --mount-proc bash starts a shell in a new PID namespace where ps shows only two processes: the shell is PID 1 there and ps is PID 2.

Namespaces nest. A PID namespace has a parent, and a process has a PID in its own namespace and in every ancestor: inside the container nginx may be PID 1 while the host knows it as PID 48213. The parent can see and signal processes in child namespaces; the child cannot see the parent's. And if the PID 1 of a namespace exits, the kernel kills every other process in that namespace (pid_namespaces(7)).

User namespaces remap IDs the same way through /proc/<pid>/uid_map. Rootless Podman, for example, maps root inside the container to your own UID and the other container UIDs to a range from /etc/subuid:

Root in the container (UID 0) is plain UID 1000 on the host, and www-data (33) is 100032, a UID nobody owns. A process that escapes the container lands with those powerless IDs. Try mapping 0 to 0 to see why a container without a user namespace is riskier.

# /proc/<pid>/uid_map of a rootless container, one range per line:
#   ID-inside-namespace  ID-outside-namespace  length
uid_map = """
0      1000    1
1      100000  65536
"""
ranges = [tuple(map(int, line.split())) for line in uid_map.strip().splitlines()]

def to_host(uid):
    for inside, outside, length in ranges:
        if inside <= uid < inside + length:
            return outside + (uid - inside)
    return None                      # unmapped IDs show up as the overflow ID 65534

for uid in [0, 1, 33, 1000, 65536, 70000]:
    host = to_host(uid)
    shown = host if host is not None else "unmapped (seen as 65534)"
    print(f"container uid {uid:>5} -> host uid {shown}")

Open in Code Lab

Lab exercise 1

PythonGraded in Code Lab

Simulate PID namespaces: one process can have a different PID in every namespace it is visible from.

Input: one command per line. At the start only init exists, PID 1 of the root namespace.

spawn NAME PARENT: start NAME as a child of PARENT, in PARENT's PID namespace
spawn NAME PARENT newns: the same, but NAME becomes the first process of a new PID namespace nested inside PARENT's namespace (like clone() with CLONE_NEWPID)
pids NAME: print NAME's PID in its own namespace, then in each ancestor namespace up to the root
ps NAME: list the processes NAME can see, those in its namespace and in every namespace nested inside it, with their PIDs as seen from NAME's namespace
exit NAME: NAME exits. If NAME is PID 1 of a nested namespace, the kernel also kills every process of that namespace and of the namespaces nested inside it

Rules: in every namespace PIDs are handed out 1, 2, 3, ... in creation order, counting every process created in that namespace or in any namespace nested inside it; they are never reused. The input is valid: names exist, new names are unique, init never exits and nothing is spawned into a namespace that was killed. Blank lines are ignored.

Output:

pids NAME: <own> <parent namespace> ... <root>
ps NAME: <pid>=<name> ... sorted by PID
exit NAME, followed by : also killed <names in creation order> when others died with it

Example:

spawn sshd init
spawn box init newns
spawn nginx box
ps init
ps nginx
pids nginx
exit box
ps init

prints

ps init: 1=init 2=sshd 3=box 4=nginx
ps nginx: 1=box 2=nginx
pids nginx: 2 4
exit box: also killed nginx
ps init: 1=init 2=sshd

Visible tests (5)

Example from the prompt
Namespaces inside namespaces
A normal exit kills nobody else and PIDs are not reused
A nested init takes down only its own namespace
Only init

Example run

Input: spawn sshd init spawn box init newns spawn nginx box ps init ps nginx pids nginx exit box ps init
Expected output: ps init: 1=init 2=sshd 3=box 4=nginx ps nginx: 1=box 2=nginx pids nginx: 2 4 exit box: also killed nginx ps init: 1=init 2=sshd

Starter code

import sys

# Namespace 0 is the root PID namespace. parent[ns] is the namespace it was created in.
parent = {0: None}
counter = {0: 1}                     # last PID handed out in each namespace
procs = {"init": {"ns": 0, "pids": {0: 1}}}   # name -> own namespace + its PID at every level
order = ["init"]                     # creation order of the processes still alive

def levels(ns):
    """The namespace itself, then each ancestor up to the root."""
    while ns is not None:
        yield ns

More lines in Code Lab: 24

Try in Code Lab

Hints and instant feedback are in Code Lab. Optional practice: it does not block finishing the lesson.

JavaScript and Python also run in your browser without an account; other languages use the secure server runner and need a free account.

Images: layers stacked with OverlayFS 🧅

A container image is not one big disk file. It is a stack of read-only layers, each a tarball of the files a build step added or changed (OCI image spec). At run time the mount namespace gets a root file system assembled by OverlayFS:

merged view (what the container sees)       /app/server.py  /bin/sh  /etc/os-release
                                                   ▲             ▲          ▲
upperdir   (writable, per container)               │             │          │
layer 3    (read-only)   /app/server.py v2  ───────┘             │          │
layer 2    (read-only)   /app/server.py v1, /etc/.wh.motd        │          │
layer 1    (read-only)   /bin/sh, /etc/os-release, /etc/motd ────┴──────────┘

A file in a higher layer hides the same path in lower layers
Layers are never modified. The first write to a lower file copies it up into the container's writable upper layer, so a hundred containers can share one image on disk
Deleting needs a marker, because a lower layer cannot change: a whiteout. In image layers it is an empty file named .wh.<name>; /etc/.wh.motd deletes /etc/motd from the layers below. The special name .wh..wh..opq makes a directory opaque: nothing below it in lower layers shows through
A whiteout only applies to lower layers, never to files of its own layer

This is why RUN rm big.tar in a later build step does not shrink an image: the file still sits in the earlier layer, only hidden.

Lab exercise 2

GoGraded in Code Lab

Merge the layers of a container image the way OverlayFS presents them, including OCI whiteouts.

Input: a line layer starts a new layer; the first layer is the bottom one. Inside a layer each line is an entry with an absolute path:

<path> <content>: a regular file (the content is the rest of the line, possibly empty)
<dir>/.wh.<name>: a whiteout: <dir>/<name> and everything below it disappear from the lower layers
<dir>/.wh..wh..opq: an opaque marker: everything below <dir> disappears from the lower layers (/.wh..wh..opq hides everything)

Blank lines are ignored. Only files are listed; directories exist implicitly.

Rules: apply the layers bottom to top. For each layer, first apply its whiteouts and opaque markers to the result of the lower layers, then add its files (a file replaces the same path from below). A whiteout never hides a file of its own layer.

Output: every file of the merged view sorted by path (Go's sort.Strings), one per line as <path> <content>, then files: <count>.

Example:

layer
/etc/os-release debian 12
/etc/motd welcome
/bin/sh dash
layer
/etc/.wh.motd
/app/server.py v1
layer
/app/server.py v2

prints

/app/server.py v2
/bin/sh dash
/etc/os-release debian 12
files: 3

Visible tests (5)

Example: an upper file wins, a whiteout deletes
An opaque directory and a directory whiteout
A whiteout hides /a and /a/b but not /ab
A whiteout never hides a file of its own layer
No layers at all

Example run

Input: layer /etc/os-release debian 12 /etc/motd welcome /bin/sh dash layer /etc/.wh.motd /app/server.py v1 layer /app/server.py v2
Expected output: /app/server.py v2 /bin/sh dash /etc/os-release debian 12 files: 3

Starter code

package main

import (
	"bufio"
	"fmt"
	"os"
	"path"
	"sort"
	"strings"
)

const whiteoutPrefix = ".wh."

More lines in Code Lab: 54

Try in Code Lab

Hints and instant feedback are in Code Lab. Optional practice: it does not block finishing the lesson.

JavaScript and Python also run in your browser without an account; other languages use the secure server runner and need a free account.

cgroups: how much a group may use 📏

Namespaces control what a process can see; control groups (cgroups) control how much it can use. Modern distributions mount the unified cgroup v2 hierarchy at /sys/fs/cgroup: every directory is a cgroup, mkdir creates one and writing a PID to its cgroup.procs file moves a process into it. Each controller exposes plain files:

cpu: cpu.max holds QUOTA PERIOD in microseconds. 50000 100000 allows 50 ms of CPU time per 100 ms period, half a CPU; max 100000 (the default) means no limit. cpu.weight sets a relative share instead of a hard cap
memory: memory.current is the usage of the cgroup and all its descendants; memory.high throttles and reclaims above a soft limit; memory.max is the hard limit. When usage would exceed it and the kernel cannot reclaim enough, the OOM killer kills a process inside that cgroup
pids: pids.max caps the number of processes and threads, which stops a fork bomb
io: io.max limits bytes or operations per second per device

Limits are hierarchical: a cgroup can never use more than any of its ancestors allows, because its usage is charged to every ancestor too. docker run --memory=512m --cpus=1.5 --pids-limit=64 just writes memory.max, cpu.max = 150000 100000 and pids.max. The Code Lab runner that grades your labs sets the same kind of limits for every program you submit: --memory, --cpus and --pids-limit, plus --network none and a read-only root file system. See what cpu.max does to a CPU-bound job:

With half a CPU the job runs 50 ms, waits 50 ms, and finishes after 450 ms. A quota of 1.5 CPUs does not help a single thread, which can only use one CPU at a time. The last line is the same half CPU with a 10 ms period: the job finishes after 495 ms, but its longest pause shrinks from 50 ms to 5 ms. Shorter periods make throttling smoother.

// cpu.max is "QUOTA PERIOD" in microseconds: the cgroup may use QUOTA of CPU time
// in every PERIOD, then it is throttled until the next period starts.
function run(cpuMax, cpuMs) {
  const [quota, period] = cpuMax.split(" ");
  if (quota === "max") return { wall: cpuMs, pause: 0 }; // no limit
  const p = Number(period) / 1000;
  const q = Math.min(Number(quota) / 1000, p);           // one thread can use at most one CPU
  const throttledPeriods = Math.ceil(cpuMs / q) - 1;     // every period but the last ends throttled
  return {
    wall: throttledPeriods * p + (cpuMs - throttledPeriods * q),
    pause: throttledPeriods > 0 ? p - q : 0,             // the wait at the end of a throttled period
  };
}

const job = 250; // a single-threaded job that needs 250 ms of CPU time
for (const cpuMax of ["max 100000", "50000 100000", "25000 100000", "150000 100000", "5000 10000"]) {
  const [quota, period] = cpuMax.split(" ");
  const cpus = quota === "max" ? "unlimited" : (Number(quota) / Number(period)) + " CPU";
  const { wall, pause } = run(cpuMax, job);
  console.log(("cpu.max \"" + cpuMax + "\"").padEnd(24) + ("(" + cpus + ")").padEnd(14) + "-> done after " + wall + " ms, longest pause " + pause + " ms");
}

Open in Code Lab

Lab exercise 3

JavaScript (Node.js)Graded in Code Lab

Enforce hierarchical memory limits like cgroup v2's memory.max.

Input: one command per line (blank lines are ignored). The root cgroup / always exists and has no limit. Sizes are whole MiB.

mkdir PATH MAX: create a cgroup; MAX is a number or max (no limit). Errors: mkdir PATH: File exists if it exists, mkdir PATH: No such file or directory if its parent does not
charge PATH N: a process in PATH wants N more MiB. The usage counts against PATH and every ancestor. If any of them would go above its MAX, the charge is refused and charge PATH N: OOM in <cgroup> names the closest such cgroup, starting from PATH and walking up; otherwise every cgroup on the way grows by N and charge PATH N: ok is printed
uncharge PATH N: PATH and every ancestor shrink by N (the input never frees more than was charged)
show: one line per cgroup in creation order, root first: <path> current=<MiB> max=<MAX>

Reaching the limit exactly is allowed. (In the kernel a charge over memory.max first triggers reclaim, and if that fails the OOM killer kills a process in the cgroup; here the charge is simply refused.)

Example:

mkdir /web 512
mkdir /web/api 300
charge /web/api 200
charge /web/api 120
show

prints

charge /web/api 200: ok
charge /web/api 120: OOM in /web/api
/ current=200 max=max
/web current=200 max=512
/web/api current=200 max=300

Visible tests (5)

Example: the cgroup's own limit
A parent's limit stops a child without one
mkdir errors and reaching the limit exactly
Three levels: the closest full cgroup is reported
Only the root

Example run

Input: mkdir /web 512 mkdir /web/api 300 charge /web/api 200 charge /web/api 120 show
Expected output: charge /web/api 200: ok charge /web/api 120: OOM in /web/api / current=200 max=max /web current=200 max=512 /web/api current=200 max=300

Starter code

const lines = require("fs").readFileSync(0, "utf8").split("\n");

// path -> { max: MiB (Infinity = no limit), current: MiB used by the cgroup and its descendants }
const groups = new Map([["/", { max: Infinity, current: 0 }]]);

function parentOf(p) {
  const i = p.lastIndexOf("/");
  return i === 0 ? "/" : p.slice(0, i);
}

// TODO 1: chain(p) returns [p, parent, grandparent, ..., "/"]

More lines in Code Lab: 16

Try in Code Lab

Hints and instant feedback are in Code Lab. Optional practice: it does not block finishing the lesson.

JavaScript and Python also run in your browser without an account; other languages use the secure server runner and need a free account.

What isolation do you really get? 🛡️

Virtual machines

Each guest has its own kernel, so a kernel bug or a root compromise inside the guest stays inside it. The boundary is the hypervisor and the CPU's virtualization hardware, a much smaller interface than a full system-call table
Costs: a whole OS to boot (seconds), patch and keep in memory for every VM
This is why public clouds separate different customers with VMs

Containers

Every container shares the host kernel. Namespaces hide things; they do not make the kernel any harder to attack. A kernel vulnerability reachable through a system call can become an escape for every container on the host
Without a user namespace, root in the container is UID 0 on the host; rootless containers remove that risk
Docker and Podman reduce the attack surface by default: a reduced set of capabilities, a seccomp filter that blocks rarely needed system calls, and an AppArmor or SELinux profile. The security lesson covers these tools
Settings that throw the isolation away: --privileged (all capabilities, all host devices, no seccomp), mounting /var/run/docker.sock (control of the host's container engine, which is root-equivalent), --pid=host or --network=host, and writable mounts of host directories
Without cgroup limits one container can starve the others of memory, CPU or PIDs. Isolation includes resources, not just visibility

In between

gVisor puts an application kernel, written in Go, between the container and the host and implements most system calls itself, so far fewer reach the host kernel
Kata Containers and Firecracker run each container or function in a lightweight microVM that boots in a fraction of a second; Firecracker runs AWS Lambda and Fargate workloads

And one thing that is not a boundary at all: chroot only changes how paths are resolved, and a root process can leave it.

Quick quiz

Question 1 of 5

What is the most important isolation difference between a VM and a container?

Question 2 of 5

Which kernel feature gives a container its own PID 1 and hides the host's processes?

Question 3 of 5

A container's cgroup has memory.max = 512M. A process inside needs more, and the kernel cannot reclaim enough. What happens?

Question 4 of 5

The top layer of an image contains an empty file /etc/.wh.motd, and a lower layer has /etc/motd. What does the container see?

Question 5 of 5

Why is mounting /var/run/docker.sock into a container dangerous?

Summary

A hypervisor runs complete VMs, each with its own kernel; trap-and-emulate became practical on x86 with VT-x/AMD-V, nested paging and virtio. Type 1 runs on the hardware, type 2 on a host OS, and KVM turns Linux into a hypervisor
A container is a normal process in a set of namespaces (mount, UTS, IPC, PID, network, user, cgroup, time). PID namespaces nest, so a process has one PID per level, and the death of a namespace's PID 1 kills the namespace
Images are read-only layers merged by OverlayFS; writes copy up, and whiteouts (.wh., .wh..wh..opq) hide lower files
cgroups v2 limit usage hierarchically: cpu.max, memory.max (then the OOM killer), pids.max, io.max
VMs give a stronger boundary at a higher cost; containers share the kernel, so add user namespaces, dropped capabilities, seccomp and resource limits, and never --privileged or the Docker socket without need. gVisor and microVMs sit in between
You simulated PID namespaces, merged image layers with whiteouts and enforced hierarchical memory limits

Practice: on a Linux machine run lsns, ls -l /proc/self/ns and cat /proc/self/cgroup, then look at /sys/fs/cgroup and a running container's memory.max. Next: OS security and hardening, where capabilities, seccomp and memory protections get a lesson of their own.

Practice

Reading is not learning until you run something. Pick one.

Graded lab exercises

Lab exercise 1PythonNot passed yet Try in Code Lab
Lab exercise 2GoNot passed yet Try in Code Lab
Lab exercise 3JavaScript (Node.js)Not passed yet Try in Code Lab

Open an example in Code Lab

Blank Code Lab in C

File Systems & Storage — From a Name to Disk Blocks

Next lesson

OS Security & Hardening — Least Privilege in Practice