Skip to main content

Running a workload on Linux: FluxVM container or microVM

On Linux you choose how an image runs, behind one front door:

fluxctl sandbox run alpine:3.22 --runtime container --trust dev -- echo hi # FluxVM's own container runtime
fluxctl sandbox run alpine:3.22 --runtime vm -- echo hi # a microVM (see "What is not there yet")
fluxctl sandbox run alpine:3.22 --runtime auto --trust agent -- echo hi # auto: the isolation follows the trust level

container is FluxVM's own runtime: no Docker, no runc, nothing to install besides FluxVM. It is shaped like systemd services and systemd-nspawn, not like a Docker clone: every container is a transient systemd unit, and the profile is written in systemd's hardening vocabulary. vm is a microVM with its own kernel. The default auto picks by trust and always says what it picked.

What already existed (read before this change)​

PieceWhat it isLinux host runtime?
oci-sandboxes.md, fluxvm-oci-init, oci_sandbox.rsone microVM per container image, on vz (macOS) only; refused elsewhere (OCI sandboxes run on the vz backend)no
docker.sock subset (docker-engine.md)Docker Engine API subset, each container a microVM (macOS); fluxctl docker = a real dockerd in one managed VMno: both run in VMs
fluxvm-container-agent / -client / -protocol / -containerd-shimin-guest OCI supervisor (pivot_root, namespaces, cgroup v2, seccomp, capabilities) for containerd/Kubernetes Pod VMs. It runs inside a guest, over vsockno: guest side only, not a host runtime
fluxvm-procbox, procbox_sandbox.rs (procbox.md)rootless Landlock + seccomp process sandbox on the host kernel: no image, no rootfs, no cgroupspartly: a process tier, not a container
QEMU, Cloud Hypervisor, Firecracker, the in-tree KVM engine, the jailer, fluxvm-cgroup, spec_lintmicroVM backends on Linux; per-VM cgroup v2yes, for VM sandboxes (a Debian image), not for OCI images
fluxvm-image Puller/BlobStore, fluxvm-oci-init unpacknative registry pull and layer unpacking (diff_id check, whiteouts, symlink-safe)reused here
linux-containers-research.mdrecommends both tiers behind one front doorthis document implements the front door for Linux

So Linux had microVMs for VM sandboxes and a weak process tier, but no way to run an OCI image on the host.

The decision​

  • Both runtimes, one front door (runtime on the sandbox API, --runtime in fluxctl, sandbox.container.default_runtime in config).
  • container is FluxVM's own native runtime: fluxvm-ctr (this change). Pull and unpack reuse the existing image code; the launcher fluxvm-container-run does namespaces, cgroup v2 (through systemd), seccomp and capability drop. It is built on the pieces already in the repo (procbox's seccomp compiler, oci-init's unpacker), not on Docker or runc.
  • systemd runs the container, as in nspawn: lifecycle, restart policy, cgroup delegation and limits, ordering and the lifetime come from a transient unit, not from FluxVM re-implementing a supervisor. Without systemd as PID 1 the same runtime runs with the daemon managing the cgroup directly.
  • Safe by default: auto gives dev/trusted workloads a container and untrusted/agent workloads a microVM; container for untrusted code is refused unless the operator allows it and the caller asks for it. auto never falls from vm back to container.
  • Not Docker API compatible, by design. Docker API clients keep using the docker.sock subset or fluxctl docker (see the matrix).

Support matrix​

container (native, this change)vm, one microVM per containershared-VM engine (fluxctl docker)
Linux hostyes (needs the host checks below)not yet for OCI images (vz/macOS only today); VM sandboxes without an image workyes (a real dockerd in one VM)
Isolationshares the host kernel: namespaces, cgroup v2, seccomp, dropped caps. A kernel bug reachable through an allowed syscall is an escapeown kernel behind KVM; an escape needs a hypervisor bugcontainers share the engine VM's kernel; the VM is the boundary to the host
Start timetens of ms to start the process once the image is unpacked (first use pays pull + unpack)about a second on vz; seconds for a cold Linux VM boot (snapshot restore is the fast path, see below)one VM start (minutes the first time, it installs Docker), then container-fast
Docker APInothe docker.sock subset (macOS)full
Rootlessyes, with an unprivileged user namespace (uid 0 inside = your uid; see AppArmor note)needs /dev/kvm access, no rootengine VM needs /dev/kvm
Kernel featurescgroup v2, seccomp, mount/pid/ipc/uts/net namespaces; user namespaces for rootless; systemd >= 240 for append: logs/dev/kvm (KVM), a backend (QEMU/Cloud Hypervisor/Firecracker)/dev/kvm
Limitsmemory, cpu, pids, io weight (cgroup v2 via systemd or direct)vcpus, memoryengine VM
Networknone (loopback) by default, or the host namespace for trusted code. No published ports, no egress policy yetNIC with the egress policy (vz)docker networks
exec / files / logs / TTL / deleteyes (same sandbox API)yesdocker CLI

Which one should I pick​

  • Your own build and test loops, CI steps, dev tools: container (fast, rootless, systemd-managed).
  • Code you did not write, anything an agent chooses to run, multi-tenant: vm.
  • You need docker build, Compose, networks, the Docker API: fluxctl docker.

One front door​

fluxctl sandbox run IMAGE [--runtime auto|container|vm] [--trust dev|trusted|untrusted|agent]
[--allow-container-untrusted] [--memory-mib N] [--cpu-percent N] [--pids N] [--io-weight N]
[--net none|host] [--writable-root] [--restart on-failure] [--print-unit] [-- CMD...]
fluxctl sandbox inspect ID [--unit] # the runtime decision, the container as a systemd unit, its state
fluxctl sandbox runtimes # what this host supports, with fix hints (also in `fluxctl doctor`)
fluxctl sandbox logs ID # console tail and the exit code

The same fields exist on POST /v1/sandboxes (runtime, trust, allow_container_untrusted, container), on the MCP sandbox_create tool, and as defaults in config:

[sandbox.container]
default_runtime = "auto" # auto | container | vm
default_trust = "untrusted" # a request that does not say gets the safe default
allow_untrusted = false # lets callers pass allow_container_untrusted

auto explains itself in every output (stderr of sandbox run, the labels fluxvm.runtime and fluxvm.runtime.reason on the record):

runtime: container (auto: trust=dev gets a native container (fast start, shares the host kernel))
container: the workload shares the host kernel. Namespaces, cgroup v2, seccomp and dropped capabilities confine it, ...

Refusal for untrusted code (HTTP 403 on the API):

refusing runtime=container for untrusted code. container: the workload shares the host kernel... vm: the workload runs in its
own kernel behind hardware virtualization... Use --runtime vm, or have the operator set sandbox.container.allow_untrusted = true
and then pass --allow-container-untrusted

--print-unit: the container is transparent​

fluxctl sandbox run alpine --runtime container --trust dev --memory-mib 256 --print-unit runs nothing and prints the unit the daemon would start, plus the equivalent systemd-run command. fluxctl sandbox inspect ID --unit prints the real one. Example:

[Unit]
Description=FluxVM container fx-1a2b3c4d (alpine:3.22)

[Service]
Type=exec
Delegate=yes
MemoryMax=256M
MemorySwapMax=0
TasksMax=512
Restart=no
KillMode=control-group
NoNewPrivileges=yes
StandardOutput=append:/var/lib/fluxvm/instances/<id>/console.log
ExecStart=/usr/bin/fluxvm-container-run run --spec /var/lib/fluxvm/instances/<id>/spec.json

# Enforced by fluxvm-container-run inside the container, in systemd's own vocabulary.
[X-FluxVM-Container]
RootDirectory=... ReadOnlyPaths=/ PrivateTmp=yes PrivateDevices=yes PrivateNetwork=yes PrivateUsers=yes
CapabilityBoundingSet=CAP_CHOWN CAP_DAC_OVERRIDE ... SystemCallFilter=~ptrace mount umount2 pivot_root ... unshare setns

The [Service] section is real systemd configuration (cgroup delegation and limits, restart, lifetime, logs, NoNewPrivileges). The [X-FluxVM-Container] section uses the same directive names for what the launcher enforces inside the container; systemd ignores X- sections, so the file loads as is, and a hardening directive can be lifted from it into your own units. (RootDirectory= and PrivateUsers= cannot be given to systemd directly: it would need its own pivot into a rootfs it did not prepare, and the user manager refuses it, see "Verified".)

How the native container runs​

  1. oci_images::pull fetches the image (existing registry code, per-digest blob store). ensure_cache unpacks the layers once per manifest digest into ctr/rootfs/<digest>-<user|root> (diff_ids verified, whiteouts honoured, no path escapes), with the mountpoints a read-only root needs. Read-only root (the default) shares that directory between containers; --writable-root takes a private copy (cp --reflink=auto: a reflink on btrfs/XFS, a plain copy elsewhere).
  2. The spec (spec.json) is derived by one pure function (build_spec): image process rules (Entrypoint/Cmd, env, workdir, user) as in the vz path, limits against the operator caps, refusals for what is not implemented.
  3. systemd starts fluxvm-container-run run as a transient unit (systemd-run, --user when the daemon is not root). The launcher unshares mount, pid, ipc, uts and net (and user when unprivileged), bind-mounts the rootfs (read-only on request), mounts /proc, a minimal /dev (null zero full random urandom tty), /tmp, /run, pivot_roots, drops capabilities from the bounding set, sets no_new_privs, installs the seccomp filter (the procbox denylist plus unshare/setns; shown in the unit as SystemCallFilter=), and execs the entrypoint as PID 1.
  4. exec joins the container's cgroup and namespaces (setns) and runs the command with the same restrictions; fs read/write join the mount namespace and do the file I/O there, so symlinks resolve inside the container. logs read the console file and the exit code. delete stops the unit, removes the cgroup and the workspace. TTL becomes RuntimeMaxSec= and the sandbox TTL reaper.
  5. Limits are never silently dropped: a limit that cannot be enforced fails the start.

Cgroup delegation and the "failed to delegate cgroup controllers" warning​

The daemon used to warn when it was not given a delegated subtree. packaging/systemd/fluxvm.service now sets Delegate=yes, the warning explains the fix, and fluxctl doctor reports which controllers are available. Containers do not depend on the daemon's delegation: each is its own unit with Delegate=yes.

Without systemd as PID 1 (Alpine/OpenRC, containers, WSL)​

The same runtime, with manager = "direct" (chosen automatically): the daemon starts the launcher itself (new process group, console file as stdout), the launcher creates <daemon cgroup>/fx-XXXX, writes the cgroup v2 files (memory.max, memory.swap.max, memory.oom.group, cpu.max, pids.max, io.weight), and moves itself in. Restart policy and RuntimeMaxSec come from the daemon's TTL instead of systemd. fluxctl doctor reports this as manager: no systemd manager available.

Rootless, and the AppArmor catch on Ubuntu​

Rootless containers map your uid to root inside a user namespace (PrivateUsers=yes); other uids need a root daemon (or newuidmap, not implemented). Ubuntu 23.10 and later restrict unprivileged user namespaces with AppArmor (kernel.apparmor_restrict_unprivileged_userns=1): a program without an AppArmor profile that says userns, gets a namespace without capabilities, and the launcher cannot mount or pivot. Fix, either:

  • install deploy/apparmor/fluxvm-container-run (sudo cp into /etc/apparmor.d/, apparmor_parser -r), the way Ubuntu does for runc, crun and podman; or
  • run the daemon as root (rootful containers need no user namespace); or
  • sysctl kernel.apparmor_restrict_unprivileged_userns=0 (a system-wide relaxation, not recommended).

fluxctl doctor and fluxctl sandbox runtimes detect this and print the fix. The launcher itself refuses with the same hint instead of failing obscurely.

What is not there yet (refused, never faked)​

  • vm for an OCI image on Linux. The one-VM-per-container flow needs the vz kernel/initramfs and a virtiofs meta share. Extending it to QEMU/Cloud Hypervisor/Firecracker needs an OCI boot image (kernel, initramfs with fluxvm-oci-init), a block device built from the image (the same unpack, into ext4) and the vsock agent. The request is refused with that reason; runtime: vm without an image creates the usual VM sandbox. Next step: build the ext4 rootfs from the shared unpack cache and boot it with the existing Linux backends.
  • Published ports, private networks, the egress allow-list, volumes, health checks, secrets for containers: refused with a reason. --offline (the default) is a network namespace with only loopback. Networking plan: veth + bridge only when the daemon is root and configured; rootless user-mode networking (slirp4netns style; fluxvm-usernet is the likely base) otherwise; the egress policy then applies at the bridge.
  • Non-root image users under a rootless daemon; user names are resolved from the image's /etc/passwd for a root daemon.
  • Stacks (after/requires exist on the spec and in the unit; the stack file format does not drive them yet).
  • A status update when a container exits (the record stays running; sandbox logs reports the exit code and stopped).
  • Overlay rootfs (copy or shared read-only today; overlayfs is available on the test host).

Next tier: faster starts (design only, not in the first slice)​

The rootfs layer is a function image -> directory (ensure_cache) and the runtime only needs a rootfs path, so these slot in without changing the runtime API (spec.rootfs stays a directory or, later, a mount the launcher is given).

  • Lazy pulling. A FAST '26 paper (https://www.usenix.org/system/files/fast26-wang-li.pdf) is reported to find that image pull is about 76% of container start time while only about 6.4% of that data is read. Not verified here: I could fetch the PDF but not read its text. Candidates: Nydus (on-demand chunk download; can serve through in-kernel EROFS), SOCI (keeps the original OCI image and adds a separate index artifact in the registry, so existing images work).
  • EROFS layers. containerd 2.3.0 (30 April 2026) release notes list "Add EROFS layer media type", "Support zstd-wrapped EROFS layers", "Add os.features support for EROFS native container images" and "Add dmverity support to the erofs snapshotter" (verified against the release page; the claim that layers are received straight from a registry is implied by the media type, not stated, and I did not test it). No composefs-with-containerd-2.x source was found.
  • Pre-baked, snapshot-restored microVMs are the fast path for vm: FluxVM already has snapshots and vm-templates; restore replaces cold boot (about two seconds against about eight on a Mac with the warm pool).
  • Needs real hardware or kernels to verify. The test host's kernel is 7.0.0-43 (Ubuntu): CONFIG_EROFS_FS=m with zstd, lzma, xattr and file-backed mounts, CONFIG_FS_VERITY=y, overlayfs built as a module and loaded. The erofs module exists but is not loaded, and loading it or mounting needs root, so nothing EROFS was exercised. EROFS on-demand (fscache) support, dm-verity, and the lazy-pull candidates are untested here.

Verification status​

What ran for real, and what is only unit-tested. Nothing here was run on a Mac.

Run live, in a Debian 13 guest (own QEMU/KVM VM on the Linux test host, kernel 6.12.117, cgroup v2, systemd, statically linked launcher, a static busybox as the image rootfs), both as an unprivileged user (rootless) and as root:

  • run: PID 1 inside, own hostname, uid 0, exit code propagated (7) and recorded; read-only root (Read-only file system), tmpfs /tmp; NoNewPrivs: 1, Seccomp: 2, bounding set a80425fb (the Docker default set); mount(2) refused; only lo in the network namespace.
  • exec (joins namespaces and cgroup; pid 2 inside, same restrictions), fs write/fs read through the mount namespace (mode kept), a write to the read-only root refused.
  • cgroup v2 limits through the direct manager: memory.max, memory.swap.max, pids.max, cpu.max written and the container in the cgroup; an 80 MB allocation under memory.max 32 MiB was OOM-killed (oom_kill counted; memory.oom.group makes the kill take the whole container, a deliberate choice).
  • A transient system unit (systemd-run --property=Delegate=yes MemoryMax=48M TasksMax=24 ...) running the launcher: the properties applied, exec worked inside the unit's cgroup, systemctl stop ended it.
  • Two bugs found and fixed by this (the child shared the launcher's mount namespace; /proc changed under setns).

Run live, on the Ubuntu 26.04 test host (kernel 7.0, systemd 259 with a user manager, AppArmor restricting user namespaces):

  • systemd-run --user with Delegate=yes, MemoryMax, TasksMax, NoNewPrivileges, StandardOutput=append: works; writing the cgroup v2 limit files in the delegated user subtree works (memory.max=67108864, pids.max=32, cpu.max=50000 100000).
  • A rootless container cannot start there: unshare -Ur and the launcher fail at setgroups/uid_map (AppArmor unprivileged_userns denies capabilities), RootDirectory= is refused by the user manager (EOPNOTSUPP), PrivatePIDs=yes fails (226/NAMESPACE). The launcher reports the AppArmor cause and the fix. The shipped AppArmor profile was not installed or tested there (it needs root and changes system policy).
  • fluxctl sandbox run ... --print-unit printed the unit and the systemd-run command for the host's user manager; fluxctl doctor --host-only printed the Linux runtime checks (manager, cgroup delegation writable: false, rootless FAIL with the AppArmor fix, seccomp, launcher, each runtime).
  • Runtime choice through the real CLI against a local manager: --runtime container --trust untrusted refused with the isolation difference; auto with the default trust and --runtime vm for an image refused with the reason; --trust dev refused with the AppArmor cause.
  • A real registry pull of alpine:3.22, layer unpack with diff_id verification into the shared rootfs (mountpoints added), the second call reusing the cache, the writable copy, and the image process resolved (/bin/sh): the ignored test live_pull_and_unpack.

Unit-tested only (golden strings and fakes): runtime selection and refusal rules, spec generation, unit/systemd-run/cgroup file rendering, host verdicts and doctor lines, user resolution, the request/CLI/REST fields, --print-unit.

Not verified at all: the daemon end to end (create through the REST API, pull, start, exec, delete) with a real image; the systemd user manager running the launcher (blocked by AppArmor on the only host with one); restart policy, TTL (RuntimeMaxSec), journald; the direct manager as a non-root user; reflink copies; anything on macOS; EROFS, lazy pulling, snapshot-restored microVMs.