REST API reference
Start the server:
sudo /usr/local/bin/fluxctl --config /etc/fluxvm.toml serve
Default bind address:
127.0.0.1:7788
Endpoints
GET /healthz
GET /readyz
GET /metrics
GET /v1/openapi.json # OpenAPI 3.1, VM surface (no auth)
POST /v1/vms # ?ready=exec adds first_command_ms + phases; Idempotency-Key honored
GET /v1/vms
GET /v1/vms?name=<name>
GET /v1/vms?tenant=<tenant>
GET /v1/vms?label=<selector> # env=prod,team!=x,gpu,!tmp
GET /v1/vms/{uuid}
PATCH /v1/vms/{uuid} # {"name": "...", "labels": {"k": "v", "gone": null}}
POST /v1/vms/{uuid}/start
POST /v1/vms/{uuid}/restart
POST /v1/vms/{uuid}/clone # {"name": "..."}; source must be stopped
POST /v1/vms/{uuid}/fork # {"count": 1-32, "namePrefix": "..."}; admin; running flux-vm source; 201 {"items": [...], "elapsed_ms": n}; ?ready=exec; Idempotency-Key honored
POST /v1/vms/{uuid}/backup # {"name": "...", "compress": bool, "all_disks": bool, "quiesce": "auto"|"required"|"never"}
POST /v1/vms/{uuid}/restore-backup # {"name": "..."}; VM must be stopped
GET /v1/backups # newest first
DELETE /v1/backups/{name}
GET /v1/backups/{name}/root # the backup's standalone root qcow2 (download)
POST /v1/vms/{uuid}/start-from-snapshot
POST /v1/vms/{uuid}/snapshot # Idempotency-Key honored
POST /v1/vms/{uuid}/restore # {"tag": "..."}; admin; running flux-vm restores in place (memory + disk)
GET /v1/vms/{uuid}/snapshots
DELETE /v1/vms/{uuid}/snapshots/{tag}
GET /v1/vms/{uuid}/disks
POST /v1/vms/{uuid}/disks # {"name": "data", "size_gib": 10}, {"name": "pvc", "path": "/dev/rbd0"} or {"name": "disk1", "backing": "/var/lib/fluxvm/images/imported/web01/disk1.raw"} (qcow2 overlay); optional "io" (QEMU disk I/O, below)
PATCH /v1/vms/{uuid}/disks/{name} # {"size_gib": 20}; name "root" = boot disk
DELETE /v1/vms/{uuid}/disks/{name} # deletes the file (or overlay); only unlinks a path-attached disk
PUT /v1/vms/{uuid}/disks/{name}/throttle # {"throttle": {"iops_total": 500}}; null lifts the limits (QEMU)
# vz: also "kind" (image|block|nbd), "url", "read_only", "caching", "sync", "controller"
# (virtio|nvme|usb); applies at the next start (a USB image disk at once); see macos.md#extra-disks
POST /v1/vms/{uuid}/cdroms/{name}/eject # remove install media (live when running); the empty drive stays
GET /v1/vms/{uuid}/serial # websocket, raw serial bytes (QEMU)
# GET /v1/vms leaves out container warm-pool slots unless ?all=true or label=fluxvm.oci-warm...
GET /v1/vms/{uuid}/ports/{name} # websocket, raw bytes to and from a vz console port (apple.console_ports; admin)
POST /v1/vms/{uuid}/stop
POST /v1/vms/{uuid}/pause
POST /v1/vms/{uuid}/resume
POST /v1/vms/{uuid}/hibernate # qemu; save RAM/CPU/devices, stop QEMU; next start resumes; admin; see hibernation.md
DELETE /v1/vms/{uuid}/hibernation # drop the saved state; next start boots from disk; admin
POST /v1/vms/import-hibernated # {"id"}; adopt a VM hibernated on another node from the shared dir; admin
GET /v1/stats # one sample of this host and every VM (cumulative counters) for the fleet's metrics collector; see "Node stats"
GET /v1/volumes # persistent volumes on this host; admin; see "Volumes and tunnels (fleet platform)"
POST /v1/volumes # {"name": "data", "size_gib": 50}; sparse raw file; 201; 409 when the name is taken; admin
GET /v1/volumes/{name} # admin
DELETE /v1/volumes/{name} # 409 while any VM on this host references it; admin
POST /v1/volumes/{name}/resize # {"size_gib": 100}; grow only, and only while no VM references it; admin
GET /v1/vms/{uuid}/tunnel?port=N # websocket: one TCP connection to port N of a running VM (binary messages); admin; see "Volumes and tunnels (fleet platform)"
GET /v1/host/hibernation # {facts: {qemu_version, machine_type, cpu_vendor, cpu_flags, state_dir}}
POST /v1/host/hibernation/check # {"manifest"} -> {compatible, reasons}
POST /v1/vms/{uuid}/diagnostics # {pcap_seconds, pcap_filter, memory_dump, redact, log_lines} -> 202 bundle job; 409 if one is running; admin; see diagnostics-bundle.md
GET /v1/vms/{uuid}/diagnostics # bundles kept for the VM (last 5); admin
GET /v1/vms/{uuid}/diagnostics/{token} # job: state running|done|failed, sections, sha256, memory_dump; admin
DELETE /v1/vms/{uuid}/diagnostics/{token} # remove a finished bundle and its dump; 409 while running; admin
GET /v1/vms/{uuid}/diagnostics/{token}/bundle # tar.gz; 409 while running or failed; admin
GET /v1/vms/{uuid}/diagnostics/{token}/memory-dump # ELF guest memory dump; admin
POST /v1/vms/{uuid}/resources
POST /v1/vms/{uuid}/hotplug/cpu
POST /v1/vms/{uuid}/hotplug/memory
POST /v1/vms/{uuid}/hotplug/nic
POST /v1/vms/{uuid}/hotplug/nic/unplug # {"mac": ...} or {"tap": ...}; extra NICs only
POST /v1/vms/{uuid}/hotplug/share
GET /v1/vms/{uuid}/cpuset
GET /v1/vms/{uuid}/numa # qemu; guest NUMA nodes, live vCPU thread affinity, memdev binding; see host-topology.md
POST /v1/vms/{uuid}/freeze
POST /v1/vms/{uuid}/thaw
GET /v1/vms/{uuid}/frozen
GET /v1/vms/{uuid}/stats
GET /v1/vms/{uuid}/pressure
GET /v1/vms/{uuid}/balloon # flux-vm KVM engine, vz, qemu with balloon; {memory_mib, target_mib, actual_mib}
POST /v1/vms/{uuid}/balloon # {"balloon_mib": N}; admin; 0 deflates
GET /v1/vms/{uuid}/memory # VMM-process PSS/private/shared + balloon; see memory-density.md
GET /v1/vms/{uuid}/logs
GET /v1/vms/{uuid}/console
POST /v1/vms/{uuid}/agent # {"command", "timeout_seconds"?, "policy"?}; see guest-exec-policy.md
POST /v1/vms/{uuid}/agent/ping
POST /v1/vms/{uuid}/agent/put-file
POST /v1/vms/{uuid}/agent/get-file
GET /v1/vms/{uuid}/screenshot # vz; ?max_width=N; image/png of the guest display; admin
POST /v1/vms/{uuid}/input # vz; {"action": "type"|"key"|"click"|...} or {"actions": [...]}; admin
PUT /v1/vms/{uuid}/signin # vz; {"username"?, "password"} into the host login Keychain; admin
GET /v1/vms/{uuid}/signin # {"configured", "username"?}; never the password; admin
DELETE /v1/vms/{uuid}/signin # {"deleted": bool}; admin
POST /v1/vms/{uuid}/signin # {"mode"?: "password"|"username"|"username_tab", "submit"?: bool}; types it; admin
GET /v1/vms/{uuid}/qga/network-interfaces
DELETE /v1/vms/{uuid} # Idempotency-Key honored
GET /v1/vm-templates
POST /v1/vm-templates # {"name", "description"?, "spec", "replace"?}
GET /v1/vm-templates/{name}
DELETE /v1/vm-templates/{name}
POST /v1/vm-templates/{name}/instantiate # {"name": "vm-1", "labels"?: {...}}
GET /v1/events # ?vm=&event=<prefix>&since=<rfc3339>&limit=
GET /v1/events/stream # Server-Sent Events, same filters
GET /v1/quotas/me
POST /v1/images/build
POST /v1/images/import # {"source": "/path/vm.ova", "name": "...", "repair": true, "remove_vmware_tools": false}; admin; 201; see import-vmware.md
GET /v1/images/catalog
POST /v1/images/catalog
DELETE /v1/images/catalog/{name}
POST /v1/images/catalog/{name}/rename
POST /v1/images/catalog/{name}/clone
POST /v1/images/catalog/{name}/export
POST /v1/images/catalog/{name}/read-only
POST /v1/images/catalog/clean
GET /v1/oci/images # cached OCI rootfs images for vz sandboxes; see oci-sandboxes.md
POST /v1/oci/images # {"image": "alpine:3.22", "platform"?: "linux/amd64"}; admin; pull and build the rootfs
DELETE /v1/oci/images/{digest|reference} # admin
POST /v1/oci/prune # admin; drop images no sandbox was started from, and unused blobs
GET /v1/vznets # private vz networks (apple.networks): subnet, members, switch_running; see macos.md
POST /v1/pools
GET /v1/pools
GET /v1/pools/{name}
DELETE /v1/pools/{name}
POST /v1/pools/{name}/claim # ?ready=exec adds first_command_ms + phases
POST /v1/pools/{name}/resize
POST /v1/sandboxes # optional volumes; vz: "offline", "host_gpu" (host-inference.md), "oci": {...} (oci-sandboxes.md)
GET /v1/sandboxes
POST /v1/sandboxes/{id}/snapshot
POST /v1/sandboxes/{id}/fs/read
POST /v1/sandboxes/{id}/fs/write
POST /v1/sandboxes/{id}/process # {"command", "timeout_seconds"?, "policy"?} or {"process": {"argv", "env"?, "cwd"?}} (no shell; agent); see guest-exec-policy.md
GET /v1/sandboxes/{id}/logs # ?lines=N (200); console tail + container exit_code / init_error / restarts / health
POST /v1/sandboxes/{id}/dry-run # {"command", "paths"}; procbox: workspace copy; flux-vm/qemu/firecracker: snapshot/restore
POST /v1/sandboxes/{id}/speculate # {"command", "timeout_seconds"?, "paths"?, "ttl_seconds"?}; admin; returns a pending changeset
GET /v1/sandboxes/{id}/changesets # {"items": [...]}, newest first
GET /v1/sandboxes/{id}/changesets/{cs}
POST /v1/sandboxes/{id}/changesets/{cs}/approve
POST /v1/sandboxes/{id}/changesets/{cs}/reject
POST /v1/sandboxes/{id}/changesets/{cs}/apply # approved only; 409 on conflict
POST /v1/sandboxes/{id}/grants # credential broker grant; admin; 201 GrantInfo
GET /v1/sandboxes/{id}/grants # {"grants": [...]}, no secrets
DELETE /v1/sandboxes/{id}/grants # revoke all; {"revoked": bool}
DELETE /v1/sandboxes/{id}/grants/{grant_id} # revoke one; 404 if unknown
ANY /v1/sandboxes/{id}/http/{port}/{*path}
ANY /sandbox/{id}/{*path}
GET /v1/vms/{uuid}/network/policy
POST /v1/vms/{uuid}/network/policy
GET /v1/vms/{uuid}/network/effective
GET /v1/vms/{uuid}/network/egress # QEMU-process egress policy (user-mode guests): mode + rules, source, what the kernel enforces
PUT /v1/vms/{uuid}/network/egress # {"mode":"loopback|allowlist|audit|open","rules":[{"cidr","proto","port","note"}]} (admin)
DELETE /v1/vms/{uuid}/network/egress # back to the node default (admin)
GET /v1/vms/{uuid}/network/egress/observed # destinations refused (or, in audit mode, that would be)
POST /v1/vms/{uuid}/network/egress/suggest # rules that would allow what was observed; ?apply=true adds them (admin)
GET /v1/network/egress/defaults # node default policy
PUT /v1/network/egress/defaults # ?apply_running=true re-programs running VMs without their own policy (admin)
GET /v1/vms/{uuid}/network/status
GET /v1/vms/{uuid}/network/stats
GET /v1/vms/{uuid}/network/flows
POST /v1/vms/{uuid}/network/edge # Kairon VM-edge spec (admin); enforced and persisted, see vm-edge-contract.md
GET /v1/vms/{uuid}/network/conntrack # live conntrack export (admin); empty entries when not attached
POST /v1/vms/{uuid}/network/conntrack # restore (admin); identity mismatch is rejected, applied on attach if not attached
GET /v1/vms/{uuid}/network/learned-ip # {"ip","source"}: arp|nd from the datapath, else dhcp|fluxvm
GET /v1/vms/{uuid}/network/drops?limit=N # attributed drops with Kairon reason names (default 100, max 4096)
POST /v1/vms/{uuid}/network/capture # starts a 1-30s tcpdump capture on the VM edge (admin)
GET /v1/vms/{uuid}/network/capture # capture sessions and their state (admin)
GET /v1/vms/{uuid}/network/capture/{token} # the finished pcap; 409 while running (admin)
GET /v1/network/groups
POST /v1/network/groups
GET /v1/network/groups/{name}
DELETE /v1/network/groups/{name}
GET /v1/network/cnp
POST /v1/network/cnp
GET /v1/network/cnp/{name}
DELETE /v1/network/cnp/{name}
GET /v1/network/identities
GET /v1/network/observe
GET /v1/network/health
GET /v1/network/ipcache
GET /v1/network/ipam
GET /v1/network/vnets # virtual networks applied on this host; see virtual-network.md
GET /v1/network/vnets/{vni}
PUT /v1/network/vnets/{vni} # admin; replaces the config for the VNI
DELETE /v1/network/vnets/{vni} # admin; refused while a VM here is attached
POST /v1/network/refresh-dns
GET /console/ # FluxVM Studio web UI (static, no auth; it asks for a token); /console redirects here; see ui.md
POST /v1/vms/{id}/ws-ticket # {"kind":"console"|"serial"|"vnc"} -> single-use 30 s ticket for a browser WebSocket (vnc: QEMU machines only)
GET /console/ws/{ticket} # redeem a ticket (WebSocket upgrade; no Authorization needed)
Sandbox routes are the agent-sandbox surface on the FluxVm backend — see agent-sandbox-gaps.md and the Feature highlights.
GET /v1/vms?name=<name> exact-matches on VmRecord.name server-side. POST /v1/vms/{uuid}/start
relaunches a Stopped VM from its existing disk/seed, skipping the image-clone/cloud-init/token-inject
work create does — for a name-keyed register-then-start caller that already has a VM on disk it just
needs running again.
GET /metrics returns Prometheus text-exposition-format gauges: fluxvm_vms_total{status="..."},
fluxvm_vms_by_backend{backend="..."}, and fluxvm_vms_agent_enabled — point a Prometheus
scrape_config at it directly, no exporter needed.
GET /v1/vms/{uuid}/logs?lines=N&follow=true streams the VM's captured console output (raw serial,
no per-line structure) as chunked plain text — lines (default 100) controls how much history to
send before either ending (default) or switching to a live tail (follow=true, polling the log file
every 300ms for new lines). Verified against a real booting VM: both the initial tail and the live
follow stream return real, growing boot output.
Day-2 VM routes
Semantics, limits and CLI equivalents for restart, PATCH/labels, snapshots, clone, disks, serial, backup, scheduled snapshots, VM templates and events are in operations.md → Day-2 VM operations. Highlights:
GET /v1/vms/{uuid}/serialis a websocket to the QEMU serial port: binary and text frames are written to the guest verbatim, guest output comes back as binary frames. One client at a time; no guest agent needed.POST /v1/vms/{uuid}/backupalways writes understate_dir/backups/(the API never takes a destination path) and returns{name, path, size_bytes, live, quiesced, disks[]}. An optionalnamereplaces the default<vm>-<utc>so callers can find their backup again after a timeout.all_disksskips linked images and block devices, which FluxVM doesn't own.quiesce(defaultauto) freezes the guest's filesystems through QGA around the snapshot when the agent answers;requiredfails without it.GET /v1/backupslists backups with that metadata;POST /v1/vms/{uuid}/restore-backupcopies one back into a stopped VM, in the disk's own format. Any engine, on default storage or ashareddisk file; a running VM can be backed up only on QEMU with default storage.POST /v1/vm-templates/{name}/instantiategoes through the same tenant,created_by_tokenand token-quota handling asPOST /v1/vms. VM templates are separate from the sandbox/v1/templatesroutes.POST /v1/vms/{uuid}/forksnapshots a running flux-vm VM once and startscountchildren from that snapshot. Children share the read-only memory file and get a copy-on-write copy of the disk, a new CID, vsock socket and tap. They keep the parent's MAC and guest IP, so the source must usenetwork = none,user, ortapwith a per-VM netns, and no extra NICs. Either all children start or none are left behind. Token quotas are charged per child.scripts/bench-fork.shtimes it.GET /v1/vms/{uuid}/screenshotandPOST /v1/vms/{uuid}/inputlet an agent drive the display of a runningvzVM (Linux or macOS guest) without a console window or host Screen Recording/Accessibility permissions. The PNG's size is inx-fluxvm-screen-width/-height;x-fluxvm-screen-blank: truemeans the guest has drawn nothing yet. Input actions:type(text, US layout, printable ASCII plus newline and tab, up to 4096 characters),key(key: a character orenter,tab,escape,backspace,delete, arrows,home,end,page_up,page_down,f1-f12;modifiers:shift,control,option,command),move,click(withmodifiers),double_click,right_click,middle_click,down,up,drag(to_x,to_y) andscroll(dx,dywheel notches, optionalx,y). Coordinates are screenshot pixels from the top left; after?max_width=Npass"screen_width": Nso they are scaled back. A batch is validated as a whole before anything is sent (at most 100 actions)./v1/vms/{uuid}/signinkeeps a guest sign-in for avzVM in the host's login Keychain (servicedev.zyvor.fluxvm.vm-signin, account: the VM id) and types it into the display onPOST, so an agent can get past a login screen without ever seeing the password.modeispassword(default; the focused field),username(username, Enter, password: a text console) orusername_tab(username, Tab, password: a graphical form);submit(default true) presses Enter afterwards. Username and password are printable ASCII, at most 256 characters. Deleting the VM removes the item.GET /v1/events/streamemitsevent: <name>/data: <json>frames; tenant-scoped tokens only see their tenant's VMs.GET /v1/openapi.jsonis exempt from auth, like/healthzand/readyz.
Auth / RBAC
[[auth.tokens]] entries in the config (see config.example.toml) enable bearer-token auth on every
route except GET /healthz, GET /readyz (liveness vs readiness), GET /v1/openapi.json, the Studio pages under GET /console/,
and the two routes whose path carries their own credential: /preview/{token}/... (a signed, expiring agent-session preview URL) and
GET /console/pty/{ticket} (a single-use ticket minted by an authenticated request). Absent or empty auth.tokens
(the default) leaves the API exactly as open as the pre-auth MVP — every request is treated as
admin — but only on a loopback listener with no OIDC configured: auth.require = true, a non-loopback listen or OIDC make
authentication mandatory, and fluxctl serve refuses to start in that case with no token and no OIDC. Two roles:
admin— everything: create/stop/pause/resume/exec/delete/build-image/resources/hotplug/freeze/thaw.read-only— anyGETroute (/v1/vms,/v1/vms/{uuid},/metrics,/frozen,/stats,/pressure, pool list/get) only; any mutating route (includingresources/hotplug/freeze/thaw) returns 403.
POST /v1/egress/check is admin-only despite looking like a read-only diagnostic. It returns
{"allow": bool, "reason": "..."} for a host. A matching [sandbox] credential_vault
inject_authorization secret is not echoed back: the field is skip_serializing, and its
Debug output is [redacted], so the response cannot leak it. The route stays admin-only because
it reveals the egress allowlist decision for arbitrary hosts, and an earlier version had no role check
at all (see CHANGELOG). Per-sandbox credentials use a separate mechanism, see
credential-broker.md.
Optional per-token tenant is authoritative on create (inherited when the body omits it;
mismatch → 403) and scopes list/get/mutate to that tenant.
List with GET /v1/vms?tenant=acme.
Three credential types are accepted, in this order: a static bearer token, an agent-session token (fxst_..., confined to one
session's routes), then an OIDC JWT when auth.oidc_issuer and auth.oidc_audience are both set (role, tenant and actor come from
claims). With no Authorization header, a client certificate verified by tls.client_ca supplies the identity (CN, OU=fluxvm-role=admin,
OU=fluxvm-tenant=<name>), and the X-Client-Cert-* headers are read only with tls.trust_proxy_headers = true
(since 0.4.21). Details, claim mapping and the upgrade notes: auth-oidc-mtls.md.
The same tenant scoping applies to /v1/sandboxes (a sandbox is a VmRecord like any other):
POST /v1/sandboxes inherits/enforces the caller's token tenant on the resolved spec exactly
like POST /v1/vms does (a template's own tenant, if any, is subject to the same check as an
explicit spec.tenant would be), GET /v1/sandboxes only ever returns the caller's own tenant's
sandboxes, and every /v1/sandboxes/{id}/... route (snapshot, fs/read, fs/write, process)
404s for a different tenant's token exactly like /v1/vms/{id}/... already did — this was
previously missing entirely (see CHANGELOG).
/v1/pools gets the same treatment, name-keyed rather than Uuid-keyed: POST /v1/pools
inherits/enforces the caller's token tenant on template.tenant (every member ever backfilled
from that pool inherits it unchanged), GET /v1/pools only returns the caller's own tenant's
pools, and GET/DELETE /v1/pools/{name}, POST /v1/pools/{name}/claim, and
POST /v1/pools/{name}/resize all 404 for a different tenant's token — also previously missing
entirely (see CHANGELOG).
POST /v1/pools/{name}/resize (admin-only, body {"size": N}) changes a pool's target size after
creation — previously the only way to change a running pool's size at all was to DELETE it
(discarding every still-ready warm member) and POST /v1/pools again from the same spec. Growing
only updates the stored target and fires the same background backfill create/claim already use
(the reaper's own per-tick top-up is the backstop, as always); shrinking deletes excess ready
members immediately and synchronously, not on the next reaper tick — a caller asking for a smaller
pool is asking to give resources back right away. See docs/operations.md's "Warm VM pools"
section for a worked example.
Every pool-returning route (POST/GET /v1/pools, GET/POST .../resize /v1/pools/{name})
returns a PoolView, not the bare stored record: alongside size (the target) and members (ids
of members ready right now) it adds ready (members.len(), named explicitly so you don't have to
know that's what members counts), pending (size minus ready, floored at 0), and
claimed_total (a lifetime count of members this pool has actually handed out via
POST .../claim). All three are additive — nothing existing was renamed or removed.
POST /v1/pools/{name}/claim accepts name, ttl_seconds and pod_uid (all optional). pod_uid is
the Kubernetes Pod UID a Secure Containers claim is for: it is recorded as the VM's request.pod_uid,
so the dataplane attached when the NIC is hot-added mints the Pod's eBPF identity (Pod-scoped network
policy, POST /v1/vms/{id}/network/pod-policy) exactly as for a VM created for the Pod. It must be 1–128
characters of [A-Za-z0-9._-] (400 otherwise, checked before a member is taken from the pool).
curl -sS http://127.0.0.1:7788/v1/vms -H 'Authorization: Bearer <token>'
curl -sS 'http://127.0.0.1:7788/v1/vms?tenant=acme' -H 'Authorization: Bearer <token>'
A token listed in [[auth.mfa]] also needs a current TOTP code on guest exec, shells and (optionally) writes. Send it in X-Fluxvm-Totp; a missing or wrong code returns 401, repeated failures 429. See host-hardening.md.
curl -sS http://127.0.0.1:7788/readyz # no token; "ok" requires state_dir (+ dataplane if required)
No token, or a token not in the config, gets 401. A read-only token on a mutating route gets 403.
Token comparison is constant-time. Verified on real hardware: 401 with no/wrong token, 200 for
read-only on GET /v1/vms, 403 for read-only on POST /v1/vms, 400 for admin on the same route
with an invalid body (proving auth let it through to the actual handler), 200 on /healthz and
/readyz with no token at all even with auth enabled.
Create through REST:
curl -sS http://127.0.0.1:7788/v1/vms \
-H 'content-type: application/json' \
--data-binary @examples/qemu.json | jq
# Production-shaped example (tenant + tap/netns):
curl -sS http://127.0.0.1:7788/v1/vms \
-H 'content-type: application/json' \
--data-binary @examples/create-vm-prod.json | jq
Whole-stack production checklist: PRODUCTION.md ·
./scripts/release-checklist.sh.
Exec through REST (agent.enabled: true required, see operations.md):
curl -sS http://127.0.0.1:7788/v1/vms/<uuid>/agent \
-H 'content-type: application/json' \
-d '{"command": "echo hello", "timeout_seconds": 30}' | jq
VM JSON contract
backend is one of "qemu", "cloud-hypervisor", "firecracker", or "auto" (see
Auto backend selection — the persisted/returned record
always shows the resolved concrete backend, never "auto").
{
"name": "job-123",
"tenant": "acme",
"backend": "qemu",
"image": "/var/lib/fluxvm/images/ubuntu.qcow2",
"vcpus": 2,
"memory_mib": 2048,
"disk_size_gib": 20,
"network": {
"mode": "user",
"forwards": [
{"host_port": 2222, "guest_port": 22, "protocol": "tcp"}
]
},
"cloud_init": {
"hostname": "job-123",
"user": "zyvor",
"ssh_authorized_keys": ["ssh-ed25519 AAAA..."],
"packages": ["curl"],
"runcmd": ["echo hello > /tmp/hello"]
},
"agent": {"enabled": true, "port": 17777},
"ttl_seconds": 600,
"extra_args": [],
"storage": "default"
}
Optional tenant is a first-class string for multi-team hosts (GET /v1/vms?tenant=). See
examples/create-vm-prod.json.
Optional data_disks (QEMU only), e.g. [{"name": "disk1", "backing": "/var/lib/fluxvm/images/imported/web01/disk1.raw"}],
creates each disk as a per-VM qcow2 overlay on backing before the first boot. backing is
checked like a POST /v1/vms/{uuid}/disks path and never written to. An image import's
suggested body fills it for every disk after the boot disk.
QEMU disk I/O. A data disk's io (in data_disks or on POST /v1/vms/{uuid}/disks) and the
create body's root_disk_io take the same object; every field is optional:
{"bus": "virtio-blk", "aio": "io_uring", "cache": "none", "iothread": true, "queues": 4}
bus:scsi(default for data disks),virtio-blk, orvhost-user-blk. The root disk is always virtio-blk, soroot_disk_io.busmust be unset orvirtio-blk.aio:io_uring,native(needscachenoneordirectsync) orthreads.cache:none,writeback(QEMU's default),writethrough,directsyncorunsafe.iothread: a dedicated QEMU iothread for the disk (virtio buses).queues: virtio queues, 1-64 (virtio buses).throttle: IOPS and bandwidth limits QEMU enforces (block_set_io_throttle) onscsi,virtio-blkand root disks:iops_total,iops_read,iops_write,bps_total,bps_read,bps_write(bytes/s), plusiops_total_max/bps_total_maxbursts lastingburst_seconds(default 1). A*_totallimit excludes its read/write pair. Limits are set right after QEMU starts (including on a migration target, since QEMU does not migrate them; a failure stops the launch) and when a disk is hot-added.PUT /v1/vms/{uuid}/disks/{name}/throttlechanges them live and keeps them for later starts. A root throttle alone keeps the root'sif=virtiodrive, so adding one does not move the guest's PCI layout.GET .../disksreports the configuredthrottle. vhost-user-blk disks are refused: their I/O does not pass QEMU's block layer.
A vhost-user-blk disk is opened by a per-disk qemu-storage-daemon, not by QEMU. Its queues and
iothread are the daemon's. Hot-adding one needs shareable guest memory, so create the VM with
"shared_memory": true. A stopped VM gets it at the next boot either way. These disks carry no serial
number. Live resize and hibernation refuse them. CBT backups and live backups leave them out; CBT
lists them as skipped.
Hot-added virtio-blk and vhost-user-blk disks go onto PCIe root ports reserved when the VM is created:
disk_hotplug_ports (default 4, at most 16). VMs created before this field existed have none, so their
device layout, live migration and saved state are unchanged. They can still attach virtio disks while
stopped. SCSI disks need no port.
The QEMU options are stored next to the disk as disks/<name>.io.json, so a disk keeps them across
reboots. GET /v1/capabilities reports host.io_uring (the kernel allows io_uring),
host.vhost_user_blk (qemu-storage-daemon is installed) and host.nvme_multipath (native NVMe
multipath is on, so an NVMe-oF namespace with several paths is a single /dev/nvmeXnY).
A disk on a host block device (a path under /dev, or a storage: shared root on one) is added to
the VM's device-cgroup allowlist on the eBPF dataplane. This happens at boot, and also before a live
attach. The allowlist holds 8 entries: /dev/kvm, /dev/vhost-vsock, /dev/net/tun, one per VFIO
group, then one per block device. An attach that would overflow it is refused.
Optional cdroms (QEMU only, at most 4), e.g.
[{"name": "install", "path": "/var/lib/fluxvm/images/win2022.iso"}, {"name": "virtio", "path": "/var/lib/fluxvm/images/virtio-win.iso"}],
attaches each ISO read-only as a SATA CD-ROM on the q35 AHCI controller (ide.0, ide.1, …), so
Windows Setup sees it with no extra drivers. path is checked like a data disk backing. No boot
order is forced: firmware skips a blank root disk and boots the first CD-ROM, and once an OS is
installed it boots from the disk. To install onto a blank disk, point image at an empty raw or
qcow2 file and set disk_size_gib.
Ejecting install media. A VM with media in a CD-ROM can't be live-migrated, because the ISO path is host-local. Once the OS is installed, eject the medium:
curl -X POST -H "Authorization: Bearer $TOKEN" \
http://127.0.0.1:7788/v1/vms/$ID/cdroms/install/eject
- Needs an admin-role token and is tenant-scoped like every
/v1/vms/{uuid}route (a tenant token can eject only its own tenant's VMs). - If the VM is running on QEMU, the medium is removed live over QMP:
blockdev-open-tray(forced, so a guest lock doesn't block it),blockdev-remove-medium,blockdev-close-tray, then the read-onlycd-<name>block node is deleted. The guest sees an empty drive. Nothing is hot-unplugged. - The drive is recorded as empty (
"path": ""incdroms[]). Later starts, restarts and migration receivers create a bareide-cddevice with no medium, so the guest's device layout (and Windows drive letters) stays the same. - Ejecting an already-empty drive succeeds and changes nothing. An unknown name returns 400
cdrom "<name>" not found. - The response is the updated VM record. The action is audited as
vm.cdrom.eject. - Creating a VM with an empty
pathis still rejected (cdrom "<name>" needs a path). An empty drive comes only from an eject.
Live migration rejects only CD-ROMs that still hold media, with
eject cdrom "<name>" first (POST /v1/vms/{id}/cdroms/{name}/eject). Inserting new media into an
empty drive isn't supported yet; recreate the VM with the ISO instead.
agent.enabled turns on the vsock guest agent (fluxctl exec) for this VM — the guest image must
have fluxvm-guest-agent installed and enabled (see the
Quick start section and build-image-tutorials.md).
agent.port is the AF_VSOCK port the guest listens on (not a host TCP port); it defaults to 17777
and rarely needs changing, since each VM already gets its own host-unique vsock CID.
storage is one of "default" (the implicit default when the field is omitted entirely — qcow2/raw,
exactly as before this existed), "lvm-thin", "nbd", "ceph-rbd", or "ceph-rbd-in-place" — see
Storage backends.
Networking modes
none:
{"mode":"none"}
QEMU user networking:
{
"mode":"user",
"forwards":[{"host_port":2222,"guest_port":22,"protocol":"tcp"}]
}
TAP/bridge (all VMMs):
{
"mode":"tap",
"bridge":"vmbr0",
"mac":"06:00:AC:10:00:02"
}
When tap_name is omitted, the manager creates one from the VM UUID.
macvtap (QEMU and Cloud Hypervisor only — see below):
{
"mode": "macvtap",
"parent": "eth0",
"macvtap_mode": "bridge",
"mac": "52:54:00:aa:bb:cc"
}
Gives the VM its own MAC directly on parent's link — no host bridge involved. macvtap_mode is
the macvtap link mode: bridge (default — siblings on the same parent can reach each other, but
not the parent itself directly), vepa, private, or passthru. The manager creates a per-VM
macvtap device on parent, opens its /dev/tapN character device, and passes that file descriptor
directly to the VMM (-netdev tap,fd=N for QEMU, --net fd=N for Cloud Hypervisor) — there's no
persistent named tap the VMM opens itself, which is why Firecracker doesn't support this mode:
its API only accepts a host device name it opens via /dev/net/tun, with no fd-passing option.
Direct — bridge-less tap (opt-in; direct-datapath.md):
{
"mode": "tap",
"mac": "02:00:00:00:0a:0a",
"direct": {"outer": "enp1s0", "mode": "l2-uplink", "guest_ips": ["192.168.1.50"]}
}
An eBPF redirect pairs the VM's tap with outer instead of a bridge. mode is l2-uplink (a physical
NIC that is not a bridge/bond port; frames are steered by destination MAC, and ARP requests by the
declared guest_ips) or peer-veth (a Pod's veth, with netns_path naming the Pod network namespace
the tap is created in; used by the Secure Containers shim). bridge, netns: true and extra NICs
cannot be combined with direct. It requires sandbox.dataplane.mode = "ebpf" or "cilium"
(legacy is rejected), a failed dataplane attach fails the create, and the tap is handed to QEMU or
Cloud Hypervisor (by name, or as a descriptor when it lives in another namespace); Firecracker and the
native hypervisor accept only the host-namespace form because they cannot take a descriptor.
Policy.allowed_network_modes sees it as tap.
Virtual networks (Linux)
A tap NIC with vnet joins an eBPF-routed virtual network instead of a bridge:
{"mode":"tap","mac":"52:54:00:12:34:56","vnet":{"vni":100,"ip":"10.50.0.11","prefix_len":24,"gateway":"10.50.0.1"}}
mac is required; bridge, netns, direct and extra are refused, and NIC hotplug is refused
on such VMs. The network itself (CIDR, gateway, endpoints on every host, uplink, SNAT address,
floating IPs) is applied with PUT /v1/network/vnets/{vni}. Capability host.vnet. See
virtual-network.md.
NIC hotplug (QEMU)
POST /v1/vms/{uuid}/hotplug/nic (admin) attaches a NIC to a running VM, used by Secure Containers after
a warm-pool claim (the pool template boots with network.mode=none). On a netns: true VM, QEMU runs
inside the namespace, so the daemon opens the new bridge tap and passes it over QMP (getfd); on
relaunch every extra NIC's tap is inherited as -netdev tap,fd=. Send either a bridged NIC:
{"bridge": "fvbhab12cd", "mac": "02:00:00:00:00:01"}
or a bridge-less one (the daemon creates the tap, applies the dataplane, then passes the tap to QEMU
over QMP with getfd + netdev_add fd= on one session):
{"mac": "02:00:00:00:00:01",
"direct": {"outer": "eth0", "netns_path": "/run/netns/fvcni-ab12cd", "mode": "peer-veth"}}
bridge and direct are mutually exclusive (400 otherwise). A direct hotplug needs a VM booted with
network.mode=none; on any failure the daemon removes the tap and dataplane it created.
POST /v1/vms/{uuid}/hotplug/nic/unplug (admin) removes an extra bridged NIC from a running VM, named
by {"mac": "..."} or {"tap": "..."}, and deletes its tap. The guest has 10 s to release the device.
The primary NIC can't be hot-removed.
Share hotplug (QEMU)
POST /v1/vms/{uuid}/hotplug/share (admin) hot-adds a virtiofs share to a running VM, so a warm-pool
member (which keeps its template's shares) can receive a Pod's rootfs and volumes after a claim:
{"host_path": "/var/lib/fluxvm/some/dir", "read_only": false}
The reply is {"tag": "fs<N>"}, N being the share's index in the VM's shared_folders; mount it in the
guest with mount -t virtiofs fs<N> <dir>. The path must be absolute, without .., and a directory the
daemon can see (a sandboxed daemon with PrivateTmp does not see /tmp). The VM must have been created with
"shared_memory": true (vhost-user-fs needs shareable guest memory); up to four shares can be hot-added
(reserved root ports). On failure the virtiofsd is killed and nothing is recorded.
Speculation, grants, exec policy and memory density
These routes came with the crash-safe lifecycle work. They are implemented and unit tested; none has
been run end-to-end on real VMs yet (each guide has a status table). The OpenAPI document served at
GET /v1/openapi.json is generated by openapi::spec() in crates/fluxvm-api/src/openapi.rs; it
has not been updated for these routes, headers or query parameters, and there is no
hand-maintained spec file. Until it is, this page is the contract.
Every route here needs an admin token except GET /v1/vms/{uuid}/balloon and
GET /v1/vms/{uuid}/memory. They are tenant-scoped like the rest of /v1/vms/{uuid} and
/v1/sandboxes/{id}.
Speculation and changesets
Guide: speculative-execution.md.
POST /v1/sandboxes/{id}/speculate runs a command in an isolated copy and returns a pending
changeset. The sandbox is started on demand like other sandbox routes.
{"command": "make build", "timeout_seconds": 120, "paths": ["/work"], "ttl_seconds": 600}
command(required).timeout_seconds(optional) bounds the command.paths(optional for procbox sandboxes, default["/"]; required for VM sandboxes, 400 otherwise) limits the diff to those directories.ttl_seconds(optional) is how long the changeset stays decidable: default 3600, clamped to 1..=86400.
The response is a changeset. The same shape is returned by GET .../changesets/{cs} and by approve,
reject and apply:
{
"id": "5b0f6c1e-0f0e-4a63-9f58-0b2f1e0f7a11",
"sandbox_id": "0c1d2e3f-0000-4000-8000-000000000001",
"state": "pending",
"created_at": 1790000000,
"updated_at": 1790000000,
"expires_at": 1790000600,
"command": "make build",
"exit_code": 0,
"stdout": "...",
"stderr": "",
"paths": ["/work"],
"run_via": "fork",
"changes": {"added": ["/work/out.bin"], "modified": [], "deleted": [], "unchanged": 41},
"side_effects": {
"egress": "blocked",
"destinations": [],
"replayable": ["1 file change(s) under /work"],
"non_replayable": []
},
"base_files": 42,
"staged": {"/work/out.bin": {"blob": "000000", "mode": 33188, "size": 1048576}},
"unstaged": [],
"apply_started_at": null,
"error": null
}
Timestamps are Unix seconds. run_via is workspace-copy (procbox), fork or snapshot (VM
sandboxes). side_effects.egress is blocked, allow_listed or unrestricted, and is declared,
not enforced (see the guide). stdout and stderr are capped at 64 KiB each, with a
[truncated N bytes] marker.
GET /v1/sandboxes/{id}/changesetsreturns{"items": [changeset, ...]}, newest first.POST .../changesets/{cs}/approveand.../rejectdecide apendingchangeset; no body.POST .../changesets/{cs}/applywrites anapprovedchangeset's file changes into the real sandbox.
Status codes: 404 unknown changeset (or one that belongs to another sandbox); 409 invalid state transition, conflict (the real files changed since the changeset's base; nothing is written), expired, or another request is already working on that changeset; 422 the changeset cannot be applied (for example files that could not be staged); 403 a non-admin token.
Credential grants
Guide: credential-broker.md.
POST /v1/sandboxes/{id}/grants returns 201:
{"secret_ref": "github-token", "hosts": ["api.github.com"], "ttl_seconds": 900}
secret_ref(required, 1-128 chars). Withoutvalue, the daemon readsFLUXVM_SECRET_GITHUB_TOKENfrom its own environment.value(optional, write-only): the full header value, for example"Bearer abc". Never returned.hosts(required, at least one): bare host names, exact or domain suffix.*,/,:,@,?and whitespace are rejected.ttl_seconds(1..=86400, default 3600) orexpires_at(RFC 3339, at most 24 h ahead). If both are sent,expires_atwins.- At most 32 live grants per sandbox.
Response (GrantInfo, which has no secret field):
{
"id": "7e0f0c52-5d57-4b35-8d3f-6a1a3d2b9c10",
"sandbox_id": "0c1d2e3f-0000-4000-8000-000000000001",
"secret_ref": "github-token",
"hosts": ["api.github.com"],
"created_at": "2026-10-08T12:00:00Z",
"expires_at": "2026-10-08T12:15:00Z",
"bound": true
}
bound: false means the sandbox has no known guest IP, so the grant cannot match any traffic.
GET /v1/sandboxes/{id}/grantsreturns{"grants": [GrantInfo, ...]}(live grants only).DELETE /v1/sandboxes/{id}/grantsrevokes all and returns{"revoked": true|false}.DELETE /v1/sandboxes/{id}/grants/{grant_id}revokes one and returns{"revoked": true}, or 404.
Every verb is admin-only, GET included. Validation failures (bad host, bad TTL, unresolvable
secret_ref, more than 32 grants) return 400.
Host overview and capabilities (macOS)
Two read-only routes feed a dashboard without scraping /metrics:
GET /v1/host/history[?since=<rfc3339>]: the last hour of host memory and per-VM runner usage at 5-second spacing:{interval_seconds, capacity, items: [{ts, memory_available_mib, memory_total_mib, memory_pressure, vms: [{vm_id, cpu_seconds_total, rss_bytes}]}]}, oldest first,sinceexclusive. In memory only: it is empty after a daemon restart. A dashboard uses it to draw full sparklines on connect.- The
vm.forkandvm.branch.startevents (GET /v1/events?event=vm.) carrytotal_msandreport, a JSON array with one entry per child:{id, name, prepare_ms, restore_ms, identity_ms, identity_via, hooks_ms, ready_ms, total_ms}. POST /v1/vms/{id}/fork/planandPOST /v1/branches/{id}/start/plan(admin; the same body as the real call): what the request would be refused for and what it is expected to cost, without doing it.{ok, count, parallelism, refusals: [{code, reason}], predicted: {extra_rss_mib_low, extra_rss_mib_high, extra_rss_basis, disk_logical_bytes_each, est_total_ms, est_basis, host_available_mib, host_available_after_low_mib, host_available_after_high_mib}}. Refusals come from the checks the real call runs (count,parallelism,hooks,forkable,network,quota,name), all collected. The prediction is a bound, not a promise: low is the per-child resident size: the larger of this source's running fork children's average and the source's own (young children understate it), else a 256 MiB fresh-restore floor; high is every child using all its memory. Children fault pages back in after a restore, so a child's size grows towards (and on the M4 slightly past) its parent's. The time is the median of forks recorded on this Mac and is absent until there is one.GET /v1/vms/{id}/usernet/policyandPUT /v1/vms/{id}/usernet/policy(admin): the egress policy of anapple.usernetguest.PUTtakes any ofallow_cidrs,deny_cidrs,allow_ports,deny_ports,dns_allow,dns_deny,allow_private,allow_cidrs6,deny_cidrs6(absent keeps, a present empty list clears), validates them like create, stores them in the VM record (so a restart, a fork or a branch start uses them) and, when the gateway runs, applies them at once: the daemon rewrites the gateway's config and signals it, and new connections and DNS queries follow the new policy while open connections are not cut. Reply: the policy plusappliedand anote. Port forwards, MTU and addresses are not changed this way.GET /v1/host/images:{"builtin_cached": ["debian-13", ...]}, the built-in images with a ready copy on this host (no network access).fluxvm-agent nodesends it in the heartbeat andfluxvm-agent centralprefers a Mac that already has the image.GET /v1/host/storage: where disk space goes.volume(path,total_bytes,free_bytesof the filesystem holding the state directory),vms[](vm_id,name,status,diskandotheras{allocated_bytes, logical_bytes, files}: the root disk, and everything else in the workspace),branches[](id,name,usage),images(the image cache) andstate_dir(everything). Allocated is the blocks a file really occupies (a sparse 3 GiB disk holding 700 MiB counts 700 MiB), but copy-on-write clones (fork, branches,vm clone) share blocks and each clone reports them in full, so the parts can add up to more than the volume holds.volume.free_bytesis the truth about what is left.GET /v1/host/overview:daemon(version,uptime_seconds),host(os_version,cpu_count,memory_total_mib,memory_available_mib,memory_pressurenormal|warn|critical,nested_virtualization,rosetta),vms(total,by_status,by_backend),branches,usage[],helpers(runner,switch,blockd,usernetpresent or not) andnetworking(vm_networking_entitlement,netobs_enabled).usage[]has one row per running or pausedvzVM:vm_id,name,status,cpu_seconds_total(cumulative: take two samples for a rate),rss_bytes,service_attributedandstarted_at(Unix seconds, the runner process start). CPU and RSS are the runner plus its Virtualization XPC service whenservice_attributedis true, otherwise the runner alone (the guest's own usage is not included). Verified againstpson an M4:started_atexact, RSS equal to runner plus service.GET /v1/host/apple/capabilities: the capability table judged against this Mac: perfeaturesupported,note,min_macos,available_here(nullwhen the host has not been probed) andreason(macOS version, chip generation, missingcom.apple.vm.networking, missing Rosetta).
Node stats
GET /v1/stats is one cheap sample of this host and every VM on it, for the fleet's metrics collector
(fleet-metrics.md); anything that moves is a cumulative counter, so a reader takes two samples and
divides by the time between them. Gauges (memory, disk free) are as of now, and a figure this host cannot measure is
null, never a made-up zero.
host:cpus,cpu_busy_ticksandcpu_total_ticks(only the ratio of two samples' differences means anything),memory_total_mib,memory_available_mib,disk_total_bytesanddisk_free_bytesof the volume holding the state directory,vm_count,vms_running.tsis the sample time. Per VM:id,name,status,backend,vcpus,memory_mib,cpu_seconds_total,memory_used_bytes,disk_read_bytes_total,disk_write_bytes_total,net_rx_bytes_total,net_tx_bytes_totaland asourcethat says which of these it can supply.vz-process(avzguest, read from the runner plus its Virtualization XPC service: CPU, resident memory, disk bytes, no network because Virtualization.framework has no per-VM counters),vz-runner(the XPC service could not be attributed, so the runner process only),cgroup(a Linux backend, from the VM's cgroup v2 and its TAP device) ornone(stopped, or nothing readable).
Volumes and tunnels (fleet platform)
These routes are the node half of fleet volumes and
tunnels. They are admin-only: tenants reach them through
fluxvm-agent central, which authenticates the tenant, applies quotas and policy, and calls them with the fleet token.
Volumes (/v1/volumes) are sparse raw disk files under <state_dir>/volumes/ that outlive the VMs using them. A volume
is {name, path, format: "raw", size_gib, size_bytes, allocated_bytes, attached_to: [vm ids]}.
POSTtakes{"name", "size_gib"}. A name is 1 to 63 lowercase letters, digits and-, starting with a letter or digit (it becomes a file name);size_gibis 1 to 16384. At most 1024 volumes per host.- A VM uses a volume as an
apple.extra_disksimage disk (pathis what it names).attached_tolists the VM records that reference the file;DELETEandresizeanswer409while it is not empty. A resize only grows. - Deleting the VM leaves the volume alone: that is the point of it.
Tunnels (GET /v1/vms/{uuid}/tunnel?port=N) upgrade to a WebSocket that carries one TCP connection to port N of a
running VM; each binary message is raw bytes (at most 64 KiB from the guest, 256 KiB from the caller), and each direction
waits for its write, so a slow end slows the other. The node connects only to the VM:
- an
apple.usernetVM: the host listener of its own TCP port forward for that guest port (409when it publishes none; add one toapple.usernet.port_forwards); - a
tap+netnsVM: from inside its network namespace, to the address its dnsmasq leased (link-local169.254.x.yis allowed there; the cloud metadata address is not); - a
vzVM on the shared network: to the address the guest reported on its serial console, only if it lies on this Mac's own VM network (abridge*interface's subnet) and no other VM on the node reports the same address. The address comes from the guest, so anything outside that (the Mac's LAN, another network) is refused with409.
Loopback, unspecified, multicast and the node's own interface addresses are always refused, so a tunnel never reaches the
node's own services. The VM must be running (409 otherwise) and the port 1 to 65535; central applies the tenant's port
policy before it connects here.
Host topology and NUMA binding (Linux)
GET /v1/host/topology: NUMA nodes (CPUs, thread siblings, memory, hugepage pools, distances) and placeable PCI devices (GPU, NIC, NVMe, accelerator) with NUMA node, PCIe root, IOMMU group and SR-IOV VFs.POST /v1/host/pci/bind/POST /v1/host/pci/release:/v1/host/gpus/bind|releasefor any PCI function; admin.CreateVmRequest.numabinds guest NUMA nodes to host nodes (QEMU only). See host-topology.md.GET /v1/host/mdev: mediated device parents, their profiles (free instances, framebuffer, MIG slices) and existing instances.POST /v1/host/mdev(admin,{"parent", "type", "uuid"?}) creates an instance (201;409for a UUID on another parent or type,503when the profile has none free).DELETE /v1/host/mdev/{uuid}(admin,204) refuses an instance a VM lists or holds.vfio_devicestakes instance UUIDs. See vgpu.md.GET /v1/host/gpus/health: per GPUhealthy,degradedorunhealthywith reasons, AER counters, PCIe link andnvidia-smicounters when installed.GET /v1/host/infer: whether the host OpenAI-compatible model server answers on[apple].host_infer_portforapple.host_gpusandboxes (see host-inference.md).fluxctl infer status/infer serve --stand-in.
Balloon and memory
Guide: memory-density.md.
GET /v1/vms/{uuid}/balloon, and POST /v1/vms/{uuid}/balloon (admin) with {"balloon_mib": 256}
(0 deflates), both return:
{"memory_mib": 1024, "target_mib": 256, "actual_mib": 192}
target_mib is what was requested; actual_mib is what the guest driver has reached so far. A
running VM on the flux-vm backend's KVM engine, the vz backend, or the qemu backend created with balloon has a
balloon; anything else returns 400, as does a balloon that would leave the guest under 64 MiB. For QEMU,
memory_mib includes hot-added DIMMs.
CreateVmRequest.balloon (QEMU only) adds a virtio-balloon-pci device:
{"balloon": {"free_page_reporting": true, "deflate_on_oom": true, "stats_interval_secs": 5}}
All three fields are optional and default to the values shown. balloon is refused with hugepages,
vfio_devices and confidential security profiles.
GET /v1/host/memory samples the host now:
{"mem_available_mib": 20480, "mem_total_mib": 31744, "psi_some_avg10": 0.4, "psi_full_avg10": 0.0}
A field is null when the host does not expose it (no PSI in the kernel). host.balloon in
GET /v1/runtime/capabilities is true when both routes are available.
GET /v1/host/pressure samples whole-host PSI from /proc/pressure now, in the same shape as
GET /v1/vms/{uuid}/pressure:
{
"cpu_some": {"avg10": 20.78, "avg60": 19.86, "avg300": 18.66, "total": 76804200790},
"memory_some": {"avg10": 0.0, "avg60": 0.0, "avg300": 0.0, "total": 6194},
"memory_full": {"avg10": 0.0, "avg60": 0.0, "avg300": 0.0, "total": 6164},
"io_some": {"avg10": 40.67, "avg60": 30.67, "avg300": 25.89, "total": 286750114310},
"io_full": {"avg10": 22.75, "avg60": 15.68, "avg300": 14.5, "total": 220501129495}
}
Averages are percentages of wall time some (or all) tasks stalled on the resource; total is
microseconds. A field is null when the kernel has no PSI. Kairon's kairon-node publishes these on
its Node for DRS.
GET /v1/vms/{uuid}/memory:
{
"vm_id": "0c1d2e3f-0000-4000-8000-000000000001",
"configured_mib": 1024,
"usage": {"rss_kib": 310000, "pss_kib": 180000, "private_kib": 120000, "shared_kib": 190000, "swap_kib": 0},
"balloon": {"memory_mib": 1024, "target_mib": 0, "actual_mib": 0}
}
usage is null when the VM has no VMM process or /proc/<pid>/smaps_rollup is unreadable;
balloon is null when the VM has no balloon. A ballooned QEMU VM also reports guest
(total_mib, free_mib, available_mib, disk_caches_mib, last_update) once the guest
driver has sent statistics; a statistic the guest did not send is null.
First-command latency: ?ready=exec
POST /v1/vms, POST /v1/vms/{uuid}/fork and POST /v1/pools/{name}/claim accept ?ready=exec (or
?wait_first_command=true). The call then blocks until a trivial guest-agent exec (true) succeeds,
which needs the guest agent enabled, and adds timings measured from the start of the request. For
create and claim they are merged into the VM record:
{"id": "...", "first_command_ms": 412, "phases": {"create_done_ms": 180, "agent_wait_ms": 231, "first_exec_ms": 412}}
create_done_ms is when create or claim returned, agent_wait_ms the extra wait for the first
successful exec, and first_exec_ms the total. For fork, the body keeps items and elapsed_ms and
adds first_command_ms (the slowest child) and first_command (one
{first_command_ms, phases} object per child). A vm.first_command event is recorded for each VM.
scripts/bench-first-command.sh uses it.
Errors
Every error is JSON: {"error": "<message>", "code": "<CODE>"}. Branch on code (it does not change); the
message is for people and may be reworded. The code follows the HTTP status:
| Status | code | Typical cause |
|---|---|---|
| 400 | BAD_REQUEST | Malformed body, query or Idempotency-Key |
| 401 | UNAUTHORIZED | Missing or invalid credential: a bearer token, or (with mTLS) a client certificate with no CN |
| 403 | FORBIDDEN | Role or session token does not allow the route |
| 404 | NOT_FOUND | No such VM or session, or one that belongs to another tenant (never 403, so ids do not leak) |
| 409 | CONFLICT | State conflict: VM not running, a duplicate in flight, a changeset whose files moved |
| 412 | PRECONDITION_FAILED | If-Match no longer matches |
| 413 | PAYLOAD_TOO_LARGE | Request body over the limit |
| 422 | UNPROCESSABLE | Same Idempotency-Key reused with a different request |
| 428 | PRECONDITION_REQUIRED | The request needs a condition first (an operator approval, an If-Match) |
| 429 | RATE_LIMITED | Honour Retry-After |
| 501 | NOT_IMPLEMENTED | The backend does not support the route |
| 503 | UNAVAILABLE | Capacity (for example no free GPU); retry later |
| other 5xx | INTERNAL | Server fault |
Two special codes ride on the generic statuses: APPROVAL_REQUIRED (a 428 that also carries approval_id) and
STEP_UP_REQUIRED (a 401 asking for a TOTP code). Tunnel, idempotency and fleet-central errors send a code too.
The fleet central's sign-up and SSO browser flows still send only error; treat a missing code as unknown and fall
back to the status.
Not sure what a code means or what to do about it? fluxctl explain answers offline, from the same table:
fluxctl explain RATE_LIMITED # meaning, usual causes, what to do (case, - and _ do not matter)
fluxctl explain 428 # every code on an HTTP status
fluxctl explain required # part of a code name
fluxctl explain --list [--json] # all of them
It exits 1 for an unknown code (with a did you mean). The table is fluxvm_core::error_codes; a test in
fluxvm-api fails when the API can send a code that the table does not explain.
Idempotency-Key
POST /v1/vms, DELETE /v1/vms/{uuid}, POST /v1/vms/{uuid}/snapshot and
POST /v1/vms/{uuid}/fork honor an Idempotency-Key request header (1-255 visible ASCII characters,
else 400). Other routes ignore it, and requests without it behave as before.
- The key is scoped by token tenant (or caller) and by method plus path.
- The first 2xx response is fsynced under
<state_dir>/idempotencybefore it is returned. A retry with the same key and an identical method, path and body gets the stored status and body back with the response headerIdempotent-Replayed: true, and nothing runs again. - Same key with a different method, path or body: 422.
- Same key while the first request is still running: 409.
- Only 2xx responses are stored, so a failed attempt can be retried under the same key.
- Records expire after 24 hours. Request bodies over 16 MiB get 413. A response over 8 MiB returns 500 and is not stored.
- The tenant guard runs before a stored response is replayed.
Exec policy and enforcement
Guide: guest-exec-policy.md. POST /v1/vms/{uuid}/agent and
POST /v1/sandboxes/{id}/process accept an optional policy object (the procbox policy shape) next
to command and timeout_seconds. With a policy, the guest confines the command with Landlock and
seccomp, and the exec response carries enforcement:
{"command": "ls /usr", "policy": {"read": ["/bin", "/usr", "/lib", "/lib64"]}}
{"result": "exec", "exit_code": 0, "stdout": "...", "stderr": "", "enforcement": {"landlock_abi": 3, "filesystem": true, "not_enforced": []}}
If the policy cannot be enforced, the command is not run and the response is an error. Procbox
sandboxes reject a per-exec policy (400). Without policy, the response is unchanged and has no
enforcement key.
GET /v1/vms/{id}/guard(read-only): the VMM Guard state for a VM — mode, exec allow-list, applied rule packs andexpired_packs. Seedocs/host-hardening.md.