FluxVM runtime boundary
FluxVM is the node-local execution runtime. Zyvor Fabric is the distributed private-cloud control plane. GuestKit owns offline guest-disk internals.
The rule for new features is:
- If the feature manipulates a VMM, KVM memory/device state, a VM-local cgroup, TAP/netns, or the VM-edge eBPF hook, it belongs in FluxVM.
- If the feature decides which host, which tenant/policy, when to evacuate, how to replicate, or how to present the operation to users, it belongs in Zyvor Fabric.
Runtime contract v1
GET /v1/runtime/capabilities advertises stable node-local primitives.
Live migration primitives:
POST /v1/vms/{id}/migration/startGET /v1/vms/{id}/migration/statusPOST /v1/vms/{id}/migration/cancelPOST /v1/vms/{id}/migration/tune(QEMU only)POST /v1/vms/{id}/migration/postcopy(QEMU only)
fluxctl migrate start|status|cancel is the CLI equivalent of the first
three routes above -- for the standalone mode mentioned below, where there
is no Fabric orchestrator to drive them over HTTP.
Adaptive controls (QEMU)
These let the caller steer a migration that is not converging. FluxVM only applies them; deciding when to use them is the orchestrator's job.
startalso acceptsauto_converge {initial_pct, increment_pct, max_pct}(QEMU'sauto-convergecapability and its throttle parameters, 1-99 each),compression {algorithm: none|zlib|zstd, level}(multifd compression; needsmultifd_channels > 1) andpost_copy: disabled|immediate|armed.armedenablespostcopy-rambut stays in pre-copy.mode: post-copywithoutpost_copymeansimmediate, as before.tune {bandwidth_mbps, max_downtime_ms, cpu_throttle_initial_pct, cpu_throttle_increment_pct, max_cpu_throttle_pct}changes those parameters on the in-flight migration (migrate-set-parameters). QEMU cannot change capabilities mid-flight, so auto-converge has to be enabled atstart. To keep it effectively off until needed, start with all three percentages at 1.postcopyswitches an armed, active migration to post-copy. From then on the destination owns guest memory: if the stream breaks, the VM is lost, not left running on the source.statusalso reportsdirty_pages_rate,dirty_sync_count,page_size,duplicate_pages,normal_bytes,mbps,expected_downtime_ms,cpu_throttle_pct,setup_time_msandpostcopy_requestswhen QEMU provides them. QEMU reports no compression ratio for multifd compression.- A receiver must be created with matching
capabilities {multifd_channels, compression, post_copy}. They are applied beforemigrate-incoming, and QEMU refuses a stream whose capabilities differ between the two ends. - Every
migration/startstates multifd, post-copy and auto-converge (on or off), multifd compression and TLS, and every tunable (max-bandwidth,downtime-limit, multifd channels, the auto-converge throttle), falling back to QEMU's own default for any the request leaves out. QEMU keeps all of them across attempts, so a retry would otherwise inherit the previous attempt's stream shape or a throttle (a cancelled 1 Mbps attempt left the next one unable to finish). A TLS credentials object left by a failed attempt is replaced. migration/cancelafter the stream completed finds the guest stopped inpostmigrate; with no post-copy it resumes the guest on the source. With post-copy enabled it stays stopped, because the destination may hold newer pages.GET /v1/runtime/capabilitiesreportsautoConverge,compression,postCopySwitch,liveTuneandpostCopyWithMultifd. The last is true only when the QEMU binary reports version 10.0 or later.
Contract v1 is intentionally narrow:
- QEMU and Cloud Hypervisor. Firecracker and the in-tree FluxVM hypervisor have neither.
- Pre-copy, post-copy and multifd are supported by both source runtimes.
tcp:andunix:migration URIs are accepted for both.exec:is rejected (QEMU has this scheme at all; Cloud Hypervisor doesn't, but the same shared allowlist covers both so the rule can't drift between them).- VM disks are not copied by this contract. Fabric must confirm shared storage or prepare storage independently before starting migration.
- FluxVM does not select the destination node, reserve capacity, perform fencing, update tenant inventory, or run DRS. Those are Fabric concerns.
status/cancelremain QEMU-only. QEMU's QMPmigrateis asynchronous with aquery-migrateprogress query and amigrate_cancelprimitive; Cloud Hypervisor'ssend-migrationis fire-and-forget -- verified live, it returns success as soon as the request is accepted, before the transfer (or its failure) is known, and Cloud Hypervisor exposes no API to poll or cancel what happens next.GET /v1/runtime/capabilities's migration entry carries this asstatusPollable(truefor QEMU,falsefor Cloud Hypervisor) so Fabric can tell up front rather than discovering it from astartcall that never resolves into a pollable status. For Cloud Hypervisor, Fabric must instead infer the outcome the same way it already watches for any other node-local state change: the VM disappearing from this node'sGET /v1/vmsonce its process exits (migration succeeded) versus it stayingRunningthere (nothing has succeeded yet).
The legacy fluxvm-agent multi-host registry remains a lightweight standalone
mode for users who do not run Fabric. It must not grow Fabric-class HA, DRS,
site recovery, tenant scheduling, or datacenter semantics.
Threat containment (Firecracker figures)
FluxVM maps Firecracker's host-integration and threat-containment figures to node-local layers:
| Barrier | Owner |
|---|---|
| KVM + minimal virtio device model | VMM backends (FC / in-tree / CH / QEMU) |
| Virtio I/O rate limiters | Firecracker rate_limiter from create fields; in-tree token buckets; Fabric Mbps/PPS |
| Jailer + cgroups + seccomp | [jailer] + fluxvm-cgroup + stock FC / in-tree seccomp |
| Host egress filter | Network Fabric or network.mode=none (required outside the VMM) |
Capability numbers (Track A isolation vs optional Track B density): capability-figures.md.
Target receiver
POST /v1/migration/receivers arms a QEMU -incoming defer process on the
target node. Two forms exist:
- Bare (
{vcpus, memory_mib, disk, ...}): a disk-only q35 QEMU. The source must have the same minimal device model; it cannot be adopted. - Adopt-mode (
{record, listen_host, advertise_host, ...}whererecordis the source'sGET /v1/vms/{id}): the receiver is launched from the record's own device model (network, seed, agent vsock, firmware) with its own workspace, tap/netns and vsock CID, so the source and receiver can run side by side, even on one node. It is stored as aCreatingVM labelledfluxvm.dev/migrating-from=<source id>, with the receiver's id and the source's name. Contract v1 accepts QEMU VMs onstorage: sharedorceph-rbd-in-placewithout TPM, VFIO, virtiofs, data disks, hot-added NICs or a direct/macvtap datapath.
Disks are never copied. Full order:
- target:
POST /v1/migration/receiverswithrecord, thenPOST /v1/migration/receivers/{id}/activate {token}; - source:
POST /v1/vms/{id}/network/migration/quiesceandexport(when a dataplane is attached), thenPOST /v1/vms/{id}/migration/start {destination: <receiver uri>}; pollGET /v1/vms/{id}/migration/statusuntilcompleted; - source:
POST /v1/vms/{id}/migration/finishstops the paused source QEMU and removes its record and workspace (409 until the phase iscompleted; the disk stays); - target:
POST /v1/migration/receivers/{id}/adopt {token}turns the receiver into arunningVM labelledfluxvm.dev/migrated-from=<source id>(409 until the incoming side has finished), thennetwork/migration/restoreandresumeon the adopted id.
The adopted VM keeps the source's name but has the receiver's id: on one node
both processes exist at once, so they cannot share an id, tap or netns. A
caller that tracks VMs by id swaps it at adopt. DELETE /v1/migration/receivers/{id} before adopt kills the receiver and removes its
VM record.
Shared-disk storage
storage: shared uses request.image (a raw file or block device on storage
every node can open) in place. Delete never removes it. <disk>.fluxvm-lock
names the VM and node that have it open; a launch is refused while another VM
holds it, unless that holder is on this node and gone (missing record or dead
process). A holder on another node is never assumed dead: after fencing it,
re-create with POST /v1/vms?shared_takeover=true, which breaks the lock first
(HA re-create).
Read-only serial
GET /v1/vms/{id}/serial is an interactive socket on QEMU. Every other backend
writes its console to console.log, so the same route streams that file
read-only: the last 64 KiB, then new bytes as binary frames; inbound frames are
ignored. A jailed Firecracker VM has no UART and gets 400.