Skip to main content

FluxVM runtime boundary

FluxVM is the node-local execution runtime. Zyvor Fabric is the distributed private-cloud control plane. GuestKit owns offline guest-disk internals.

The rule for new features is:

  • If the feature manipulates a VMM, KVM memory/device state, a VM-local cgroup, TAP/netns, or the VM-edge eBPF hook, it belongs in FluxVM.
  • If the feature decides which host, which tenant/policy, when to evacuate, how to replicate, or how to present the operation to users, it belongs in Zyvor Fabric.

Runtime contract v1​

GET /v1/runtime/capabilities advertises stable node-local primitives.

Live migration primitives:

  • POST /v1/vms/{id}/migration/start
  • GET /v1/vms/{id}/migration/status
  • POST /v1/vms/{id}/migration/cancel
  • POST /v1/vms/{id}/migration/tune (QEMU only)
  • POST /v1/vms/{id}/migration/postcopy (QEMU only)

fluxctl migrate start|status|cancel is the CLI equivalent of the first three routes above -- for the standalone mode mentioned below, where there is no Fabric orchestrator to drive them over HTTP.

Adaptive controls (QEMU)​

These let the caller steer a migration that is not converging. FluxVM only applies them; deciding when to use them is the orchestrator's job.

  • start also accepts auto_converge {initial_pct, increment_pct, max_pct} (QEMU's auto-converge capability and its throttle parameters, 1-99 each), compression {algorithm: none|zlib|zstd, level} (multifd compression; needs multifd_channels > 1) and post_copy: disabled|immediate|armed. armed enables postcopy-ram but stays in pre-copy. mode: post-copy without post_copy means immediate, as before.
  • tune {bandwidth_mbps, max_downtime_ms, cpu_throttle_initial_pct, cpu_throttle_increment_pct, max_cpu_throttle_pct} changes those parameters on the in-flight migration (migrate-set-parameters). QEMU cannot change capabilities mid-flight, so auto-converge has to be enabled at start. To keep it effectively off until needed, start with all three percentages at 1.
  • postcopy switches an armed, active migration to post-copy. From then on the destination owns guest memory: if the stream breaks, the VM is lost, not left running on the source.
  • status also reports dirty_pages_rate, dirty_sync_count, page_size, duplicate_pages, normal_bytes, mbps, expected_downtime_ms, cpu_throttle_pct, setup_time_ms and postcopy_requests when QEMU provides them. QEMU reports no compression ratio for multifd compression.
  • A receiver must be created with matching capabilities {multifd_channels, compression, post_copy}. They are applied before migrate-incoming, and QEMU refuses a stream whose capabilities differ between the two ends.
  • Every migration/start states multifd, post-copy and auto-converge (on or off), multifd compression and TLS, and every tunable (max-bandwidth, downtime-limit, multifd channels, the auto-converge throttle), falling back to QEMU's own default for any the request leaves out. QEMU keeps all of them across attempts, so a retry would otherwise inherit the previous attempt's stream shape or a throttle (a cancelled 1 Mbps attempt left the next one unable to finish). A TLS credentials object left by a failed attempt is replaced.
  • migration/cancel after the stream completed finds the guest stopped in postmigrate; with no post-copy it resumes the guest on the source. With post-copy enabled it stays stopped, because the destination may hold newer pages.
  • GET /v1/runtime/capabilities reports autoConverge, compression, postCopySwitch, liveTune and postCopyWithMultifd. The last is true only when the QEMU binary reports version 10.0 or later.

Contract v1 is intentionally narrow:

  • QEMU and Cloud Hypervisor. Firecracker and the in-tree FluxVM hypervisor have neither.
  • Pre-copy, post-copy and multifd are supported by both source runtimes.
  • tcp: and unix: migration URIs are accepted for both. exec: is rejected (QEMU has this scheme at all; Cloud Hypervisor doesn't, but the same shared allowlist covers both so the rule can't drift between them).
  • VM disks are not copied by this contract. Fabric must confirm shared storage or prepare storage independently before starting migration.
  • FluxVM does not select the destination node, reserve capacity, perform fencing, update tenant inventory, or run DRS. Those are Fabric concerns.
  • status/cancel remain QEMU-only. QEMU's QMP migrate is asynchronous with a query-migrate progress query and a migrate_cancel primitive; Cloud Hypervisor's send-migration is fire-and-forget -- verified live, it returns success as soon as the request is accepted, before the transfer (or its failure) is known, and Cloud Hypervisor exposes no API to poll or cancel what happens next. GET /v1/runtime/capabilities's migration entry carries this as statusPollable (true for QEMU, false for Cloud Hypervisor) so Fabric can tell up front rather than discovering it from a start call that never resolves into a pollable status. For Cloud Hypervisor, Fabric must instead infer the outcome the same way it already watches for any other node-local state change: the VM disappearing from this node's GET /v1/vms once its process exits (migration succeeded) versus it staying Running there (nothing has succeeded yet).

The legacy fluxvm-agent multi-host registry remains a lightweight standalone mode for users who do not run Fabric. It must not grow Fabric-class HA, DRS, site recovery, tenant scheduling, or datacenter semantics.

Threat containment (Firecracker figures)​

FluxVM maps Firecracker's host-integration and threat-containment figures to node-local layers:

BarrierOwner
KVM + minimal virtio device modelVMM backends (FC / in-tree / CH / QEMU)
Virtio I/O rate limitersFirecracker rate_limiter from create fields; in-tree token buckets; Fabric Mbps/PPS
Jailer + cgroups + seccomp[jailer] + fluxvm-cgroup + stock FC / in-tree seccomp
Host egress filterNetwork Fabric or network.mode=none (required outside the VMM)

Capability numbers (Track A isolation vs optional Track B density): capability-figures.md.

Target receiver​

POST /v1/migration/receivers arms a QEMU -incoming defer process on the target node. Two forms exist:

  • Bare ({vcpus, memory_mib, disk, ...}): a disk-only q35 QEMU. The source must have the same minimal device model; it cannot be adopted.
  • Adopt-mode ({record, listen_host, advertise_host, ...} where record is the source's GET /v1/vms/{id}): the receiver is launched from the record's own device model (network, seed, agent vsock, firmware) with its own workspace, tap/netns and vsock CID, so the source and receiver can run side by side, even on one node. It is stored as a Creating VM labelled fluxvm.dev/migrating-from=<source id>, with the receiver's id and the source's name. Contract v1 accepts QEMU VMs on storage: shared or ceph-rbd-in-place without TPM, VFIO, virtiofs, data disks, hot-added NICs or a direct/macvtap datapath.

Disks are never copied. Full order:

  1. target: POST /v1/migration/receivers with record, then POST /v1/migration/receivers/{id}/activate {token};
  2. source: POST /v1/vms/{id}/network/migration/quiesce and export (when a dataplane is attached), then POST /v1/vms/{id}/migration/start {destination: <receiver uri>}; poll GET /v1/vms/{id}/migration/status until completed;
  3. source: POST /v1/vms/{id}/migration/finish stops the paused source QEMU and removes its record and workspace (409 until the phase is completed; the disk stays);
  4. target: POST /v1/migration/receivers/{id}/adopt {token} turns the receiver into a running VM labelled fluxvm.dev/migrated-from=<source id> (409 until the incoming side has finished), then network/migration/restore and resume on the adopted id.

The adopted VM keeps the source's name but has the receiver's id: on one node both processes exist at once, so they cannot share an id, tap or netns. A caller that tracks VMs by id swaps it at adopt. DELETE /v1/migration/receivers/{id} before adopt kills the receiver and removes its VM record.

Shared-disk storage​

storage: shared uses request.image (a raw file or block device on storage every node can open) in place. Delete never removes it. <disk>.fluxvm-lock names the VM and node that have it open; a launch is refused while another VM holds it, unless that holder is on this node and gone (missing record or dead process). A holder on another node is never assumed dead: after fencing it, re-create with POST /v1/vms?shared_takeover=true, which breaks the lock first (HA re-create).

Read-only serial​

GET /v1/vms/{id}/serial is an interactive socket on QEMU. Every other backend writes its console to console.log, so the same route streams that file read-only: the last 64 KiB, then new bytes as binary frames; inbound frames are ignored. A jailed Firecracker VM has no UART and gets 400.