Skip to main content

Compatibility matrix

Short, dated evidence for Kairon’s differentiating features on real hardware. Historical narrative belongs in runbooks; this file is the pass/fail surface operators and releases should cite.

Do not claim production live migration in README until the live row below is green from an automated nightly run.

Lab profile (Zyvor)​

ComponentExpectation
Nodes≥3 KVM hosts with FluxVM
StorageShared filesystem or Ceph RBD at identical paths
NetworkL2/L3 reachability for guest + migration data plane
CNICilium (when network-policy / identity tests run)
RunnerSelf-hosted GitHub Actions runner with lab access

Automation entry points:

Lab enablement: the matrix job runs only on runs-on: [self-hosted, kairon-lab]. A kairon-lab runner is online (setup and the variable list for the per-feature drills: runbook-lab-setup.md). The workflow reads these repository secrets/vars (KAIRON_HW_LAB, KAIRON_KUBE_URL, KAIRON_KUBE_TOKEN and KAIRON_KUBE_CA are set, the rest are not): KAIRON_HW_LAB=1, KAIRON_KUBE_URL, KAIRON_KUBE_TOKEN, KAIRON_KUBE_CA (PEM content; the workflow writes it to a file), KAIRON_HW_MACHINE, KAIRON_HW_TARGET_NODE, plus optional KAIRON_HW_EBPF_MACHINE (live-eBPF case), KAIRON_HW_IMAGE (real create/stop/start/delete smoke), KAIRON_HW_SOURCE_SSH / KAIRON_HW_TARGET_SSH (failure drills; the runner needs passwordless SSH and sudo for iptables / systemctl) and KAIRON_HW_CONTROLLER_SSH (systemd controller instead of Pods), and KAIRON_UI_URL (var) / KAIRON_UI_TOKEN (secret) for the guest-agent case and for the live case, which starts a memory-dirtying load in the guest over the guest agent (KAIRON_HW_LOAD_MIB, default 256) and records not run without them. The adaptive cases take KAIRON_HW_ADAPTIVE_LOAD_MIB (default 1024) and KAIRON_HW_ADAPTIVE_BANDWIDTH_MBPS (default 200), and the post-copy source-failure drill needs KAIRON_HW_POSTCOPY_MACHINE, a disposable Machine the lab recreates before each run (sudo pkill on the source host). Until those are set, push-triggered runs succeed on the validate job only and leave every migration row below as not run.

Single-host deploy smoke (manual)​

Bare-metal systemd path (scripts/deploy-remote.sh), not a substitute for the multi-host matrix above.

CaseLast resultDateNotes
Deploy node+controller+ui (v0.6.0-24-g0020758)pass2026-09-22Host 80.79.5.173 / nldw4-4-16-36 (k3s). Health: node :8081, controller :8083 (host :8080 occupied by krytond), ui :8082
CRD server-side applypass2026-09-22deploy/crd.yaml
Machine create / Runningpass2026-09-22v07-smoke then deleted
Machine stop → startpass2026-09-22deploy-smoke
UI overview + machines APIpass2026-09-22host 80.79.5.173 port 8082 (recorded before the dashboard served HTTPS by default)
Network effective (console relay)pass2026-09-22Needs matching KAIRON_NODE_CONSOLE_TOKEN in both env files; redeploy leaves existing env untouched
Network flows/stats/dropsn/a2026-09-22deploy-smoke uses network.mode: user — FluxVM eBPF flow/drop/stats need tap+dataplaneMode: ebpf
VM edge, Kairon-scheduled Machinepass2026-10-04Host 175.110.122.71 (k3s, FluxVM cilium mode, schema 12). Machine edge-demo (tap+netns, dataplaneMode: ebpf, antiSpoof, learnIP, qos) Running; FluxVM loaded flags, egress 2000 pps, 3 allow-list names and a 100 Mbit tbf from Kairon's edge post; status.network.edge projected
VM edge enforcement (FluxVM VMs on the same host)pass2026-10-04DNS and SNI allow/deny, spoof_ip and spoof_mac (bridged tap), egress pps rate_limit, ingress police drops, learned IP via ARP, live conntrack export (14 entries), and edge + pending conntrack reapplied after FluxVM restart and VM stop/start
VM edge packet capture via kairon-uipass2026-10-04kaironctl network capture edge-demo --filter icmp -o icmp.pcap through kairon-ui, the node relay (KAIRON_NODE_CONSOLE_ADDR=:8092) and FluxVM: pcap with ICMP echo request and reply; captures lists done with packet counts; unknown token 404
VM edge metricspass2026-10-04kairon-node /metrics exports kairon_net_conntrack_restored_total and kairon_net_migration_blackhole_ms; kairon_net_drops_total appears on the first drop (unit-tested)
Native macOS node, vz Machine (7867692)pass2026-10-08Apple M4, macOS 27.2: Node Ready, Machine scheduled, Debian 13 booted by FluxVM, guest IP in status, SSH, delete (scripts/macos-e2e.sh). Not run: macOS guests, migration, CSI, multi-Mac
Policy-only edge, netns MAC defaultunit tests2026-10-04Not yet run against a live Machine
v0.10.6 (fresh install, upgrade, self-test)not run2026-11-18No cluster run recorded for this release; unit tests and CI only. The new offline commands were also run on a Linux dev host (see TEST_REPORT.md). Rows below are for earlier versions.
v0.10.5 (fresh install, upgrade, self-test)not run2026-11-17No real-host run recorded for this release; unit tests and CI only. Rows below are for earlier versions.
v0.10.4 (fresh install, upgrade, self-test)not run2026-11-12No real-host run recorded for this release; unit tests and CI only. Rows below are for earlier versions.
Published v0.10.3-rc2: fresh kaironctl install --chart oci://... with default image tags, then selftest --profile standard (FluxVM 0.4.18)pass2026-11-11Host 80.79.5.173 (k3s, KVM). kaironctl-linux-amd64 from the pre-release (SHA256SUMS verified), no image-tag overrides: controller and node pulled v0.10.3-rc2 and were Ready; doctor 7 OK; self-test 6 passed (control-plane, boot, power, pause-resume, hotplug-cpu, observability; console-log and network skipped). kaironctl upgrade against the published chart OK. The production profile without webhook values fails with a clear message and creates nothing. latest stayed on v0.10.2 for the pre-release.
Fresh Helm install + selftest --profile standard (chart 0.10.2, FluxVM 0.4.18)pass2026-11-10Host 80.79.5.173 (k3s, KVM, 12 vCPU). kaironctl install with image tags pinned to v0.10.2 and node.imageCacheDir (host dir owned by UID 65532). 6 passed: control-plane, boot, power, pause-resume, hotplug-cpu, observability. Skipped: console-log (needs the dashboard) and network (forwarded port not reachable from outside the host). The default chart tag 0.10.2 does not exist (images are v0.10.2): fixed in #202.
Redeploy node+controller keeps ports (v0.6.0-116-g1b928b9)pass2026-10-06Host 80.79.5.173. Health: node :32302, controller :32301, ui :8082, console :8090; nothing on :8080 (krytond). Needed kubectl apply -f deploy/rbac.yaml first: the cluster's ClusterRoles predated machinebackups
Deploy node+controller+ui with --apply-crds --apply-rbac (v0.7.2-17-g81911e3)pass2026-10-11Host 80.79.5.173 / nldw4-4-16-36 (k3s). CRDs now v1 (storage) + v1beta1/v1alpha1 served, deprecated (the v1alpha1 read prints the deprecation warning). RBAC regenerated from the chart; kairon-ui has the Pools/Claims/Backups grants. UI auto-picked :27231 because :18082 is held by zyvor-fabric-agent (pinned with --ui-port=27231)
Dashboard API sweep on v1pass2026-10-1126 list/diagnostic routes 200 (quotas, budgets, sets, instance types, policies, pools, claims, backups, restores, fleet, usage.csv, node capabilities/catalog/pools/templates/sandboxes); wrong password and unauthenticated calls 401; Images and /atlas/* 501 (unconfigured, by design)
Machine create + delete through the UI API on v1pass2026-10-11Created v1-smoke (apiVersion kairon.zyvor.dev/v1, Pending: node lacks the capability label), delete returned 204 and the object was gone
Tenant fencepass2026-10-06Two fenced tenants in one namespace: labels projected, tenant-fence-<tenant> groups Applied by kairon-node, each denying only the other tenant's IPs (/32, /128); opt-out deleted the group. Guest IPs were patched into status (no booted guests). Admission checks not run: no webhook on this host
Preemption by Haltpass2026-10-06Pending priority-10 Machine halted an opted-in priority-0 Machine (annotations recorded); deleting the preemptor resumed it; no preemption-policy meant no halt. Victim was pinned with spec.nodeName; node lacks kairon.zyvor.dev/capable so the preemptor stayed unschedulable
Hotplug persist across stop/startpass2026-10-06Real qemu guest (node22-agent.qcow2) 1 vCPU/1Gi, max 2/2Gi: hotplugged to 2/2048, spec lowered to 1/1Gi, Stopped → Running booted -smp cpus=2 -m 2048M; hotplug-persist=true raised spec to 2/2048Mi. Found and fixed the maxmem == memory boot failure. Halted → Running is the next row
Halted resume keeps hotplugpass2026-10-06Real qemu guest hotplugged 3/3072 → 4/4096, halted and resumed: FluxVM booted at 3/3072 and kairon-node hotplugged back to 4/4096 (status.applied* matched)
Pool claim carries templatepass2026-10-06createMachine: true claim from a FluxVM pool: Machine created with template resources, cloud-init, guest agent, TTL and forwards, pinned to the node, adopted by kairon-node; FluxVM's normalised user network and default storage produced no warnings
Snapshot schedule dailyAt/jitter/maxAgepass2026-10-06CRDs applied (CEL rejects setting both intervalSeconds and dailyAt); a dailyAt schedule fired at its 18:50 UTC slot and projected the next run a day later. The snapshot itself failed (stopped Machine, no volume, no CSI snapshotter here), so maxAgeSeconds pruning is unit-tested only
CPU-pinning deploy wiringpass2026-10-06deploy-remote.sh --reserved-cpus=0-1: kairon-cpustate.path copied the state file (root:kairon 0640), label 2-11 published; --no-cpu-pinning removed units, copy and env keys. Helm node.cpuPinning rendered only (helm template)
MCP approvalpass2026-10-06delete_machine under --allow-write refused with an ID; kaironctl approve machine/default/NAME ID wrote the annotation; the retry consumed it and deleted the Machine; audit outcomes approval-required then approved:<user>
Stale evacuationpass2026-10-06Controller with KAIRON_STALE_EVACUATION=true and a hand-made stale kairon-node-<node> Lease in kairon-system: no action until kaironctl node fence NODE --reason ..., then the evacuate=true Machine got Fenced=True StaleEvacuation and lost spec.nodeName; the other was only logged; the fenced Machine deleted cleanly; --clear removed the annotation. Single node, so nothing was re-placed. Leases outside kairon-system are forbidden to the controller by RBAC
Discovered pinnable CPUspass2026-10-0612 CPUs, kubelet none policy, KAIRON_RESERVED_CPUS=0-1: refused without reserved CPUs and on the root-only state file; with a readable copy published 2-11; simulated static entry 4-5 gave 2-3_6-11; removing the file cleared the label; an operator label was kept. Host restored to discovery off

Migration matrix​

CaseLast resultDateNotes
Create / restart / delete smoke (≤100 VMs scaled down in CI)pass2026-10-14One VM, not 100: create → healthy → stop → start → delete from a shared-storage image on the two-host lab; the 100-VM scale-down is still not run
Cold migrationpass2026-10-14Two hosts on the same QEMU 10.2.1 (nldw4-4-04-32 runs a source build under /opt with the packaged efi-virtio.rom; nldw4-4-16-36 Ubuntu 26.04 package), NFS shared root disk, mTLS adapter (FluxVM 0.4.8 on both hosts; an earlier run the same day also moved hw-cold-labm1 both ways): hwm-live moved nldw4-4-04-32 → nldw4-4-16-36, healthy with a guest IP on the target
Live pre-copy under loadfail2026-10-14Two runs, both directions of QEMU pairing: plain pre-copy with the guest rewriting 256 MiB ran the full 600 s without converging (throughput about 930 Mbps against a dirty rate of about 930 Mbps, expected downtime 2.4 s). The adaptive rows below are the same lab with a heavier load and pass
Guest agent exec after migrationpass2026-10-14Same lab: echo through kairon-ui (admin session) → kairon-node → qemu-guest-agent in the migrated Machine, exit 0
Live + eBPF dataplanenot run—Machine with spec.network.dataplaneMode=ebpf; set KAIRON_HW_EBPF_MACHINE. Needs two hosts; also exercises the VM-edge conntrack export/restore
Source failure during transferpass2026-10-14Two hosts on the same QEMU 10.2.1 (nldw4-4-04-32 runs a source build under /opt with the packaged efi-virtio.rom; nldw4-4-16-36 Ubuntu 26.04 package), NFS shared root disk, mTLS adapter: kairon-node stopped on the source mid-transfer and restarted; the migration still reached Succeeded without operator action
Ambiguous commit → NeedsRecoverypass2026-10-14Same matched-QEMU lab, nldw4-4-04-32 → nldw4-4-16-36, firewalling the control-plane port during Running, then ConfirmDestinationNotCommitted: passed 5 of 6 runs on a freshly created Machine, plus 3 of 3 after the destination-health guard below (cutover completed, Machine Running on the target, recorded diagnosis destinationSessionPhase: Prepared, destinationRuntimeStatus: creating). The one failing run came right after a cascade of timed-out migrations that left stale FluxVM runtimes behind: its diagnosis showed destinationRuntimeStatus: failed, and after the action the migration sat in Adopting with the Machine Unknown. Not reproduced on a clean lab and not explained. ConfirmDestinationCommitted and ConfirmDestinationNotCommitted now refuse while the destination runtime is failed/stopped (and NotCommitted re-checks after its retried commit) instead of deleting the source: with the destination QEMU killed during NeedsRecovery, the action stayed NeedsRecovery with recovery refused, was not recorded as applied, and the source runtime stayed running; ForceAbort then ended it Failed
Controller failover mid-migrationpass2026-10-14Two hosts on the same QEMU 10.2.1 (nldw4-4-04-32 runs a source build under /opt with the packaged efi-virtio.rom; nldw4-4-16-36 Ubuntu 26.04 package), NFS shared root disk, mTLS adapter: kairon-controller restarted mid-migration; the migration still reached Succeeded
Adaptive: auto-converge under loadpass2026-10-14Same lab, guest dirtying 1024 MiB behind a 200 Mbps link: converged, downtime 47 ms, total 158 s (the 8.2.2 → 10.2.1 run: 61 ms, 427 s)
Adaptive: post-copy switch under loadpass2026-10-14Same lab, same load, nldw4-4-16-36 → nldw4-4-04-32: switched to post-copy, downtime 9 ms, total 25 s
Adaptive: zstd multifd compressionpass2026-10-14Same lab, idle guest: downtime 33 ms, total 5.2 s, compression ratio 142%
Source failure after post-copy → NeedsRecoverypass2026-10-14Same lab: source QEMU killed after the post-copy switch; NeedsRecovery with post-copy latched, source not resumed, destination not aborted, then ForceAbort recovered it

Direction and QEMU versions. Run 1 (2026-10-14) was nldw4-4-04-32 on QEMU 8.2.2 to nldw4-4-16-36 on 10.2.1, one way only: FluxVM refuses the reverse ("this host's QEMU cannot boot the source's machine type pc-q35-10.2"). Run 2 had both hosts on QEMU 10.2.1 and migrated in both directions (a manual round trip: 151 ms and 125 ms downtime, about 4.5 s total). Matching versions was not enough on its own: a source-built QEMU's efi-virtio.rom (160 KB) differs from the packaged one (512 KB), and the receiver then fails with Size mismatch: virtio-net-pci.rom; both hosts must ship the same option ROM. Live + eBPF dataplane was not run (no eBPF Machine in the lab).

hardware-migration-matrix.sh --write-compat rewrites these rows from a run (the workflow uploads the result as the compatibility-md artifact); without the flag it prints markdown rows for copy/paste. Every migration case checks that the Machine ends up Running and Ready on the expected node, and the script exits non-zero if any case fails. For the eBPF live case, the lab Machine must set dataplaneMode: ebpf and dataplaneRequired: true so migration network quiesce/export/restore exercises FluxVM's TC/eBPF path (see network-fabric.md).

The four adaptive rows need FluxVM with the adaptive migration controls on both hosts: each node's migration adapter GET /v1/capabilities must report tune, autoConverge, postCopySwitch and zstd in compression. Adaptive live migration is Experimental until all four pass on the nightly workflow alongside the base live rows.

Incremental S3 backups​

Single node plus MinIO, driven by scripts/lab-cbt-backup.sh against a disposable image-booted Machine with spec.guestAgent.enabled on a node labelled kairon.zyvor.dev/cbt=true (FluxVM with QEMU 5.1+). --minio-ssh starts MinIO with an Object Lock bucket and creates the labelled Secret; the script prints these rows. Incremental S3 backups stay Experimental until all four pass. FluxVM's own QEMU-gated engine test (stopped_vm_chain_round_trips_through_qemu, passed on QEMU 10.2.1) covers a stopped VM only, through qemu-storage-daemon, without Kairon.

CaseLast resultDateNotes
CBT: changed-bytesnot run—Full, dirty --dirty-mib (64) in the guest, incremental: changedBytes within --dirty-mib .. + --slack-mib
CBT: restorenot run—Corrupt the guest file, halt, restore full + incremental, start: sha256 matches
CBT: synthesizenot run—Synthetic full of the incremental, delete the full (S3 objects too), restore the incremental again: sha256 matches
CBT: forced-fullnot run—SIGKILL the Machine's QEMU (--node-ssh): the next incremental fails with status.disk.fullRequired, a full succeeds

Topology-aware placement​

Driven by scripts/lab-topology.sh against one node with the kairon.zyvor.dev/pinnable-cpus label and a FluxVM that reports host.topology; the script prints these rows and reports a case "not run" when the node's hardware cannot exercise it. Topology-aware placement stays Experimental until the multi-NUMA, GPU+NIC and SR-IOV rows pass. FluxVM's QEMU-gated numa_qemu test (passed on QEMU 10.2.1) checks the memory backend's policy=bind and vCPU thread affinity without Kairon.

CaseLast resultDateNotes
Topology: nodetopologypass2026-10-11Single-socket node (12 CPUs, 1 NUMA node, QEMU 10.2.1, k3s 1.35): NodeTopology with 1 NUMA node, 3 PCI devices, numaBinding true
Topology: single-numapass2026-10-11Same node, pinnable CPUs 2-11: guest node 0 on host numa0, vCPU threads on CPUs 2-3, 1024MiB policy=bind (status.appliedTopology)
Topology: unsatisfiablepass2026-10-11Same node: a single-numa-node Machine asking for an NVIDIA GPU stayed Pending, reason TopologyUnsatisfiable (one NUMA node, so the oversize-CPU variant is refused by the plain capacity check first)
Topology: multi-numanot run2026-10-11Lab host has one NUMA node
Topology: gpu-nicnot run2026-10-11No allowlisted GPU and NIC under one PCIe root in the lab
Topology: sriovnot run2026-10-11No SR-IOV VF in the lab

Elastic memory​

Driven by scripts/lab-elastic-memory.sh against one node with a FluxVM that reports host.balloon. Pressure is forced by setting KAIRON_MEMORY_LOW_WATERMARK_MIB above the host's MemAvailable and restarting kairon-node; relief removes it. Elastic memory stays Experimental until it has run under real (not forced) memory pressure and on more than one node, including evacuation, which is not covered here.

CaseLast resultDateNotes
Elastic memory: balloon-devicepass2026-10-11Single node (31424MiB, QEMU 10.2.1, k3s 1.35, Ubuntu 24.04 guest): 2Gi Machine with min 1Gi; status.memory written; QEMU has virtio-balloon-pci,deflate-on-oom=on,free-page-reporting=on
Elastic memory: inflatepass2026-10-11Same node: forced pressure moved the balloon target to 256MiB in the first pass, reason HostPressure
Elastic memory: exhaustedpass2026-10-11Same node: target and guest actual 1024MiB (at min), KaironMemoryPressure=True; a new Machine pinned to the node stayed unscheduled: "node reports KaironMemoryPressure"
Elastic memory: reliefpass2026-10-11Same node: target back to 0, reason Relief, KaironMemoryPressure=False; the guest was still deflating (255MiB actual) when the case ended
Elastic memory: overcommitpass2026-10-11Same node: two 18854Mi Machines (min 512Mi) on 31424Mi allocatable; the second stayed unscheduled ("insufficient memory") until the node got memory-overcommit-ratio: 1.5, then scheduled

Hibernate​

Driven by scripts/lab-hibernate.sh against one node with a FluxVM that reports host.hibernate. The guest runs a cloud-init probe that writes a random marker to /dev/shm and prints a tick counter, its boot id and the marker to the serial console, so the image needs no guest agent. Hibernate stays Experimental until resume on another node has run on two real hosts sharing the hibernation directory and an RBD root disk.

CaseLast resultDateNotes
Hibernate: hibernatepass2026-10-12Single node (QEMU 10.2.1, pc-q35-10.2, k3s 1.35, Ubuntu 24.04 guest, 1Gi): Hibernated in 10s, 460 MiB of state, FluxVM record stopped with no pid
Hibernate: capacitypass2026-10-12Same node: no QEMU process left; host MemAvailable 10494 -> 11140 MiB
Hibernate: resumepass2026-10-12Same node: Running in 16s with resumeNode set to the node; tick 4 -> 6, same boot id and /dev/shm marker; status.hibernation cleared
Hibernate: discardpass2026-10-12Same node: discard-hibernation annotation gave a new boot id; annotation removed, status.hibernation cleared
Hibernate: resume on another nodenot run2026-10-12Needs two nodes sharing the hibernation directory and a shared root disk

Lab pair re-run (2026-10-15)​

Run after regenerating the migration CA and per-host certificates and giving each node its own kairon-migration-adapter-fluxvm over the shared /srv/kairon-shared NFS export. Cluster: k3s on nldw3-4-32-26 (controller, UI, node) plus nldw4-4-04-32 (a second kairon-node-lab unit and its own adapter). Machine: the lab's hw-live (2 vCPU, 2 GiB, idle guest). QEMU 10.2.1 is installed on both hosts (a source build at /opt/qemu-10.2.1 on nldw4-4-04-32, the Ubuntu package on nldw3-4-32-26); which binary FluxVM launched on nldw4-4-04-32 was not recorded. Driven from a workstation with hardware-migration-matrix.sh using a short-lived token for the kairon-lab service account.

CaseResultNotes
Create / restart / delete smokepass (health only)KAIRON_HW_IMAGE unset: checked hw-live Running/Ready, no create/delete
Cold migrationpassnldw3-4-32-26 → nldw4-4-04-32, about 38 s
Live migration, idle guest, nldw4-4-04-32 → nldw3-4-32-26passFour runs (two plain, one during the source-failure case, one during the NeedsRecovery drill): 4.1–4.4 s transfer, 148–242 ms downtime, dataPlaneEncrypted: true
Live migration, idle guest, nldw3-4-32-26 → nldw4-4-04-32passTwo runs (one plain, one during the controller-failover case): 4.2–4.6 s transfer, 64–262 ms downtime, dataPlaneEncrypted: true
Source failure during transferpasskairon-node-lab stopped on the source for 20 s mid-transfer, then restarted; migration Succeeded, Machine healthy on the target
Controller failover mid-migrationpasskairon-controller restarted mid-migration; migration Succeeded
Ambiguous commit → NeedsRecoverypassDROP on the destination's tcp/9445 until NeedsRecovery, then ConfirmDestinationNotCommitted completed the cutover; diagnosis showed destination session Prepared, runtime creating; no rule left behind
Live pre-copy under load, guest agent exec, adaptive (converge, post-copy, zstd), post-copy source failurenot runNeed a dashboard URL and token for guest load, a disposable Machine, and FluxVM adaptive support on both hosts
Live + eBPF dataplanenot runNo eBPF Machine in the lab

One run per case, a single Machine and an idle guest, so this shows the control plane, certificates and failure handling work on this pair; it says nothing about convergence under load.

Repeatability, dead node, rotation and scale (2026-10-15/16)​

Same pair, same idle hw-live guest. These runs are what the single-run table above does not show, including the failures.

CheckResultNotes
Repeated live migrationpartial13 of 15 live migrations in the early repeat and soak runs succeeded. Soaks stopped after 8 and 10 clean hops. Failures were (1) a FluxVM race that reaped the receiver's Creating placeholder before QEMU launch (about 1 in 10; fixed in FluxVM #417, not yet deployed to the lab), (2) a transient API timeout the controller treated as terminal (fixed, #152), (3) a commit timeout on a host with load 22/54/57 on 12 CPUs
Dead node (power-off simulation)passNodeUnreachable after 51 s; Machine restarted on the other node 13 s after kaironctl node fence attestation. Machines not opted in to evacuation were left alone. Simulates a dead agent and killed VM, not a shared-disk split brain
Migration CA rotationpassNew 10-year CA, 90-day leaf certificates, mTLS verified both ways, two live migrations afterwards (runbook-migration-cert-rotation.md)
Scale smokepass at cap8 small Machines ramped in batches of 2, all Running in 97 s (7 on one node, 1 on the other; lowest free memory 8.7 GiB), then deleted. The cap was 8 because the lab has no memory for more; no limit was found

Defects found by these runs and fixed: a lost VM when a stale same-name FluxVM record hid an adopted VM and a cold migration ran during Adopting (one migration per Machine is now enforced, #149); a source left paused after ForceAbort (#148); a destination whose runtime failed before commit (#150).

Not shown: more than 8 Machines, migration under load or with many Machines, guest IPs (user-mode networking), and the FluxVM #417 fix in the soak.

Container sandboxes​

spec.sandbox.oci and kaironctl container (see container-sandboxes.md) run an OCI image on FluxVM's container track, which exists on macOS hosts only (Virtualization.framework).

Run on 2026-10-15 on one Apple-silicon Mac (FluxVM 0.4.7, local daemon on 127.0.0.1:7788) with the OCI boot artifacts from FluxVM's oci-boot CI run of 2026-10-11 (older than the host's FluxVM; they worked).

CaseLast resultDateNotes
Run an image to completion (fluxctl sandbox run alpine:3.22 --rm)pass2026-10-15Printed the container's output, exit 0, powered off; about 7 s including the image pull
Create through Kairon's client and read the resultpass2026-10-15CreateSandboxForMachine with spec.sandbox.oci → container ran sh -c 'echo …; exit 3'; SandboxLogs returned exit code 3 and the console output in about 4 s (TestContainerSandboxAgainstLiveFluxVM, opt-in with KAIRON_FLUXVM_OCI_URL)
Through kairon-node and the controller on a Macnot run—The node reconcile path (status projection, admission webhook) is unit-tested with fakes only
linux/amd64 image under Rosettanot run—
Published ports, restart policy, health checknot run—Request mapping is unit-tested; FluxVM behaviour was not exercised
Linux hostnot supported—FluxVM's container track does not run on the Linux hypervisors

Agent sandboxes​

Driven by scripts/bench-sandbox.sh on one node with an image that runs qemu-guest-agent (see agent-sandboxes.md). It needs kairon-ui with the console relay configured (exec, wake on connect, vnc screenshot and guestfs go through it), kairon-node --image-cache-dir (clone and cdrom) and libguestfs-tools on the node for guestfs.

Run on 2026-10-14 against nldw4-4-16-36 (k3s 1.36, kairon-hw Ubuntu guest, 1 vCPU / 1Gi, image on NFS). That node's FluxVM build has no hibernate or fork endpoint and its kairon-node had no libguestfs or image cache during the run, so those rows are environment failures, not Kairon results. The existing Hibernate rows passed on the other lab node.

CaseLast resultDateNotes
Sandbox: idle pausepass2026-10-14--idle-timeout 1m: Paused by kairon-controller 212s after the agent came up (the guest stayed busy for its first 2.5 minutes; last activity 60s before the pause)
Sandbox: wake on connect (Paused)pass2026-10-14kaironctl guest exec on the Paused Machine returned in 1.6s including the resume; phase and powerState Running afterwards
Sandbox: wake on connect (Hibernated)not run2026-10-14This node's FluxVM returns 404 for POST /v1/vms/{id}/hibernate
Sandbox: pool claim, Running poolpass2026-10-14status.bindMillis 2267 against a warm Running member
Sandbox: pool claim, Paused poolpass2026-10-14status.bindMillis 4193 against a warm Paused member (includes the resume)
Sandbox: forknot run2026-10-14This node's FluxVM returns 404 for POST /v1/vms/{id}/fork
Sandbox: lifetimepass2026-10-14--max-lifetime 1m Machine deleted by kairon-controller (seen within 4 minutes in an earlier partial run; the script now times it)
Sandbox: freeze auto-thawpass2026-10-14freeze --unfreeze-timeout 1m confirmed in 4.2s; kairon-node thawed it 63s after the request
Sandbox: freeze/unfreezepass2026-10-14Round trip confirmed in 3.8s; the guest writes again afterwards
Sandbox: vnc screenshotfail2026-10-14kairon-node got permission denied on FluxVM's vnc.sock on this node; same relay path as kaironctl vnc
Sandbox: create clone from Haltedpass2026-10-14qemu-img convert of the source disk took 117s (675 MB image on NFS), the clone booted and its guest agent answered 40s later
Sandbox: guestfs read-onlynot run2026-10-14libguestfs-tools is not installed on this node
Sandbox: cdrom insert/ejectpartial2026-10-14Stop cleared status.runtimeID in 13s and cdrom insert was admitted; the start then needs --image-cache-dir, which this node lacked; cdrom eject removed the entry and the Machine then started. Live eject not exercised

Agent sessions​

Needs a Mac node running kairon-node natively against FluxVM 0.4.8 or later (FluxVM creates agent sessions only on its Apple Virtualization backend), kairon-ui with the console relay, and ui.agentSessions.previews: true for the preview row. The lab has no Mac, so nothing below has run on real hardware. Unit tests cover the same paths against a fake FluxVM (see agent-sessions.md).

CaseLast resultDateNotes
Session: capability labelnot run-Mac node gets kairon.zyvor.dev/agent-sessions=true; a Linux node with FluxVM 0.4.8 does not
Session: run with git workspacenot run-kaironctl session run reaches Running; status.workspace.cloned true
Session: offline policynot run---preset offline: outbound request from session exec fails; status.policy.offline true
Session: allow-host policynot run-Only the allowed host answers; status.policy.unenforced empty on vz
Session: exec and jobsnot run-session exec exit code matches; bg + logs -f streams to the end
Session: files and gitnot run-put/cat round trip; git commit shows in git log
Session: checkpoint and restorenot run-A file written after checkpoint is gone after restore
Session: fork and promotenot run-Each child becomes an AgentSession on the parent's node; promote applies a child's change to the parent
Session: rolloutnot run---count 4: four results, children deleted afterwards
Session: previewnot run-URL opens through kairon-ui; CSP sandbox header present; 404 after unexpose
Session: OTLP exportnot run-Trace shows in the collector with one span per step
Session: approvalsnot run---process-ask 'git push*': exec answers 428; an operator's approve gets 403; after an admin approves, --approval-id runs it once; note names the approver
Session: chunked uploadnot run-put of a 50 MiB file: same sha256 in the guest; an interrupted put leaves no file
Session: shellnot run-session shell echoes input, follows a window resize, exits with the shell's code; refused when the session has process rules
Session: probenot run-Lists the guest's git and node; ready_for.git true
Session: screennot run-FluxVM after 0.4.8: session screenshot saves a PNG of the guest display; input click and type change it; session_screenshot returns an image to an MCP client; the audit log has no typed text
Session: enginenot run---engine agent with a stock image: Failed with FluxVM's 409; with agent-micro: Running and status.capabilities.engine is agent
Session: fork clone reportnot run-Fork on APFS: files_copied 0, no warning
Session: credential grantnot run-grant --host api.github.com: an authenticated API call from the guest succeeds, the guest's environment has no token; fails after revoke-grant
Session: quotanot run-A session over MachineQuota stays Pending with the quota's reason
Session: deletenot run-FluxVM session gone; finalizer released

Incident bundles​

Driven by scripts/lab-incident-bundle.sh against one node, through kairon-ui and the node console relay. The guest gets two planted secrets through cloud-init (an api_key= file and a password= runcmd) and a probe printing a tick counter and its boot id to the serial console.

CaseLast resultDateNotes
Incident bundle: authpass2026-10-12Single node (QEMU 10.2.1, k3s 1.35, Ubuntu 24.04 guest, 1Gi): POST diagnostics without a token is 401
Incident bundle: bundlepass2026-10-12Same node: kaironctl diagnose machine --deep --memory-dump --pcap 5s in 5s, 43 KB, 56 files; 48 of 52 node sections ok (no balloon device; no eBPF dataplane, so no stats, flows or pcap)
Incident bundle: redactpass2026-10-12Same node: neither planted secret in any of the 56 files; 3 redacted:sha256: fingerprints
Incident bundle: memory dumppass2026-10-12Same node: 1040 MiB ELF core, guest paused 1099 ms; downloaded with kaironctl diagnose memory-dump through kairon-ui, sha256 matches, contains the guest kernel banner
Incident bundle: continuitypass2026-10-12Same node: probe ticked on (3 -> 9) in the same boot after the dump
Incident bundle: busypass2026-10-12Same node: a second start while a dump job runs is 409
Incident bundle: admin sessionpass2026-10-12Same node: an admin login session reads the job
Incident bundle: cleanuppass2026-10-12Same node: deleting the Machine removed its FluxVM diagnostics directory
Incident bundle: packet capturenot run2026-10-14Machine with dataplaneMode: ebpf failed: 175's FluxVM is pinned to mode = "cilium" with no Cilium socket (old k3s removed); 80's FluxVM 0.3.0 returns 404 on /network/edge (no dataplane schema 12)
Incident bundle: non-admin account refusedpass2026-10-14Two-node k3s (80, kairon-ui self-signed TLS): non-admin viewer (KAIRON_UI_USERS_JSON, admin:false) POST …/diagnostics → 403 incident bundles require an admin account, GET …/diagnostics/<token> → 403, plain machine read still 200; the admin account's same POST → 202

Virtual networks​

Driven by scripts/lab-virtual-network.sh on one node with a FluxVM that reports host.vnet and kairon-node started with --vnet-uplink. It creates two VirtualNetworks with the same CIDR (one with egress SNAT) and three Machines. The guests run a cloud-init probe loop and report to their serial console, so the image needs no guest agent. The uplink program ran on the node's SSH uplink ahead of Cilium's cil_from_netdev for the whole run, and was detached when the last network was removed. FluxVM's root-gated vnet_netns test separately routes across two netns-simulated hosts over Geneve, and checks floating-IP DNAT and egress, anti-spoofing and SNAT for TCP. Virtual networks stay Experimental until the cross-host, floating-IP and BGP rows run on real nodes.

CaseLast resultDateNotes
Virtual network: ipampass2026-10-12Single node (kernel 7.0, QEMU 10.2.1, k3s 1.35, Ubuntu 24.04 guests, Cilium on the uplink): two networks on 10.88.0.0/24 got VNIs 4096 and 4097; a1 10.88.0.10, a2 10.88.0.11, b1 10.88.0.10 (overlapping) as requested; status.network.virtualNetwork set
Virtual network: same-hostpass2026-10-12Same node: a1 and a2 ping each other on VNI 4096
Virtual network: gatewaypass2026-10-12Same node: a1 pings the distributed gateway 10.88.0.1
Virtual network: isolationpass2026-10-12Same node: b1 (VNI 4097, a1's address) failed all 6 pings to a2's 10.88.0.11 while a1 reached it
Virtual network: snatpass2026-10-12Same node: a1 reached 1.1.1.1 by ping and HTTP, translated to the node's uplink address
Virtual network: closedpass2026-10-12Same node: b1 (no egressSNAT) failed all 6 pings to 1.1.1.1
Virtual network: cleanuppass2026-10-12Same node: after the Machines were deleted, kairon-node removed both VNIs from FluxVM and the uplink program was detached
Virtual network: cross-hostnot run—Needs two nodes; Geneve between hosts has only run between netns-simulated hosts (FluxVM vnet_netns)
Virtual network: floating-ipnot run—Needs a public address routed to the node; covered only by FluxVM vnet_netns
Virtual network: bgpnot run—Needs a BGP router; the speaker is unit-tested against a fake peer

NVMe-oF storage​

Driven by scripts/lab-nvmeof.sh on one node, which is both the storage node and the Machine host. A dummy interface carries two tcp portals, and soft-RoCE (rdma_rxe) on the same interface carries an rdma portal. The first four cases drive the CSI Controller and Node servers in-process against the real kernel target and initiator. The rest go through kairon-csi-node, a host-installed kairon-node and FluxVM. The Machine downloads its qcow2 image from a loopback HTTP server (spec.image.source), and kairon-node writes it onto the empty Block volume with qemu-img. The guest prints a tick counter, its root device and its disk sizes to the serial console, so the image needs no guest agent.

CaseLast resultDateNotes
NVMe-oF: tcppass2026-10-12Single node (kernel 7.0.0-34, nvme_core.multipath=Y): Filesystem volume over 2 tcp portals, ext4, online expand to 463 MiB; 2 of 2 paths live, ANA optimized
NVMe-oF: anapass2026-10-12Same node: one portal's ANA group set inaccessible showed 1 of 2 paths usable; a 1 MiB write+fsync went through
NVMe-oF: failoverpass2026-10-12Same node: volume unexported from one portal; a 1 MiB write+fsync took 2 ms over the other path; the path reconnected on re-export
NVMe-oF: rdmapass2026-10-12Same node: raw Block volume over nvme-rdma (soft-RoCE); a 4 KiB pattern at 64 MiB written through the published device read back with O_DIRECT
NVMe-oF: block-bootpass2026-10-12Same node (QEMU 10.2.1, k3s 1.35, Ubuntu 24.04 guest): 4 GiB nvme-tcp Block PV seeded from a qcow2 image; QEMU root on /dev/nvme0n1 (FluxVM storage: shared), guest root /dev/vda1
NVMe-oF: iopass2026-10-12Same node: over QMP, the virtio-blk data disk has an iothread, num-queues=2 and cache.direct (cache none), and FluxVM records aio: io_uring (QMP does not report aio). The vhost-user-blk disk has num-queues=2 on one qemu-storage-daemon. The guest sees both 1 GiB disks
NVMe-oF: pathspass2026-10-12Same node: StoragePaths True, then PathsDegraded with one portal unexported, then True after re-export; the guest kept ticking
NVMe-oF: raw image onto a Block volumenot run2026-10-14lab-nvmeof.sh on 80 passed tcp, ana, failover and rdma again, then the block-boot Machine failed: FluxVM 0.3.0 rejects storage: shared (422). Drill stopped and its devices removed
NVMe-oF: rdma on a real RDMA NICnot run2026-10-12soft-RoCE only in this lab
NVMe-oF: in-cluster DaemonSetsnot run2026-10-12The lab ran host-installed binaries; the chart is rendered and linted only
NVMe-oF: two nodesnot run2026-10-12Storage node and Machine host were the same machine

Disk QoS​

Driven by scripts/lab-qos.sh on one node with host-installed kairon-controller, kairon-node and FluxVM from this change. The Machine's data disk is a 1 GiB raw file behind a hostPath PV. The guest measures 4 KiB direct I/O on both disks for one second and prints the rates to the serial console. QEMU's default burst lets one-second samples run 5–15% over the limit. All cases below passed in one run, with load average 24–32 from other workloads on the host. In an earlier run the ch-virtiofs case failed once: Cloud Hypervisor exited with Cannot create virtio-rng device: Operation not permitted. Two manual retries and the next full run passed.

CaseLast resultDateNotes
Disk QoS: crd-validationpass2026-10-13Single node (kernel 7.0.0-34, QEMU 10.2.1, k3s 1.35): iopsTotal with iopsRead and a throttle on a vhost-user-blk disk both rejected with 422
Disk QoS: fluxvm-bootpass2026-10-13Same node, FluxVM API only: QMP shows root 300 IOPS and 50 MiB/s after create; a live PUT to 350 IOPS and a hot-added data disk with 150 read / 120 write IOPS both survive stop/start (new QEMU pid)
Disk QoS: ch-virtiofspass2026-10-13Same node, Cloud Hypervisor v53.0: a VM with a virtiofs share stays up after 20 s and ch-remote info lists fs0@virtiofs-0.sock. Before this change FluxVM passed dax=off, which v53 rejects, and still reported the VM running
Disk QoS: bootpass2026-10-13Same node: Machine with root 200 IOPS and data disk 10 MiB/s; status.appliedDiskThrottles and QMP match; guest 207 IOPS on root, 11151 KiB/s on the data disk
Disk QoS: live-changepass2026-10-13Same node: raised to 400 IOPS and 20 MiB/s; same QEMU pid; guest 428 IOPS, 22288 KiB/s
Disk QoS: liftpass2026-10-13Same node: image.io: null lifts the root limit in QMP and status; guest root 17997 IOPS
Disk QoS: recreatepass2026-10-13Same node: root limit back, then Stopped → Running; the new FluxVM runtime's create request carries root_disk_io.throttle and QMP shows 200 IOPS
Disk QoS: limits across live migrationnot run2026-10-13One lab node; QEMU does not migrate throttles, so kairon-node reapplies them on the target (unit-tested only)
virtiofs DAXnot supported2026-10-13QEMU 10.2.1 vhost-user-fs-pci has no cache-size; Cloud Hypervisor v53 --fs has no dax

Node maintenance and DRS​

Driven by scripts/lab-maintenance.sh on one node, against a host-installed kairon-controller (run with -drs-mode=recommend -drs-interval=10s through a temporary systemd drop-in) and FluxVM. Enter and exit go through kaironctl node maintenance. One QEMU Machine prints a tick counter, its boot id and a /dev/shm marker to the serial console. With one node there is never a migration target, so these cases exercise the cordon, the fallbacks and exit, not migration.

CaseLast resultDateNotes
Node maintenance: drspass2026-10-12Single node (k3s 1.35, QEMU 10.2.1): recommend mode exported kairon_drs_node_utilization_percent (8% reserved) and kairon_drs_imbalance_percent 0; kaironctl drs plan reported no moves with 1 eligible node
Node maintenance: waitpass2026-10-12Same node: cordoned and Entering in 17s with "1 with no eligible target node"; the Machine stayed Running, no MachineMigration was created, MaintenanceEntering Event recorded
Node maintenance: stoppass2026-10-12Same node: --fallback stop stopped the Machine and parked it (from Running to Stopped); its QEMU process was gone; InMaintenance in 15s with a MaintenanceReady Event and kairon_node_maintenance{state="InMaintenance"} 1
Node maintenance: exitpass2026-10-12Same node: exit uncordoned the node and the Machine was Running again in 69s in a fresh boot; parked annotation removed, MaintenanceExited Event
Node maintenance: hibernatepass2026-10-12Same node: --fallback hibernate hibernated the Machine and reached InMaintenance in 27s; exit resumed it in the same boot (same boot id and /dev/shm marker, tick 4 -> 12)
Node maintenance: cordonpass2026-10-12Same node: on a node the operator had cordoned, maintenance reached InMaintenance without marking the cordon as its own, and the node stayed cordoned after exit
Node maintenance: migrate Machines to another nodepass2026-10-14Two-host lab: kaironctl node maintenance enter nldw4-4-04-32 live-migrated both Machines to nldw4-4-16-36 (two maintenance-* MachineMigrations Succeeded), InMaintenance in 30 s; exit uncordoned the node
DRS: recommend or automatic movesnot run2026-10-12Needs two nodes
Node maintenance and DRS: in-cluster chart RBACpass2026-10-14Chart rendered (helm template) and applied under a throwaway ServiceAccount: controller role can get/list/watch/patch nodes (PATCH of a node annotation with its own token → 200) and not update nodes, delete namespaces, create clusterrolebindings or list secrets/pods cluster-wide; node role can get/list/patch, not watch. Objects removed afterwards. Controller binary itself still ran with deploy/rbac.yaml, not the chart

The pressure signal is driven by scripts/lab-drs-pressure.sh on the same node, with host-installed FluxVM, kairon-node and kairon-controller from this change. Two Machines run on the node: one burns both vCPUs, one idles. Their QEMU threads are pinned to host CPUs 10 and 11, which two host busy loops keep loaded for 150 s. Without that contention a burning guest gets all the CPU it wants and does not stall. The script creates a second, synthetic Node with no kubelet or kairon-node, and keeps its heartbeat, lease and a calm pressure annotation fresh. The lab Machines tolerate the not-ready, unreachable and Cilium agent-not-ready taints such a node gets. kairon-controller runs -drs-mode=recommend -drs-signal=pressure through a temporary systemd drop-in, so nothing is migrated to the synthetic node. Host PSI on this shared node was 20–55% in CPU or IO across runs, from other workloads.

CaseLast resultDateNotes
DRS pressure: publishpass2026-10-13Single node (kernel 7.0.0-34, QEMU 10.2.1, k3s 1.35): a 5 s old Node annotation had cpu 52.0, memory 0.1, io 18.7; /proc/pressure avg60 read right after was 52.58, 0.13, 19.26
DRS pressure: machine-stallpass2026-10-13Same node, CPUs 10–11 contended: the burning Machine's cgroup CPU PSI was 5.4% against 2.3% for the idle one. Without contention both read about 0.2%
DRS pressure: planpass2026-10-13Same node: kaironctl drs plan --signal pressure planned the burning Machine from the real node (cpu 52.0%, threshold 25%) to the synthetic node (0.0%), "this Machine stalls 5.4%"
DRS pressure: controller-recommendpass2026-10-13Same node: kairon-controller recorded a DRSRecommendation Event on the burning Machine: "cpu pressure on nldw3-4-32-26 is 51.1% (threshold 25%); … this Machine stalls 4.1%"
DRS pressure: stalepass2026-10-13Same node: with the synthetic node's sample backdated 10 minutes, the plan left it out ("no fresh sample") and planned no moves
DRS pressure: automatic move between two real nodesnot run2026-10-13One lab node; the synthetic node only receives recommendations

GPU and vGPU​

Driven by scripts/lab-vgpu.sh on one node with the kernel's mbochs sample mdev driver standing in for a vGPU host driver: a virtual parent with three display profiles (4, 16 and 64 MiB) sharing a 256 MiB pool. Host-installed kairon-controller, kairon-node and FluxVM. The guest prints the count of 1234:1111 display functions and the size of BAR0 to the serial console. KubeVirt also runs on the lab cluster, and its virt-handler removed instances it had not created, so the run had its DisableMDEVConfiguration feature gate on (reverted afterwards).

CaseLast resultDateNotes
vGPU: inventorypass2026-10-12Single node (kernel 7.0.0-34, QEMU 10.2.1, k3s 1.35): FluxVM /v1/host/mdev listed mbochs-small 4 MiB/64 free, medium 16/16, large 64/4; kairon-node published the same in NodeTopology mediatedDevices
vGPU: healthpass2026-10-12Same node: FluxVM judged the onboard VGA 0000:04:00.0 healthy and NodeTopology carried Healthy on the device
vGPU: bootpass2026-10-12Same node: a Machine asking for profile: mbochs-small was Running in 76s with allocation mbochs/UUID; FluxVM vfio_devices held the UUID, QEMU had sysfsdev=/sys/bus/mdev/devices/<uuid>, and the guest saw one display function with a 4 MiB BAR0
vGPU: accountingpass2026-10-12Same node: a second Machine asking for 4 mbochs-large stayed Pending with TopologyUnsatisfiable: needs 4 gpu profile mbochs-large, 3 free; kairon_vgpu_available_instances{mbochs-small} 63, kairon_gpu_vram_mib allocated 4 of 256
vGPU: stoppass2026-10-12Same node: stopping removed the instance; mbochs-small back to 64 free, mbochs-large 4
vGPU: startpass2026-10-12Same node: starting re-created the same UUID, QEMU held it and the guest saw the 4 MiB vGPU again
vGPU: deletepass2026-10-12Same node: deleting the Machine removed the instance; mbochs-small 64 free
vGPU: NVIDIA vGPU profiles on a real GPUnot run2026-10-12Needs an NVIDIA GPU with the vGPU host driver
vGPU: MIG-backed profiles, nvidia-smi health and VRAMnot run2026-10-12Needs an NVIDIA MIG-capable GPU
GPU health: Unhealthy GPU skipped by the schedulernot run2026-10-12Needs a GPU with AER or ECC errors; unit-tested

Accelerator profiles​

Driven by scripts/lab-accel.sh on the same node and mbochs driver as the GPU and vGPU rows: four 64 MiB mbochs-large instances fit in its 256 MiB pool. Host-installed kairon-controller from this change, kairon-node and FluxVM. KubeVirt's DisableMDEVConfiguration feature gate was on for the run and reverted afterwards. An earlier run without it failed: virt-handler removed one of a running Machine's two instances from sysfs. The guest probe then never printed, and a later run counted the missing instance as still pending, so the profile reported capacity 0 instead of 1.

CaseLast resultDateNotes
Accelerator profile: profile-statuspass2026-10-12Single node (kernel 7.0.0-34, QEMU 10.2.1, k3s 1.35): pair profile (2 mbochs-large, p2pRequired, sameNuma) Ready=True, Available=True, capacity 2 on the node; migration.required profile Ready=False/LiveMigrationUnsupported; 1Gi-per-device profile Available=False/NoCapacity: needs 1 gpu profile mbochs-large with 1Gi, 0 free
Accelerator profile: crd-validationpass2026-10-12Same node: class accelerator with memory.minimumPerDevice refused with 422
Accelerator profile: expand-bootpass2026-10-12Same node: a Machine with accelerators: [{name: vga, profile: <pair>}] got annotation vga=<pair>@1 and one spec.devices entry (2 mbochs-large, sameNuma + samePcieRoot), two instances on mbochs, Running in 78s; the guest saw 2 display functions with a 64 MiB BAR0
Accelerator profile: capacitypass2026-10-12Same node: the pair profile then had capacity 1 and 1 Machine, in status and in kairon_accelerator_profile_capacity / _machines
Accelerator profile: unavailablepass2026-10-12Same node: Machines naming a missing profile and the live-migration profile stayed Pending with AcceleratorProfileUnavailable and no node
Accelerator profile: unsatisfiablepass2026-10-12Same node: a Machine with the 1Gi profile was expanded and stayed Pending with TopologyUnsatisfiable: ... needs 1 gpu profile mbochs-large with 1Gi, 0 free
Accelerator profile: snapshotpass2026-10-12Same node: changing the pair profile's count from 2 to 1 (generation 2) left the running Machine's spec.devices, annotation and both instances as they were
Accelerator profile: deletepass2026-10-12Same node: deleting the Machine removed both instances; the profile (now count 1) reported capacity 4 and 0 Machines
Accelerator profile: whole GPUs with p2pRequired under one PCIe rootnot run2026-10-12Needs two allowlisted GPUs; unit-tested
Accelerator profile: RDMA NIC next to the GPUsnot run2026-10-12Needs a GPU and a NIC under one PCIe root; unit-tested
Accelerator profile: sameNuma on a multi-NUMA hostnot run2026-10-12Lab host has one NUMA node
Accelerator profile: NVIDIA vGPU or MIG profilenot run2026-10-12Needs an NVIDIA GPU

Cross-cluster DR​

Driven by scripts/lab-dr.sh on one node against a host-installed kairon-controller (with --dr-secret-namespace and --dr-test-bridges=drtest0), kairon-node and FluxVM. The two sites are two namespaces, dr-a and dr-b, on the same k3s cluster: each one's peer is the same API server, reached with a kairon-dr-peer ServiceAccount token. Both share one bucket on versitygw 1.8.0 (POSIX backend). Site loss is simulated by unlabelling the peer credentials Secret, not by a network partition. Site A runs db and web (web depends on db) on a VirtualNetwork app (10.77.0.0/24, same name and CIDR on both sites); web is created first, so A's addresses (web .2, db .3) are the reverse of what B would allocate in failover order. Each guest fsyncs a generation and timestamp every 2s, so a restored guest reports the instant its disk returns to. The plans set network.preserveIP and DNS records for both Machines; the lab installs a minimal external-dns DNSEndpoint CRD and runs no external-dns.

CaseLast resultDateNotes
DR: replicatepass2026-10-12Single node (k3s 1.35, QEMU 10.2.1), policy every 5m: dr-b imported 2 restore points 65s after the plan was created; RPO 49s, RPOMet=True
DR: network readiness and DNS on the active sitepass2026-10-12Same node: dr-a published DNSEndpoint dr-shop (db.shop.drlab.example=10.77.0.3, web.shop.drlab.example=10.77.0.2), dr-b none; dr-b's mirrored specs pinned db=10.77.0.3, web=10.77.0.2 and reported NetworkReady=True
DR: testpass2026-10-12Same node: kaironctl dr test started copies of db and web on the isolated bridge drtest0 in 194s; their last writes were within 1s of the restore points; dr-a kept running; dr test-end removed the copies
DR: scheduled testpass2026-10-12Same node, test.every: 5m after a manual test: dr-b started a scheduled test by itself, created and restored both copies (Running in 173s), ended it and removed the copies by itself; lastTest.succeeded=True, LastTestSucceeded=True
DR: consolidatepass2026-10-12Same node: after a policy synthetic full of web, the next replication was an incremental on it and retention made no further synthetic full
DR: failoverpass2026-10-12Same node, sites cut off: db then web Running on dr-b, RTO 208s (objective 10m), data loss 63s; restored disks within 3s of the restore points; dr-a kept running while cut off and was fenced 7s after reconnecting (Machines Halted, policy deleted, epoch 2)
DR: preserve VirtualNetwork addressespass2026-10-12Same node: db and web came up on dr-b with dr-a's addresses (db 10.77.0.3, web 10.77.0.2) although dr-b created db first
DR: DNS follows the active sitepass2026-10-12Same node: after the failover the DNSEndpoint was on dr-b only and the fenced dr-a had deleted its own; after the failback it was on dr-a only again
DR: failbackpass2026-10-12Same node: planned failover back to dr-a in 164s; dr-b halted and took final backups, db's last write on dr-b (generation 378) was on dr-a, data loss 0s; dr-b standby at epoch 3
DR: two real clustersnot run2026-10-12Needs a second cluster; the lab's sites share one API server
DR: separate or replicated buckets per sitenot run2026-10-12Both sites used one bucket
DR: real network partition between sitesnot run2026-10-12Site loss was simulated by unlabelling the peer Secret
DR: external-dns writing to a real zonenot run2026-10-12No external-dns ran; only the DNSEndpoint objects were checked
DR: floating IP moving between clustersnot run2026-10-12Floating IPs are unique per cluster, so the one-cluster lab cannot carry one over; needs two clusters with BGP

Mac Fabric​

Driven by scripts/lab-mac-fabric.sh on the one-node Linux lab cluster with a host-installed kairon-controller. The lab has no Macs. The script creates three Mac Nodes through the API, with the labels, annotations, vm-only taint and Ready heartbeat kairon-node publishes:

  • mac-a: M4 Max, macOS 26, Rosetta, nested virtualization, Thunderbolt fabric lab.
  • mac-b: M3, macOS 15, Rosetta, nested virtualization, fabric lab.
  • mac-c: M1, macOS 14, no Rosetta, no Thunderbolt.

No agent runs for them, so nothing boots. The overlay cases run macnode.OverlaySync, the code kairon-node runs on a Mac, once for each simulated Mac, writing into temporary directories.

CaseLast resultDateNotes
Mac Fabric: capability schedulingpass2026-10-12k3s 1.35: minChipGeneration: 4 placed on mac-a; M3+, macOS 15+ and nested placed on mac-b; minChipGeneration: 5 and Rosetta pinned to mac-c stayed Pending with the per-node reason
Mac Fabric: backend isolationpass2026-10-12Same cluster: the API rejected spec.apple on qemu (422); a vz Machine pinned to the Linux node stayed Pending ("the vz backend needs a Mac"); a qemu Machine pinned to a Mac stayed Pending ("a Mac runs only the vz backend", or the vm-only taint without a toleration)
Mac Fabric: MacNetwork allocationpass2026-10-12Same cluster: a fabric network got 10.89.192.0/24, port 48192, mac-a .2-.31 and mac-b .32-.61 over their Thunderbolt addresses; a plain one got 10.89.193.0/24, port 48193 and all three Macs over InternalIP, 20 hosts each
Mac Fabric: membership schedulingpass2026-10-12Same cluster: a Machine on the fabric network was placed on a member; pinned to mac-c it stayed Pending ("not a member of MacNetwork fab"); a missing network stayed Pending ("not found")
Mac Fabric: overlay filespass2026-10-12Same cluster: the three Macs' overlay.json had the right subnet, slice, port and the other members as peers; key files matched between Macs and were 0600; an entry and key Kairon did not write were kept
Mac Fabric: key rotationpass2026-10-12Same cluster: spec.keyEpoch: 1 gave status keyEpoch 1 and previousKeyEpoch 0; both members' key files listed the new key first and the unchanged epoch-0 key second
Mac Fabric: a Mac leavingpass2026-10-12Same cluster: removing mac-b's fabric label left mac-a the only member with the same slice and subnet; mac-b's entry and key were removed. This run found "peers": null for a lone member, which FluxVM's parser rejects; fixed to []
Mac Fabric: kaironctl macpass2026-10-12Same cluster: mac nodes listed chip, cores, macOS, Rosetta, nested, Thunderbolt and fabric; mac network showed subnet, port and members
Mac Fabric: vz Machines booting on several Macsnot run2026-10-12Needs Macs
Mac Fabric: Rosetta and nested virtualization inside a guestnot run2026-10-12Needs an M3 or later Mac with macOS 15 or later
Mac Fabric: MacNetwork traffic between Macs over Thunderboltnot run2026-10-12Needs two cabled Macs running FluxVM
Mac Fabric: capability and Thunderbolt discovery on real Macsnot run2026-10-12discover_darwin.go is vetted for darwin/arm64 but has not run on a Mac since this change

Component versions (fill per run)​

PieceVersion
Kairon
Kubernetes
FluxVM
Cilium
CSI / storage