Verified on real hardware · 15 October 2026
What the lab has actually run.
Two real hosts, real QEMU, a re-keyed migration CA and a shared NFS root. Everything below was run; what was not run is listed beside it.
One run per case, one Machine, an idle guest. This shows the control plane, certificates and failure handling work on this pair. It says nothing about convergence under load.
Results
Eight cases, eight passes.
Seven are the migration matrix on the two-host lab; the eighth is a container sandbox on a Mac.
| Case | What was done | Result | Read more |
|---|---|---|---|
| Smoke | Machine hw-live checked Running and Ready. Health only: no create or delete. | pass | Guide › |
| Cold migration | nldw3-4-32-26 → nldw4-4-04-32, about 38 s. | pass | Guide › |
| Live migration, nldw4 → nldw3 | Four runs: 4.1–4.4 s transfer, 148–242 ms downtime, data plane encrypted. | pass | Guide › |
| Live migration, nldw3 → nldw4 | Two runs: 4.2–4.6 s transfer, 64–262 ms downtime, data plane encrypted. | pass | Guide › |
| Source failure mid-transfer | kairon-node-lab stopped on the source for 20 s during the transfer, then restarted. Migration Succeeded. | pass | Guide › |
| Controller failover mid-migration | kairon-controller restarted while the migration ran. Migration Succeeded. | pass | Guide › |
| Ambiguous commit → NeedsRecovery | Destination port blocked until the migration parked in NeedsRecovery; ConfirmDestinationNotCommitted completed the cutover; block removed. | pass | Guide › |
| Dead node: restart on the other node | Agent stopped and VM killed: NodeUnreachable after 51 s, restarted 13 s after fencing attestation. Machines not opted in were left alone. Not a shared-disk split-brain test. | pass | Guide › |
| Migration CA rotation | New 10-year CA, mTLS verified both ways, two live migrations afterwards. | pass | Guide › |
| Scale smoke: 8 small Machines | Ramped in batches of 2, all Running in 97 s, lowest free memory 8.7 GiB, then deleted. 8 was the cap; no limit was found. | pass | Guide › |
| Repeated live migration | 13 of 15 succeeded in early repeat runs; soaks stopped after 8 and 10 clean hops. Causes found and fixed or fixed upstream; the FluxVM fix is not yet deployed to the lab. | partial | Guide › |
| Container sandbox on a Mac | alpine:3.22 run through FluxVM, then created through Kairon’s client: exit code 3 and console output read back in about 4 s. | pass | Guide › |
Numbers
Six live migrations, all encrypted.
Measured from the MachineMigration objects of that day: a 2 vCPU, 2 GiB guest, idle.
live migrations
Four one way, two back. All Succeeded.
worst downtime
Best was 64 ms; five of six were 148–262 ms.
slowest transfer
Fastest was 4.1 s, about 0.5 GB of the guest’s 2 GiB.
of 6 encrypted
Data plane TLS on the freshly generated CA.
How it was run
The repo’s own scripts, from a workstation.
Nothing here is special to the lab beyond the topology. The matrix script drives the cluster with a short-lived token for a dedicated service account.
Re-key the pair
A new migration CA and per-host certificates; each node gets its own migration adapter.
Share the disk root
One NFS export mounted on both hosts, so a live move transfers memory only.
Run the matrix
Smoke, cold and both live directions, then the three failure cases.
Check nothing is left
No firewall rule, no stuck migration, every node service active.
Not run yet
What this page does not claim.
These stay marked Experimental or unverified until a run is recorded.
- Live migration under guest load, and the three adaptive cases (auto-converge, post-copy switch, zstd): they need a dashboard URL and token to generate load.
- Guest-agent exec after migration, and post-copy source failure (needs a disposable Machine).
- Live migration with the eBPF dataplane: no eBPF Machine in this lab.
- The create, stop, start, delete smoke with an image: only health was checked.
- Container sandboxes through kairon-node and the controller on a Mac, Rosetta images, published ports, restart policy and health checks. Only the FluxVM call path was run.
- More than 8 Machines, migration under many-Machine load, a shared-disk split-brain, and anything with a GPU.
- Reliable repeated live migration: the receiver race fix (FluxVM #417) is merged but not yet running in the lab.
Read the record.
Dates, hosts and caveats for every row.