Skip to main content

Verified on real hardware · 15 October 2026

What the lab has actually run.

Two real hosts, real QEMU, a re-keyed migration CA and a shared NFS root. Everything below was run; what was not run is listed beside it.

One run per case, one Machine, an idle guest. This shows the control plane, certificates and failure handling work on this pair. It says nothing about convergence under load.

Results

Eight cases, eight passes.

Seven are the migration matrix on the two-host lab; the eighth is a container sandbox on a Mac.

CaseWhat was doneResultRead more
SmokeMachine hw-live checked Running and Ready. Health only: no create or delete.passGuide ›
Cold migrationnldw3-4-32-26 → nldw4-4-04-32, about 38 s.passGuide ›
Live migration, nldw4 → nldw3Four runs: 4.1–4.4 s transfer, 148–242 ms downtime, data plane encrypted.passGuide ›
Live migration, nldw3 → nldw4Two runs: 4.2–4.6 s transfer, 64–262 ms downtime, data plane encrypted.passGuide ›
Source failure mid-transferkairon-node-lab stopped on the source for 20 s during the transfer, then restarted. Migration Succeeded.passGuide ›
Controller failover mid-migrationkairon-controller restarted while the migration ran. Migration Succeeded.passGuide ›
Ambiguous commit → NeedsRecoveryDestination port blocked until the migration parked in NeedsRecovery; ConfirmDestinationNotCommitted completed the cutover; block removed.passGuide ›
Dead node: restart on the other nodeAgent stopped and VM killed: NodeUnreachable after 51 s, restarted 13 s after fencing attestation. Machines not opted in were left alone. Not a shared-disk split-brain test.passGuide ›
Migration CA rotationNew 10-year CA, mTLS verified both ways, two live migrations afterwards.passGuide ›
Scale smoke: 8 small MachinesRamped in batches of 2, all Running in 97 s, lowest free memory 8.7 GiB, then deleted. 8 was the cap; no limit was found.passGuide ›
Repeated live migration13 of 15 succeeded in early repeat runs; soaks stopped after 8 and 10 clean hops. Causes found and fixed or fixed upstream; the FluxVM fix is not yet deployed to the lab.partialGuide ›
Container sandbox on a Macalpine:3.22 run through FluxVM, then created through Kairon’s client: exit code 3 and console output read back in about 4 s.passGuide ›

Numbers

Six live migrations, all encrypted.

Measured from the MachineMigration objects of that day: a 2 vCPU, 2 GiB guest, idle.

6

live migrations

Four one way, two back. All Succeeded.

262 ms

worst downtime

Best was 64 ms; five of six were 148–262 ms.

4.6 s

slowest transfer

Fastest was 4.1 s, about 0.5 GB of the guest’s 2 GiB.

6

of 6 encrypted

Data plane TLS on the freshly generated CA.

How it was run

The repo’s own scripts, from a workstation.

Nothing here is special to the lab beyond the topology. The matrix script drives the cluster with a short-lived token for a dedicated service account.

Re-key the pair

A new migration CA and per-host certificates; each node gets its own migration adapter.

Share the disk root

One NFS export mounted on both hosts, so a live move transfers memory only.

Run the matrix

Smoke, cold and both live directions, then the three failure cases.

Check nothing is left

No firewall rule, no stuck migration, every node service active.

Not run yet

What this page does not claim.

These stay marked Experimental or unverified until a run is recorded.

  • Live migration under guest load, and the three adaptive cases (auto-converge, post-copy switch, zstd): they need a dashboard URL and token to generate load.
  • Guest-agent exec after migration, and post-copy source failure (needs a disposable Machine).
  • Live migration with the eBPF dataplane: no eBPF Machine in this lab.
  • The create, stop, start, delete smoke with an image: only health was checked.
  • Container sandboxes through kairon-node and the controller on a Mac, Rosetta images, published ports, restart policy and health checks. Only the FluxVM call path was run.
  • More than 8 Machines, migration under many-Machine load, a shared-disk split-brain, and anything with a GPU.
  • Reliable repeated live migration: the receiver race fix (FluxVM #417) is merged but not yet running in the lab.

Read status and production gaps

Read the record.

Dates, hosts and caveats for every row.