Skip to main content

What ships today

Full feature inventory. The root README is the short product landing; guides linked below remain authoritative for how-to detail.

Machine lifecycle & placement

  • Machine CRD — CPU, memory, image, network, power state, volumes, DRA device claims
  • MachineSet — a Deployment/ReplicaSet-shaped fleet of identical Machines from one template, with RollingUpdate/Recreate rollout — guide
  • MachineInstanceType — a reusable named CPU/memory shape (spec.instanceTypeName), resolved into spec.resources once instead of inlining it every time — guide
  • NUMA topology, CPU set, hugepages (spec.resources.numaNode/.cpuSet/.hugepages, qemu-only) — direct passthroughs to FluxVM's own existing support, refused with a clear error on a non-qemu backend rather than silently ignored — guide
  • Real CPU pinning (spec.resources.cpuPinning, qemu-only) — exclusive host-core allocation, a real scheduler capacity model against operator-asserted kairon.zyvor.dev/pinnable-cpus node labels — atomically patched alongside node assignment, applied via the same cgroup cpuset.cpus write spec.resources.limits already uses — guide
  • Windows guests — legacy-BIOS Windows Server/10 with a pre-built, virtio-driver + cloudbase-init image works today (spec.cloudInit unchanged); spec.security.secureBoot/.tpm (Windows 11) now pass through to FluxVM's own real OVMF Secure Boot (qemu-only) and emulated TPM 2.0 (qemu or cloud-hypervisor) — Kairon enforces the same backend restriction FluxVM's own scheduler does, refusing an unsupported combination with a clear error rather than relaying a bare FluxVM 400. A real Secure Boot chain still needs the FluxVM node itself configured with an OVMF vars template (an operator responsibility, not something Kairon manages) — guide
  • MigrationPolicy — scopes migration bandwidth/concurrency to Machines matching a selector, independent of the global migration.maxConcurrentPerNode/maxConcurrentCluster caps — guide
  • PVC-backed boot disk — spec.volumes[0] resolves through a Bound PersistentVolumeClaim to a real host directory (hostPath/local PersistentVolumes directly, or a network-block volume via Kairon's own first-cut CSI driver) instead of a hand-placed image file — guide · CSI guide
  • Multi-volume (first cut) — spec.volumes[1+] attach as FluxVM virtiofs shared folders (QEMU); boot remains volumes[0] / spec.image
  • CSI node CHAP (node.csi.chap.enabled, opt-in) — resolve a PV's nodeStageSecretRef for iSCSI login, Secrets allowlisted to the release namespace only — CSI guide
  • Third-party CSI client (node.thirdPartyCSIDrivers, first cut) — kairon-node acts as its own generic CSI client against an operator-allowlisted third-party driver's socket (validated conceptually against Ceph-CSI/RBD), including attachRequired drivers via VolumeAttachment and node-stage/publish secrets from one allowlisted namespace — guide
  • Atlas-provisioned volumes (atlas.enabled, first cut) — spec.volumes[].atlas asks the Atlas storage control plane for a disk by size and policy; the controller creates it, waits on its job, writes the PVC into claimName, schedules only once Ready, and deletes it after node runtime cleanup. atlas.mode: rbd boots the Atlas image in place through FluxVM ceph-rbd-in-place (node pool allowlist, image name bound to the Machine) — guide
  • Image import (spec.image.source, opt-in via node.imageCacheDir) — kairon-node downloads a remote qcow2/raw image URL into its own digest-keyed cache instead of requiring a hand-placed file, shared across every Machine naming the same digest — guide
  • Image catalog (spec.image.catalogName, admin-only for mutations, API-only) — reference a named, checksummed (optionally signed) FluxVM image-catalog alias instead of a raw path; node-scoped register/rename/clone/export/read-only API — guide
  • Egress check (admin-only, API-only) — ask a node whether a sandbox's outbound request to a host would be allowed under its static egress allowlist, with any matched credential-vault secret reported only as a boolean — guide
  • Warm pools (admin-only, API-only) — create a FluxVM pool of pre-booted, Paused VMs for instant claiming instead of a cold create; node-scoped create/list/get/delete/claim API — guide
  • Runtime diagnostics (freeze/thaw admin-only, rest any-operator, API-only) — a node's real FluxVM capability manifest, a Machine's cgroup-derived PSI pressure stats and effective host CPU set, and a kernel-level cgroup freeze/thaw distinct from spec.powerState: Paused — guide
  • Network observability (any-operator) — a Machine's fully-resolved effective network policy, real drop-reason/flow/stats data straight from FluxVM's eBPF dataplane — kaironctl network flows|drop-reasons|stats|effective, dashboard Network panel, and the uiapi relays — guide ("Troubleshooting" section)
  • eBPF dataplane (recommended) — spec.network.dataplaneMode: ebpf (production profile / docs default recommendation); Kairon declares, FluxVM owns TC/eBPF — reference
  • VM edge — spec.network.antiSpoof / learnIP / qos plus a policy's allowSNI / allowDNS, enforced by FluxVM's TC program (dataplane schema 12); policy allow lists also turn the edge on for a plain tap Machine; netns Machines get a generated MAC; attributed drops via kaironctl network drops and kairon_net_drops_total; packet capture with pcap download (kaironctl network capture --output); conntrack moved on live migration — reference
  • AI agents over MCP — kaironctl mcp serve: Model Context Protocol tools for Hermes Agent and other MCP clients (Machines, policies, VM-edge network data; power, snapshot and capture only with --allow-write) — guide
  • Placement — least-loaded scheduling across Ready, capable-labeled nodes, deterministic tie-break, required (hard) spec.placement.affinity/antiAffinity, weighted soft scoring (preferredAffinity/preferredAntiAffinity, topologySpreadConstraints), a best-effort DRA topology-awareness hint, and topologySpreadConstraints.maxSkew hard enforcement via whenUnsatisfiable: DoNotSchedule — guide
  • Scheduling priority (spec.priority, first cut) — when a reconcile tick can't fit every still-pending Machine (not enough eligible nodes, or a MachineQuota at its cap), higher-priority Machines are attempted first; purely an admission-order signal for that tick's pending Machines, never preemption of an already-scheduled one however low its own priority — kaironctl create/create machineset --priority N, kaironctl edit machine NAME --priority N — guide
  • DRA → VFIO — an allocated ResourceClaim's PCI BDF against the node's vfio_devices administrator allowlist, fail-closed (an empty allowlist denies all passthrough); device-type-agnostic, so this also covers SR-IOV NIC passthrough once a VF is bound to vfio-pci on the host — the same mechanism GPU passthrough already uses, not a Multus integration (Machines have no Pod for Multus to attach to) — guide
  • CPU/memory hotplug — grow a running Machine's spec.resources via FluxVM's real QMP device_add/object-add, no reboot — guide
  • Pause and resume (spec.powerState: Paused) — suspend a running Machine's guest CPUs via FluxVM's real QMP stop/cont (backend-agnostic), RAM and device state fully resident, Kairon's virtctl pause/unpause equivalent — kaironctl pause/resume, or the dashboard — guide
  • Halt (spec.powerState: Halted) — power a Machine's VMM process off while FluxVM keeps its own VM record/disk intact, freeing node capacity like Stopped but resuming via FluxVM's own kept config instead of a full Kairon-side recreate — kaironctl halt, or the dashboard — guide
  • Live host resource limits (spec.resources.limits) — a real, kernel-enforced cgroup v2 cap (CPU quota %, memory ceiling, I/O weight, PID count) on a running Machine's own VMM process, freely raisable/lowerable at any time, backend-agnostic — guide
  • Live resource usage (status.resourceUsage) — real, cgroup-derived CPU/memory/disk-I/O usage refreshed every reconcile tick, shown inline in the dashboard next to each Machine — guide
  • kaironctl top machines/top nodes — a kubectl top-style live usage view over that same status.resourceUsage data, no new metrics pipeline: top machines prints each Machine's own CPU%/memory/disk-I/O, top nodes rolls those same per-Machine samples up by spec.nodeName (model.AggregateUsageByNode) into a per-host hotspot view kaironctl get machines alone can't answer without manually grouping rows. The same rollup backs the dashboard's Nodes page (GET /api/v1/nodes/usage) below — guide
  • Guest agent (spec.guestAgent) — real qemu-guest-agent-reported status.guestIP, including for user/SLIRP networking, which has no DHCP lease to parse at all — guide
  • Guest exec (spec.guestAgent.enabled, admin-only) — run a one-shot command inside the guest via kairon-ui's dashboard, no SSH/console needed — real qemu-guest-agent guest-exec, synchronous, exit code + stdout/stderr — guide
  • Text console (spec.guestAgent.console) — an interactive shell in-browser (xterm.js), Kairon's virtctl console equivalent, working on every backend — requires FluxVM's own proprietary fluxvm-guest-agent baked into the guest image — guide
  • Guest file access (spec.guestAgent.console, admin-only) — read or write a file inside a Running guest at runtime, no SSH/shared volume needed — same fluxvm-guest-agent channel as the text console — guide
  • Backend-agnostic guest exec (spec.guestAgent.console, admin-only, API-only) — run a command over FluxVM's own vsock agent instead of qemu-guest-agent — the only guest-exec path that works on Cloud Hypervisor/Firecracker/FluxVm-backend sandboxes, not just QEMU — guide
  • Sandboxes (spec.sandbox, admin-only, API-only) — run a Machine on FluxVM's own lightweight, fast-boot in-tree hypervisor track for short-lived ephemeral workloads, optionally from a pre-built template; includes an HTTP proxy relay straight into the guest's own web server — guide
  • Machine logs — kubectl logs/-f equivalent: the VM's real captured serial console output, with a live-tailing Follow toggle, same any-operator authorization as the VNC console — guide
  • VM-state snapshot/restore (admin-only, API-only) — a full hypervisor-level checkpoint of a running Machine's RAM/CPU/device state via FluxVM's real savevm/Cloud Hypervisor snapshot, restored back into the same Machine in place; distinct from MachineSnapshot's disk-content-only CSI snapshot — guide

Migration

  • Cold migrate & evacuate — stop → reassign → restart, restart-safe in the API
  • Secure live handshake — TLS 1.3 mTLS prepare → transfer → commit, never a user-supplied tcp: URI
  • Split-brain guards — rollback on transfer failure, NeedsRecovery on ambiguous commit, adopt-only cutover
  • A real migration adapter ships — cmd/kairon-migration-adapter-fluxvm against real FluxVM endpoints, not just a test double
  • Node fencing is detection + operator-attested action, never a guess — kairon-controller flags a Machine's node NodeUnreachable every reconcile tick (Node-Ready, and when node.livenessLease.enabled is on, Ready + stale agent Lease → AgentLivenessStale) but never reschedules it itself (that risks running the same VM twice if the node isn't actually dead); kaironctl fence --reason ... is the explicit, attested step that clears it for rescheduling — same shape as NeedsRecovery. The same opt-in Lease also lets fence refuse when kairon-node still looks alive despite NodeUnreachable=True. Opt-in kairon.zyvor.dev/storage-domain/network-domain node labels also let migration preflight block a target when it's confirmed to not share storage/network with the source — guide

Guarding the fleet

  • MachineDisruptionBudget — kaironctl evacuate throttles itself against minAvailable/maxUnavailable instead of taking a whole node's Machines at once; kairon-controller now reconciles real status every tick too (kaironctl get budgets/kubectl get mdb), purely observational. kaironctl create budget NAME --selector k=v (--min-available X | --max-unavailable X)/edit budget NAME [--selector k=v] [--min-available X] [--max-unavailable X] create and mutate one directly — no more dropping to kubectl apply/YAML just to stand one up — guide
  • Cordon-triggered automatic evacuation (controller.cordonEvacuation.enabled, opt-in) — kairon-controller itself migrates every Machine off a Node the moment it's cordoned, budget-throttled exactly like kaironctl evacuate, mirroring KubeVirt's LiveMigrateIfPossible — guide
  • MachineQuota — a namespace-scoped maxMachines/maxTotalCpu/maxTotalMemory cap. kaironctl create quota NAME [--max-machines N] [--max-total-cpu N] [--max-total-memory SIZE]/edit quota NAME [flags] create and mutate one directly — guide
  • Admission webhook (webhook.enabled, opt-in) — both of the above can now be enforced at admission, not just in the reconcile loop or kaironctl: a Machine create that would blow a quota, or a MachineMigration create that would violate a budget, gets rejected outright instead of just parked Pending or silently allowed. See Guarding the fleet below.
  • Network Fabric — spec.network, MachineNetworkPolicy, NetworkSecurityGroup, Service Fabric VIP membership → FluxVM eBPF edge — reference
  • Cilium CNI (opt-in, three paths) — spec.network.dataplaneMode: cilium requests FluxVM's cilium dataplane; spec.network.ciliumAttach (Helm network.ciliumAttach.enabled) reconciles a cluster-scoped CiliumExternalWorkload and writes status.network.cilium plus podUID; MachineNetworkPolicy.spec.cilium.sync (Helm network.ciliumPolicySync.enabled) upserts a namespaced CiliumNetworkPolicy so Hubble sees the same intent. Controller stays free of the Cilium Go SDK (raw REST). kaironctl network status MACHINE prints emoji lines for dataplane, attach, and CNP sync — reference · CLI

Snapshots

  • CSI VolumeSnapshot — MachineSnapshot orchestrates a standard snapshot object; MachineSnapshotRestore restores one into a new PersistentVolumeClaim via the standard CSI dataSource flow (deliberately doesn't also create a Machine — see the guide for why) — guide
  • Guest quiesce for snapshots (spec.guestAgent.enabled) — a real guest-fsfreeze/-thaw around MachineSnapshot's VolumeSnapshot creation for an application-consistent snapshot, coordinated between kairon-controller and kairon-node since only the node has a network path to the Machine's FluxVM instance — guide
  • MachineSnapshotSchedule (first cut) — periodically creates a MachineSnapshot for every Machine matching a label selector on a plain spec.intervalSeconds, Kairon's own backup-automation primitive — deliberately not real cron syntax. spec.keepLast optionally retains only the N most recent ready-to-use snapshots per Machine, pruning older ones the schedule itself created (never a manually-created one). status.nextRunTime projects when a schedule's next round is expected (kaironctl get snapshotschedules' NEXTRUN column, kubectl get machinesnapshotschedules' own NextRun printer column, and the dashboard's Next run column all show it) — a simple as-of-last-fire projection, not a live countdown; kaironctl/the dashboard both check the live spec.suspend flag before trusting it, so a schedule suspended after its last run never shows a stale already-passed timestamp. The dashboard's own Snapshot schedules page lists every schedule and can suspend/resume one with a click — the one write action exposed there so far; spec.selector/.intervalSeconds/.keepLast still need kaironctl/kubectl to edit. kaironctl describe snapshotschedule NAME previews exactly which Machines its selector currently matches and whether the next reconcile tick would actually fire a new round of snapshots for them, evaluated with the same Spec.Due check the controller itself uses — a real "what would this do right now" question worth answering before loosening/tightening a selector or interval, without waiting for the next tick to find out. spec.startingDeadlineSeconds (opt-in, unset by default) is Kairon's analog of Kubernetes CronJob's field of the same name: a run discovered more than that many seconds late — e.g. after kairon-controller was down, or this CRD's status was reset by a reinstall — is skipped rather than fired as a stale catch-up, though the schedule's own clock still advances so it doesn't re-detect the same missed window forever; status.lastRunError records the skip, surfaced the same way any other run error already is (kaironctl get/describe, the dashboard's Last error column). kaironctl trigger snapshotschedule NAME requests an immediate, out-of-band run — "back up these Machines right now" without waiting for intervalSeconds to elapse, or temporarily touching suspend/intervalSeconds to force one — implemented as a durable request annotation (kairon.zyvor.dev/trigger-now) the controller notices and acts on its very next reconcile tick, bypassing both spec.suspend and spec.startingDeadlineSeconds (a paused schedule can still be asked for one snapshot without permanently unpausing it, and "right now" is never "too late"); status.lastHandledTriggerTime records which request was already handled so the same one is never re-fired — guide

Operate it

  • kaironctl — Cobra CLI with colored tables and emoji install progress; install/upgrade/uninstall use the embedded Helm chart (no separate helm binary). kubectl kairon is the same tree (Krew manifest in deploy/krew/). network status covers Cilium attach/CNP sync. Full reference: CLI
  • kairon-ui — an optional web dashboard, real per-operator login with in-place password change/reset (not "regenerate a hash and redeploy"), optional OIDC/SSO, a NeedsRecovery recovery workflow, an optional VNC console, and more than one replica once you need it — see The dashboard
  • kairon-controller HA — controller.replicaCount can go above 1: a coordination.k8s.io/v1 Lease (internal/leaderelection, on by default) coordinates replicas so only the elected leader reconciles, while the admission webhook and health/metrics endpoints keep serving from every replica regardless — see guide
  • Prometheus metrics + alert rules, a per-node/cluster migration concurrency quota, status.dataPlaneEncrypted visibility into whether a live transfer is actually encrypted
  • Opt-in OTel reconcile spans (otel.enabled / KAIRON_OTEL_ENDPOINT) — tiny OTLP/HTTP JSON exporter on controller and node item loops, not the full OpenTelemetry SDK — guide
  • PodDisruptionBudget on by default, opt-in NetworkPolicy, digest-pinned + Trivy-scanned + cosign-signed + SBOM'd container images, Helm + raw manifests + CI

Live memory transfer needs a node-local migration adapter deployed and configured on each node — without one, a live request blocks before the source is touched. Cold migration needs nothing extra and works today.

Snapshots & devices (examples)​

spec:
  volumes:
    - {name: data, claimName: database-data}
kaironctl snapshot database --name before-upgrade --class csi-snapclass
spec:
  deviceClaims:
    - name: gpu-claim   # administrator allowlist on the node -- empty allowlist denies all

More worked examples: examples/.