Status
Release status and production gaps formerly maintained in the root README.
v0.10.4 is the current release (security and reliability hardening patch, see RELEASE_NOTES.md); v0.10.3 fixed install and upgrade and added hardening; v0.10.2 was a patch on v0.10.1 that fixes the release pipeline; v0.10.1 added the MCP server image and the MCP Registry listing, and v0.10.0 added HTTPS by default for the dashboard and the console relay, kaironctl selftest, virtctl parity in kaironctl, and experimental agent sessions, agent sandboxes, Mac Fabric, cross-cluster DR, DRS on measured pressure, per-disk I/O QoS, accelerator and vGPU profiles, node maintenance, NVMe-oF, incremental S3 backups, topology-aware placement, elastic memory, virtual networks, hibernate, adaptive live migration and incident bundles; v0.9.1 was a patch on v0.9.0: all-namespaces Migrations and Snapshots, kaironctl doctor that no longer fails binary/systemd installs, Machine phase Events, complete SHA256SUMS; v0.9.0 added the kaironctl lifecycle and diagnostics commands, dashboard namespace selector, all-namespaces Machines view, bulk power actions and sign-in fixes; v0.8.0 promoted kairon.zyvor.dev to v1 and shipped the redesigned dashboard; v0.7.0 was never published, v0.7.1 rebuilt it on Go 1.27.2, v0.7.2 updated golang.org/x/net) and open source (native macOS node, developer ecosystem kit, enforced VM edge, MCP approvals, tenant fence, preemption, MachineImage catalog, experimental fleet automation; see RELEASE_NOTES.md). v0.6.0 added (production foundations: status skip-patch, node-scoped watches, capacity scheduling, production Helm profile, release artifacts, expanded CI). Prior v0.5.0 absorbed, absorbing everything the sections above describe: admission webhooks, dashboard password management, console TLS automation, Helm/CI hardening, MachineSet/instance types/Windows/NUMA parity, pause/resume/halt, VM-state snapshot/restore, a full second wave of FluxVM route wrapping (both guest-exec channels, guest file access, sandboxes/templates/warm pools/image catalog, runtime/network diagnostics), the MachineSnapshotSchedule CRD, kaironctl top, and a hardening pass aimed at untrusted multi-tenant traffic (opt-in namespace-scoped kairon-ui authorization, opt-in network default-deny) -- see RELEASE_NOTES.md for the full per-release changelog. Cold relocation, snapshots, DRA bridging, the secure live control plane, and a real FluxVM migration adapter are all real and tested — real two-host live migration has not yet been exercised against real hardware in this repository's own CI (see Operability).
Pre-v0.10.3 hardening evidence (2026-11-10): a fresh install of v0.10.2 with FluxVM 0.4.18 passed kaironctl selftest --profile standard (6 passed, 2 skipped), and the hardening changes on main were built and tested together; what was and was not verified is itemised in TEST_REPORT.md. Experimental features keep their status below.
The published v0.10.3-rc2 was installed from the chart with default image tags on a real host (see the report). The final v0.10.3 tag is the same code apart from dependency bumps and version files, and has not itself been installed on a real host. v0.10.4, v0.10.5 and v0.10.6 have had unit tests and CI only (v0.10.6's offline commands were also run on a Linux dev host): no install, upgrade or self-test of any of them on a cluster has been recorded. After the tag, its images, OCI chart and release assets were checked for existence and consistency (see TEST_REPORT.md); that is not a run.
Lab coverage of v0.10.0: each experimental feature's lab rows are in COMPATIBILITY.md; agent sessions and Mac Fabric have not run on a real Mac. Verified on the single-node lab: boot, power, network, vCPU hotplug, snapshot-restore. Cold migration passed on the two-host lab on 2026-10-14 and 2026-10-15 (see below). FluxVM on the lab host lacks the fork and disk-hotplug endpoints (those checks skip) and fails NIC hotplug with a QMP Duplicate ID error, which is tracked upstream in FluxVM.
Still open after v0.9
v0.7.0 shipped without the multi-host live-migration claim. Still open:
-
Green Zyvor lab matrix — cold, source-failure, failover and adaptive cases passed on a two-host lab on 2026-10-14 (both directions, matched QEMU 10.2.1; see
COMPATIBILITY.md). Still open: theNeedsRecoverydrill passed 5 of 6 runs and failed once unexplained (the recovery action left the migration inAdoptingand the MachineUnknown; seeCOMPATIBILITY.md), live + eBPF, and plain pre-copy under heavy write load (did not converge in 600 s). Thekairon-labrunner has been online since 2026-10-16; the first green nightly is pending. Workflow:.github/workflows/hardware-migration.yml; needs the self-hostedkairon-labrunner and theKAIRON_HW_*/KAIRON_KUBE_*secrets. An earlier run the same day (80.79.5.173↔175.110.122.71, shared/srv/kairon-sharedover NFS) passed smoke and cold migration both ways but could not run live rows: FluxVM 0.3.0 on 80 rejectsstorage: shared. With FluxVM 0.4.8 on both hosts the live rows ran (see above); the single-host systemd smoke alone does not unlock the multi-host live claim. On 2026-10-15 the same pair, re-keyed with a new migration CA and one adapter per node, passed smoke (health only), cold, live in both directions (six live migrations, 64–262 ms downtime, data plane encrypted), source failure, controller failover and the NeedsRecovery drill, one run each on an idle guest; seeCOMPATIBILITY.md. Follow-up runs the same day: repeated live migration is not yet reliable (13 of 15 succeeded; the FluxVM receiver race behind most failures is fixed upstream but not deployed to the lab), a dead-node drill passed (restart on the other node 13 s after fencing), CA rotation passed, and a ramped scale smoke held 8 small Machines (the cap, not a limit); seeCOMPATIBILITY.md. -
macOS beyond one Mac — not verified: macOS guests, live migration, CSI, the VNC relay, kairon-ui, multi-Mac clusters, Intel Macs (
macos.md). -
Fleet automation is experimental and opt-in (
guides/enterprise-fleet.md). -
Adaptive live migration (
MigrationPolicystrategy: adaptive) is experimental until the lab matrix's adaptive cases (live-adaptive-converge,live-postcopy-switch,live-zstd,postcopy-source-failure) run green on the nightly workflow; they passed once by hand on a two-host lab on 2026-10-14. The engine, post-copy safety and adapter contract are unit- and integration-tested only (guides/migration-policies.md). -
Incremental S3 backups (
MachineBackupspec.target.s3,MachineBackupPolicy) are experimental until thescripts/lab-cbt-backup.shrows inCOMPATIBILITY.mdpass. FluxVM's backup engine is tested against real QEMU; Kairon's controller, agent and policy planner are unit-tested only (guides/machine-backup-incremental.md). -
Topology-aware placement (
spec.topology,spec.devices,NodeTopology) is experimental until the multi-NUMA, GPU+NIC and SR-IOV rows inCOMPATIBILITY.mdrun; only single-NUMA hosts have run in the lab. NUMA memory binding is enforced by QEMU, vCPU thread pinning is best-effort (guides/machine-topology.md). -
Elastic memory (
spec.resources.elasticMemory, thekairon.zyvor.dev/memory-overcommit-rationode annotation,KaironMemoryPressure) is experimental until thescripts/lab-elastic-memory.shrows inCOMPATIBILITY.mdpass. QEMU only; reclaim is balloon-only and NUMA accounting still counts full memory (guides/machine-elastic-memory.md). -
Virtual networks (
VirtualNetwork,spec.network.virtualNetwork, kairon-node--bgp-*) are experimental until the cross-host rows inCOMPATIBILITY.mdrun on real nodes; cross-host Geneve forwarding has only run between netns-simulated hosts. IPv4 only, no in-network load balancer, no edge policy on vnet Machines, announce-only BGP (guides/virtual-networks.md). -
Hibernate (
spec.powerState: Hibernated) is experimental until the cross-node row inCOMPATIBILITY.mdruns on two real hosts; same-node hibernate, resume and discard have run. QEMU only; cross-node resume needs a shared hibernation directory and an Atlas RBD root disk (guides/machine-hibernate.md). -
Incident bundles (
kaironctl diagnose machine --deep, kairon-ui/api/v1/machines/{ns}/{name}/diagnostics) are experimental until the packet-capture row inCOMPATIBILITY.mdruns on a Machine on the eBPF dataplane (the 2026-10-14 attempt was blocked by FluxVM: 175's host dataplane is pinned tociliumwithout a Cilium socket, and 80's FluxVM 0.3.0 has no/network/edge); the bundle, redaction, memory dump and non-admin refusal have run. Redaction is pattern-based; QMP state and memory dumps are QEMU only (guides/machine-incident-bundle.md). -
Node maintenance mode and DRS (
kaironctl node maintenance,kairon.zyvor.dev/maintenance,-drs-mode) are experimental until the two-node rows inCOMPATIBILITY.mdrun; on one node the cordon, the wait, stop and hibernate fallbacks, exit and DRS recommend mode have run. On the two-host lab (2026-10-14)node maintenance enterlive-migrated both Machines away. No DRS move has run on two real nodes yet. The chart's controller role (patchon nodes) was checked on 2026-10-14 against a real API server. DRS balances reserved CPU and memory, not measured demand (guides/node-maintenance-drs.md). -
vGPU profiles and GPU health (
spec.devices[].profile,NodeTopologymediatedDevicesand devicehealth) are experimental until the NVIDIA rows inCOMPATIBILITY.mdrun; the end-to-end lab used the kernel'smbochssample mdev driver, not an NVIDIA GPU. MIG partitioning is left to NVIDIA tooling, and a GPU turningUnhealthyunder a running Machine is only reported (guides/gpu-vgpu.md). -
Cross-cluster DR (
MachineProtectionPlan,kaironctl dr) is experimental until the two-cluster rows inCOMPATIBILITY.mdrun; the lab ran both sites as two namespaces of one cluster with one S3 gateway. It builds on incremental S3 backups (item 5), protects image-booted Machines only, and does not stop the old site's Machines while it is cut off (guides/cross-cluster-dr.md). Kept addresses cover VirtualNetwork Machines only, DNS moves through external-dns (Kairon writes aDNSEndpoint, not provider records), and a floating IP moves only if both sites route it. -
Mac Fabric (
spec.apple,MacNetwork,kaironctl mac, kairon-node--thunderbolt-fabricand--mac-fabric-secret-file) is experimental until the real-Mac rows inCOMPATIBILITY.mdrun; the lab has no Macs, so scheduling, network allocation and the overlay files ran against simulated Mac Nodes and no vz Machine has booted on more than one Mac. The overlay is FluxVM's encrypted UDP mesh, not Thunderbolt RDMA (guides/mac-fabric.md). -
Accelerator profiles (
MachineAcceleratorProfile,spec.accelerators) are experimental until the whole-GPU, RDMA NIC and multi-NUMA rows inCOMPATIBILITY.mdrun; the lab used the kernel'smbochssample vGPU driver on one single-NUMA node. Peer to peer is approximated by a shared PCIe root (no NVLink or switch detection), there is no driver or firmware matching, and capacity counts devices only (guides/accelerator-profiles.md). -
Agent sandboxes (
spec.lifecycle, pausedMachinePools,MachineClaimstatus.usage,ui.auth.sandboxOperatorExec) are experimental until the rows inCOMPATIBILITY.mdrun on the lab. Idle detection counts guest CPU, disk and relay sessions only, can act up to a minute late, and a Stop idle action discards a disk not onspec.volumes(guides/agent-sandboxes.md). -
Agent sessions (
AgentSession,kaironctl sessionincluding approvals, chunked uploads,shell,probe, credential grants and the Mac screen andengine(which need a FluxVM newer than 0.4.8), thesession_*MCP tools, the SDKAgentSessionclasses,ui.agentSessions.previews) are experimental until the rows inCOMPATIBILITY.mdrun on a Mac. FluxVM 0.4.8 creates agent sessions on macOS only, so only Mac nodes advertise them, and the lab has no Macs; kairon-node, the relays and the CLI ran against a fake FluxVM. Pinned and forked sessions count againstMachineQuotabut are not held back by it (guides/agent-sessions.md).
Out of band (not a code gate): CII Best Practices badge still needs human OAuth (Scorecard / issue tracker).
Production gaps
Closed in the v0.6 foundations track (agent status path): kairon-node no longer patches Machine status every reconcile tick. Conditions use shared helpers that preserve LastTransitionTime unless Status/Reason/Message actually change; observedGeneration is set on meaningful writes; volatile ResourceUsage alone does not force an etcd write (live samples go to kairon_machine_resource_usage Prometheus gauges). kaironctl top / UI node-usage therefore reflect the last meaningful status write.
Closed in the v0.6 foundations track (node-scoped lists): each kairon-node lists Machines with kairon.zyvor.dev/assigned-node=<thisNode> and migrations with kairon.zyvor.dev/migration-source-node=<thisNode>, driven by watches with a periodic safety resync (default 5m), instead of listing every Machine/migration in the cluster every few seconds. The controller stamps and repairs those labels whenever spec.nodeName / status.sourceNode change.
Closed in the v0.6 foundations track (capacity-aware scheduling): the scheduler hard-filters on node status.allocatable CPU/memory (and hugepages when requested), scores by remaining capacity percentage instead of equal VM counts, and the controller reserves capacity within a reconcile tick so simultaneous placements cannot overcommit one node. Memory overcommit is opt-in per node (kairon.zyvor.dev/memory-overcommit-ratio) and counts only elastic-memory Machines' guaranteed minimum (guides/machine-elastic-memory.md). Unschedulable outcomes set a Scheduled=False Condition with a structured reason.
Closed in the v0.6 foundations track (packaging): Helm image tags default to .Chart.AppVersion (not latest); charts/kairon/values-production.yaml enables webhook, namespace isolation, network default-deny, and migration dataplane TLS. Release workflow publishes CLI binaries, checksums, OCI Helm chart, and a GitHub Release. Hardware migration evidence is tracked in docs/COMPATIBILITY.md and driven by .github/workflows/hardware-migration.yml.
Still genuinely open, and why:
- Cilium cluster-network attach is opt-in and not a Pod CNI.
network.ciliumAttach/network.ciliumPolicySyncdefault to off so bare-metal clusters without Cilium are unchanged. ExternalWorkload identity and IPv4 stay empty until Cilium's own agent registers the workload — Kairon projects status, it does not program BPF. Multus NAD is still not a primary attach path. There is no embedded Hubble UI;kaironctl network status --flowsis a pass-through to the existing uiapi whenKAIRON_UI_URLis set. Seedocs/network-fabric.md. - Fencing detection is dual-signal when opted in, still not exhaustive. With
node.livenessLease.enabled,kairon-controllerfeeds each node's owncoordination.k8s.io/v1Lease intoNodeUnreachabledetection (Ready + stale agent lease →AgentLivenessStale) in addition to Node-Ready, andkaironctl fence --liveness-lease-namespace ...still cross-checks before clearing runtime state. Lease lookup errors fail open (do not inventAgentLivenessStale). Opt-in--stale-evacuationautomates the fence only for Machines annotatedkairon.zyvor.dev/evacuate=trueon a node an operator or power-fencing tool attested dead (kaironctl node fence); Kairon still cannot tell a dead node from a partitioned one by itself, so the attestation is trusted as is. Migration preflight can only catch a confirmed storage/network mismatch when both nodes are labeled withkairon.zyvor.dev/storage-domain/network-domain— it can't prove compatibility when the labels are unset. Real multi-host fencing/preflight behavior hasn't been exercised against real hardware in this repo's own CI. Seedocs/guides/machine-fencing.md. topologySpreadConstraints.maxSkewhard enforcement is opt-in per constraint (whenUnsatisfiable: DoNotSchedule) — omitted orScheduleAnyway(the default) stays scoring-only, as before. ADoNotScheduledomain is only counted among currently-eligible candidate nodes, not every node cluster-wide, and — like affinity/anti-affinity — is evaluated against the reconcile-time Machine list, not a live watch, so a same-tick scheduling race can transiently see stale domain counts (resolved on the next tick). Seedocs/guides/machine-placement.md.maxSkewbelow 1 withDoNotScheduleis rejected at admission (create, and update when placement changed).- DRA topology-awareness is a scoring hint unless the Machine asks for hard topology —
kairon-controllerhas no role in DRA device allocation itself. Underspec.topology.policy: restrictedorsingle-numa-nodean allocated claim pins the Machine to the claim's node and NUMA node, and an unallocated claim keeps it Pending (DRAAllocationPending); otherwise it stays a small score bonus. Seedocs/guides/machine-topology.mdanddocs/guides/machine-placement.md. - Confidential-compute enforcement (SEV-SNP/TDX), large-scale hardware qualification — hardware-dependent, not exercisable in CI.
- PVC-backed boot disks are a first cut: one boot volume per Machine.
hostPath/localsources resolve directly; a network-block volume now works too, through Kairon's own first-cut CSI driver (csiNode.enabled, iSCSI or NVMe-oF) — dynamic provisioning, CHAP (iSCSI only; controller path always; node-side consumption via opt-innode.csi.chap.enabledallowlisted to the release namespace), volume expansion, and volume snapshots are all supported (csiController.enabled/.snapshotter.enabled). No other backends (Ceph/EBS/etc.) yet — seedocs/guides/machine-storage-csi.md. - NVMe-oF and raw Block volumes are Experimental.
nvme-tcp/nvme-rdmathrough the kernel target with native multipath, Block-mode boot in place and Block data disks with per-disk QEMU I/O (virtio-blk, vhost-user-blk, io_uring, iothreads, queues), and aStoragePathscondition. All of it ran on one node that was both storage node and Machine host, with rdma only over soft-RoCE. Subsystems allow any host (no DH-CHAP), and nothing has run in-cluster or across two nodes yet — seedocs/guides/machine-storage-nvmeof.md. A PV naming any other CSI driver is still refused unless listed innode.thirdPartyCSIDrivers. MachineSnapshotRestorerestores into a new PVC only, deliberately not also a Machine (see its guide for why), and needs a real CSI snapshotter behind your StorageClass — Rancher'slocal-path-provisioner, a common default, doesn't have one. Creating the restore before itsMachineSnapshotfinishes is not an error:status.phaseparks atPendingand retries automatically once the snapshot (and its underlyingVolumeSnapshot) becomes ready — this used to instead land in a permanent, non-retryingFailedrequiring a delete-and-recreate, an inconsistency with how every other "waiting on an external condition" status in this same feature (e.g. a PVC stillPendingunderWaitForFirstConsumer) was already handled correctly.- CPU/memory hotplug is grow-only (FluxVM has no CPU/DIMM unplug), bounded by headroom reserved at creation (
spec.resources.maxCpu/.maxMemory, itself fixed once set — not something a running Machine can grow), and lost across any stop/start — expected QEMU behavior, not a bug. kairon-uimulti-replica state propagation is eventually-consistent, not instant:ui.replicaCount > 1is now supported (session revocation, login lockout, console tickets, and password changes propagate via a sharedConfigMap, deliberately not Redis), but cross-replica visibility lands within ~15s, login-lockout's failure count is per-replica not cluster-wide-atomic, and concurrent password changes to two different accounts on two different replicas can still race. Seedocs/guides/kairon-ui-ha.md.kairon-controllerleader-election takeover is coarse, not instant:controller.replicaCount > 1is now safe (acoordination.k8s.io/v1Lease, on by default, ensures only one replica ever reconciles), but a crashed or partitioned leader costs up to ~15s before another replica takes over and reconciliation resumes. Seedocs/guides/kairon-controller-ha.md.- Distributed tracing is opt-in and narrow. Every component exposes reconcile-loop and apiserver-call health metrics.
kairon_reconcile_item_errors_total{kind}counts per-item failures. Opt-inotel.enabled(Helm) /KAIRON_OTEL_ENDPOINTposts OTLP/HTTP JSON reconcile spans fromkairon-controllerandkairon-nodevia a tiny exporter (internal/oteltrace), not the full OpenTelemetry Go SDK — enough to see which Machine/migration timed out without a third large stdlib exception. Seedocs/guides/observability.md. - Rate limiting covers
kairon-uionly.kairon-uinow rate-limits every route per client address (-rate-limit-rps/-rate-limit-burst, on by default), on top of the existing per-username login lockout — keyed by the direct TCP peer by default, butui.rateLimit.trustedProxyHeader/.trustedProxyCidrs(opt-in, both empty by default) let it key by a forwarded-for header instead, once trusted, so aService/Ingress/load balancer no longer collapses every real client into one bucket. The header is only ever trusted from a direct peer matchingtrustedProxyCidrs— set to the load balancer's own address range — otherwise any client could spoof it to evade throttling or collide with someone else's bucket. Since v0.10.4 the client address is the rightmost header entry that is not insidetrustedProxyCidrs(the leftmost, used before, was client-controlled), sotrustedProxyCidrsmust list every proxy hop; this has not been run behind a real proxy (guides/kairon-ui-dashboard.md). The admission webhook and inter-component RPCs (migration control-plane mTLS, the console relay) deliberately have no rate limiting added here: the webhook backstops real Kubernetes writes underfailurePolicy: Fail, so throttling it risks rejecting a legitimate bulkkubectl apply; the RPCs are already restricted to mTLS/bearer-token-authenticated peers, not open to arbitrary traffic. See SECURITY.md. - Every
kairon-uiJSON request body is now size-capped. Nothing inkairon-ui's HTTP stack ever bounded request body size before this — a caller, even a legitimately authenticated one, could send an arbitrarily large body and havejson.Decoderbuffer all of it into memory.decodeJSON(internal/uiapi/server.go) now wraps every request body inhttp.MaxBytesReaderat a 1MiB default, comfortably above every ordinary request this API accepts; guest file writes (POST .../agent-file/put) get an explicit 6MiB override, since a real file's content travels base64-encoded (~4/3 expansion) and the read side of the same feature was already capped around 3MB of real content. Rejected with a plain 400, not a broken connection — seedocs/guides/machine-guest-agent-files.md. - Live-migration data-plane identity is opt-in, and still two places to configure at once.
migration.dataplaneTlsSecretName(opt-in) gives every node a real, distinguishable cert for the QEMU RAM/state stream instead of the shared control-plane cert every node could otherwise present there —scripts/gen-migration-mtls-certs.shgenerates both the certs and a ready-to-apply Secret. Left unset (the default), the data plane still falls back to the shared cert, same as before this existed. Either way, actually turning data-plane encryption on also requires each node'skairon-migration-adapter-fluxvmsystemd unit to pass-migration-data-tls=trueindependently — outside Helm's control, easy to half-upgrade a fleet. The control-plane cert (migration.tlsSecretName) stays deliberately shared regardless — it proves cluster membership, not host identity, by design. Seedocs/guides/machine-migration-tls.md. - Cluster backup/restore covers Kairon's Kubernetes-level state; disk content is per Machine.
scripts/backup-crds.sh/restore-crds.sh(seedocs/runbook-backup-restore.md) save thekairon.zyvor.devCRDs and, opt-in, the chart-managed Secrets, not VM disks or FluxVM's per-host runtime state. Disk content is covered per Machine byMachineBackup: on the node (guides/machine-backup.md), or to S3 with incremental chains that restore on any node (guides/machine-backup-incremental.md, experimental). PVC-backed disks remain your CSI driver's or storage backend's job. Not yet drilled against a real full cluster-loss scenario in this repo's own CI. - OIDC/SSO group-to-admin mapping is opt-in and login-time-fresh, not IdP-live:
ui.oidc.adminGroups(unset by default, matching prior behavior exactly) grants admin capability to an OIDC session whose ID token carries one of the configured groups, re-checked against current Kairon config on every request — but the group membership itself only reflects the IdP's state as of the caller's last login, not a live check, so removing someone from an IdP group doesn't revoke their admin session early; it takes effect at their next sign-in. Seedocs/guides/kairon-ui-oidc.md. - Every
kairon.zyvor.devCRD servesv1(storage),v1beta1andv1alpha1with an identical schema, so no conversion webhook is live; the conversion webhook scaffold exists for the next real schema change:kairon-controller's webhook server exposes a testedPOST /convert/machinequotas(internal/conversion), proven end to end against a real workedMachineQuotafield-rename example — on the same TLS listener/certificate/Serviceas the existing admission webhook, no new trust boundary. Cutting a real schema-changing version still needs a CRD manifest change and a real converter for whatever's actually changing, plus solving one still-open gap:charts/kairon/crds/*.yamllives in Helm's special, never-templatedcrds/directory, so wiring a livecaBundleinto a CRD'sspec.conversionneeds a deliberate choice (move that CRD intotemplates/, or a separate patch step) not yet made. Seedocs/guides/crd-versioning.md. - The admission webhook's
MachineDisruptionBudgetand Machine-CREATE-MachineQuotachecks only ever evaluateCREATE, matching the reconcile-loop/kaironctlchecks they backstop exactly.MachineQuotaalso evaluatesUPDATE(a hotplug resize past quota) at the webhook, but only whenwebhook.enabledis true. Independent of that:kairon-nodenow also self-checks a hotplug resize againstMachineQuotaitself before actually growing a running Machine's live vCPUs/memory — reusing the exact same admission logic as the webhook (internal/controller.QuotaTrackersForNamespace/AdmitQuotaResize) — so a resize past quota is caught even on a deployment that has never turned the webhook on. This self-check is effectively on by default (no new Helm flag) but fails open: any lookup error, most commonly akairon-nodeServiceAccount that hasn't yet been granted the new read-onlymachinequotasRBAC (a binary upgraded ahead of its chart), causes it to log a warning and proceed with the resize exactly as before this existed, rather than newly blocking on account of infrastructure the check itself depends on. - The VNC console inherits FluxVM's own unauthenticated VNC socket as-is — Kairon can't add auth/encryption FluxVM itself doesn't have. Its real security rests on kairon-ui's operator auth, a single-use per-session ticket now bound to the one Machine it was issued for, an opt-in per-Machine allowlist (
kairon.zyvor.dev/console-allowed-users, unset means unchanged all-operator access), and the shared cluster-wide token to kairon-node (optionally now over TLS). See SECURITY.md. kaironctlcovers virtctl's verbs, with FluxVM's limits.console,vnc,restart,soft-reboot,stop --grace-period/--force,export create|list|download|delete,memory-dump,guest osinfo|fslist|userlist,credentials,expose,create --dry-run,expand,wait,open,start --paused,--dry-runon power verbs,freeze/unfreeze,vnc screenshot,objectgraph,permitted-devices,adm log-verbosity,create clone,guestfs,cdromandexport manifestexist; the mapping is in CLI.resetisrestart: FluxVM has no reset, and its stop is already a hard power-off. A graceful stop needs the guest agent.exposewrites an EndpointSlice with the guest IP, so it only reaches guests whose IP is routable from the cluster's nodes, and an IP change shows up on kairon-controller's next tick.credentialsedits Linux guests only (authorized_keys,chpasswd), andmemory-dumpneeds the qemu backend. FluxVM cannot insert media into a created VM, socdrom insertneeds a Stopped Machine;create clonecopies only the boot disk;vnc screenshotneeds a display without a VNC password;guestfsneeds libguestfs on the node.usbrediris not available: FluxVM has no USB redirection.- The sandbox HTTP proxy is still API-only. The other runtime features (Halted, both guest-exec channels, sandboxes/templates/warm pools, the image catalog, egress check, runtime and network diagnostics) now have dashboard controls (the machine inspector's operations panel, including incident bundles, and the node tools panel) and CLI verbs (
kaironctl guest ...,kaironctl runtime ...,kaironctl diagnose machine). kairon-uinow has a dashboard page forMachineQuota/MachineDisruptionBudget/MachineSet/MachineInstanceType/MigrationPolicy.GET /api/v1/quotas/disruption-budgets/machinesets/instancetypes/migration-policieswrap the same read-onlyList*Namespacecallskaironctl getalready uses — any-authenticated-operator, matching every other read-only cross-checkable resource — and each now has a matching list page (web/src/pages/{Quotas,DisruptionBudgets,MachineSets,InstanceTypes,MigrationPolicies}.tsx), closing what was previously a real visibility gap: an operator using only the dashboard had no way to see quota usage or disruption-budget state at all, both first-class admission-time concerns (see Guarding the fleet). Four of the five stay list-only — no create/update/delete route or dashboard form;kaironctl/kubectlremain how they get mutated.MachineSetis the one exception:DELETE /api/v1/machinesets/{namespace}/{name},PATCH /api/v1/machinesets/{namespace}/{name}/scale, and matching "Delete"/"Scale" row actions now let an operator remove or resize one straight from the dashboard, the same any-authenticated-operator gatehandleDeleteMachinealready has (no separate admin check). Picked over the other four because it's the one an operator manages as a day-to-day fleet-sizing operation (kaironctl create/scale/edit machinesetall shipped this same session) rather than a GitOps-managed policy object or a mostly-static reference value. Deleting a MachineSet from the dashboard also cascades to delete every Machine it created — see Deleting a MachineSet deletes its replicas too — so the dashboard's confirm prompt says so explicitly.kairon-uinow has a dashboard page for the real KubernetesNode, not just a kairon.zyvor.dev CRD. Node taints started actually gating scheduling, thenkaironctl get/describe nodegave an operator a way to see them from the CLI (see Guarding the fleet) -- butGET /api/v1/nodes(added earlier for the Overview tile's plain node count) had no dashboard page rendering the list itself until now. The new Nodes page shows the same columnskaironctl get nodesprints -- name,Readycondition,spec.unschedulable, akey[=value]:Effecttaint summary, and addresses -- read-only likekaironctl's own support, since Node's create/delete lifecycle stayskubectl's job either way. No new REST route, no new RBAC: the route and the ClusterRole grant it needs both already existed for the Overview tile. Seedocs/guides/machine-placement.md's "Taints and tolerations" section.- Fixed a real gap:
kairon-ui's ownClusterRolenever actually grantedget/list/watchonMachineQuota/MachineDisruptionBudget/MachineInstanceType/MigrationPolicy, or the extra verbsMachineSet's delete andMachineSnapshotSchedule's suspend/resume route need (delete,patch) — every one of those dashboard/REST routes above has 403'd against a real Kubernetes apiserver since the day it shipped. Nothing caught it because everyinternal/uiapitest runs its handlers against a fakehttptest.Serverdouble, which never enforces RBAC at all — a real cluster was the only way this would surface.charts/kairon/templates/all.yaml'skairon-uiClusterRolenow grants exactly the verbs each route actually needs, no more. kairon-uiandkaironctlnow also coverMachineNetworkPolicy/NetworkSecurityGroup— the two CRDs that drive FluxVM's real eBPF/TC network enforcement (see the network-policy guide) had zero human-facing tooling beyond rawkubectl, unlike every other kind in this project: nokaironctl get/describe/delete, and no dashboard/REST route at all (the existing network-observability endpoints only ever show a Machine's effective, already-merged policy, not the policy objects that produced it).kaironctl get networkpolicies/securitygroups,describe, anddeletenow work like every other kind;GET /api/v1/network-policies/security-groupsback two new list-only dashboard pages (Network policies, Security groups). The dashboard stays list-only;kaironctl create networkpolicy/create securitygroupandkaironctl edit networkpolicy/edit securitygroup(see CLI) have since closed the other half of this gap —kaironctl/kubectl(not the dashboard) are how these get created or edited.- A warm-pool claim only becomes a Kairon-managed
Machinewhen the caller opts in. By default, claiming hands back FluxVM's own VM record directly with no automaticMachineobject — a claimed VM sits outside admission-time guards (MachineQuota, the webhook) until one exists.POST .../claim'screateMachine: true(plus a requiredmachineName) closes that gap for a given claim: it forces the FluxVM-side VM name to matchMachine.RuntimeName()and creates the Machine object from the pool's own template, so kairon-node's existing adoption path picks up the exact runtime on its next reconcile tick. The Machine carries the template's resources and hotplug headroom, kernel, cloud-init, NUMA/cpuset/hugepages, secure boot/TPM, guest agents and TTL, and is pinned to the claimed VM's node; VFIO devices, the raw network map, shared folders, data disks and storage overrides are not carried and are reported asmachineWarnings. Seedocs/guides/machine-sandboxes.md's "Warm pools" section. - Real CPU pinning (
spec.resources.cpuPinning) needs an accuratekairon.zyvor.dev/pinnable-cpusnode label. An operator can set it, or kairon-node can discover it: online CPUs minus--reserved-cpusminus kubelet's exclusive pod CPUs fromcpu_manager_state. Discovery is off until--reserved-cpusis set. kubelet's state file is0600 root, sodeploy-remote.sh --reserved-cpus=LISTinstalls a path-triggered copier unit and Helmnode.cpuPinning.enabledadds a copier sidecar; either way the copy can lag kubelet by a moment. An operator label is never overwritten, so it can still drift from kubelet's real allocations. Seedocs/guides/machine-cpu-pinning.md. - The third-party CSI client (
node.thirdPartyCSIDrivers) is a first cut, not a general "any CSI driver" integration.attachRequireddrivers work through aVolumeAttachment(the driver's external-attacher must be running) and node secrets resolve from one allowlisted namespace, but no live third-party driver has been run end to end yet; behavior is unit-tested against fakes. Seedocs/guides/machine-storage-thirdparty-csi.md. kairon-uinow has opt-in per-namespace authorization (ui.auth.namespaceScoping.enabled, defaultfalse) — closes a real gap:kairon-ui's ownClusterRoleis cluster-scoped and its HTTP layer previously had no namespace authorization axis at all, onlyisAdminIdentity's admin-vs-not split, so every authenticated operator (password, legacy shared token, or OIDC) could see and mutate every namespace's Machines/quotas/policies regardless of intent. Once enabled, a non-admin session-token operator is restricted to the namespaces in their ownui.auth.users[].namespacesor reachable via a matchingui.oidc.namespaceGroupsentry — everything else 403s. A request with no named identity (the legacy sharedui.token) is refused with a 403, the overview is filtered to the caller's grants, and node routes need an administrator. Off by default: flipping it on for an existing deployment with nonamespaces/namespaceGroupsconfigured moves every non-admin operator from "sees every namespace" to "sees none," a deliberate fail-closed default, not a silent no-op. Honest first-cut limits: the dashboard's namespace inputs stayed free-text in this first cut (v0.9.0 added a nav namespace selector fed byGET /api/v1/namespaces), a scoped-out operator gets the existing generic 403 rather than a constrained dropdown;GET /api/v1/overviewstill aggregates cluster-wide regardless;POST /api/v1/migrations/evacuate(a node-shaped bulk action spanning whatever namespaces that node's Machines happen to live in) isn't scoped by it either; and Node-scoped routes aren't namespace-scoped at all, since Node isn't a namespaced kairon object. The legacy shared token has no per-caller identity to scope by, so it stays fully unrestricted regardless of this setting, by design.Machine.spec.tenantis a network fence, not a tenancy boundary. Withkairon.zyvor.dev/tenant-fence=true, kairon denies other tenants' observed guest addresses in the same namespace (seedocs/guides/tenant-fence.md). It is not a VRF: unobserved addresses and other namespaces are not denied. The rename/clear check needs the admission webhook. Kubernetes Namespace is still the quota and RBAC boundary (MachineQuota,MachineDisruptionBudget, and the namespace-scoped authorization above).kubectl explain machine.spec.tenantandkaironctl describe machinesay so.- Preemption only halts. An opted-in pending Machine (
kairon.zyvor.dev/preempt=true) halts at most one opted-in lower-priority Machine (kairon.zyvor.dev/preemption-policy=Halt) per tick in the same namespace; there is no eviction, migration, or cross-namespace preemption. RejectingPauseas a policy needs the admission webhook. Seedocs/guides/preemption.md. - Hotplug after a Halted resume is best effort. Resume from Halted boots at FluxVM's kept create-time size, and kairon-node hotplugs back to the pre-halt size right after. If headroom or the guest refuses, the Machine runs smaller and
status.applied*says so. Seedocs/guides/machine-hotplug.md. - MCP approval trusts whoever can annotate. Destructive MCP tools wait for
kaironctl approve, which is an annotation write; anyone withpatchon the Machine or MachineBackup can approve. The approver name is self-reported (KAIRON_APPROVERor the OS user). Seedocs/guides/hermes-mcp.md.
Report vulnerabilities privately to security@zyvor.dev — see SECURITY.md.