Skip to main content

Status

Release status and production gaps formerly maintained in the root README.

v0.10.4 is the current release (security and reliability hardening patch, see RELEASE_NOTES.md); v0.10.3 fixed install and upgrade and added hardening; v0.10.2 was a patch on v0.10.1 that fixes the release pipeline; v0.10.1 added the MCP server image and the MCP Registry listing, and v0.10.0 added HTTPS by default for the dashboard and the console relay, kaironctl selftest, virtctl parity in kaironctl, and experimental agent sessions, agent sandboxes, Mac Fabric, cross-cluster DR, DRS on measured pressure, per-disk I/O QoS, accelerator and vGPU profiles, node maintenance, NVMe-oF, incremental S3 backups, topology-aware placement, elastic memory, virtual networks, hibernate, adaptive live migration and incident bundles; v0.9.1 was a patch on v0.9.0: all-namespaces Migrations and Snapshots, kaironctl doctor that no longer fails binary/systemd installs, Machine phase Events, complete SHA256SUMS; v0.9.0 added the kaironctl lifecycle and diagnostics commands, dashboard namespace selector, all-namespaces Machines view, bulk power actions and sign-in fixes; v0.8.0 promoted kairon.zyvor.dev to v1 and shipped the redesigned dashboard; v0.7.0 was never published, v0.7.1 rebuilt it on Go 1.27.2, v0.7.2 updated golang.org/x/net) and open source (native macOS node, developer ecosystem kit, enforced VM edge, MCP approvals, tenant fence, preemption, MachineImage catalog, experimental fleet automation; see RELEASE_NOTES.md). v0.6.0 added (production foundations: status skip-patch, node-scoped watches, capacity scheduling, production Helm profile, release artifacts, expanded CI). Prior v0.5.0 absorbed, absorbing everything the sections above describe: admission webhooks, dashboard password management, console TLS automation, Helm/CI hardening, MachineSet/instance types/Windows/NUMA parity, pause/resume/halt, VM-state snapshot/restore, a full second wave of FluxVM route wrapping (both guest-exec channels, guest file access, sandboxes/templates/warm pools/image catalog, runtime/network diagnostics), the MachineSnapshotSchedule CRD, kaironctl top, and a hardening pass aimed at untrusted multi-tenant traffic (opt-in namespace-scoped kairon-ui authorization, opt-in network default-deny) -- see RELEASE_NOTES.md for the full per-release changelog. Cold relocation, snapshots, DRA bridging, the secure live control plane, and a real FluxVM migration adapter are all real and tested — real two-host live migration has not yet been exercised against real hardware in this repository's own CI (see Operability).

Pre-v0.10.3 hardening evidence (2026-11-10): a fresh install of v0.10.2 with FluxVM 0.4.18 passed kaironctl selftest --profile standard (6 passed, 2 skipped), and the hardening changes on main were built and tested together; what was and was not verified is itemised in TEST_REPORT.md. Experimental features keep their status below.

The published v0.10.3-rc2 was installed from the chart with default image tags on a real host (see the report). The final v0.10.3 tag is the same code apart from dependency bumps and version files, and has not itself been installed on a real host. v0.10.4, v0.10.5 and v0.10.6 have had unit tests and CI only (v0.10.6's offline commands were also run on a Linux dev host): no install, upgrade or self-test of any of them on a cluster has been recorded. After the tag, its images, OCI chart and release assets were checked for existence and consistency (see TEST_REPORT.md); that is not a run.

Lab coverage of v0.10.0: each experimental feature's lab rows are in COMPATIBILITY.md; agent sessions and Mac Fabric have not run on a real Mac. Verified on the single-node lab: boot, power, network, vCPU hotplug, snapshot-restore. Cold migration passed on the two-host lab on 2026-10-14 and 2026-10-15 (see below). FluxVM on the lab host lacks the fork and disk-hotplug endpoints (those checks skip) and fails NIC hotplug with a QMP Duplicate ID error, which is tracked upstream in FluxVM.

Still open after v0.9​

v0.7.0 shipped without the multi-host live-migration claim. Still open:

  1. Green Zyvor lab matrix — cold, source-failure, failover and adaptive cases passed on a two-host lab on 2026-10-14 (both directions, matched QEMU 10.2.1; see COMPATIBILITY.md). Still open: the NeedsRecovery drill passed 5 of 6 runs and failed once unexplained (the recovery action left the migration in Adopting and the Machine Unknown; see COMPATIBILITY.md), live + eBPF, and plain pre-copy under heavy write load (did not converge in 600 s). The kairon-lab runner has been online since 2026-10-16; the first green nightly is pending. Workflow: .github/workflows/hardware-migration.yml; needs the self-hosted kairon-lab runner and the KAIRON_HW_* / KAIRON_KUBE_* secrets. An earlier run the same day (80.79.5.173 ↔ 175.110.122.71, shared /srv/kairon-shared over NFS) passed smoke and cold migration both ways but could not run live rows: FluxVM 0.3.0 on 80 rejects storage: shared. With FluxVM 0.4.8 on both hosts the live rows ran (see above); the single-host systemd smoke alone does not unlock the multi-host live claim. On 2026-10-15 the same pair, re-keyed with a new migration CA and one adapter per node, passed smoke (health only), cold, live in both directions (six live migrations, 64–262 ms downtime, data plane encrypted), source failure, controller failover and the NeedsRecovery drill, one run each on an idle guest; see COMPATIBILITY.md. Follow-up runs the same day: repeated live migration is not yet reliable (13 of 15 succeeded; the FluxVM receiver race behind most failures is fixed upstream but not deployed to the lab), a dead-node drill passed (restart on the other node 13 s after fencing), CA rotation passed, and a ramped scale smoke held 8 small Machines (the cap, not a limit); see COMPATIBILITY.md.

  2. macOS beyond one Mac — not verified: macOS guests, live migration, CSI, the VNC relay, kairon-ui, multi-Mac clusters, Intel Macs (macos.md).

  3. Fleet automation is experimental and opt-in (guides/enterprise-fleet.md).

  4. Adaptive live migration (MigrationPolicy strategy: adaptive) is experimental until the lab matrix's adaptive cases (live-adaptive-converge, live-postcopy-switch, live-zstd, postcopy-source-failure) run green on the nightly workflow; they passed once by hand on a two-host lab on 2026-10-14. The engine, post-copy safety and adapter contract are unit- and integration-tested only (guides/migration-policies.md).

  5. Incremental S3 backups (MachineBackup spec.target.s3, MachineBackupPolicy) are experimental until the scripts/lab-cbt-backup.sh rows in COMPATIBILITY.md pass. FluxVM's backup engine is tested against real QEMU; Kairon's controller, agent and policy planner are unit-tested only (guides/machine-backup-incremental.md).

  6. Topology-aware placement (spec.topology, spec.devices, NodeTopology) is experimental until the multi-NUMA, GPU+NIC and SR-IOV rows in COMPATIBILITY.md run; only single-NUMA hosts have run in the lab. NUMA memory binding is enforced by QEMU, vCPU thread pinning is best-effort (guides/machine-topology.md).

  7. Elastic memory (spec.resources.elasticMemory, the kairon.zyvor.dev/memory-overcommit-ratio node annotation, KaironMemoryPressure) is experimental until the scripts/lab-elastic-memory.sh rows in COMPATIBILITY.md pass. QEMU only; reclaim is balloon-only and NUMA accounting still counts full memory (guides/machine-elastic-memory.md).

  8. Virtual networks (VirtualNetwork, spec.network.virtualNetwork, kairon-node --bgp-*) are experimental until the cross-host rows in COMPATIBILITY.md run on real nodes; cross-host Geneve forwarding has only run between netns-simulated hosts. IPv4 only, no in-network load balancer, no edge policy on vnet Machines, announce-only BGP (guides/virtual-networks.md).

  9. Hibernate (spec.powerState: Hibernated) is experimental until the cross-node row in COMPATIBILITY.md runs on two real hosts; same-node hibernate, resume and discard have run. QEMU only; cross-node resume needs a shared hibernation directory and an Atlas RBD root disk (guides/machine-hibernate.md).

  10. Incident bundles (kaironctl diagnose machine --deep, kairon-ui /api/v1/machines/{ns}/{name}/diagnostics) are experimental until the packet-capture row in COMPATIBILITY.md runs on a Machine on the eBPF dataplane (the 2026-10-14 attempt was blocked by FluxVM: 175's host dataplane is pinned to cilium without a Cilium socket, and 80's FluxVM 0.3.0 has no /network/edge); the bundle, redaction, memory dump and non-admin refusal have run. Redaction is pattern-based; QMP state and memory dumps are QEMU only (guides/machine-incident-bundle.md).

  11. Node maintenance mode and DRS (kaironctl node maintenance, kairon.zyvor.dev/maintenance, -drs-mode) are experimental until the two-node rows in COMPATIBILITY.md run; on one node the cordon, the wait, stop and hibernate fallbacks, exit and DRS recommend mode have run. On the two-host lab (2026-10-14) node maintenance enter live-migrated both Machines away. No DRS move has run on two real nodes yet. The chart's controller role (patch on nodes) was checked on 2026-10-14 against a real API server. DRS balances reserved CPU and memory, not measured demand (guides/node-maintenance-drs.md).

  12. vGPU profiles and GPU health (spec.devices[].profile, NodeTopology mediatedDevices and device health) are experimental until the NVIDIA rows in COMPATIBILITY.md run; the end-to-end lab used the kernel's mbochs sample mdev driver, not an NVIDIA GPU. MIG partitioning is left to NVIDIA tooling, and a GPU turning Unhealthy under a running Machine is only reported (guides/gpu-vgpu.md).

  13. Cross-cluster DR (MachineProtectionPlan, kaironctl dr) is experimental until the two-cluster rows in COMPATIBILITY.md run; the lab ran both sites as two namespaces of one cluster with one S3 gateway. It builds on incremental S3 backups (item 5), protects image-booted Machines only, and does not stop the old site's Machines while it is cut off (guides/cross-cluster-dr.md). Kept addresses cover VirtualNetwork Machines only, DNS moves through external-dns (Kairon writes a DNSEndpoint, not provider records), and a floating IP moves only if both sites route it.

  14. Mac Fabric (spec.apple, MacNetwork, kaironctl mac, kairon-node --thunderbolt-fabric and --mac-fabric-secret-file) is experimental until the real-Mac rows in COMPATIBILITY.md run; the lab has no Macs, so scheduling, network allocation and the overlay files ran against simulated Mac Nodes and no vz Machine has booted on more than one Mac. The overlay is FluxVM's encrypted UDP mesh, not Thunderbolt RDMA (guides/mac-fabric.md).

  15. Accelerator profiles (MachineAcceleratorProfile, spec.accelerators) are experimental until the whole-GPU, RDMA NIC and multi-NUMA rows in COMPATIBILITY.md run; the lab used the kernel's mbochs sample vGPU driver on one single-NUMA node. Peer to peer is approximated by a shared PCIe root (no NVLink or switch detection), there is no driver or firmware matching, and capacity counts devices only (guides/accelerator-profiles.md).

  16. Agent sandboxes (spec.lifecycle, paused MachinePools, MachineClaim status.usage, ui.auth.sandboxOperatorExec) are experimental until the rows in COMPATIBILITY.md run on the lab. Idle detection counts guest CPU, disk and relay sessions only, can act up to a minute late, and a Stop idle action discards a disk not on spec.volumes (guides/agent-sandboxes.md).

  17. Agent sessions (AgentSession, kaironctl session including approvals, chunked uploads, shell, probe, credential grants and the Mac screen and engine (which need a FluxVM newer than 0.4.8), the session_* MCP tools, the SDK AgentSession classes, ui.agentSessions.previews) are experimental until the rows in COMPATIBILITY.md run on a Mac. FluxVM 0.4.8 creates agent sessions on macOS only, so only Mac nodes advertise them, and the lab has no Macs; kairon-node, the relays and the CLI ran against a fake FluxVM. Pinned and forked sessions count against MachineQuota but are not held back by it (guides/agent-sessions.md).

Out of band (not a code gate): CII Best Practices badge still needs human OAuth (Scorecard / issue tracker).

Production gaps​

Closed in the v0.6 foundations track (agent status path): kairon-node no longer patches Machine status every reconcile tick. Conditions use shared helpers that preserve LastTransitionTime unless Status/Reason/Message actually change; observedGeneration is set on meaningful writes; volatile ResourceUsage alone does not force an etcd write (live samples go to kairon_machine_resource_usage Prometheus gauges). kaironctl top / UI node-usage therefore reflect the last meaningful status write.

Closed in the v0.6 foundations track (node-scoped lists): each kairon-node lists Machines with kairon.zyvor.dev/assigned-node=<thisNode> and migrations with kairon.zyvor.dev/migration-source-node=<thisNode>, driven by watches with a periodic safety resync (default 5m), instead of listing every Machine/migration in the cluster every few seconds. The controller stamps and repairs those labels whenever spec.nodeName / status.sourceNode change.

Closed in the v0.6 foundations track (capacity-aware scheduling): the scheduler hard-filters on node status.allocatable CPU/memory (and hugepages when requested), scores by remaining capacity percentage instead of equal VM counts, and the controller reserves capacity within a reconcile tick so simultaneous placements cannot overcommit one node. Memory overcommit is opt-in per node (kairon.zyvor.dev/memory-overcommit-ratio) and counts only elastic-memory Machines' guaranteed minimum (guides/machine-elastic-memory.md). Unschedulable outcomes set a Scheduled=False Condition with a structured reason.

Closed in the v0.6 foundations track (packaging): Helm image tags default to .Chart.AppVersion (not latest); charts/kairon/values-production.yaml enables webhook, namespace isolation, network default-deny, and migration dataplane TLS. Release workflow publishes CLI binaries, checksums, OCI Helm chart, and a GitHub Release. Hardware migration evidence is tracked in docs/COMPATIBILITY.md and driven by .github/workflows/hardware-migration.yml.

Still genuinely open, and why:

  • Cilium cluster-network attach is opt-in and not a Pod CNI. network.ciliumAttach / network.ciliumPolicySync default to off so bare-metal clusters without Cilium are unchanged. ExternalWorkload identity and IPv4 stay empty until Cilium's own agent registers the workload — Kairon projects status, it does not program BPF. Multus NAD is still not a primary attach path. There is no embedded Hubble UI; kaironctl network status --flows is a pass-through to the existing uiapi when KAIRON_UI_URL is set. See docs/network-fabric.md.
  • Fencing detection is dual-signal when opted in, still not exhaustive. With node.livenessLease.enabled, kairon-controller feeds each node's own coordination.k8s.io/v1 Lease into NodeUnreachable detection (Ready + stale agent lease → AgentLivenessStale) in addition to Node-Ready, and kaironctl fence --liveness-lease-namespace ... still cross-checks before clearing runtime state. Lease lookup errors fail open (do not invent AgentLivenessStale). Opt-in --stale-evacuation automates the fence only for Machines annotated kairon.zyvor.dev/evacuate=true on a node an operator or power-fencing tool attested dead (kaironctl node fence); Kairon still cannot tell a dead node from a partitioned one by itself, so the attestation is trusted as is. Migration preflight can only catch a confirmed storage/network mismatch when both nodes are labeled with kairon.zyvor.dev/storage-domain/network-domain — it can't prove compatibility when the labels are unset. Real multi-host fencing/preflight behavior hasn't been exercised against real hardware in this repo's own CI. See docs/guides/machine-fencing.md.
  • topologySpreadConstraints.maxSkew hard enforcement is opt-in per constraint (whenUnsatisfiable: DoNotSchedule) — omitted or ScheduleAnyway (the default) stays scoring-only, as before. A DoNotSchedule domain is only counted among currently-eligible candidate nodes, not every node cluster-wide, and — like affinity/anti-affinity — is evaluated against the reconcile-time Machine list, not a live watch, so a same-tick scheduling race can transiently see stale domain counts (resolved on the next tick). See docs/guides/machine-placement.md. maxSkew below 1 with DoNotSchedule is rejected at admission (create, and update when placement changed).
  • DRA topology-awareness is a scoring hint unless the Machine asks for hard topology — kairon-controller has no role in DRA device allocation itself. Under spec.topology.policy: restricted or single-numa-node an allocated claim pins the Machine to the claim's node and NUMA node, and an unallocated claim keeps it Pending (DRAAllocationPending); otherwise it stays a small score bonus. See docs/guides/machine-topology.md and docs/guides/machine-placement.md.
  • Confidential-compute enforcement (SEV-SNP/TDX), large-scale hardware qualification — hardware-dependent, not exercisable in CI.
  • PVC-backed boot disks are a first cut: one boot volume per Machine. hostPath/local sources resolve directly; a network-block volume now works too, through Kairon's own first-cut CSI driver (csiNode.enabled, iSCSI or NVMe-oF) — dynamic provisioning, CHAP (iSCSI only; controller path always; node-side consumption via opt-in node.csi.chap.enabled allowlisted to the release namespace), volume expansion, and volume snapshots are all supported (csiController.enabled/.snapshotter.enabled). No other backends (Ceph/EBS/etc.) yet — see docs/guides/machine-storage-csi.md.
  • NVMe-oF and raw Block volumes are Experimental. nvme-tcp/nvme-rdma through the kernel target with native multipath, Block-mode boot in place and Block data disks with per-disk QEMU I/O (virtio-blk, vhost-user-blk, io_uring, iothreads, queues), and a StoragePaths condition. All of it ran on one node that was both storage node and Machine host, with rdma only over soft-RoCE. Subsystems allow any host (no DH-CHAP), and nothing has run in-cluster or across two nodes yet — see docs/guides/machine-storage-nvmeof.md. A PV naming any other CSI driver is still refused unless listed in node.thirdPartyCSIDrivers.
  • MachineSnapshotRestore restores into a new PVC only, deliberately not also a Machine (see its guide for why), and needs a real CSI snapshotter behind your StorageClass — Rancher's local-path-provisioner, a common default, doesn't have one. Creating the restore before its MachineSnapshot finishes is not an error: status.phase parks at Pending and retries automatically once the snapshot (and its underlying VolumeSnapshot) becomes ready — this used to instead land in a permanent, non-retrying Failed requiring a delete-and-recreate, an inconsistency with how every other "waiting on an external condition" status in this same feature (e.g. a PVC still Pending under WaitForFirstConsumer) was already handled correctly.
  • CPU/memory hotplug is grow-only (FluxVM has no CPU/DIMM unplug), bounded by headroom reserved at creation (spec.resources.maxCpu/.maxMemory, itself fixed once set — not something a running Machine can grow), and lost across any stop/start — expected QEMU behavior, not a bug.
  • kairon-ui multi-replica state propagation is eventually-consistent, not instant: ui.replicaCount > 1 is now supported (session revocation, login lockout, console tickets, and password changes propagate via a shared ConfigMap, deliberately not Redis), but cross-replica visibility lands within ~15s, login-lockout's failure count is per-replica not cluster-wide-atomic, and concurrent password changes to two different accounts on two different replicas can still race. See docs/guides/kairon-ui-ha.md.
  • kairon-controller leader-election takeover is coarse, not instant: controller.replicaCount > 1 is now safe (a coordination.k8s.io/v1 Lease, on by default, ensures only one replica ever reconciles), but a crashed or partitioned leader costs up to ~15s before another replica takes over and reconciliation resumes. See docs/guides/kairon-controller-ha.md.
  • Distributed tracing is opt-in and narrow. Every component exposes reconcile-loop and apiserver-call health metrics. kairon_reconcile_item_errors_total{kind} counts per-item failures. Opt-in otel.enabled (Helm) / KAIRON_OTEL_ENDPOINT posts OTLP/HTTP JSON reconcile spans from kairon-controller and kairon-node via a tiny exporter (internal/oteltrace), not the full OpenTelemetry Go SDK — enough to see which Machine/migration timed out without a third large stdlib exception. See docs/guides/observability.md.
  • Rate limiting covers kairon-ui only. kairon-ui now rate-limits every route per client address (-rate-limit-rps/-rate-limit-burst, on by default), on top of the existing per-username login lockout — keyed by the direct TCP peer by default, but ui.rateLimit.trustedProxyHeader/.trustedProxyCidrs (opt-in, both empty by default) let it key by a forwarded-for header instead, once trusted, so a Service/Ingress/load balancer no longer collapses every real client into one bucket. The header is only ever trusted from a direct peer matching trustedProxyCidrs — set to the load balancer's own address range — otherwise any client could spoof it to evade throttling or collide with someone else's bucket. Since v0.10.4 the client address is the rightmost header entry that is not inside trustedProxyCidrs (the leftmost, used before, was client-controlled), so trustedProxyCidrs must list every proxy hop; this has not been run behind a real proxy (guides/kairon-ui-dashboard.md). The admission webhook and inter-component RPCs (migration control-plane mTLS, the console relay) deliberately have no rate limiting added here: the webhook backstops real Kubernetes writes under failurePolicy: Fail, so throttling it risks rejecting a legitimate bulk kubectl apply; the RPCs are already restricted to mTLS/bearer-token-authenticated peers, not open to arbitrary traffic. See SECURITY.md.
  • Every kairon-ui JSON request body is now size-capped. Nothing in kairon-ui's HTTP stack ever bounded request body size before this — a caller, even a legitimately authenticated one, could send an arbitrarily large body and have json.Decoder buffer all of it into memory. decodeJSON (internal/uiapi/server.go) now wraps every request body in http.MaxBytesReader at a 1MiB default, comfortably above every ordinary request this API accepts; guest file writes (POST .../agent-file/put) get an explicit 6MiB override, since a real file's content travels base64-encoded (~4/3 expansion) and the read side of the same feature was already capped around 3MB of real content. Rejected with a plain 400, not a broken connection — see docs/guides/machine-guest-agent-files.md.
  • Live-migration data-plane identity is opt-in, and still two places to configure at once. migration.dataplaneTlsSecretName (opt-in) gives every node a real, distinguishable cert for the QEMU RAM/state stream instead of the shared control-plane cert every node could otherwise present there — scripts/gen-migration-mtls-certs.sh generates both the certs and a ready-to-apply Secret. Left unset (the default), the data plane still falls back to the shared cert, same as before this existed. Either way, actually turning data-plane encryption on also requires each node's kairon-migration-adapter-fluxvm systemd unit to pass -migration-data-tls=true independently — outside Helm's control, easy to half-upgrade a fleet. The control-plane cert (migration.tlsSecretName) stays deliberately shared regardless — it proves cluster membership, not host identity, by design. See docs/guides/machine-migration-tls.md.
  • Cluster backup/restore covers Kairon's Kubernetes-level state; disk content is per Machine. scripts/backup-crds.sh/restore-crds.sh (see docs/runbook-backup-restore.md) save the kairon.zyvor.dev CRDs and, opt-in, the chart-managed Secrets, not VM disks or FluxVM's per-host runtime state. Disk content is covered per Machine by MachineBackup: on the node (guides/machine-backup.md), or to S3 with incremental chains that restore on any node (guides/machine-backup-incremental.md, experimental). PVC-backed disks remain your CSI driver's or storage backend's job. Not yet drilled against a real full cluster-loss scenario in this repo's own CI.
  • OIDC/SSO group-to-admin mapping is opt-in and login-time-fresh, not IdP-live: ui.oidc.adminGroups (unset by default, matching prior behavior exactly) grants admin capability to an OIDC session whose ID token carries one of the configured groups, re-checked against current Kairon config on every request — but the group membership itself only reflects the IdP's state as of the caller's last login, not a live check, so removing someone from an IdP group doesn't revoke their admin session early; it takes effect at their next sign-in. See docs/guides/kairon-ui-oidc.md.
  • Every kairon.zyvor.dev CRD serves v1 (storage), v1beta1 and v1alpha1 with an identical schema, so no conversion webhook is live; the conversion webhook scaffold exists for the next real schema change: kairon-controller's webhook server exposes a tested POST /convert/machinequotas (internal/conversion), proven end to end against a real worked MachineQuota field-rename example — on the same TLS listener/certificate/Service as the existing admission webhook, no new trust boundary. Cutting a real schema-changing version still needs a CRD manifest change and a real converter for whatever's actually changing, plus solving one still-open gap: charts/kairon/crds/*.yaml lives in Helm's special, never-templated crds/ directory, so wiring a live caBundle into a CRD's spec.conversion needs a deliberate choice (move that CRD into templates/, or a separate patch step) not yet made. See docs/guides/crd-versioning.md.
  • The admission webhook's MachineDisruptionBudget and Machine-CREATE-MachineQuota checks only ever evaluate CREATE, matching the reconcile-loop/kaironctl checks they backstop exactly. MachineQuota also evaluates UPDATE (a hotplug resize past quota) at the webhook, but only when webhook.enabled is true. Independent of that: kairon-node now also self-checks a hotplug resize against MachineQuota itself before actually growing a running Machine's live vCPUs/memory — reusing the exact same admission logic as the webhook (internal/controller.QuotaTrackersForNamespace/AdmitQuotaResize) — so a resize past quota is caught even on a deployment that has never turned the webhook on. This self-check is effectively on by default (no new Helm flag) but fails open: any lookup error, most commonly a kairon-node ServiceAccount that hasn't yet been granted the new read-only machinequotas RBAC (a binary upgraded ahead of its chart), causes it to log a warning and proceed with the resize exactly as before this existed, rather than newly blocking on account of infrastructure the check itself depends on.
  • The VNC console inherits FluxVM's own unauthenticated VNC socket as-is — Kairon can't add auth/encryption FluxVM itself doesn't have. Its real security rests on kairon-ui's operator auth, a single-use per-session ticket now bound to the one Machine it was issued for, an opt-in per-Machine allowlist (kairon.zyvor.dev/console-allowed-users, unset means unchanged all-operator access), and the shared cluster-wide token to kairon-node (optionally now over TLS). See SECURITY.md.
  • kaironctl covers virtctl's verbs, with FluxVM's limits. console, vnc, restart, soft-reboot, stop --grace-period/--force, export create|list|download|delete, memory-dump, guest osinfo|fslist|userlist, credentials, expose, create --dry-run, expand, wait, open, start --paused, --dry-run on power verbs, freeze/unfreeze, vnc screenshot, objectgraph, permitted-devices, adm log-verbosity, create clone, guestfs, cdrom and export manifest exist; the mapping is in CLI. reset is restart: FluxVM has no reset, and its stop is already a hard power-off. A graceful stop needs the guest agent. expose writes an EndpointSlice with the guest IP, so it only reaches guests whose IP is routable from the cluster's nodes, and an IP change shows up on kairon-controller's next tick. credentials edits Linux guests only (authorized_keys, chpasswd), and memory-dump needs the qemu backend. FluxVM cannot insert media into a created VM, so cdrom insert needs a Stopped Machine; create clone copies only the boot disk; vnc screenshot needs a display without a VNC password; guestfs needs libguestfs on the node. usbredir is not available: FluxVM has no USB redirection.
  • The sandbox HTTP proxy is still API-only. The other runtime features (Halted, both guest-exec channels, sandboxes/templates/warm pools, the image catalog, egress check, runtime and network diagnostics) now have dashboard controls (the machine inspector's operations panel, including incident bundles, and the node tools panel) and CLI verbs (kaironctl guest ..., kaironctl runtime ..., kaironctl diagnose machine).
  • kairon-ui now has a dashboard page for MachineQuota/MachineDisruptionBudget/MachineSet/MachineInstanceType/MigrationPolicy. GET /api/v1/quotas/disruption-budgets/machinesets/instancetypes/migration-policies wrap the same read-only List*Namespace calls kaironctl get already uses — any-authenticated-operator, matching every other read-only cross-checkable resource — and each now has a matching list page (web/src/pages/{Quotas,DisruptionBudgets,MachineSets,InstanceTypes,MigrationPolicies}.tsx), closing what was previously a real visibility gap: an operator using only the dashboard had no way to see quota usage or disruption-budget state at all, both first-class admission-time concerns (see Guarding the fleet). Four of the five stay list-only — no create/update/delete route or dashboard form; kaironctl/kubectl remain how they get mutated. MachineSet is the one exception: DELETE /api/v1/machinesets/{namespace}/{name}, PATCH /api/v1/machinesets/{namespace}/{name}/scale, and matching "Delete"/"Scale" row actions now let an operator remove or resize one straight from the dashboard, the same any-authenticated-operator gate handleDeleteMachine already has (no separate admin check). Picked over the other four because it's the one an operator manages as a day-to-day fleet-sizing operation (kaironctl create/scale/edit machineset all shipped this same session) rather than a GitOps-managed policy object or a mostly-static reference value. Deleting a MachineSet from the dashboard also cascades to delete every Machine it created — see Deleting a MachineSet deletes its replicas too — so the dashboard's confirm prompt says so explicitly.
  • kairon-ui now has a dashboard page for the real Kubernetes Node, not just a kairon.zyvor.dev CRD. Node taints started actually gating scheduling, then kaironctl get/describe node gave an operator a way to see them from the CLI (see Guarding the fleet) -- but GET /api/v1/nodes (added earlier for the Overview tile's plain node count) had no dashboard page rendering the list itself until now. The new Nodes page shows the same columns kaironctl get nodes prints -- name, Ready condition, spec.unschedulable, a key[=value]:Effect taint summary, and addresses -- read-only like kaironctl's own support, since Node's create/delete lifecycle stays kubectl's job either way. No new REST route, no new RBAC: the route and the ClusterRole grant it needs both already existed for the Overview tile. See docs/guides/machine-placement.md's "Taints and tolerations" section.
  • Fixed a real gap: kairon-ui's own ClusterRole never actually granted get/list/watch on MachineQuota/MachineDisruptionBudget/MachineInstanceType/MigrationPolicy, or the extra verbs MachineSet's delete and MachineSnapshotSchedule's suspend/resume route need (delete, patch) — every one of those dashboard/REST routes above has 403'd against a real Kubernetes apiserver since the day it shipped. Nothing caught it because every internal/uiapi test runs its handlers against a fake httptest.Server double, which never enforces RBAC at all — a real cluster was the only way this would surface. charts/kairon/templates/all.yaml's kairon-ui ClusterRole now grants exactly the verbs each route actually needs, no more.
  • kairon-ui and kaironctl now also cover MachineNetworkPolicy/NetworkSecurityGroup — the two CRDs that drive FluxVM's real eBPF/TC network enforcement (see the network-policy guide) had zero human-facing tooling beyond raw kubectl, unlike every other kind in this project: no kaironctl get/describe/delete, and no dashboard/REST route at all (the existing network-observability endpoints only ever show a Machine's effective, already-merged policy, not the policy objects that produced it). kaironctl get networkpolicies/securitygroups, describe, and delete now work like every other kind; GET /api/v1/network-policies/security-groups back two new list-only dashboard pages (Network policies, Security groups). The dashboard stays list-only; kaironctl create networkpolicy/create securitygroup and kaironctl edit networkpolicy/edit securitygroup (see CLI) have since closed the other half of this gap — kaironctl/kubectl (not the dashboard) are how these get created or edited.
  • A warm-pool claim only becomes a Kairon-managed Machine when the caller opts in. By default, claiming hands back FluxVM's own VM record directly with no automatic Machine object — a claimed VM sits outside admission-time guards (MachineQuota, the webhook) until one exists. POST .../claim's createMachine: true (plus a required machineName) closes that gap for a given claim: it forces the FluxVM-side VM name to match Machine.RuntimeName() and creates the Machine object from the pool's own template, so kairon-node's existing adoption path picks up the exact runtime on its next reconcile tick. The Machine carries the template's resources and hotplug headroom, kernel, cloud-init, NUMA/cpuset/hugepages, secure boot/TPM, guest agents and TTL, and is pinned to the claimed VM's node; VFIO devices, the raw network map, shared folders, data disks and storage overrides are not carried and are reported as machineWarnings. See docs/guides/machine-sandboxes.md's "Warm pools" section.
  • Real CPU pinning (spec.resources.cpuPinning) needs an accurate kairon.zyvor.dev/pinnable-cpus node label. An operator can set it, or kairon-node can discover it: online CPUs minus --reserved-cpus minus kubelet's exclusive pod CPUs from cpu_manager_state. Discovery is off until --reserved-cpus is set. kubelet's state file is 0600 root, so deploy-remote.sh --reserved-cpus=LIST installs a path-triggered copier unit and Helm node.cpuPinning.enabled adds a copier sidecar; either way the copy can lag kubelet by a moment. An operator label is never overwritten, so it can still drift from kubelet's real allocations. See docs/guides/machine-cpu-pinning.md.
  • The third-party CSI client (node.thirdPartyCSIDrivers) is a first cut, not a general "any CSI driver" integration. attachRequired drivers work through a VolumeAttachment (the driver's external-attacher must be running) and node secrets resolve from one allowlisted namespace, but no live third-party driver has been run end to end yet; behavior is unit-tested against fakes. See docs/guides/machine-storage-thirdparty-csi.md.
  • kairon-ui now has opt-in per-namespace authorization (ui.auth.namespaceScoping.enabled, default false) — closes a real gap: kairon-ui's own ClusterRole is cluster-scoped and its HTTP layer previously had no namespace authorization axis at all, only isAdminIdentity's admin-vs-not split, so every authenticated operator (password, legacy shared token, or OIDC) could see and mutate every namespace's Machines/quotas/policies regardless of intent. Once enabled, a non-admin session-token operator is restricted to the namespaces in their own ui.auth.users[].namespaces or reachable via a matching ui.oidc.namespaceGroups entry — everything else 403s. A request with no named identity (the legacy shared ui.token) is refused with a 403, the overview is filtered to the caller's grants, and node routes need an administrator. Off by default: flipping it on for an existing deployment with no namespaces/namespaceGroups configured moves every non-admin operator from "sees every namespace" to "sees none," a deliberate fail-closed default, not a silent no-op. Honest first-cut limits: the dashboard's namespace inputs stayed free-text in this first cut (v0.9.0 added a nav namespace selector fed by GET /api/v1/namespaces), a scoped-out operator gets the existing generic 403 rather than a constrained dropdown; GET /api/v1/overview still aggregates cluster-wide regardless; POST /api/v1/migrations/evacuate (a node-shaped bulk action spanning whatever namespaces that node's Machines happen to live in) isn't scoped by it either; and Node-scoped routes aren't namespace-scoped at all, since Node isn't a namespaced kairon object. The legacy shared token has no per-caller identity to scope by, so it stays fully unrestricted regardless of this setting, by design.
  • Machine.spec.tenant is a network fence, not a tenancy boundary. With kairon.zyvor.dev/tenant-fence=true, kairon denies other tenants' observed guest addresses in the same namespace (see docs/guides/tenant-fence.md). It is not a VRF: unobserved addresses and other namespaces are not denied. The rename/clear check needs the admission webhook. Kubernetes Namespace is still the quota and RBAC boundary (MachineQuota, MachineDisruptionBudget, and the namespace-scoped authorization above). kubectl explain machine.spec.tenant and kaironctl describe machine say so.
  • Preemption only halts. An opted-in pending Machine (kairon.zyvor.dev/preempt=true) halts at most one opted-in lower-priority Machine (kairon.zyvor.dev/preemption-policy=Halt) per tick in the same namespace; there is no eviction, migration, or cross-namespace preemption. Rejecting Pause as a policy needs the admission webhook. See docs/guides/preemption.md.
  • Hotplug after a Halted resume is best effort. Resume from Halted boots at FluxVM's kept create-time size, and kairon-node hotplugs back to the pre-halt size right after. If headroom or the guest refuses, the Machine runs smaller and status.applied* says so. See docs/guides/machine-hotplug.md.
  • MCP approval trusts whoever can annotate. Destructive MCP tools wait for kaironctl approve, which is an annotation write; anyone with patch on the Machine or MachineBackup can approve. The approver name is self-reported (KAIRON_APPROVER or the OS user). See docs/guides/hermes-mcp.md.

Report vulnerabilities privately to security@zyvor.dev — see SECURITY.md.