User guide: metrics and alerts
Until now, Kairon's Prometheus metrics were kairon-controller-only and
migration-lifecycle-only (internal/metrics's original scope) -- a real
observability gap this page closes the worst of: there was no signal at
all if kairon-controller/kairon-node themselves were unhealthy or
wedged, no metrics from kairon-node/kairon-ui at all, and every alert
in charts/kairon/alerts.yaml was migration-specific.
What each component exposes
Every metric name and its exact meaning lives in internal/metrics's
package doc and Observe* method comments -- this table is a map to that,
not a duplicate of it.
| Component | Port | New in this pass | Already existed |
|---|---|---|---|
kairon-controller | controller.healthPort, default 32301 (/metrics) | kairon_reconcile_duration_seconds, kairon_reconcile_errors_total, kairon_reconcile_item_errors_total, kairon_webhook_decisions_total (only if webhook.enabled), kairon_apiserver_request_duration_seconds | kairon_migration_phase_count, kairon_migration_phase_age_seconds, kairon_migration_completed_total, kairon_migration_transfer_duration_seconds, kairon_migration_cutover_downtime_seconds, kairon_migration_dataplane_encrypted |
kairon-node | node.healthPort , default 32302 (/metrics) | kairon_reconcile_duration_seconds, kairon_reconcile_errors_total, kairon_reconcile_item_errors_total, kairon_apiserver_request_duration_seconds | (none -- internal/health.Server.Metrics existed but was never wired to anything, so /metrics 404'd) |
kairon-ui | its own listen port (GET /metrics, unauthenticated -- see SECURITY.md) | kairon_ui_request_duration_seconds, kairon_apiserver_request_duration_seconds | (none -- no health/metrics endpoint of any kind existed) |
kairon_apiserver_request_duration_seconds is the one metric all three
share, by design: every component talks to the Kubernetes API through the
same internal/kube.Client, so instrumenting it once (Client.Observe,
wired to each component's own Recorder in cmd/*/main.go) gives every
component apiserver call visibility for free, labeled method
(GET/POST/PATCH/PUT/DELETE) and outcome (ok/error).
kairon-controller also reports live migration telemetry from each MachineMigration's status.progress and status.adaptive. Adaptive migration is experimental until the lab migration matrix runs green.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
kairon_migration_progress | gauge | namespace, name, field | Telemetry for a migration in Starting or Running. field is dirty_rate_mbps, ram_remaining_bytes, ram_transferred_bytes, throughput_mbps, expected_downtime_ms, cpu_throttle_pct, estimated_completion_seconds, compression_ratio_percent or duplicate_pages. |
kairon_migration_adaptive_stage | gauge | namespace, name, stage | 1 for the adaptive engine's current stage: PreCopy, AutoConverge, RelaxDowntime, PostCopy, Abort or Unsupported. |
kairon_migration_adaptive_actions_total | counter | action | Adaptive engine actions, such as auto-converge, throttle, relax-downtime, bandwidth, postcopy, abort, tune-failed and postcopy-refused. |
kairon_topology_allocation_failures_total | counter | reason | Scheduling attempts for spec.topology/spec.devices Machines that found no placement; reason is TopologyUnsatisfiable, DRAAllocationPending or Unschedulable. |
kairon_node_numa_free_cpus | gauge | node, numa | Pinnable CPUs not held by a Machine, per NUMA node, from each NodeTopology. See topology-aware placement. |
kairon_gpu_vram_mib | gauge | node, state | GPU memory per node: total from NodeTopology, allocated to Machines' whole GPUs and vGPU framebuffers. See vGPU. |
kairon_gpu_devices | gauge | node, health | GPUs per health state: Healthy, Degraded, Unhealthy or Unknown. |
kairon_accelerator_profile_capacity | gauge | profile, node | How many more Machines with a MachineAcceleratorProfile the node's free devices could take (devices only). See accelerator profiles. |
kairon_accelerator_profile_machines | gauge | profile | Machines expanded from the profile. |
kairon_vgpu_available_instances | gauge | node, parent, profile | Free mediated-device instances per profile, as last reported by the node. |
kairon_node_maintenance | gauge | node, state | 1 while a node is in maintenance mode, state Entering or InMaintenance. See node maintenance. |
kairon_node_maintenance_remaining_machines | gauge | node | Machines still holding a VMM on a node in maintenance mode. |
kairon_drs_imbalance_percent | gauge | — | Gap between the most and least utilized node (reserved CPU or memory, or worst host PSI with -drs-signal pressure) in the last DRS round. See DRS. |
kairon_drs_node_utilization_percent | gauge | node, resource | Reserved cpu, memory and max share per node in the last DRS round, and pressure (worst host PSI) with -drs-signal pressure. |
kairon_dr_rpo_seconds | gauge | namespace, plan | Age of the oldest of a protection plan's newest restore points. See cross-cluster DR. |
kairon_dr_active | gauge | namespace, plan | 1 on the site that runs the plan's Machines. |
kairon_dr_peer_reachable | gauge | namespace, plan | 1 when the last call to the peer cluster succeeded. |
kairon_dr_last_rto_seconds | gauge | namespace, plan, kind | Start to every Machine Running for the last failover or test. |
kairon_dr_last_data_loss_seconds | gauge | namespace, plan, kind | Start of the last failover or test minus the oldest restore point it used. |
kairon_dr_last_test_success | gauge | namespace, plan | 1 when the last finished DR test (manual or scheduled) succeeded. |
kairon_dr_last_test_timestamp_seconds | gauge | namespace, plan | Unix time the last DR test finished; alert when it gets older than test.every. |
kairon_dr_dns_records | gauge | namespace, plan | Records in the plan's external-dns DNSEndpoint on this site (0 on the standby). |
kairon_drs_moves_total | counter | result | DRS moves: recommended, started or blocked (disruption budget). |
kairon_node_psi_some_avg60_percent | gauge | node, resource | kairon-node only. Host PSI some avg60 (cpu, memory, io) last published on the Node for DRS's pressure signal. See DRS. |
kairon_node_memory_pressure | gauge | node, state | kairon-node only. 1 while host memory is under pressure (state=pressure), and while it is with every elastic-memory balloon at its minimum (state=exhausted, the KaironMemoryPressure condition). See elastic memory. |
kairon_node_virtual_networks | gauge | — | kairon-node only. Virtual networks applied to the local FluxVM. See virtual networks. |
kairon_node_virtual_network_endpoints | gauge | location | kairon-node only. Endpoints of the applied networks on this node (local) and on other nodes (remote). |
kairon_node_virtual_network_sync_errors_total | counter | — | kairon-node only. Failed virtual network applies or listings against FluxVM. |
kairon_node_bgp_session_up | gauge | peer | kairon-node only. 1 while the BGP session to peer is Established. |
kairon_node_bgp_advertised_prefixes | gauge | peer | kairon-node only. Prefixes announced to peer. |
kairon_machine_balloon_mib | gauge | namespace, machine, kind | kairon-node only. Balloon size of an elastic-memory Machine: kind=target (asked) or actual (given back by the guest). |
kairon_memory_reclaim_actions_total | counter | direction | kairon-node only. Balloon changes, inflate or deflate. |
kairon-node also registers the VM-edge metrics below (not registered on the controller or kairon-ui). Details, including how drops are read from FluxVM, are in docs/ebpf-edge.md.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
kairon_net_drops_total | counter | namespace, machine, reason, policy | VM-edge drops attributed to a MachineNetworkPolicy or dataplane guard; policy is - when not tied to a policy. No series exists until the first drop. |
kairon_net_conntrack_restored_total | counter | none | Conntrack entries restored onto a destination during live migration. |
kairon_net_migration_blackhole_ms | histogram | none | Milliseconds between conntrack export and restore. Buckets 5 to 2500 ms. |
kairon-node's metrics needed no chart changes -- its health port was
already exposed as a container port, just never serving /metrics.
kairon-ui's did: it has no separate health-check port at all, so its
/metrics route shares the same listen port as the dashboard and its
/api/v1/... routes, on the same "unauthenticated, like /healthz"
convention -- see SECURITY.md's kairon-ui section for the trust
boundary that implies (aggregate request counts/latencies, not sensitive
data, but reachable wherever the dashboard itself is).
kairon_quota_resource (kairon-controller only)
A later addition on top of the above: kairon-controller's /metrics now
also exposes kairon_quota_resource{namespace, quota, resource, type},
one gauge per MachineQuota dimension it caps or tracks (resource is
machines/cpu_cores/memory_mib; type is used or hard) --
mirroring kube-state-metrics' own kube_resourcequota{resource, type}
shape so the familiar used / hard ratio pattern applies directly.
Recorded once per Reconcile tick (internal/metrics.Recorder.ObserveQuotas),
from the exact same tallies that tick patches onto each MachineQuota's
own status.used* fields -- never a second, independently-computed
number that could drift from what kubectl get machinequota shows for
the same tick. A dimension the MachineQuota doesn't cap at all (e.g. no
maxTotalCpu set) never gets a type="hard" series for it, only
type="used" -- there's no limit to report, not a limit of zero that
would misleadingly read as "already over quota."
Why kairon-controller-only, unlike kairon_apiserver_request_duration_seconds:
this needs the same cluster-wide MachineQuota/Machine listing the
reconcile loop's own quota enforcement already does (internal/controller/quota.go),
which neither kairon-node (no cluster-wide visibility, only ever
self-checks quota for its own node's hotplug resizes) nor kairon-ui (no
reconcile loop at all) has an equivalent of. See docs/guides/machine-quotas.md
for the full picture, including the KaironQuotaNearLimit alert below.
kairon_disruption_budget_status (kairon-controller only)
The same follow-on applied to the other resource that already computed
real status every tick but never exposed it as a metric:
kairon-controller's /metrics now also exposes
kairon_disruption_budget_status{namespace, budget, field}, one gauge per
MachineDisruptionBudget status field (field is expected_machines,
current_healthy, desired_healthy, or disruptions_allowed) --
mirroring kube-state-metrics' own kube_poddisruptionbudget_status_*
gauges (four separate metric names there, collapsed here into one vector
via the field label, the same collapsing kairon_quota_resource already
does with its own type label). Recorded once per Reconcile tick
(internal/metrics.Recorder.ObserveDisruptionBudgets, called from
internal/controller/disruption.go's reconcileDisruptionBudgetsStatus),
from the exact same BudgetState.Status() values that tick patches onto
each MachineDisruptionBudget's own status.* fields -- never a second,
independently-computed number that could drift from what
kubectl get machinedisruptionbudget shows for the same tick.
Why kairon-controller-only: same reason as kairon_quota_resource above
-- this needs the cluster-wide MachineDisruptionBudget/Machine/
MachineMigration listing LoadBudgetStates already does
(internal/controller/disruption.go), which neither kairon-node nor
kairon-ui has an equivalent of. See
docs/guides/machine-disruption-budgets.md for the full picture,
including the KaironDisruptionBudgetExhausted alert below.
kairon_machineset_status (kairon-controller only)
The same follow-on applied to a third resource that already computed real
status every tick but never exposed it as a metric:
kairon-controller's /metrics now also exposes
kairon_machineset_status{namespace, machineset, field}, one gauge per
MachineSet rollout status field (field is replicas, ready_replicas,
or updated_replicas) -- mirroring kube-state-metrics' own
kube_replicaset_status_replicas/kube_replicaset_status_ready_replicas/
kube_deployment_status_replicas_updated gauges (three separate metric
names there), collapsed here into one vector via the field label, the
same collapsing kairon_quota_resource/kairon_disruption_budget_status
already do with their own type/field labels. Recorded once per
Reconcile tick (internal/metrics.Recorder.ObserveMachineSets, called
from internal/controller/machineset.go's reconcileMachineSets), from
the exact same tally that tick's status.replicas/.readyReplicas/
.updatedReplicas patch uses -- never a second, independently-computed
number that could drift from what kubectl get machineset shows for the
same tick. Unlike kairon_quota_resource/kairon_disruption_budget_status,
this is observed even when the tick's own create/delete step failed (the
tally is computed from the Machine snapshot before that step ever runs, so
it stays accurate regardless of whether the attempted mutation succeeded).
Why kairon-controller-only: same reason as kairon_quota_resource and
kairon_disruption_budget_status above -- kairon-node never lists
MachineSets cluster-wide at all (it only reconciles individual Machines
already assigned to it), and kairon-ui has no reconcile loop of its own,
so neither has an equivalent per-tick tally to report here. See
docs/guides/machine-sets.md for the full picture, including the
KaironMachineSetRolloutStuck alert below.
Alerts
charts/kairon/alerts.yaml now has five rule groups:
kairon-migrations(unchanged) -- the original four alerts, all pointing atdocs/runbook-migration-failures.md.kairon-health(new) -- four cross-component alerts with no runbook of their own (check the named component's own logs):KaironReconcileErrorRateHigh-- more than 5 failed reconcile ticks in 15 minutes, onkairon-controllerorkairon-node.KaironReconcileItemErrorRateHigh-- more than 5 per-item reconcile failures (kairon_reconcile_item_errors_total) in 15 minutes.KaironAPIServerErrorRateHigh-- more than 10 failed apiserver calls in 15 minutes, from any component.KaironWebhookDenyRateHigh-- more than 20 admission-webhook denials in 15 minutes (webhook.enabledonly).severity: info, notwarn-- a denial is the webhook doing its job; a spike usually means aMachineQuota/MachineDisruptionBudgetwas set stricter than expected, not that something is broken.
kairon-quotas(new) -- one alert,KaironQuotaNearLimit: aMachineQuota'sused/hardratio (kairon_quota_resource, see above) has stayed at or above 90% for 10 minutes on some namespace/quota/resource. Fires before the limit is actually hit -- previously the only signal was a Machine already stuckPending(or, withwebhook.enabled, an outright denied create/resize) with a "MachineQuota ... reached" message.severity: warn, pointed atdocs/guides/machine-quotas.mdrather than a dedicated runbook (there's nothing to debug -- raise the limit or free capacity).kairon-disruption-budgets(new) -- one alert,KaironDisruptionBudgetExhausted: aMachineDisruptionBudget'sdisruptions_allowed(kairon_disruption_budget_status, see above) has stayed at0for 10 minutes. UnlikeKaironQuotaNearLimit, this fires at exhaustion rather than approaching it --disruptions_allowedswings between0and a positive number as Machines matching the budget's selector come and go, so a "near" threshold on a small integer count would either never fire (rounding) or fire on every ordinary dip;== 0sustained for 10 minutes is the meaningful signal instead ("this budget is currently blocking everykaironctl evacuate/webhook-admitted disruption against it," not "it dipped to 0 for one reconcile tick").severity: info, same rationale asKaironWebhookDenyRateHigh: a disruption budget sitting at 0 is working-as-intended during, say, a rolling node drain, not necessarily a problem -- pointed atdocs/guides/machine-disruption-budgets.mdrather than a dedicated runbook.kairon-machinesets(new) -- one alert,KaironMachineSetRolloutStuck: aMachineSet'sready_replicas(kairon_machineset_status, see above) has stayed below itsreplicasfor 30 minutes. Previously the only signal a rollout had stalled was pollingkaironctl get machinesets/kubectl get machinesetsand noticingREADYnever catches up toREPLICAS-- this fires proactively instead, the same 30-minute "sustained, not a transient blip" thresholdKaironMigrationStuckInFlightalready uses for an in-flight migration.severity: warn, pointed atdocs/guides/machine-sets.mdrather than a dedicated runbook.
Neither rule file distinguishes which kairon-controller or
kairon-node instance fired, since kairon_reconcile_*/
kairon_apiserver_* use the same metric names on every instance's own
/metrics -- that's your scrape config's job/instance labels to add,
same as any other multi-target Prometheus rule.
If you run prometheus-operator, metrics.prometheusRule.enabled=true
renders charts/kairon/alerts.yaml as a real PrometheusRule named
kairon-alerts -- renamed from kairon-migrations now that it covers
more than migrations. On helm upgrade from an older release, the
old-named object is orphaned (Helm creates the new name rather than
renaming in place): kubectl delete prometheusrule kairon-migrations -n <namespace> cleans it up. Otherwise, copy the rule groups into your own
Prometheus rule_files config directly.
Scraping with prometheus-operator
metrics.serviceMonitor.enabled=true renders:
- a
PodMonitorforkairon-controllerand one forkairon-node(neither has a Service;/metricsis on thehealthcontainer port), and - with
ui.enabled, aServiceMonitorforkairon-ui(/metricson its dashboard port;schemefollowsui.insecureHttp, and with HTTPS the scrape setsinsecureSkipVerifybecause the default certificate is self-signed; turn it off withmetrics.serviceMonitor.ui.insecureSkipVerify=falseonce you have a trusted certificate).
Each object is skipped, without an error, when the cluster does not report
its CRD (.Capabilities.APIVersions). helm template does not know your
cluster's CRDs, so pass --api-versions monitoring.coreos.com/v1/PodMonitor
(and .../ServiceMonitor) or set metrics.serviceMonitor.requireCRD=false.
Use metrics.serviceMonitor.labels when your Prometheus selects monitors by
label (for example release: kube-prometheus-stack), and .controller,
.node and .ui .enabled to drop one target. With networkPolicy.enabled
remember allowIngressFrom for the Prometheus namespace. The existing
metrics.prometheusRule.enabled alerts are unchanged.
Grafana dashboards
metrics.grafanaDashboards.enabled=true renders one ConfigMap per file in
charts/kairon/dashboards/, labelled grafana_dashboard: "1" for the
Grafana sidecar (grafana-sc-dashboard; label, namespace and folder
annotation are values, see metrics.grafanaDashboards). The files are also
plain Grafana JSON you can import by hand. Each has a datasource variable
(any Prometheus data source).
| Dashboard | Covers | Metrics used |
|---|---|---|
| Fleet overview | Reconcile and apiserver health, webhook decisions, quotas, MachineSets, disruption budgets, DRS, placement failures, node maintenance, PSI and memory pressure, GPUs, NUMA, kairon-ui | kairon_reconcile_*, kairon_apiserver_request_duration_seconds, kairon_webhook_decisions_total, kairon_quota_resource, kairon_machineset_status, kairon_disruption_budget_status, kairon_drs_*, kairon_topology_allocation_failures_total, kairon_node_maintenance*, kairon_node_psi_some_avg60_percent, kairon_node_memory_pressure, kairon_gpu_*, kairon_node_numa_free_cpus, kairon_ui_request_duration_seconds |
| Machines | CPU, memory, disk, network edge, balloon, sample age, lifecycle actions | kairon_machine_resource_usage (cpu_percent, memory_bytes, disk_read_bytes, disk_write_bytes), kairon_machine_usage_sample_timestamp_seconds, kairon_net_drops_total, kairon_net_conntrack_restored_total, kairon_net_migration_blackhole_ms, kairon_machine_balloon_mib, kairon_memory_reclaim_actions_total, kairon_lifecycle_actions_total |
| Migrations | Phases, age, results, encryption, timings, live telemetry, adaptive engine | kairon_migration_* |
| Snapshots, backups and DR | S3 backup age, chain length, duration, bytes, reconcile errors for snapshot and backup kinds, DR RPO, RTO, peer and test state | kairon_backup_*, `kairon_reconcile_item_errors_total{kind=~"snapshot.* |
Every query uses only metric names registered in this repository's Go code
(internal/metrics); each panel's description names its metric. Known gaps,
so you are not surprised by an empty panel:
- No per-Machine network throughput. Kairon exports no rx/tx byte counters; the network panels show VM-edge drops and live-migration conntrack metrics only. Use node-level metrics from your node exporter for bandwidth.
- No snapshot-state metric. There is no metric for MachineSnapshot readiness or age; the dashboard shows reconcile errors for those kinds, and the object state is covered by the GitOps health checks.
- Per-Machine usage comes from
kairon-node(kairon_machine_resource_usage), so the node PodMonitor must be on. The disk panels applyrate()on the assumption that FluxVM reports cumulative byte counts; this has not been checked against a running VM. - Migration transfer and downtime histograms use Prometheus's default buckets (up to 10 s), so their quantiles saturate at 10 s for longer migrations.
- The shipped
KaironMigrationDataPlaneUnencryptedalert selectskairon_migration_dataplane_encrypted{phase=~...}, but that metric has onlynamespaceandnamelabels, so the selector matches nothing. The dashboard uses the unfiltered series. The alert is left as is here.
The dashboards were checked as JSON and as chart output only; they have not been loaded into a real Grafana, and the queries have not been run against a Prometheus holding real Kairon data.
Argo CD and Flux health
Neither tool understands Kairon's status.phase. Ready-made Argo CD health
Lua, a Flux healthCheckExprs example and an Argo CD Application are in
examples/gitops/.
Real limits today (first cut)
- Distributed tracing is opt-in via
otel.enabled/KAIRON_OTEL_ENDPOINT(internal/oteltrace): OTLP/HTTP JSON reconcile spans forkairon-controllerandkairon-nodeitem loops. Not the full OpenTelemetry Go SDK — a deliberate narrow exporter so enabling tracing does not pull a large dependency tree. Off by default. kairon-ui'skairon_ui_request_duration_secondsnow carries aroutelabel -- the registered mux pattern a request matched (e.g./api/v1/machines/{namespace}/{name}), never the raw request path, so cardinality stays bounded regardless of how many distinct Machines/namespaces are ever actually requested, the same waystatus_classalready bounds it against the full HTTP status range instead of the raw numeric code. A request matching nothing registered under/api/v1/(a genuine 404) reportsroute="unmatched".kairon_reconcile_errors_totalonly counts a whole reconcile tick returning an error outright (aList*call itself failing) -- the far more common case, one Machine/MachineMigration/MachineSnapshot/ MachineSnapshotRestore among many failing its own reconcile step, is caught, logged, and status-patched individually without the tick itself failing, so it never incremented this counter at all.kairon_reconcile_item_errors_total{kind}now counts exactly those,kindbeing one ofmachine/migration/snapshot/snapshotrestore-- labeled by resource kind only, never a specific name/namespace, to keep cardinality bounded. Still not a substitute for a real trace: it can tell you "machine reconciliation is failing repeatedly," not which Machine -- check the accompanying "... reconcile failed" log line (which does carry namespace/name) for that, or enableotel.enabled.
Migration concurrency caps
migration.maxConcurrentPerNode/migration.maxConcurrentCluster (both 0 = unlimited) bound how many non-terminal migrations may touch one node or the cluster at once — a lightweight admission control, mainly useful to cap the blast radius of a bulk kaironctl evacuate.
Workload hardening defaults
All three workloads set CPU/memory resources: by default, kairon-controller/kairon-ui each get a PodDisruptionBudget (on by default — voluntary-eviction protection only, not HA), and an opt-in NetworkPolicy (*.networkPolicy.enabled) restricts their ingress once you tell it which other namespace needs to reach in (*.networkPolicy.allowIngressFrom — get this wrong and Prometheus scraping or your ingress controller breaks silently, so it's opt-in rather than default-on). Image supply-chain (digest pins, Trivy, cosign, SBOM) is documented in SECURITY.md's "Images" section.
Real two-host live-migration testing and a live NeedsRecovery drill are documented as runbooks with helper scripts, since they need hardware this repository's own CI doesn't have: runbook-multi-host-migration-test.md, runbook-recovery-drill.md.