Skip to main content

Per-second metrics platform

Netra collects per-second host, network, workload and application metrics on every node, streams them to netrad, keeps them in a tiered store, scores them for anomalies, evaluates health rules against them, and charts them in the console. It is the Netdata-style "every metric, every second, zero config" layer, joined to Netra's eBPF evidence: a spike on a chart links straight to the flows, drop reasons, TCP events and captures behind it.

Related pages: metric alerts, anomaly detection, application collectors.

The platform is observe-only. Collectors read /proc, /sys, cgroup files and the counters the agent already read from its own eBPF maps; they never write to any of them. Metric alerts only notify. Nothing here touches enforce mode, rules or policies.

How it fits together​

node: collectors ─▶ tier-0 store (1 h, replay buffer) ─▶ anomaly flags
│
└─ gzip JSON stream, POST /api/v1/agents/metrics
netrad: per-node stores (tier 0 1 s, tier 1 1 min, tier 2 1 h)
├─ query, WebSocket stream, contexts GET /api/v1/metrics/*
├─ metric alert engine ─▶ notification channels
├─ exporters ─▶ Prometheus remote write, OTLP, Graphite
└─ evidence, AI briefs, fleet roll-up
  • Collectors run once a second in the agent. A collector that fails or is slow is reported in netra.collector_duration, netra.collector_events and netra.collector_disabled and never blocks the others.
  • The agent store keeps one hour of 1-second samples. It is also the replay buffer: when netrad is unreachable the sender resumes from its cursor and catches up, so a restart or a network blip leaves no gap.
  • netrad keeps one store per node with three tiers. Rollups keep min, max, sum, count and the anomaly count per bucket, so a 30-day chart still shows peaks and the anomaly ribbon.

What is collected​

CollectorContextsSource
proc.statsystem.cpu, cpu.cpu, system.intr, system.ctxt, system.forks, system.processes, system.softirqs/proc/stat, /proc/softirqs
proc.meminfosystem.ram, mem.available, mem.used_percent, mem.swap, mem.committed, mem.kernel, mem.hugepages, mem.writeback, mem.pgfaults, mem.oom_kill/proc/meminfo, /proc/vmstat
proc.systemsystem.load, system.uptime, system.file_nr_used, system.file_nr_utilization, system.entropy/proc/loadavg, /proc/uptime, /proc/sys
proc.pressuresystem.cpu_some_pressure, system.memory_full_pressure, system.io_full_pressure (and the other some/full pairs, plus stall time)/proc/pressure
proc.diskstatsdisk.io, disk.ops, disk.qops, disk.util, disk.await, disk.backlog, system.io/proc/diskstats
diskspacedisk.space, disk.space_utilization, disk.inodes, disk.inodes_utilizationstatfs on host mounts
proc.net.devnet.net, net.packets, net.errors, net.drops, net.fifo, net.frames, net.carrier, net.speed, net.mtu, net.operstate, system.net/proc/net/dev, /sys/class/net
proc.net.snmp, proc.net.netstatipv4.packets, ipv4.tcppackets, ipv4.tcpsock, ipv4.tcperrors, ipv4.tcp_retrans_segments, ipv4.udppackets, ipv4.icmp, ipv6.*, ip.tcp_retransmits, ip.tcp_accept_queue, ip.tcp_syn_queue, ip.tcpsyncookies, ip.tcp_backlog_drops, ip.tcpconnaborts/proc/net/snmp*, /proc/net/netstat
proc.net.sockstatipv4.sockstat_tcp_sockets, ipv4.sockstat_tcp_mem, ipv4.sockstat_udp_*, ipv6.sockstat6_*/proc/net/sockstat*
proc.net.softnet_statsystem.softnet_stat (processed, dropped, squeezed)/proc/net/softnet_stat
netfilter.conntracknetfilter.conntrack_sockets, conntrack_max, conntrack_utilization, conntrack_changes, conntrack_errors/proc/sys/net/netfilter, /proc/net/stat/nf_conntrack
cgroupscgroup.cpu, cgroup.throttled, cgroup.mem_usage, cgroup.mem_utilization, cgroup.mem, cgroup.mem_events, cgroup.io, cgroup.serviced_ops, cgroup.*_pressurecgroup v2 files of each pod Netra already tracks (pod cgroup, which includes its containers; per container with NETRA_METRICS_CGROUP_CONTAINERS=true)
apps.groupsapp.cpu_utilization, app.mem_usage, app.processes, app.threads, app.disk_io, app.page_faults/proc/<pid>/stat and io, grouped by comm
ebpf.datapathebpf.packets, ebpf.bandwidth, ebpf.iface_*, ebpf.kernel_drops, ebpf.tcp_events, ebpf.shield, ebpf.policy_drops, ebpf.conntrack_entriescounters already in the agent report
workload.redworkload.red_rate, red_bandwidth, red_errors, red_http, red_dns, red_durationworkload-attributed counters in the agent report
apps<kind>.*, apps.upapplication collectors

The process collector never reads /proc/<pid>/cmdline or environ; groups are named by the 15-byte kernel comm. The busiest 40 groups (by CPU and memory) get their own charts and the rest are summed into other; once a group is charted it stays charted until it exits, so series do not churn.

Workload RED series​

workload.red_* turns the same counters behind GET /api/v1/insights/red into per-second series per workload (namespace, workload, workload_kind labels), so anomaly detection and metric alerts cover every workload with no configuration. Rate is datapath packets and bandwidth; errors are blocked packets, TCP retransmissions and RTOs, cleartext HTTP/1 5xx and DNS failures; duration is TCP smoothed RTT. The busiest 200 workloads by bytes are kept (NETRA_METRICS_RED_WORKLOADS_MAX). The agent builds its report every few seconds; between reports the last rates are repeated rather than dropping to zero, and a report that stops changing stops being repeated after 15 seconds.

Querying​

EndpointWhat
GET /api/v1/metrics/nodesstreaming nodes, last ingest, series and bytes per node
GET /api/v1/metrics/contexts?nodes=&family=every context with its charts, dimensions, labels, units and time range
GET /api/v1/metrics/data?context=...one query (below)
GET /api/v1/metrics/streamWebSocket: send {"subscribe":[{id, context, charts, nodes, labels, window, points, groupBy, group, aggregate}]}; each subscription is pushed once a second
GET /api/v1/metrics/anomaliesanomaly rate per node, ranked anomalous metrics, correlation for a highlighted range (anomaly detection)
GET /api/v1/metrics/evidenceflows, drop reasons, captures and co-anomalies behind a metric window (below)
GET /api/v1/metrics/alertsactive metric alerts, transitions, rules, silences (metric alerts)
GET /api/v1/metrics/exportersexporter status (below)
GET /api/v1/metrics/summarythis cluster at a glance
GET /api/v1/metrics/fleetthis cluster plus every NETRA_FLEET_PEERS peer

/data parameters, Netdata-style:

ParameterMeaning
contextrequired, for example net.net
nodes, charts, dimensionscomma-separated globs
labelskey:value,... (values may be globs)
after, beforeunix seconds, or negative seconds relative to now (default -600, 0)
pointsnumber of buckets returned
groupreduction inside a bucket: avg (default), min, max, sum, last, p50, p90, p95, p99 (percentiles need tier 0)
group_bydimension (default), chart, node, instance, all, label:<key>
aggregatehow grouped series combine: sum, avg, min, max
tierforce tier 0, 1 or 2; otherwise the finest tier covering the window is used

Each dimension carries values and anomalyRate per bucket. NaN (no data) is null.

Evidence behind a metric​

GET /api/v1/metrics/evidence?context=&node=&labels=&after=&before= answers "what was happening on the network when this chart moved". It picks evidence by the kind of metric:

Metric looks likeEvidence
drops, softnet, shield, policy, interface errorsdrop reasons, drop sites, flows ranked by blocked packets, captures
TCP, retransmits, workload.red_errors, red_duration, listen queuesTCP events, kernel network diagnostics, flows ranked by retransmissions, captures
DNSDNS findings, flows
HTTP, nginx, apache, haproxy, envoyworkload RED, flows
conntrackdatapath summary, flows
anything elseflows ranked by bytes, captures

The answer holds the top 10 flow-history peers in the window (narrowed to a workload when the chart's namespace/workload labels are passed), the node's current kernel drop reasons for drop metrics, capture sessions that overlapped the window, other metrics that were anomalous at the same time, and links to the matching Netra endpoints. The console shows it under a chart (Evidence button, or drag across the chart to pick the range); netractl metrics evidence CONTEXT and MCP netra_metrics_evidence return the same.

CLI and MCP​

netractl metrics nodes
netractl metrics contexts --family net
netractl metrics query net.net --node worker-1 --after -300 --group max
netractl metrics top --node worker-1
netractl metrics anomalies --after -3600
netractl metrics evidence ebpf.kernel_drops --node worker-1 --after -900
netractl metrics fleet

MCP read tools: netra_metrics_contexts, netra_metrics_query, netra_metrics_anomalies, netra_metrics_evidence, netra_metrics_fleet, netra_metrics_alerts, plus the netra://metrics/contexts, netra://metrics/anomalies and netra://metrics/alerts resources.

Console​

Metrics is generated from /contexts: families in a sidebar, one canvas chart per context (or per instance when a node has a handful), live at one second over the WebSocket with polling as the fallback, windows from 5 minutes to 30 days, split by dimension or by node across the fleet. Metric anomalies shows the anomaly-rate timeline per node; drag a range to rank the metrics whose distribution changed most. Metric alerts lists active alerts with Ack and Silence actions.

Fleet roll-up​

GET /api/v1/metrics/summary reports this cluster's nodes, stale nodes, series, anomaly rate over the last 15 minutes, raised alerts by severity, the top alerts and the top anomalous dimensions. GET /api/v1/metrics/fleet adds the same summary from every peer in NETRA_FLEET_PEERS (see fleet clusters), fetched in parallel and best-effort; a peer's own peers are not followed. Totals weight the anomaly rate by node count.

Exporting​

netrad can ship stored metrics elsewhere. Each sink is off until its target is set.

VariableMeaning
NETRA_EXPORT_PROMETHEUS_RW_URLPrometheus remote write v1 endpoint (protobuf, snappy)
NETRA_EXPORT_PROMETHEUS_RW_HEADERSK=V,K2=V2, for example an Authorization header
NETRA_EXPORT_OTLP_ENDPOINTOTLP/HTTP base URL; gauges are posted to /v1/metrics as JSON
NETRA_EXPORT_OTLP_HEADERSK=V,...
NETRA_EXPORT_GRAPHITE_ADDRhost:port for Graphite plaintext over TCP
NETRA_EXPORT_RESOLUTIONbucket size, default 10s (average per bucket)
NETRA_EXPORT_CONTEXTS, NETRA_EXPORT_EXCLUDEcomma-separated context globs
NETRA_EXPORT_PREFIXmetric name prefix, default none (netra in Helm)

Prometheus names are <prefix>_<context> with dots turned into underscores (netra_net_net), labelled instance (the node), chart, dimension, family and the chart's own labels; OTLP sets k8s.cluster.name from NETRA_CLUSTER_NAME. Each exporter keeps a cursor per series, sends only complete buckets, retries on failure, and skips ahead rather than fall more than 15 minutes behind. GET /api/v1/metrics/exporters shows the last success, last error, points sent and lag per sink. Credentials come only from these variables (in Helm, from metricsPlatform.export.existingSecret).

Configuration​

Agent​

VariableDefaultMeaning
NETRA_METRICS_ENABLEDtruecollectors and stream on/off
NETRA_METRICS_TIER0_RETENTION1hlocal 1-second history and replay buffer
NETRA_METRICS_MAX_SERIES20000series cap on the node
NETRA_METRICS_AGENT_DIRempty (memory)persist the agent store
NETRA_METRICS_PROC, NETRA_METRICS_SYS, NETRA_METRICS_FS_ROOT/proc, /sys, /proc/1/root when visiblehost roots
NETRA_METRICS_PROCESS_GROUPS, NETRA_METRICS_PROCESS_GROUPS_MAXtrue, 40process groups
NETRA_METRICS_CGROUP_CONTAINERSfalsechart each container cgroup as well as each pod
NETRA_METRICS_RED_WORKLOADS_MAX200workload RED series
NETRA_METRICS_ML, NETRA_METRICS_ML_TRAIN_WINDOW, NETRA_METRICS_ML_TRAIN_EVERYtrue, 1h, 30manomaly models on the node
NETRA_APPS_CONFIG, NETRA_APPS_DISCOVERY, NETRA_APPS_EVERY/etc/netra/apps.yaml, false, 5sapplication collectors

netrad​

VariableDefaultMeaning
NETRA_METRICS_ENABLEDtrueingest, store and APIs on/off
NETRA_METRICS_DIR<state dir>/metricsstore directory; - keeps everything in memory
NETRA_METRICS_TIER0_RETENTION1h1-second samples (memory)
NETRA_METRICS_TIER1_RETENTION336h1-minute rollups
NETRA_METRICS_TIER2_RETENTION8760h1-hour rollups
NETRA_METRICS_DISK_QUOTA_MB1024oldest rollup blocks are dropped first above it
NETRA_METRICS_MAX_SERIES50000per node
NETRA_METRICS_MAX_NODES2000stores kept
NETRA_METRICS_PARENT_MLfalsescore anomalies on netrad for agents running with ML off

Helm: metricsPlatform.* (separate from metrics.*, which is netrad's own Prometheus /metrics). Keep metricsPlatform.diskQuotaMb below persistence.size; the state file shares the volume.

Honest limits​

  • Per-second resolution lives in memory for tier0Retention (one hour by default); older windows come from 1-minute and 1-hour rollups, where percentiles fall back to max.
  • A series that stops reporting keeps its history but stops growing; the series cap rejects new series (counted) rather than evicting old ones.
  • Pods in their own network namespace are seen through their cgroup, their host-side veth and Netra's eBPF counters, not through their own /proc/net.
  • Workload RED is network-level (packets, retransmissions, SRTT, cleartext HTTP/1 status), not an application request trace; see flow log for the same caveats.
  • Tests: ./scripts/ci-metrics-unit.sh (fixtures, no root) and sudo ./scripts/ci-metrics-veth.sh (real collectors and netrad, iperf3 load on a veth, anomaly flagged, alert fired).