Changelog
The latest 20 entries. The full history is in CHANGELOG.md.
Unreleased
- Toolchain Go 1.27.4. CI and
go.modtoolchain move past GO-2026-6809 (crypto/x509) and GO-2026-6859 (net/http); both are fixed in 1.27.3 / 1.27.4. - Security: no built-in admin key.
scripts/deploy-remote.shno longer defaultsNETRA_API_KEYto the publishedAdmin@321; it mints a random key (still written to~/.netra/api-key) unlessNETRA_API_KEYis set. The dashboard login no longer mapsadmin/Admin@321to a baked-in token: the password is the API key. Controllers deployed with the old default should rotate their key. Docs:docs/dashboard-login.md.
0.32.0 — 2026-10-19
- Per-node agent mTLS identity (opt-in). With
NETRA_AGENT_MTLS_BIND_NODE, the reported node must match the client certificate (URI SANspiffe://netra.local/node/<name>, else DNS SAN, else CN). OptionalNETRA_AGENT_MTLS_DENY_SERIALShot-reloads revoked serials. Shared-cert mode stays the default. Helm:mtls.bindNode,mtls.denySerialsSecret. Docs:docs/agent-mtls.md. - Scoped authorization. OIDC claims (
NETRA_OIDC_NAMESPACES_CLAIM, clusters, tenants) andNETRA_SERVICE_CREDENTIALS_FILEadd namespace/cluster/tenant ACLs on top of viewer/operator/admin. Scoped callers are filtered on observe queries and blocked from admin-only fleet surfaces. Docs:docs/auth-oidc-rbac.md. - Optional SQLite flow history.
NETRA_FLOW_BACKEND=sqlitewrites incremental indexed deltas to<state>.flows.db(retention viaNETRA_FLOW_RETAIN/MAX/MAX_BYTES); JSON sidecar remains default.GET /api/v1/fleet/flows/historyfans out toNETRA_FLEET_PEERS. Docs:docs/flow-log.md. - Failed-connection investigation timeline.
POST /api/v1/investigate/connection(also GET),netractl investigate connection, MCPnetra_investigate_connectioncompose DNS, flows, netlink, policy explain, path/drops and a confidence scorecard with an explicit visibility gap. Observe-only. Docs:docs/investigate-connection.md. - Competitive docs no longer understate sampled HTTP/2/gRPC. Positioning and “Still not done” distinguish always-on L7 metadata from opt-in
NETRA_L7_SAMPLE. Docs:docs/competitive-observability.md.
0.31.0 — 2026-10-17
- CI and deps match the GA chart defaults. Helm CI now expects DNS/scan detect and bounded auto-capture on by default;
golang.org/x/netbumps to v0.60.0 for the HTTP/2 Trailer advisories; toolchain Go 1.27.2 + golangci-lint v2.15.0; govulncheck ignore drops the stale Cilium catch-all and documents GO-2026-6665 (Snowflake logger, no fixed release yet). Agent-image and release publish jobs run on hosted runners while the orgzyvor-buildpool sticks busy with no assigned job. - Diagnose surfaces for APIs that already existed. New console pages: Deny Preview (live + flow-history blast radius), Netlink timeline, BPF Attachments / hook drift, and Detect (DNS + scan findings). Overview Detect links to Detect; Needs action ranks detect findings there. MCP adds
netra_netlink,netra_netlink_findings,netra_ebpf_attachments,netra_listen_queues. - Auto-mitigate shadow preview.
GET /api/v1/ebpf/auto-mitigate/preview(MCPnetra_ebpf_auto_mitigate_preview) lists candidates from scan findings and UDP deltas without mutating store or datapath. Metric anomalies never feed it; live actions still need an enforce lease. - Detector HA warm-start. DNS/scan finding rings persist beside
NETRA_STATE_FILE(.dnsdetect/.scandetect) and reload on leader promote so observe continuity survives failover. Never persists auto-mitigate actions. - Capture session summaries for agents.
GET /api/v1/capture/summaries(MCPnetra_capture_summaries) returns frame/protocol/duration metadata for active and recent sessions — no PCAP bytes. - Flowlog DNS/SNI for historical deny preview. Flow deltas retain best-effort
dnsName/tlsSnijoins from the same agent report; counterfactual history preview can match name-based denies when those fields were observed. - Exe-hash leased deny.
BlockedExeHashes+exehash_pids(capgate-shaped, procmeta-sourced) withPOST /api/v1/ebpf/exehash-denyrequiringX-Netra-Confirm-Risk: high. Enforce only while leased; fails open. - Stack pressure histograms.
GET /api/v1/ebpf/stack-histograms(MCPnetra_ebpf_stack_histograms) buckets softirq/softnet and listen-queue depth from agent reports (not entry→exit softirq latency). Overview adds a notify-only Metrics chapter (anomaly rate + raised alerts). - Helm
agent.server. Optional NodePort/host override forNETRA_SERVERwhen hostNetwork agents cannot reliably reach the in-cluster Service. - Helm Envoy Gateway front door and agent scheduling. Optional Gateway API route with backend TLS policy (tested against Envoy Gateway);
agent.nodeSelectorandagent.affinityfor DaemonSet placement. - K3s/Cilium bootstrap survives host reuse.
ensure-cilium-k3s.shstarts a stopped k3s unit when the binary is already present, and points Cilium at the host apiserver so a LAN route for10.43.0.0/16cannot break the install chicken-and-egg with an x509 unknown-authority error. - GA maturity defaults and historical deny preview. README Maturity labels shipped surfaces as GA (default / optional attach) vs privacy/safety opt-in. Chart defaults turn on observe-only DNS and scan detect, lease-gated auto-mitigate, and bounded auto-capture (
dnsdetect.enabled,scandetect.enabled,automitigate.enabled,alerting.autoCapture.enabled). Sampled L7, TLS plaintext uprobes, Cilium/Hubble and MCP mutations stay opt-in. Deny preview gains a flow-history path (POST /api/v1/ebpf/deny/preview-history,netractl ebpf deny-preview --since, MCPnetra_ebpf_deny_preview_history) wired throughinternal/counterfactualover the existing 7-dayflowlog. Overview shows compact DNS/scan finding counts and a TLSFP link. - Image signature check names the renamed repository. Release images are signed by
zyvorai/zyvor-netra's workflow since the repository rename, so thecosign verifycommand indeploy/README.md(which namedzyvorai/netra) failed; it now matches the signer. Clone commands, Helm charthome/sources, docs-site and README links point atzyvorai/zyvor-netratoo (the Go module path is unchanged). - README maturity table covers 0.30.0. L7 names and query types, per-second metrics, node isolation (loaded but inert until a policy is set) and security review now say what runs by default.
securityCI job tolerates one misfiled advisory, explicitly. GO-2026-6596 (Cilium Gateway API HTTPRoutes, CVE-2026-56742) affects Cilium v1.17–v1.19, but its OSV record also carries a catch-all range that flags every version ofgithub.com/cilium/cilium, so govulncheck reported Netra's use of the flow protobuf types and no upgrade could clear it.scripts/ci-govulncheck.shnow fails on any reachable advisory except those listed with a reason inscripts/govulncheck-ignore.txt, and reports stale entries.web-e2echecks every shipped page again. Its page list lackedmetrics,metric-anomalies,metric-alertsandsecurity-review, so the job failed on its own consistency check since the metrics platform landed.
0.30.1 — 2026-10-15
netra-mcpis published to the MCP Registry.server.json(io.github.zyvorai/netra) andDockerfile.mcpship the stdio server as the signed multi-arch imageghcr.io/zyvorai/netra-mcp; tag pushes publish the registry entry with GitHub OIDC.serverInfo.versionnow reports the release version (it said0.1.0) andscripts/check-version-sync.shgates it,server.jsonand the image label. Mutating tools stay off unlessNETRA_MCP_ALLOW_MUTATIONS=true. Docs
0.30.0 — 2026-10-07
-
Security intelligence and review. Matched DNS response events carry the query type (
dnsQType, two bytes of existing event padding, so the 196-byte event ABI and every earlier offset are unchanged), which turns on the DNS detector'stxt-heavysignal. The threat feed gains canonical indicators, an optional?ttl=,X-Netra-Intel-Revisioncompare-and-set, the last 16 revisions with rollback (GET /api/v1/intel/history,POST /api/v1/intel/rollback/{revision},netractl intel history|rollback), an optional persistent journal (NETRA_INTEL_STATE_PATH) and a leader-only HTTPS refresh (NETRA_INTEL_SOURCE_URL) that keeps the last good revision on any bad download.GET /api/v1/security/optimizerreports equivalent, covered and overlapping deny predicates, andGET /api/v1/security/incidentscorrelates DNS findings, scan findings and feed hits on the same exact workload within ten minutes (netractl security optimizer|incidents, Security → Security Review, read-only MCP toolsnetra_security_optimizer,netra_security_incidents,netra_intel_history). None of it applies a deny, deletes a rule or changes mode; applying a feed still needs a lease andX-Netra-Confirm-Risk: high. Docs -
MCP tool counts corrected.
netra-mcpregisters 189 tools (129 read, 60 mutating); README, capabilities, MCP integration, brochure and website copy said 180 or 183, and one place said 52 mutating. -
L7/DNS programs load on Linux 6.8+ again.
netra_l7_cgroup_egresswas rejected by the verifier on the lab host (Linux 7.0) with "the sequence of 8193 jumps is too complex", in the image built before this change as well, soNETRA_L7=autohad been running those hosts without TLS SNI, HTTP Host, DNS query names or DNS QTYPE. The SNI and Host searches indexed packet memory with their loop counters, so the verifier tracked one state per iteration until its branch queue passed the fixed 8192 limit; with that lifted, the unrolled DNS QNAME decoder alone exceeded the 1,000,000-instruction budget. The parsers inbpf/netra_l7.hnow copy up to 640 payload bytes withbpf_skb_load_bytesinto a work area in the per-CPU scratch map and walk it withbpf_loopcallbacks (nested for name copies), keeping parser state in the map so the verifier treats it as unknown.netra_l7_cgroup_egressnow verifies in about 95,000 instructions andnetra_l7_cgroup_ingressin about 37,000 (Linux 7.0, clang 18 as inDockerfile.agent; clang 21 also loads). Behavior is unchanged: positions searched, name rules and outputs match, the existing parser tests pass on the new API, and new ones cover an invalid SNI candidate before the real one, Host after other headers with tabs, a Host value ending the payload, and a QNAME cut by a compression pointer. The parsers now also see bytes outside the skb's linear area, and the HTTP method is read from the same copy: egress request bytes usually sit outside it, so reading the method from packet memory had missed most requests. Two fixes found while testing live: a DNS response is now matched to its query by the receiving socket's cgroup (bpf_skb_cgroup_id) rather thanbpf_get_current_cgroup_id(), which on ingress is whatever task the softirq interrupted, so no response had ever matched (node-widednsHealthshowed 320 queries and 0 responses); responses, latency, RCODE and QTYPE now arrive. Live on the lab host after deploy:www.wikipedia.orgSNI andneverssl.comGETHost recorded, DNS responses matched (284 on the node within a minute), and TXT responses carrying QTYPE 16 with RCODE 3 for a nonexistent name.bpf_loopneeds Linux 5.17; older kernels drop the L7 programs throughNETRA_L7=autoas before. Docs:docs/l7-metadata.md. -
Per-second metrics platform. Every agent now collects host (CPU, memory, pressure, disks, filesystems), network stack (interfaces, SNMP/netstat, sockstat, softnet, conntrack), cgroup v2 (one chart per pod; per container with
NETRA_METRICS_CGROUP_CONTAINERS=true), comm-only process groups, eBPF datapath and per-workload RED series once a second into a Gorilla-compressed store (internal/tsdb), and streams them to netrad over a replay buffer so a restart or network blip leaves no gap. netrad keeps 1 s for an hour, 1 min for two weeks and 1 h for a year per node. Every dimension gets an unsupervised k-means anomaly model (internal/anomaly, stdlib only), with anomaly rates on every query and a Kolmogorov–Smirnov "what changed here" ranking for a highlighted range.internal/metricalertevaluates Netdata-style rules (lookup, calc/warn/crit expressions, hysteresis, delays, silences, ack) with 46 built in, notifying through the existing channels and never touching enforcement. App collectors cover nginx, Apache, HAProxy, Redis, memcached, Envoy, CoreDNS, etcd and any Prometheus endpoint, with credentials from environment variables only and optional discovery by comm and socket inode (never cmdline). Exporters: Prometheus remote write, OTLP JSON, Graphite. Netra-only:GET /api/v1/metrics/evidencejoins a chart window to flows, drop reasons, TCP/DNS/HTTP boards and captures; AI briefs cite metric alerts and anomalies;GET /api/v1/metrics/fleetrolls up peer clusters. Surfaces:/api/v1/metrics/*,netractl metrics, MCPnetra_metrics_*, console Metrics/Anomalies/Alerts pages. Tests:scripts/ci-metrics-unit.sh(jobgo) andscripts/ci-metrics-veth.sh(jobmetrics-veth-smoke: real collectors and netrad, iperf3 on a veth, anomaly flagged, alert raised, evidence answered), both passing on a live lab host. Docs:docs/metrics.md,docs/metric-alerts.md,docs/anomaly-detection.md,docs/app-collectors.md. -
Kubernetes list calls page, and pod lists skip terminated pods. On a lab cluster with 12,629 pods (over 11,000
Evicted) the pod list was 72 MB and the client cut every response at 8 MiB, so the insights boards,ebpf workloadsandpolicy simulatefailed with "unexpected end of JSON input". Pods, services and events are now listed withlimit/continue.ListPodsandListWorkloadsalso ask the API server forstatus.phase!=Failed,status.phase!=Succeeded: terminated pods cannot own traffic and their stale IPs may belong to a new pod, and on that cluster the list went from 10.7 s per page to 129 pods in under a second.GET /api/v1/podsstill lists every pod. -
Node isolation: a per-node allow-only egress filter, in shadow or enforce. Until now Netra could not say "this node may only talk to these destinations":
allowed_*maps are exceptions to deny rules, default-deny exists only per workload, and observe mode does not count what enforce would drop.bpf/netra_nodeiso.cis a standalone TCX program at the head of each agent interface's egress chain that judges new outbound TCP connections and all UDP against up to 64 CIDR + protocol + port-range rules; shadow counts would-block packets and bytes and the destinations behind them, enforce drops them. It is independent of the cluster mode and ofconfig_map. Replies to inbound connections, flows established before the policy, ICMP/ICMPv6, DHCP and non-first fragments always pass, local UDP services can be exempted by source port (SSH's 22 by default), and the agent always allows its controller. Enforce carries a lease (15 m default, at most 1 h) and falls back to shadow when it lapses, when the controller goes stale, when an apply fails, and on a controller restart (shadow policies persist). Policies are replaced atomically by generation.GET/PUT/DELETE /api/v1/ebpf/node-isolation[/{node}](PUTadmin-only), agentNETRA_NODE_ISOLATION=auto|off|required(Helmagent.nodeIsolation). Ported from FluxVM's pod-policy engine and XDP shield, including its global per-rule match function: the inlined rule scan was rejected by the Linux 7.0 verifier at 1,000,000 instructions, and the ported version verifies in 5,820. Tests:BPF_PROG_TEST_RUNverdicts with FluxVM's prefix cases plus shadow/always-pass/generation-swap cases, and a scratch-veth attach with real UDP, both passing on Linux 7.0 and wired intoscripts/ci-ebpf-tests.sh; unit tests for the ABI, store (lease expiry, restart fail-open) and API. Docs:docs/node-isolation.md. Duvora uses this for its shadow/enforce isolation stages. -
Node isolation attaches where the agent has no interfaces, and coverage reports it. The first live deploy (Linux 7.0, k3s with Cilium) loaded
netra_nodeisobut attached it nowhere: in cgroup mode beside Cilium the agent has no TC interfaces, so the list was empty. It now attaches onNETRA_NODE_ISOLATION_INTERFACES(Helmagent.nodeIsolationInterfaces) when set, else on the agent's interfaces, else on the interfaces of the IPv4 and IPv6 default routes (read from/proc/net/routeand/proc/net/ipv6_route); on that host that iseno8303.netra_nodeiso_egressalso now appears in program health and/api/v1/ebpf/coverage, which previously listed only the main datapath collection. Live test on that host, driven from Duvora: shadow counted SYNs to 1.1.1.1:443 and NTP while traffic flowed; enforce dropped them (and upstream DNS) while SSH, the controller and Duvora stayed reachable; the kill switch and release restored egress. -
Stable controller certificate. The chart minted a new self-signed certificate on every controller pod start, so no client could pin it.
tls.existingSecretserves akubernetes.io/tlsSecret instead, anddeploy-remote.shcreatesnetra-system/netra-tlsonce (SANs as before plus the host IP), reuses it on later deploys, and saves its certificate to~/.netra/tls.crtfor clients such as Duvora (NETRA_TLS_PERSIST=0keeps the per-pod certificate).
0.29.0 — 2026-09-27
- Dashboard and docs site follow one apple.com UX contract (
docs/design/APPLE-UX-CONTRACT.md). The dashboard now defaults to Apple light (white page, flat hairline cards); dark stays one click away, and a storednetra-themechoice still wins, so existing users see no change. Buttons collapse to four looks (blue primary, neutral, tinted state, red destructive) with no!important, metric tiles become a hairline metric band, the login is a single centered composition, and the Workloads "Per page" row no longer overlaps its neighbouring text. New tokens (--surface-1,--hairline-1,--shadow-1/2/3,--focus-ring) and primitives (styles/apple-story.css,components/Toolbar.tsx) are shared;website/src/css/custom.cssmaps Infima to the same values. A full-page audit of all 28 pages in both themes and at 390px (no page errors, no horizontal overflow, no sub-3:1 text) also fixed: unstyled filter inputs on Connections/Workloads, red tab pills on Surfaces (red is now only for "Remove …" chips), stretched checkboxes, oversized empty-state text, ~2.5:1 inlinecode, andPolicies/Topologycrashing on an empty response. Nav data now lives inweb/src/lib/navGroups.tswith a test that every page is reachable from the menu. A second audit against a populated controller (real API shapes, error states) also fixed: cards clipped off the right edge on phones (grid tracks are nowminmax(0, 1fr)), the Diagnostics menu running past short viewports,.card poverriding purpose-built paragraph classes (login error, snapshot time, Ask Netra meta), bright accent text at ~2:1 on light surfaces (new--accent-*-texttokens), stretched severity badges, numbers wrapping mid-digit in metric bands, and raw{"error":…}JSON showing in the UI (api()now unwraps it). Unlabelled inputs on L7, Policies and Firewall gained accessible names, and the phone nav now hints that it scrolls. Card titles are now<h2>(pageh1→ cardh2→ sub-headingh3), so no page skips a heading level; verified on all 28 pages against mock and real data. Incidents ranks each cluster's signals by severity, shows the top four, and expands the rest on request (a 15,896px page became about 4,757px, and open clusters stay open across the 20s refresh). The Policies GitOps hint no longer repeats the controller's identical message. Two more fixes surfaced while closing out the audit: closing the connection-evidence dialog now returns keyboard focus to the "Explain connection" button that opened it (it was captured after the Close button'sautoFocushad already taken focus, so focus fell to the page body; this was also the one browser-smoke assertion that had been failing), and the Ask Netra status line reads "Heuristic engine only · Mutations: never — AI endpoints are read-only" instead of splicing the server's clause after a bare "·" (and no longer prints "undefined" when fields are missing). A first deployment to a fresh K3s host then audited every page against a real agent's data (no page errors, no 5xx, no clipping, no contrast failures); that pass moved the lasth3card title on Workloads ("Unattributed traffic") toh2, raised the compact Explain buttons to 36px, and extended the Congestion "?" glossary touch targets to 36px without changing their look. Thedocs/uxlogin, Overview, Path and Congestion screenshots were re-captured from that deployment (the login page's host is redacted); the Flows, Firewall and Drops images, the policy/lockdown/capture images and the GIFs still show the previous design. No API or routing contract changed.
0.28.5 — 2026-09-22
- Netlink changes now say who made them. The recorder hears every link, address, route and neighbor change over RTNL multicast, but a multicast notification does not name its requester.
bpf/netra_rtnl.cis anfentryonrtnetlink_rcv_msg, which runs synchronously in the requesting task, so the current task is the requester. For each request that modifies state it records the process's comm, pid, cgroup id, the RTM_* message type, the interface the request names (an index: a fixed field for address and neighbor requests andRTA_OIFfor a route; or, for a link request that names its device only by name, which is how modernip link set dev Xworks, theIFLA_IFNAMEand the message flags) and, for a route, its destination (RTA_DSTand prefix length) into a ring buffer; read-only requests (RTM_GET*, dumps) are dropped in the kernel before anything is reserved. It captures no argv or environment and nothing of a message beyond the interface (index or name) and a route's destination (values the recorder already publishes about the change), and only observes; it counts what it drops when the ring is full.internal/rtnlactorjoins each recorded change to the request behind it by message type, interface and time, and every event now carriesoriginand, when a process did it,actor(comm, pid, cgroup, and the pod when the requester is in one). The join is deliberately strict about what it claims: exactly one requester isprobable; several at that moment areambiguousand listed, never picked between; andorigin: kernel(nobody requested it: carrier loss, kernel timers, router advertisements) is claimed only while the sensor was already running, the kernel has not dropped requests around then and the ring still reaches back far enough, otherwise the event is simply left unattributed. Interface matching is what stops a veth's carrier loss being credited to theip link set <peer> downissued at the same instant, and for routes the destination prefix separates two processes adding routes on one interface at the same moment (all found on real kernels: the first version reported two same-interface routes as ambiguous, correctly, and matching on the destination resolves it; a link request that names its device only byIFLA_IFNAMEnow explains only that device, with a create exempt because it also creates a peer; and it credited a veth's carrier loss toip link set <peer> downbecause everyip linkcommand first sends a bare 32-byteRTM_NEWLINKprobe with no index, name or attributes, which a join read as "names no interface" and matched to everything, so the kernel program now records the message length, that probe explains nothing, and a request with attributes but no readable name is uncertain: it names nobody and does not let the change be called the kernel's). Findings name the requester in their message (default route (via 10.0.0.1 on eth0) was removed from the main table by calico-node (pid 42, pod kube-system/calico-node-x), or... went down: eth1 (down) by the kernel (no process requested it), which separates an administrator'sip link set downfrom a pulled cable) and say nothing when the sensor could not tell; a finding built from events with different requesters is not summarised as one. Events are attributed on the copies a report carries, held back about 300 ms so their request record has arrived, and the ring stays raw.NETRA_RTNL_ACTOR=auto|off(Helmagent.rtnlActor); it needs kernel BTF andfentry, and where the kernel cannot run it the report saysunavailablewith the reason and the recorder carries on unattributed. Tests:TestRTNLActoron a real kernel loads the object and changes the network from the test process and from a childip, requires each record to name its requester, a read-only dump to record nothing and nothing to be dropped; and runs the real recorder, sensor and joiner together, requiring the routeipadded to be attributed toip, this process's route to this process, an admin link-down toip, and the veth's carrier loss to the kernel and not to thatip. Unit tests cover the decoder, every window edge, interface mismatch, ambiguity and each situation wherekernelmust not be claimed (scripts/ci-netlink-unit.sh). Limits: the join is by message type, interface, destination and time, so it is a probable match and not proof; two requests for the very same prefix, or two requesters for one non-route object in the same instant, are reported as ambiguous; a multipath route names no single interface and matches any; a link request that names its device by an alternative name is read as nobody's (kernel); the program records only the agent's own network namespace (interface indexes are per namespace, so a pod's request for itsifindex 3must not be credited with a host change on the host'sifindex 3): it reads the requester's namespace with three CO-RE member reads, the agent names the one to record through a one-element map, requests from any other namespace are dropped in the kernel before anything is reserved (which also keeps pod churn out of the ring buffer), and the joiner refuses to match across namespaces as a second guard. An earlier version of these notes claimed only host-namespace matches were used before that was true; this is the change that makes it so. deploy-remote.shcan no longer leave the controller down while helm reports success. Deploying 0.28.4 to the live host took the controller down for about ten minutes. The script builds the images on the host and imports them into k3s's containerd under a fixed tag (ghcr.io/zyvorai/netra:0.27.81), then runshelm upgrade. Kubelet garbage-collects images no pod uses once the disk passes 85%, and the host was at 84%: the freshly imported controller image was collected before its new pod started, the pod stuck inImagePullBackOff("NotFound": the fixed tag is on no registry),helm --waitstill passed, and the script died at a 180 srollout statustimeout, so the agent restart was skipped too. Rolling back would not have helped, because the previous revision uses the same tag. The fix is inscripts/lib/deploy-guards.sh, sourced on the host: the deploy warns when the disk is at 80% or more and refuses at 95% (NETRA_DEPLOY_WARN_DISK_PCT,NETRA_DEPLOY_MAX_DISK_PCT,NETRA_DEPLOY_SKIP_DISK_CHECK=1); builds both images first and imports them only right before helm, so the GC window is seconds instead of the whole agent build; ensures each image is present beforehelm upgradeand before each restart, re-importing the local build if it is missing; and waits for the pods to be Ready (NETRA_DEPLOY_READY_TIMEOUT, default 600 s), detectingImagePullBackOff/ErrImagePull, repairing it (re-import, delete only the stuck pod so it restarts at once) and failing loudly with the pods printed if they never come up. The first live use of the guards found a third, which the offline test could not: the wait was "every matching pod is Ready", but right afterrollout restartthe old pod is still Ready and its replacement does not exist yet, so it returned at once and the deploy reported "NETRA ready" while the new pods were still starting. It now waits forkubectl rollout status(polled in short slices so a stuck pull can be repaired between them), with a regression test that keeps a Ready old pod in view while the rollout is incomplete and requires the wait to keep polling; mutation-checked. Two bugs in the first version of the guards were caught by the new offline test and fixed before it shipped: underset -o pipefailan unreadabledfwould have aborted the deploy despite being meant as a warning, andctr images ls | grep -qcan SIGPIPEctron a long image list and report a present image as missing.scripts/ci-deploy-guards.sh(jobgo) runs the guards against stubbeddf,k3s,podmanandkubectl, reproduces the outage (image GC'd, pod stuck, repaired), and renders the real remote script to check it parses and orders builds, imports, ensure and helm correctly; it is mutation-checked. Not covered: this cannot make a full disk healthy, and a controller image that is genuinely gone from the host's build cache still needs a rebuild.
0.28.4 — 2026-09-21
- BPF attachment inventory: Netra can now tell when its own hooks have silently gone. Netra resolves its interface list once at agent start and attaches with
bpf_link, so a NIC or bond that is deleted and recreated under the same name loses its hooks (a link dies with its device) while the agent's own hook list still says they are attached; the traffic on that interface is then no longer observed and nothing said so.internal/bpfattachasks the kernel what is attached, read-only: XDP from the interface's netlink attributes, TCX in execution order from the kernel's program query, and classiccls_bpffilters from the netlink filter list, keeping only each program's id, kernel name and an owner class (netra,cilium,other). It never reads bytecode or a map and never attaches, detaches or replaces anything, so the Cilium coexistence rules are unchanged.internal/bpfattachdiagcompares it with the agent's hook list:bpf-netra-hook-missingandbpf-xdp-replaced(warning),bpf-no-interface-coverage(info, for an agent withagent.interfacesunset, which sees cgroup-level traffic only) andbpf-inventory-unavailable. It never guesses: no finding when the kernel cannot list TCX (before 6.6), when the query for that interface failed, when the list was capped, or when the controller does not hold the list the summary refers to, and it accepts the kernel's program name in either form, full or truncated to 15 characters (netra_edge_ingressornetra_edge_ingr): a real-kernel test showed current x86_64 and arm64 kernels return the full name, and matching only the truncated form would have reported every long-named Netra program (XDP, edge, capture) as missing. Findings are level-triggered and flow throughinternal/health, so they reach the alert poller, incidents, the AI brief and the SIEM export. The kernel is read every 30 s and the list is sent only when it changed or every 5 minutes (a hash-checked summary otherwise, restored by the controller).NETRA_BPF_ATTACH=auto|off(Helmagent.bpfAttach),GET /api/v1/ebpf/attachments,netractl ebpf attachments,netra_bpf_attach_*gauges (the only label is the finding severity). It reports drift and does not repair it: the fix is still an agent restart. Tests:scripts/ci-bpfattach-unit.sh(jobgo) and, on a real kernel,scripts/ci-ebpf-tests.shattaches Netra's real programs to a scratch veth beside another program (TCX, XDP, a classiccls_bpffilter), detaches a hook, and deletes and recreates the interface; that step has not run in CI yet, so treat the kernel behaviour as unverified until theebpfjob is green. Docs:docs/bpf-attachments.md.
0.28.3 — 2026-09-21
- Netlink findings: the recorder now says what is wrong, and alerts on it.
internal/netlinkdiagderives evidence-backed findings from the recorded changes: default route removed (critical for IPv4), default gateway unreachable (its neighbor entry isFAILED, critical), other failed neighbors, a host uplink down or deleted (critical if it carried the default route), an uplink's MTU changed, and the recorder's own overruns. They are level-triggered, so a finding clears when the latest snapshot shows the network recovered, and a fresh neighbor or link change waits up to 45 s for a snapshot to confirm it. Noise is designed out: only host uplinks (device,bond,team,vlan) can go "down" (pod veths, tunnels and bridges such as libvirt'svirbr0are normal churn),INCOMPLETEis ARP in flight and never a failure, one failed neighbor is info and three or more a warning, and the subject is always the node. They flow throughinternal/health, so they reach the alert poller (sourcehealth, so existing channels and severity filters apply), incidents, the AI brief and the SIEM export with no extra wiring, and two on one node correlate; they do not move the health score.GET /api/v1/netlink/findings(window,node;evaluatedandskippedso silence cannot pass for health),netractl netlink findings, andnetra_netlink_findings{severity}. Messages carry interface names and IPs, never MAC addresses; the evidence, which does, stays in the authenticated API. Tests cover every detector and its false-positive case, including thatlower-layer-downis recognised (the first design matched a string the library never emits), the health integration, and the alert poller end to end (one critical notification, de-duplicated, and none once the route returns). Nothing here says why a change happened.
0.28.2 — 2026-09-21
- Netlink change recorder: which link, address, route or neighbor changed on a node, and when. A read-only RTNL sensor in the agent (
internal/netlinkwatch,NETRA_NETLINK=auto|off, Helmagent.netlink, ring sizeagent.netlinkEventBuffer) records link, address, route (every routing table, not just main) and neighbor changes plus a full snapshot every 30 s and after any loss.GET /api/v1/netlink(view=events|state|all,node,kind,since,limit),netractl netlink state|events, and fixed-cardinalitynetra_netlink_*gauges (the only label is the five-value event kind). It never changes a link, address, route, rule, neighbour or qdisc, loads no eBPF program, and touches no Cilium object. Built from the externalNetra-Netlink-Recorder-v1design, which could not ship as written and was reworked rather than applied (git applyfailed on every modified file, and the new code had these faults): it was never wired into the agent or the report, so it recorded nothing; the netlink library ends a subscription on the first receive error (ENOBUFSin a route storm) and never reopens it, so one storm blinded it until restart, and kernel-side loss was not counted; the link-down check compared againstlowerlayerdownwhile the library sayslower-layer-down, and would have flagged every new veth as critical; the cursor advanced on read so a failed POST lost events; the full snapshot rode every 3 s report; and addresses, routes and neighbors carried an ifindex but no interface name. The recorder now resubscribes (immediately after an overflow, with capped backoff for other faults) and records anoverrunevent; delivery is committed only after the controller accepts the report and the controller dedupes by(epoch, cursor); the snapshot ships only when it changed (or every 5 min); neighbor reachable/stale/probe churn is counted (suppressed) instead of evicting real changes; anddropped,missed,overrunsandresubscribesseparate a quiet node from a lossy timeline. Limits, stated indocs/netlink-recorder.md: host network namespace only, history is controller memory (cleared on restart,historySincemarks the blind spot), and there are no findings, alerts or drop correlation yet. Tests:scripts/ci-netlink-unit.sh(jobgo) andscripts/ci-netlink-veth.sh(jobnetlink-veth-smoke: a real veth in a throwaway namespace, plus a forced overrun). The real-kernel job has not run yet; treat the kernel behaviour as unverified until it is green. kind-lifecycleno longer fails when the repository has no release tag. It skips with a warning annotation (CI setsALLOW_NO_PREVIOUS_TAG=1); a local run stays strict. The install, upgrade and rollback test does not run until a tag exists.- Verified: 0.28.1 installs and runs on a real node, and its UI is complete. The two released images were pulled from
ghcr.ioonto the K3s lab host with the chart from thev0.28.1tag (--reset-then-reuse-values); both running pods reported the exact release digests, the controller and agent reported 0.28.1, the cookie session, 12 read endpoints and/metricspassed, andscripts/check-served-ui.shconfirmed the logo and favicon are served and identical to the repository files. The install script's own digest and agent checks were fixed first: the previous run had silently skipped its digest check. - Docs quote the 0.28.1 digests and signing identity (
deploy/README.md, the website quickstart) and say to use 0.28.1 or later.
0.28.1 — 2026-09-20
- Fix: the 0.28.0 images shipped without the logo and the favicon. The Dockerfile's web stage copied
web/srcbut notweb/public, wherezyvor-logomark.svgandzyvor-mark.svglive, so every image built from it served the app's HTML fallback (HTTP 200) for those two paths: the nav logo was a broken image and the browser tab had no icon. The health probes, the release dry run and the lifecycle test all passed, because an unknown path answers 200.web/publicis now copied, andscripts/check-served-ui.shfetches every file inweb/publicand every asset the served page references from a running controller, failing on a non-200, on the HTML fallback, or on any byte that differs from the repository file. It runs in the release dry run against both the controller image and the thin runtime image, and inkind-lifecycleagainst the upgraded controller. Use 0.28.1; the 0.28.0 images have the defect (the controller itself is unaffected). - Verified: the released 0.28.0 installs and runs on a real node. On the K3s lab host the two released images were pulled from
ghcr.io(not built from source), installed with the chart from thev0.28.0tag and--reset-then-reuse-values, and both running pods reported the exact release digests as theirimageID. The controller reported 0.28.0, the agent reported within seconds (observe mode, 16 hooks), the cookie session hadHttpOnly,SecureandSameSite=Strict, the cookie held no API key, sign-out ended the session, and 12 read endpoints answered 200. The keyless signing certificate namesrelease.yml@refs/tags/v0.28.0and the tagged commit. - Docs: how to cut and install a release.
docs/ci.mdhas a "Cutting a release" section (bump, gate, dry run, tag, verify the registry, install the released images).deploy/README.mdhas "Install from a release": the release digests, the exact cosign identity, and the Helm command that keeps an install's values. The dashboard login doc lists the session cookie's attributes, its 12-hour lifetime, that rotating the API key ends every session, and the one-time migration of an oldnetra-token.docs/social/README.mdlists the social assets and how each is rebuilt. - Fix:
kind-lifecycleno longer upgrades a release to itself. It picked the newest tag whose commit was notHEAD, so any commit on top of a fresh tag (still the tag's version) selected that tag as the "previous" release and tripped the "nothing is being upgraded" assertion. It now picks the newest tag with a different version.
0.28.0 — 2026-09-20
-
First tagged release since 0.27.97, and the first published to ghcr.io. The release workflow was added after
v0.27.97was tagged, so no earlier version has a published image. Versions 0.27.98 to 0.27.111 were built and deployed but never tagged, so this release carries all of them (their entries are below) together with everything under this heading. The minor version moves because dashboard sign-in changed: the browser now holds an HttpOnly session cookie and the bearer token is no longer kept in local or session storage (a leftovernetra-tokenfrom an older UI is migrated to a session on first load and removed). -
Licence: no Apache-2.0 wording remains. The Enterprise Pricing sheets (the five JPEGs, the PDF, and the website copy) described Community as Apache-2.0. They now say what
LICENSEsays: the Zyvor Production License v1.0 is free for evaluation and non-production use, and production use needs a commercial license (Community is shown as free for non-production; the tagline "Open dataplane" became "Source-available dataplane"). The sheets are rebuilt from a new source in the repository,docs/sales/enterprise-pricing/sheets.htmlwithbuild.sh, because the old images and PDF had none. The README and website share card (docs/social/netra-share-card.png) carried an "APACHE 2.0" pill and "v0.14" in its pixels; it now shows "ZYVOR PRODUCTION LICENSE" and v0.28. Apache-2.0 now appears only ingo.mod, as the licence of third-party dependencies. -
CI now runs every use case for real. New jobs, each a script that fails on the first bad assertion and was mutation-checked on real runners:
enforce-veth(a real agent drops exactly what deny/CIDR/port rules name in enforce mode and never in observe, ingress vs egress, IPv6, lease expiry, fail-open when the controller dies),capture-live(both capture backends to a.pcapthat tcpdump reads),mcp-live,persistence-live(state surviveskill -9; an enforcement lease is never resurrected; a corrupt state file is refused and left untouched),sinks-live(webhook, Slack, Teams, HTTP bridge, SMTP, OTLP and syslog to loopback receivers, with signatures, retry, severity filter and no secret leaked),doctor-live,kind-lifecycle(previous release ->helm upgrade --reset-then-reuse-valueswith the agent DaemonSet -> restart ->helm rollback),manifests-kind(the plainkubectl apply -k deploy/path) andweb-e2e(a real browser against a real controller).ci-netractl-live.shnow also runs the mutating commands (146 pass, was 122 with 25 skipped),hack/smoke.shandexamples/siem-export.sh.docs/ci.mdmaps every use case to its job. -
Hygiene gates. gofmt, shellcheck and
go.modtidiness are enforced;golangci-lintreports only issues a change introduces; a pinnedgovulncheckfails on a reachable known vulnerability; CodeQL and dependabot are on. To get there: 17 files were formatted, six shellcheck warnings cleared, and grpc (1.83.1), x/text (0.39.0) and the AWS S3 client (1.97.3) bumped for three vulnerabilities the code could reach. -
Nightly and release dry-run.
nightly.ymlruns the whole suite under-race, 10 minutes of fuzzing, the privileged suites onubuntu-24.04,ubuntu-24.04-armandubuntu-22.04(a 6.8 azure kernel; clang 18 is installed there because the image's clang 14 cannot build the objects), multi-arch image builds and two-replica HA, and keeps one "Nightly CI is failing" issue.release-dryrun.ymlbuilds both images and boots the controller and the thin runtime image on pull requests that could break a release. -
Fix: one failing feed no longer blanks the Firewall page or zeroes the Overview. Found by the new web e2e against a controller with no Kubernetes API. Both pages loaded their feeds with
Promise.all, so a single 502 (/api/v1/ebpf/workloads,/api/v1/insights/summary) discarded every result: the Firewall page showed no rules and never displayed a rule added through it, and every Overview board read 0 while the brief beside them said "1 agents". They now usePromise.allSettled: each feed applies on its own and the first failure is still shown. -
Fix: typing in an investigation filter jumped back to the Overview, and phones scrolled sideways. A Nav click showed the page but left the URL hash empty, while the Connections and Workloads filters and "Copy link" read the page from the hash, so with an empty hash typing in "namespace (exact)" navigated to the Overview and a copied link pointed there. Nav clicks now keep the hash in step (
replaceState: no history entry, filters carried over). Separately, the decorative hero glow reached 8% past its box, so the Overview scrolled sideways at every width (8 px at 390 px, 57 px at 1440 px);mainnow clips horizontal overflow (overflow-x: clip, no scroll container, so sticky elements are unaffected). Both were found byweb/tests/investigation-smoke.cjs, which had gone stale (it waited for an Overview heading a redesign removed, and it runs only on pull requests that touchweb/); it is updated to the current UI and was mutation-checked against both fixes (including a desktop-width assertion). -
Fix: two CI flakes.
TestL7SampleIPv6returned as soon as the first sample arrived and asserted a served request, so on a slower kernel the egress sample could beat the ingress one; it now waits for the serving side.TestDumpIsCorrectWhileOtherListenersChurnassumedinet_diagdumps the listener hash atomically, which it does not (a stable listener can be skipped by one dump under extreme churn); it now asserts the listener is never missing persistently and that misses stay rare, anddocs/listen-queues.mdrecords the limit. -
Product brochure rebuilt (
docs/sales/brochure/, 17 pages, regenerated withbuild.py): how eBPF does it, the eight eBPF objects, a four-part pictorial walkthrough of a traffic-drop incident (detection, capturing the node's packets, triage, leased containment), the newest kernel sensors with measured costs, an honest privacy table, access and export, and how it is tested. It corrects claims that had gone stale: the license (Zyvor Production License v1.0, not Apache-2.0), "no application payloads" (none by default; capture and sampling are opt-in and limited), and the MCP tool count (180, not 133). The DOCX copies were removed because they still carried the old claims. The README license badge and the MCP doc's tool counts were corrected too. The remaining Apache-2.0 / "open source" wording was then fixed too:python/pyproject.toml, the field sheet (HTML and PDF), the Product Perspective deck (PPTX and PDF, regenerated with LibreOffice; only the two affected labels changed), the buyers guide,docs/packetwolf.mdanddocs/dashboard-login.md. Not changed: the Enterprise Pricing sheets, whose packaging is built on an Apache-2.0 "Community" edition and needs a business decision, not a search and replace. -
Docs. README: the MCP tool counts were stale (180 tools: 120 read, 60 mutating), the docs index was missing 38 documents, the repository tree missed the newer packages and objects, and the recent sensors, sinks and access controls are listed. Each feature doc now says which CI job proves it, and
docs/otlp-push.mdanddocs/l7-metadata.mdwere corrected to what CI now shows.
0.27.111 — 2026-09-19
- Mutual TLS between agents and the controller (opt-in, off by default).
NETRA_AGENT_MTLS=optional|requiredwithNETRA_AGENT_CLIENT_CAmakes netrad verify a client certificate from the agent; the agent presentsNETRA_CLIENT_CERT/NETRA_CLIENT_KEY(and can trust a private CA withNETRA_CA_FILE). Nothing changes until a mode is set: the chart renders byte-identical output by default. - It is a second factor, not a replacement. In
requiredmode an agent-only request (report, capture stream, and the agent-key path of/api/v1/ebpf/config) needs the agent key and a certificate signed by the agent CA with the client-auth key usage. A leaked agent key alone no longer works. People are never asked for a certificate: the browser,netractl, API keys, probes and/metricsare untouched, because the listener asks for a certificate and verifies one only if it is offered. - Fails closed, everywhere it can. netrad refuses to start on a typo'd mode, a missing or empty CA file, or
requiredover plain HTTP (a client certificate cannot be requested there, and pretending otherwise would leave an operator believing agents are authenticated). A certificate signed by another CA, expired, or without the client-auth usage fails the handshake. An agent given an unusable certificate refuses to run instead of reporting into a wall. A bad mode that reaches the server anyway meansrequired, notoff. - A staged rollout.
optionalverifies certificates when offered and records it:/api/v1/agentsitems gain"mtls": true(set by the controller from the handshake, never taken from the agent's report) and/metricsgainsnetra_agent_mtls_reports_totalandnetra_agent_mtls_rejected_total, so which agents have moved over is visible beforerequiredis switched on. - Renewal without a restart. The agent re-reads its certificate and key when the files change (cert-manager rewrites them in place), keeps using the previous pair while the two files briefly disagree mid-rotation, and applies the same certificate to the packet-capture websocket.
- Chart.
mtls.mode,mtls.secretName(a Secret withca.crt,tls.crt,tls.key) ormtls.certManager(a Certificate for a CA issuer, ECDSA P-256, renewed automatically). The controller mounts onlyca.crt, never the agents' key; template-time errors for a bad mode,tls.enabled=false, or no certificate source. - What it is not. One certificate is shared by all agents, so it proves "an agent", not which node: a compromised agent can still name another node in its report, exactly as with the shared agent key. Per-node certificates are a larger change. The controller reads its CA at startup; rotating the CA needs a controller restart. See
docs/agent-mtls.md. - Tested against real TLS, with mutation checks. Real listeners and real certificates (handshake success and failure by issuer, key usage, expiry; optional vs required; rotation and half-rotation), API tests over a real TLS listener, agent tests for the HTTP client and the capture websocket, and
scripts/ci-mtls-smoke.sh, which runs the real netrad (all five unsafe settings refused; certificate, issuer, usage and key enforced) and, on Linux, the real agent with a certificate, without one, and with the certificate replaced under it. Six enforcement points and both agent wiring points were mutated and each mutation is caught by a test.
0.27.110 — 2026-09-19
- TLS plaintext sampling: HTTPS operation counts, read where the application hands bytes to OpenSSL. Uprobes on
SSL_write,SSL_readand their_exvariants (bpf/netra_ssl.c) give HTTPS the same bounded counts the packet-level sampler gives cleartext: HTTP method, status and host, and HTTP/2 / gRPC operation names, per role (served / issued).GET /api/v1/l7/tlsand thenetra_tls_sample_*metric family. Off by default and opt-in:NETRA_TLS_UPROBES=auto|required(Helmagent.tlsUprobes,tlsUprobesComms,tlsUprobesGap). New:internal/sslprobe,docs/tls-plaintext.md. - This reverses a documented non-goal, deliberately, with limits. The docs said "no decrypt". That still holds by default and for everything else; this is the one explicit, opt-in exception, and the docs (
competitive-observability.md,p5-surfaces.md,l7-sampling.md) are updated so they no longer contradict it. It is not TLS interception: no keys, certificates or proxy; the application decrypts its own traffic and the probe copies the first 128 bytes it was handed or returned. Those bytes go kernel → agent memory → parser → discarded, never to disk, a log, or the controller. What leaves the agent is an allowlisted operation name and a coarse outcome: never a path, query string, header, cookie, bearer token or body. - Limit what it can see.
NETRA_TLS_UPROBES_COMMS=nginx,envoyrestricts it to named processes, enforced in the kernel before any byte is copied; a real-kernel test proves a process off the list produces no event, and a mutant that removes the check is caught. Enabling it setshostPIDon the agent DaemonSet (to find other workloads' libraries), which the chart renders only when this or process metadata is on; CI asserts a default render has neither. - Tested with secrets, on real kernels. Real HTTPS between real processes on the GitHub runners with secrets planted in the URL path, query string, a cookie and a bearer token: none appears in the API, the agent report,
/metricsor either process's logs, and per-role counts are exact. Verified on Linux 6.17 x86_64 and arm64 (OpenSSL 3.0.13), where one object serves both architectures because the register offsets are written at load time from the running architecture. - Two counting defects found by reading the real output, not by an assertion. The first end-to-end run passed but reported
hosts: localhost GET 30for 15 requests (the host table merged both roles, so every request counted twice) andeligible 89 / emitted 87with the rate limit off (calls skipped by the allowlist were counted as candidates, inflating the scale factor). Host counts are now per role, andeligiblecounts only observed-process calls soeligible = emitted + rateLimited + lostand the scale factor is exactly 1 when nothing is limited. This also corrects the packet-level sampler's host table, which had the same double count. - The whole privileged eBPF suite now runs on real arm64 kernels in CI (
ebpf-arm64), which it never had: the agent image ships for arm64, and the CO-RE drop attribution, the TCP event layouts, listen queues, the packet-level L7 sampler and this feature's register layout were all only ever exercised on x86_64. All of it passes. - Cost, measured (kernel accounting on real runners): about 1.0 µs of program time per probe on x86_64 and 1.4 µs on arm64; about 3 to 4% on an HTTPS request that includes a full handshake, trap included (
TestSSLCostPerCallre-measures it in CI). Long-lived connections making many small calls pay per call; the rate limit limits samples, not probe entries. Documented with the other limits: dynamically linked OpenSSL only (not static builds, Gocrypto/tls, BoringSSL, GnuTLS, NSS or Java/Node bundles). - CI.
scripts/ci-sslprobe-unit.shin thegojob; the privileged script's new TLS step (fails on a skip) runs in bothebpfandebpf-arm64; a newtlssample-smokejob runs the real agent and controller end to end; helm asserts TLS sampling renders off andhostPIDabsent by default. Mutation-checked: inverting the role inference, attaching deleted libraries, leaking the request target into the host field, and removing the in-kernel allowlist each fail a test.
0.27.109 — 2026-09-19
- Sampled L7 protocol observation: which Redis commands, SQL verbs, Kafka APIs and gRPC methods workloads use, and how often they fail. A new standalone
cgroup_skbprogram (bpf/netra_l7sample.c) copies up to 128 payload bytes of TCP segments to or from a configured service port into a ring buffer, rate-limited per flow and direction; the agent classifies them (internal/l7sample) into bounded operation names and discards the bytes.GET /api/v1/l7/sampledandnetra_l7_sample_*metrics. Off by default because it reads application payloads:NETRA_L7_SAMPLE=auto|required(Helmagent.l7Sample,l7SamplePorts,l7SampleGap). - Privacy is enforced by the output, and tested. Every field a parser can return is an allowlisted operation name, a coarse outcome or a validated pattern (Redis command, SQL verb, SQLSTATE class, Kafka API, gRPC
/pkg.Svc/Method, HTTP status): keys, SQL text, topics, paths, cookies and bearer tokens are read only to classify and never appear in a result. Tests plant secrets in every protocol and require they never surface; 200 000 random/mutated inputs and a 20 s CI fuzz require every label be bounded and nothing panic; the end-to-end smoke asserts a planted secret appears nowhere the stack exposes (API, agent report,/metrics, logs). Cardinality is bounded by construction, with caps as a backstop. - Two roles so nothing is counted twice. A request is seen leaving its client and arriving at its server, so counts carry
role=served(this node's services) orrole=issued(its workloads) and must be summed per role, never together. The first end-to-end run showedGETat 60 for 30 sent, which is what prompted this. - Sampling is honest. The kernel counts eligible, emitted, rate-limited and lost segments (they add up);
scaleFactor = eligible/emittedturns sampled counts into estimates, scaled per node. HTTP/2 header compression is stateful, so a header block that references state the sample lacks is reported asundecodable, not guessed; Kafka responses and MySQL login packets are not classified.docs/l7-sampling.mdlists the limits. - The real verifier caught a real bug. The first version was rejected (
R3 unbounded memory access): after a helper call clang recomputed a copy offset from an unbounded value and deleted the guard it could prove redundant from the source. The bounds checks are now compiler-opaque, and short payloads are copied in fixed chunks with an overlapping tail so every length from 1 to 140 bytes is captured exactly, byte for byte, on a real kernel. - Cost, measured: about 29 ns per program run for traffic to a port that is not configured (the common case) and about 65 ns on a configured port with the default rate limit (Linux 6.8 aarch64 VM), per packet in the attached cgroup; re-measured in CI.
- CI.
scripts/ci-l7sample-unit.sh(unit, race, fuzz, minimum test count) in thegojob; the privileged job runs the kernel tests against the real verifier and real TCP and fails on any skip; a newl7sample-smokejob runs the real agent and controller end to end with exact per-role counts; the object ships in the agent image (checked by the image gate); helm asserts sampling renders off by default. Mutation-checked: a compiler-visible break of the chunk copy fails the exactness test, an unbounded label fails the label tests, and a secret flowing into an output field fails the leak test.
0.27.108 — 2026-09-19
- The agent reads its big BPF maps about 15 times faster and allocates about 150 times less. On a live node the agent averaged 194% of a core and 313 MiB. Tracing its
bpf()calls showed the cause was not the newer sensors but its original flow collection: every 3 seconds it walked every entry of five 131 072-entry LRU hash maps (two syscalls per entry), formatted two IP strings for every entry, sorted them all, and kept 1 000. It now reads in batches (BPF_MAP_LOOKUP_BATCH, about a thousand entries per syscall, falling back to iteration on kernels without it) and ranks on raw counters, decoding only the top N (internal/mapscan). On 100 000 entries in each of two flow mapsreadStatsgoes from 275 ms and 253 MB per call to 18.7 ms and 1.6 MB; 60 000 entries take 59 syscalls instead of 120 000. It also stops countingconntrackentry by entry. - The report is byte-for-byte the same. The original loops are kept verbatim as a reference (
legacy_readers_test.go); tests fill real BPF maps of the production shapes, overflowing the 1 000 cap several times, and require the new readers to be deep-equal forreadStats,readTCPHealth,readUDPFlowHealthandreadQUICObserved, plus ties, a missing map, and a map with the wrong layout. They run against a real kernel in the privileged job, which fails if any skip. The one visible difference is an improvement: rows with equal scores now come back in a stable order (the old sort was unstable there). - An agent's collection cost is now visible. Each report carries
mapScans(entries, syscalls, batched or not, milliseconds per map); the controller exportsnetra_agent_map_scan_millis_max,netra_agent_map_scan_entries_maxandnetra_agent_map_scan_unbatched_nodesper map; a read over a second is logged. The problem above was found only by attachingstraceto a production process. Design and limits:docs/agent-map-reads.md. - Mutation-checked: breaking the batch
ENOSPCgrowth path, the top-N tie-break, or copying rows out of the reused scan buffer each fails a test.
0.27.107 — 2026-09-19
- Fix: the agent hung forever on its first report on any quiet host. Each report reads the kernel log (
/dev/kmsg). It opened the device withos.OpenFile(…, O_NONBLOCK), and in Go a pollable file opened that way does not returnEAGAINat the end of the data: the goroutine parks in the runtime poller until more arrives. On a quiet host that is never, so the agent's main loop blocked in its firstsyncAndReport— it never reported, and it could not be stopped by SIGTERM either (the main goroutine was the one stuck). It only escaped when the kernel ring buffer held more than the 256 KiB read cap, which a busy, long-running host does and a fresh node, a kind/dev cluster, a small VM or a CI runner does not; that is why the live cluster's agent was fine. The read now uses raw non-blockingread(2), soEAGAINends the snapshot as it should, tolerates/dev/kmsg'sEPIPE(an overwritten record), and uses the 8 KiB buffer the kernel requires for a whole record. Found becausetlsfp-smokehung a CI job for six hours: its cleanupwaited on an agent that would not die, and/api/v1/agentswas empty because it had never reported. Reproduced in a VM (goroutine dump: main goroutine inkmsg.Snapshot), then verified on Linux: the old code hangs on both a FIFO and the real/dev/kmsg, the new code returns immediately (57 KB of the real log).tlsfp-smoke,http-status-smoke,flow-observe-vethandauto-capture-vethall pass in the VM with it. - Drop attribution cost, measured. About 100 ns per dropped packet by the kernel's own BPF accounting on a 4-vCPU aarch64 VM (Linux 6.8) and about 540 ns on a shared GitHub runner (Linux 6.17, x86_64); 150 to 700 ns added to a send-and-drop loop, the worst case; roughly 1 to 5% of a core at 100k drops per second. On a real node it is about 7% of the agent's map syscalls (traced live); the agent's own older flow maps are the rest.
TestDropInfoCostPerDropre-measures it on every CI run and fails above 50 µs per drop. Seedocs/drop-info.mdfor the caveats (one VM, loopback). - CI can no longer burn hours. Every job now has a
timeout-minutes(it was the 360-minute default), the three veth smoke scripts reap their background processes with a bounded wait and SIGKILL instead ofwaiting on something that ignores SIGTERM, and the privileged job runs the real-/dev/kmsgregression case as root.
0.27.106 — 2026-09-19
- TCP listen-queue pressure: which listener is about to refuse connections. The kernel already counted listen-queue overflows, but not whose. The agent now samples, per TCP listener, the accept-queue depth and the configured backlog, plus the half-open (
SYN_RECV) connections waiting on it, and keeps each listener's peak and a fill-level histogram.GET /api/v1/listen-queues?top=&node=(viewer) andnetra_listen_queue_*metrics (bounded: six fixed histogram bins, no ports as labels). It readsinet_diag, the interface behindss -ltn: no BPF, no BTF, no root, so it works on every kernel, and the entire feature is testable against a real kernel without privilege. "Full" isqueue > backlogas the kernel defines it, so a backlog of N holds N+1; verified on a real kernel with a backlog-1 listener that never accepts (queue 2, max 1, overflows counted).NETRA_LISTEN_QUEUES=auto|off, Helmagent.listenQueues. Deliberately not BPF: the plan called for a kernel histogram, but the data is already exposed per listener, and sampling gives the same answer with none of the verifier, BTF or kernel-version risk. The limit is that it samples: a burst that fills and drains between two samples is not seen (the overflow counters still count it). - Fix: TCP event sensors now work on newer kernels. The first GitHub Actions run (Linux 6.17, x86_64) showed
tcp_send_resethas a different record from 6.8:sport/dport/family/saddr/daddrare gone, replaced by two 28-bytesockaddr_in6-sized blobs. The loader used to refuse it, so the send-reset sensor silently did not run there. It now recognises the blob shape (only whensaddris exactly 28 bytes and there is nosport, never guessing), reads family, big-endian port and address from it, and still refuses anything else with the reason. The layout tests use the real 6.17 format files, and theebpfjob now prints the runner's kernel and tracepoint formats so the next layout change is a labelled failure, not a mystery. - Drop-reason numbers differ between kernels, and the names are already right.
NETFILTER_DROPis 8 on 6.8 and 12 on 6.17 (TCP_RESET35 vs 45,UNHANDLED_PROTO56 vs 74; 94 vs 124 reasons). Names are taken from the running kernel's own table, so drop attribution was correct on both; a test now pins both real tables against each other's numbering so a "simplification" to a fixed table fails. AllTestDropInfo*cases pass on the 6.17 x86_64 runner as well as 6.8 aarch64. - Fix: the controller image no longer breaks the build from a clean checkout. The Dockerfile's web stage did not copy
package-lock.jsonand rannpm install, which floated@novnc/novncto a release whose packageexportsbreak vite. It now copies the lockfile and usesnpm ci; thewebCI job does too. This is what hadkind-e2ered on GitHub, and it would have hit any release build. - CI. New
scripts/ci-listenq-unit.sh(minimum test count per step,-race, real inet_diag) in thegojob; theebpfjob runs the root-only half-open case; helm assertsNETRA_LISTEN_QUEUES. Mutation-checked:queue >= backlog, never recording peaks, and not attributing half-open sockets each fail a test.
0.27.105 — 2026-09-19
- CI: every test is now gated, including the ones that could silently vanish. (1)
scripts/ci-kernel-sensors-unit.shgives the TCP-event and drop-attribution sensors their own unit +-racegate and asserts a minimum test count per step, so a renamed test cannot fall out of a-runfilter unnoticed. (2)scripts/ci-agent-image.sh+ a newagent-imagejob buildDockerfile.agentwith the image's own clang and check what shipped: a BPF object for everybpf/netra_*.c, each a real BPF ELF, and nothing stray — a sensor added underbpf/but not to the Dockerfile used to build, pass every test, and ship an agent that quietly never loaded it. (3)ci-ebpf-tests.shnow compiles everybpf/netra_*.cinstead of a hard-coded list, and adds a mutation guard: the drop-attribution program with its ethertype check removed must fail the non-IP test, proving that test still bites. (4) Thegojob also builds the whole module for darwin/arm64 and linux/arm64 (the Linux-only sensor loaders have stubs elsewhere, and releases ship arm64). (5) Thehelmjob asserts the agent's tracefs mount (read-only, and absent when disabled),NETRA_TCP_EVENTS/NETRA_DROP_INFO, and the 512Mi controller memory limit. Each new gate was itself broken on purpose to confirm it fails: dropping theCOPYof an object, a filter that matches no tests, a render without the mount, and the family-guessing mutant. - Kernel drop attribution: which connection, why, and which kernel function. A new optional agent object,
bpf/netra_dropinfo.c, attaches toskb:kfree_skband records, for every dropped packet, its IP 5-tuple, the drop reason, and the address of the kernel code that dropped it (resolved through/proc/kallsyms, e.g.nft_do_chain [nf_tables],__udp4_lib_rcv). The existing reason counters said a drop rate existed; this says whose packets and where.GET /api/v1/ebpf/drop-info?top=&node=&reason=(viewer) serves totals, per-reason counts, top sites and top flows tagged with their node;/metricsgetsnetra_drop_info_*with the reason as its only label (capped at 30, the rest inother). Helmagent.dropInfo, envNETRA_DROP_INFO=auto|off|required. Aggregation is in the kernel (LRU tables), not a ring buffer. - Reason names come from the running kernel. Drop-reason numbering changes between kernels, so the names are parsed from the tracepoint's own
print fmt(tpformatnow reads__print_symbolictables), not a table baked into Netra. - BTF only for this sensor. The skb is read through three CO-RE-relocated
sk_bufffields against/sys/kernel/btf/vmlinux. The core datapath and the other objects stay BTF-free; a kernel without BTF loses only this feature, and the agent reports why (unavailable) instead of just going quiet. - No invented tuples. A tuple is trusted only when the tracepoint's ethertype and the IP version nibble agree; a non-IP frame, or a packet dropped before its network header was set, is counted as a drop with no tuple. Mutation-checked: making the program guess the family from the bytes fabricates
10.9.8.7 → 6.5.4.3out of a non-IP frame and the test fails. - Verified on a real 6.8 kernel and verifier (Ubuntu 24.04, clang 18): load and CO-RE relocation, a UDP datagram to a closed port attributed to the exact tuple with
NO_SOCKET, an exact count, and__udp4_lib_rcv; annftdrop of a TCP SYN attributed to the connection withNETFILTER_DROP; IPv6; the non-IP case; a kernel without BTF; a layout it cannot read safely. The CIebpfjob runs these (they skip on a runner kernel without BTF). Not yet run in a pod or on a live node, and the per-drop cost has not been measured under a high drop rate; seedocs/drop-info.md.
0.27.104 — 2026-09-19
- TCP event tracepoints: per-connection retransmits, RSTs and the state-transition matrix. A new optional agent object,
bpf/netra_tcpevents.c, attaches four passive tracepoints (tcp:tcp_retransmit_skb,tcp:tcp_send_reset,tcp:tcp_receive_reset,sock:inet_sock_set_state). The existing sockops program reports retransmits only as a count at connection close and counts onlyTCP_CLOSE; this attributes retransmits and RSTs to the tuple that is losing packets now, in both directions, and counts every state transition.GET /api/v1/ebpf/tcp-events(viewer) serves the totals, the matrix and the top flows;/metricsgetsnetra_tcp_events_*andnetra_tcp_state_transitions{from,to}with bounded labels. It is on by default inautomode (NETRA_TCP_EVENTS=auto|off|required) and degrades to "not reporting" rather than failing the agent. - No hard-coded kernel layout. A classic tracepoint's record layout differs between kernels and even between sibling tracepoints of one kernel (on Linux 6.8
tcp_send_resethassportat offset 28,tcp_receive_resetat 16). Instead of a struct that would silently misread on some kernel, the agent reads each tracepoint's ownformatfile (internal/tpformat), checks field names and sizes, and patches the offsets into the program's read-only data before load. A tracepoint whose layout cannot be established is left out with the reason reported; the others still run. - Verified on a real kernel and verifier, not just in unit tests: Ubuntu 24.04 (Linux 6.8) with clang 18, using
bpf/integrationTestTCPEvents*— all four sensors load and attach, transitions are counted, received RSTs and abortive closes attributed to the right tuple, retransmits attributed under annftloss rule, IPv6, missing-field and missing-tracepoint handling. The CIebpfjob now compiles the object, installsnftables, and runs these. Finding recorded indocs/tcp-events.md: on 6.8tcp_send_resetdoes not fire for the kernel's RST reply to a closed port (that shows as a received RST on the dialling side). - Fix: the agent pod could not read tracefs, so the
kfree_skbdrop-reason sensor was silently off on real clusters. Reading a live k3s node's agent from the host showed its/syswas a bare sysfs with no tracefs (the agent log said "kernel drop reason tracepoint unavailable; using stack counters only"). The DaemonSet now mounts the host's/sys/kernel/tracingread-only (Helmagent.hostTracefs, default true; alsodeploy/agent.yaml), which both this sensor and the new TCP event sensors need. On upgraded nodes the drop-reason sensor therefore starts attaching where it did not before. Helm also getsagent.tcpEvents. - Status: the kernel object is verified on a real 6.8 kernel and verifier; the pod-level mount has been rendered (
helm template, on and off) but this release has not yet been rolled out to the live cluster, so end-to-end reporting there is unconfirmed.