Skip to main content

AI traces, decision router and A/B experiments (Beta)

Three additions to AI Workloads that close the loop from serving to improvement: record what the gateway serves, route each request to the right model, and measure which arm is better.

Everything here sits behind the OpenAI-compatible gateway. Clients keep one base URL and one API key.

1. Trace capture​

Capture is opt-in per endpoint.

fabricctl ai trace enable my-endpoint --token-ids --auto-score finish_stop
OptionMeaning
--sample-rateFraction of requests recorded (default 1.0)
--redactKeep only the request SHA-256, sizes and token counts. No prompt or completion text
--token-idsAdd return_token_ids, logprobs (and stream_options.include_usage for streams) to the upstream request when the client did not set them. vLLM returns the exact token IDs and logprobs the engine used, so training or evaluation never has to re-tokenize text
--auto-scorenon_empty, finish_stop, json_valid or contains:<text>. Failed requests score 0

Each recorded response carries x-fabric-trace-id. A trace holds the request, completion text, token usage, latency, status, finish reason, and (when the backend returns them) prompt/completion token IDs and per-token logprobs. Streaming responses are reassembled from the SSE chunks.

Limits and behavior:

  • Bodies larger than max_body_bytes (default 256 KiB) are not stored. A response larger than four times that keeps status and latency only.
  • The newest max_traces (default 10 000) are kept. Pruning runs every 100 writes.
  • Capture never fails a request; a failed write is logged.
  • WebSocket proxying and the FLUXVM_AI_DRY_RUN canned responses are not traced.
  • Traces can contain user data. Use --redact, and restrict /api/ai/traces (read) and DELETE /api/ai/traces (write) with RBAC.

Score, list, export​

fabricctl ai trace list --endpoint my-endpoint --limit 20
fabricctl ai trace score <trace-id> 1.0 --label good
fabricctl ai trace export --min-score 0.8 --format sft > sft.jsonl # chat fine-tuning rows
fabricctl ai trace export --format raw > traces.jsonl # full traces with token IDs

2. Decision router​

A router is attached to a front-door endpoint (the one API keys are scoped to). Candidates are other InferenceEndpoints. Apply it from JSON:

{
"endpoint": "assistant",
"strategy": "cost",
"min_quality": 0.5,
"hard_quality": 0.8,
"hard_prompt_tokens": 2000,
"candidates": [
{"name": "large", "endpoint": "qwen-72b", "quality": 0.9, "cost_per_mtok_in": 1.2, "cost_per_mtok_out": 3.6},
{"name": "small", "endpoint": "qwen-8b", "quality": 0.6, "cost_per_mtok_in": 0.1, "cost_per_mtok_out": 0.3}
]
}
fabricctl ai router apply router.json
fabricctl ai router decide assistant request.json # dry run, shows the arm and why
StrategyPicks
rulesFirst matching rule (prompt size, tools, keywords in the last user message, model, header), else default
costCheapest candidate with quality >= min_quality that fits max_context. Requests with tools or at least hard_prompt_tokens use hard_quality. If nothing clears the bar, the best-quality arm
splitWeighted A/B split, sticky per sticky_header (default x-user-id); random when the header is absent

Prompt size is estimated as characters / 4. If the chosen arm has no ready backend the next arm (by quality) is used; if none is ready the request gets 503. Responses carry x-fabric-variant. allow_override lets callers force an arm with the x-fabric-variant request header. Keep it off for experiments.

API keys scoped to a model are checked against the front-door endpoint's deployment, not the arm that serves the request.

3. A/B experiments​

Put two arms on a split router with trace capture and an --auto-score (or feedback) on the front-door endpoint, then:

fabricctl ai router report assistant --min-samples 30
fabricctl ai router promote assistant small # 100% of traffic to 'small'

The report gives, per arm: requests, errors, mean score, p50/p95 latency, tokens and estimated cost (from the candidate prices). Each non-control arm is compared with the control (control, else the first candidate) using a Welch z-test (normal approximation) at 5%. It is only run when both arms have at least min_samples scored traces. The recommendation is promote, keep or keep collecting; promoting is always an explicit call.

The test assumes independent samples and roughly continuous scores. For 0/1 scores it is the usual two-proportion approximation. Do not stop the moment a p-value dips below 0.05; fix the sample size first.

REST​

GET/POST /api/ai/trace-policies DELETE /api/ai/trace-policies/{endpoint}
GET /api/ai/traces (endpoint, router, variant, model, since, min_score, max_score, scored, limit)
GET /api/ai/traces/export (same filters, format=raw|sft)
DELETE /api/ai/traces (needs endpoint, router or since)
GET /api/ai/traces/{id}
POST /api/ai/traces/{id}/feedback {"score": 0..1, "labels": []}
GET/POST /api/ai/routers GET/DELETE /api/ai/routers/{endpoint}
POST /api/ai/routers/{endpoint}/decide body = a chat request; no side effects
GET /api/ai/routers/{endpoint}/report (since, min_samples)
POST /api/ai/routers/{endpoint}/promote {"variant": "…"}

Reads need the read role; writes, feedback, purge and promote need write.

Failover​

When the chosen arm cannot be reached (connect error, timeout, or a 5xx status before any body byte is streamed), the gateway retries the next-ranked arm that has a ready backend. Set "failover": false on a router to surface the upstream error instead, and "max_attempts" (default 2) to bound the tries.

The response carries x-fabric-variant (the arm that answered) and x-fabric-failover-from (comma-separated arms that failed first). Each failed attempt is recorded as a trace on the failed arm with its error status, so it shows up in that arm's error rate in the A/B report. Note that failover moves traffic between arms, so a flaky arm skews the split: check the error column before trusting a comparison.

LLM-as-judge scoring​

Deterministic scorers only catch format problems. To grade quality, point a trace policy at another Fabric endpoint that serves a judge model:

fabricctl ai trace enable chat --token-ids --judge judge-endpoint \
--judge-rubric "Score factual accuracy; penalise invented citations."

After each trace is stored, a background task sends the conversation and the answer to the judge and writes score, score_source: "judge" and a one-line judge_reason back to the trace. judge.sample_rate limits how many traces are judged. The judge reply must be JSON {"score": 0..1, "reason": "..."}; code fences and surrounding text are tolerated, out-of-range scores are discarded.

  • Human feedback is never overwritten, even if it arrives while the judge runs.
  • Redacted traces, non-2xx traces and traces without stored text are skipped.
  • Judge calls go straight to the judge endpoint, so they are not traced and do not count against caller API keys.
  • Score existing traces with fabricctl ai trace judge <endpoint> [--rejudge] (POST /api/ai/traces/judge, up to 200 traces, 4 at a time).
  • A judge is a model, not ground truth: spot-check its scores against human feedback before promoting an arm on them.

Bandit routing​

"strategy": "bandit" sends traffic where the scores are best, instead of a fixed split. It needs scored traces (an auto_score or a judge on the trace policy); without scores every arm stays in warm-up and traffic is spread evenly.

  • Each arm gets a Beta posterior from its newest 500 scored traces (a 0..1 score counts as a fractional success). Per request the router draws from every posterior and picks the highest draw (Thompson sampling), so a clearly better arm gets most traffic while a close one still gets enough to be measured.
  • warmup (default 20): arms with fewer scored traces than this are sampled first, uniformly among the cold ones.
  • explore (default 0.1, max 0.5): share of requests sent to a random arm, so a worse-looking arm can still recover.
  • Scores are cached for 10 seconds, so traffic shifts with a short delay.
  • It optimises score only. Cost and latency are not part of the reward; keep cost routing for that, or leave a cheap arm as a candidate you check in the report.

Sequential testing​

The A/B report now shows two p-values per challenger. p_value is the classic fixed-horizon test and is only valid if you look once, at a sample size you chose in advance. p_sequential is an always-valid p-value (mixture SPRT, effect prior sd 0.1): you may check the report as often as you like and stop the moment it drops below 0.05 without inflating false positives. The recommendation uses the sequential test, so "promote" can appear early for a large effect but will not fire on a lucky early peek. Small effects need more samples to clear it.