AI traces, decision router and A/B experiments (Beta)
Three additions to AI Workloads that close the loop from serving to improvement: record what the gateway serves, route each request to the right model, and measure which arm is better.
Everything here sits behind the OpenAI-compatible gateway. Clients keep one base URL and one API key.
1. Trace capture
Capture is opt-in per endpoint.
fabricctl ai trace enable my-endpoint --token-ids --auto-score finish_stop
| Option | Meaning |
|---|---|
--sample-rate | Fraction of requests recorded (default 1.0) |
--redact | Keep only the request SHA-256, sizes and token counts. No prompt or completion text |
--token-ids | Add return_token_ids, logprobs (and stream_options.include_usage for streams) to the upstream request when the client did not set them. vLLM returns the exact token IDs and logprobs the engine used, so training or evaluation never has to re-tokenize text |
--auto-score | non_empty, finish_stop, json_valid or contains:<text>. Failed requests score 0 |
Each recorded response carries x-fabric-trace-id. A trace holds the request,
completion text, token usage, latency, status, finish reason, and (when the
backend returns them) prompt/completion token IDs and per-token logprobs.
Streaming responses are reassembled from the SSE chunks.
Limits and behavior:
- Bodies larger than
max_body_bytes(default 256 KiB) are not stored. A response larger than four times that keeps status and latency only. - The newest
max_traces(default 10 000) are kept. Pruning runs every 100 writes. - Capture never fails a request; a failed write is logged.
- WebSocket proxying and the
FLUXVM_AI_DRY_RUNcanned responses are not traced. - Traces can contain user data. Use
--redact, and restrict/api/ai/traces(read) andDELETE /api/ai/traces(write) with RBAC.
Score, list, export
fabricctl ai trace list --endpoint my-endpoint --limit 20
fabricctl ai trace score <trace-id> 1.0 --label good
fabricctl ai trace export --min-score 0.8 --format sft > sft.jsonl # chat fine-tuning rows
fabricctl ai trace export --format raw > traces.jsonl # full traces with token IDs
2. Decision router
A router is attached to a front-door endpoint (the one API keys are scoped
to). Candidates are other InferenceEndpoints. Apply it from JSON:
{
"endpoint": "assistant",
"strategy": "cost",
"min_quality": 0.5,
"hard_quality": 0.8,
"hard_prompt_tokens": 2000,
"candidates": [
{"name": "large", "endpoint": "qwen-72b", "quality": 0.9, "cost_per_mtok_in": 1.2, "cost_per_mtok_out": 3.6},
{"name": "small", "endpoint": "qwen-8b", "quality": 0.6, "cost_per_mtok_in": 0.1, "cost_per_mtok_out": 0.3}
]
}
fabricctl ai router apply router.json
fabricctl ai router decide assistant request.json # dry run, shows the arm and why
| Strategy | Picks |
|---|---|
rules | First matching rule (prompt size, tools, keywords in the last user message, model, header), else default |
cost | Cheapest candidate with quality >= min_quality that fits max_context. Requests with tools or at least hard_prompt_tokens use hard_quality. If nothing clears the bar, the best-quality arm |
split | Weighted A/B split, sticky per sticky_header (default x-user-id); random when the header is absent |
Prompt size is estimated as characters / 4. If the chosen arm has no ready
backend the next arm (by quality) is used; if none is ready the request gets
503. Responses carry x-fabric-variant. allow_override lets callers force an
arm with the x-fabric-variant request header. Keep it off for experiments.
API keys scoped to a model are checked against the front-door endpoint's
deployment, not the arm that serves the request.
3. A/B experiments
Put two arms on a split router with trace capture and an --auto-score (or
feedback) on the front-door endpoint, then:
fabricctl ai router report assistant --min-samples 30
fabricctl ai router promote assistant small # 100% of traffic to 'small'
The report gives, per arm: requests, errors, mean score, p50/p95 latency,
tokens and estimated cost (from the candidate prices). Each non-control arm is
compared with the control (control, else the first candidate) using a Welch
z-test (normal approximation) at 5%. It is only run when both arms have at
least min_samples scored traces. The recommendation is promote, keep or
keep collecting; promoting is always an explicit call.
The test assumes independent samples and roughly continuous scores. For 0/1 scores it is the usual two-proportion approximation. Do not stop the moment a p-value dips below 0.05; fix the sample size first.
REST
GET/POST /api/ai/trace-policies DELETE /api/ai/trace-policies/{endpoint}
GET /api/ai/traces (endpoint, router, variant, model, since, min_score, max_score, scored, limit)
GET /api/ai/traces/export (same filters, format=raw|sft)
DELETE /api/ai/traces (needs endpoint, router or since)
GET /api/ai/traces/{id}
POST /api/ai/traces/{id}/feedback {"score": 0..1, "labels": []}
GET/POST /api/ai/routers GET/DELETE /api/ai/routers/{endpoint}
POST /api/ai/routers/{endpoint}/decide body = a chat request; no side effects
GET /api/ai/routers/{endpoint}/report (since, min_samples)
POST /api/ai/routers/{endpoint}/promote {"variant": "…"}
Reads need the read role; writes, feedback, purge and promote need write.
Failover
When the chosen arm cannot be reached (connect error, timeout, or a 5xx
status before any body byte is streamed), the gateway retries the next-ranked
arm that has a ready backend. Set "failover": false on a router to surface the
upstream error instead, and "max_attempts" (default 2) to bound the tries.
The response carries x-fabric-variant (the arm that answered) and
x-fabric-failover-from (comma-separated arms that failed first). Each failed
attempt is recorded as a trace on the failed arm with its error status, so it
shows up in that arm's error rate in the A/B report. Note that failover moves
traffic between arms, so a flaky arm skews the split: check the error column
before trusting a comparison.
LLM-as-judge scoring
Deterministic scorers only catch format problems. To grade quality, point a trace policy at another Fabric endpoint that serves a judge model:
fabricctl ai trace enable chat --token-ids --judge judge-endpoint \
--judge-rubric "Score factual accuracy; penalise invented citations."
After each trace is stored, a background task sends the conversation and the
answer to the judge and writes score, score_source: "judge" and a one-line
judge_reason back to the trace. judge.sample_rate limits how many traces are
judged. The judge reply must be JSON {"score": 0..1, "reason": "..."}; code
fences and surrounding text are tolerated, out-of-range scores are discarded.
- Human feedback is never overwritten, even if it arrives while the judge runs.
- Redacted traces, non-2xx traces and traces without stored text are skipped.
- Judge calls go straight to the judge endpoint, so they are not traced and do not count against caller API keys.
- Score existing traces with
fabricctl ai trace judge <endpoint> [--rejudge](POST /api/ai/traces/judge, up to 200 traces, 4 at a time). - A judge is a model, not ground truth: spot-check its scores against human feedback before promoting an arm on them.
Bandit routing
"strategy": "bandit" sends traffic where the scores are best, instead of a
fixed split. It needs scored traces (an auto_score or a judge on the trace
policy); without scores every arm stays in warm-up and traffic is spread evenly.
- Each arm gets a Beta posterior from its newest 500 scored traces (a 0..1 score counts as a fractional success). Per request the router draws from every posterior and picks the highest draw (Thompson sampling), so a clearly better arm gets most traffic while a close one still gets enough to be measured.
warmup(default 20): arms with fewer scored traces than this are sampled first, uniformly among the cold ones.explore(default 0.1, max 0.5): share of requests sent to a random arm, so a worse-looking arm can still recover.- Scores are cached for 10 seconds, so traffic shifts with a short delay.
- It optimises score only. Cost and latency are not part of the reward; keep
costrouting for that, or leave a cheap arm as a candidate you check in the report.
Sequential testing
The A/B report now shows two p-values per challenger. p_value is the classic
fixed-horizon test and is only valid if you look once, at a sample size you
chose in advance. p_sequential is an always-valid p-value (mixture SPRT, effect
prior sd 0.1): you may check the report as often as you like and stop the moment
it drops below 0.05 without inflating false positives. The recommendation uses
the sequential test, so "promote" can appear early for a large effect but will
not fire on a lucky early peek. Small effects need more samples to clear it.