Pre-Adoption Report for Model Upgrades FinOps & Governance

Before you switch a workflow to a cheaper or newer model, you usually want more than a vibe check. The mirror-adoption report mirror-tests a candidate model against a sampled fraction of your real, live traffic, has an LLM judge each comparison, and rolls the results up into one report: win/loss/tie counts, a fault-rate delta, a cost delta, and a latency delta.

It does not switch anything for you. This is a signal a human reads before deciding whether to adopt a candidate — not an automatic gate on your routing.

---

How It Works

Steps 1–2 run entirely in the proxy gateway and work the same way whether or not you have a control plane connected. Steps 3–5 need one — judging, durable verdict storage, and the report itself all live in the control plane, not the proxy.

1. Mirror sampling — the proxy gateway (packages/proxy/src/routing/mirror.rs) samples a small, capped fraction of eligible requests. For each sampled request, it serves the real response from the requested model as normal, and — off the request's latency path, after the real response has already been returned — also sends the same request to your configured candidate model.
2. Scrub and publish — the pair (request text + both responses) is DLP-scrubbed and published as a transient message. Nothing here is written to disk yet. A standalone open-core install without a control plane stops here: mirroring still runs and a local counter still tracks outcomes, but nothing judges the pair and there is no report to read.

Limits

Standalone: mirroring bills twice and reports nothing

A standalone open-core proxy (no control plane) still makes the mirrored call — and pays for it — but has nowhere to publish the comparison pair. The proxy logs a warning at startup and another the first time it discards a pair; after that the discards are silent at the default log level. If you run standalone, leave mirror_sample_rate at 0, or run with a control plane.

This feature is useful, but it is not a substitute for real evaluation. Read this section before you make a decision off of it.

• Sent to the candidate's own provider. The mirrored copy goes to the candidate model's provider with that provider's credential, exactly as the request would if the candidate were its model. A candidate the request's format cannot reach as written (an OpenAI model for a Claude Code request, say) or with no credential for its provider is not mirrored, and the skip is counted in the routing_targets_skipped metric with stage="mirror".
• Non-streaming traffic only. The proxy never mirrors a streaming request — the two responses can't be meaningfully compared turn-for-turn, and mirroring a stream would mean fully consuming a second stream nobody asked for just to score it. If your workload streams (most interactive agent harnesses do), it is not covered by this report at all.
• Sampled, not exhaustive. The mirror sample rate is hard-capped at 5% of eligible traffic, regardless of what you configure. This bounds cost — every mirrored request is billed twice — but it also means the report is built from a slice of your traffic, not all of it.
• The judge is an opinion, not a correctness check. The verdict comes from an LLM comparing two responses for correctness, completeness, and instruction-following. It is a real, useful signal, and it is not a ground-truth check — it can be wrong the same way any LLM judgment can be wrong, and it says nothing about wrong-but-well-formed answers that merely look right.
• Verdict-only storage — no response text is ever retained. The only durable trace of a mirrored comparison is the verdict plus numeric metadata (scores, cost, latency, which judge model ran). The request text and both full response bodies exist only transiently, in memory, for the moment it takes to judge them, and are never written to a database, a log line, or anywhere else. If you're evaluating this feature for a privacy- or security-sensitive workload, this is the property that matters most: mirroring a request never creates a durable copy of its contents.
• Cost and latency deltas may come back empty. The report can only compute a cost or latency delta from pairs where both the original call's and the candidate's cost/latency are known. The proxy publishes the original side with every pair; pairs from an older proxy that did not publish it are left out of these two deltas. A null delta means "no pairs yet where both sides were priced," never "zero difference." (Win/loss/tie counts and the fault-rate delta are unaffected by this.)
• The report is platform-wide, not scoped to your workspace's own traffic. A candidate model's adoption signal is deliberately aggregated across every workspace mirroring that same candidate, not filtered to yours — small per-workspace sample sizes would otherwise make the report meaningless. The verdicts it aggregates never contain response text (see above), so this doesn't cross any confidentiality boundary, but it does mean the number you see reflects more than just your own traffic.
• A ratio is refused, not guessed, below a minimum sample size. If a candidate model doesn't yet have enough recorded verdicts, the report says so explicitly rather than computing a win/loss ratio from too few samples. Treat that response as "come back later," not as a bad or a good result.

Supersedes an Earlier, Automatic-Enforcement Design

An earlier design (referred to internally as "C6/C7") called for the proxy to automatically enforce routing decisions per workspace based on a mirror-measured fault-rate delta — i.e., to have the system silently switch or block a model on its own once the numbers crossed a threshold. That automatic-enforcement design was never built, and this feature does not build it either. What ships here supersedes that plan rather than fulfilling it: mirroring now produces a report a human reads, not a control loop that acts on your routing by itself. If per-workspace automatic enforcement is something you need, it remains unbuilt, tracked as a known gap rather than an oversight.

---

Configuration

Whether you're standalone open-core or connected to a control plane, mirroring itself is configured the same way, directly in the proxy's config.yaml — no dashboard or control plane required to turn sampling on:

intutic_settings:
  routing:
    # An explicit, operator-directed candidate to mirror-test — independent
    # of the bandit router's own shadow-mode candidate selection.
    mirror_candidate_model: "gpt-4o-mini"
    # Fraction of eligible, non-streaming traffic to mirror. Clamped to 0.05
    # (5%) regardless of what you set here — every mirrored request is
    # billed twice.
    mirror_sample_rate: 0.05

Both default off (mirror_sample_rate: 0.0, mirror_candidate_model unset) — mirroring costs a second model call on every sampled request, so it is opt-in, never on by default.

Reading the Report