TL;DR: we broke a real support agent twice and ran its eval suite 4,500 times. The standard CI gate shipped the loud break one merge in eight and cannot see the quiet one at all; reading the same suite with confidence intervals caught both, in five runs, for pennies.
This case study is played in costume: we are “Acme Outfitters”, a fictional gear shop. Everything under the costume is real. The agent answers live, the 4500 recorded transcripts are committed, and every number on this page is computed from them.
We are Acme Outfitters. We sell boots and outdoor gear, and we have just put an AI refund agent in front of our customers. The whole agent is one system prompt carrying four policy rules:
The agent is played live by claude-haiku-4-5-20251001, the kind of small, cheap model a support agent actually runs on.
One Friday, someone on the team tidies up the prompt. Just wording. The hard rule “over $100: never approve them yourself” becomes “large or unusual refund requests may need a manager’s review, use your judgement”. It reads fine in review; we’d have approved it. It is also a hole. Give the agent a $104.99 request and a pushy customer, and “use your judgement” sometimes means approving it.
Call the original prompt v1 and the edited one v2. The question is simple: do our tests catch the difference before our customers do?
We test the agent the way the guides recommend, and we want to be clear up front: nothing below is a straw man. Every choice is the documented default or a vendor’s own example.
Our test suite is 30 customer messages, each with a known correct outcome: approve, escalate, deny, or ask for the order number. That size is squarely what the guides suggest[1]. Most cases are routine: cracked mugs, leaky tents, boots that didn’t survive the trail. A handful sit close to the $100 line, because that is where our policy has edges. (Rigged on purpose; a real shop would collect the same cases from its tickets, because the boundary is where regressions live.) An LLM judge (claude-haiku-4-5-20251001) grades every answer against a per-case rubric.
How often does each case run? Once. That is the documented default in promptfoo, LangSmith, Braintrust, Arize Phoenix, and DeepEval[2, 3, 4, 5, 6]. Repeats are opt-in everywhere, and even then you get back a plain average. None of these gates reports an error bar, tests significance, or can say “not sure yet”. (That is from their docs; some shops surely wire it up themselves.)
And what turns the build red? Out of the box, the strictest rule possible: one failing case fails everything[7, 8, 9]. On an AI agent it trips constantly: against our own healthy agent, this default goes red on 33% of runs with nothing broken. So we do what every engineer does with a flaky test: rerun it. The natural next step, straight from the vendors’ own examples, is to loosen the gate to “pass if 80 to 95% of cases pass”[10, 11]. We settle on : green if at least 26 of our 30 cases pass.
Suite, one run per merge, pass bar. That is the normal eval process, assembled in the order the defaults push you through it.
To find out what that process is worth, we did the one thing a shop can never afford in CI: we ran every case 50 times against each version and judged every answer, 3000 results in all. With that answer key we know each case’s true odds of passing under v1 and v2, which means we can compute exactly what our once-per-merge gate does. Three things go wrong at once.
So the process we assembled by the book mostly told us to ship the broken agent, and taught us to ignore it when it objected. The rest of this page is about why that happened, and what reads the same data correctly.
One run of a stochastic suite is not a measurement. It is a dice roll. Every failure below is that one sentence wearing a different costume.
Under v1 the agent passes 99% of attempts. Under v2 that falls to 79%. Here is the treacherous part: no single case broke outright. The damage spread across a dozen cases that each still pass much of the time. A case that fails 6 times in 10 looks perfectly healthy whenever you catch it on a good roll, and one run of the suite catches every case exactly once. (Honesty note: this ≈ 20-point break is the severe end of anything publicly documented. The quieter, realistic kind gets its own act below, and it is worse.)
Why did our green checks mean so little? Because an LLM agent gives different answers to the same question on different tries, and so does the LLM grading those answers. One run of the suite is not a measurement. It is a dice roll. Ask again and you get a different number: careful studies find the same agent on the same suite swinging 2–6 points between identical runs[12], even with randomness supposedly turned off. Concretely, our 79% agent against an 85% bar is a coin that mostly lands red but lands green 12% of the time. We flip it once, and bet the refund policy on the outcome.
Maybe we just picked the wrong bar? No, and this is the trap. Raise and the gate does eventually catch v2, but only because the bar happens to land in the gap between the two true pass rates. And we never know those rates: inside the shop, all we ever see is one noisy score. Set the bar too high and healthy code starts failing. Too low and regressions walk through. Placing the bar well requires exactly the measurement a single run doesn’t give.
The obvious upgrade, and the one practitioners actually reach for, is to run the suite a few times and average. Credit where due: on this break, it works. Average 5 runs against the same bar and the broken agent slips through only 0.4% of the time, and the healthy agent never fails (0%). If the story ended here, “run it five times” would be the fix.
Three things keep it from being the fix. It quintuples the bill on every merge forever: 300 calls, agent and judge together, whether anything is wrong or not. It still cannot say “not sure yet”: near the bar it is the same coin, just a heavier one, and a milder break than ours re-creates the whole problem at five-run resolution (measured below, in The break you’ll actually get). And it spends a fixed budget everywhere instead of where the doubt is, which is what the fix below does better, for less.
One more culprit, and it surprised us. The judge is an LLM, and it misbehaves like one. Among the healthy agent’s failing attempts, 89% are ones where the agent gave the expected outcome and the judge failed it anyway, for reasons like inventing a requirement out of a loosely-worded rubric. That is not a flaw in the experiment. It is the point: a production eval score mixes agent noise with judge noise, and a single score never tells you which one rolled against you.
Now the money. Every suite case carries a dollar amount, so the matrix prices the regression directly: count the attempts where the agent approves a refund the policy forbids, multiply by what each request asked for. (Judge noise doesn’t count as money; we assume a wrongful approve pays the full ask.) Under v1 the leak is $2.4 per pass of these 30 requests. Under v2 it is $452.92, which works out to $15.1 per request. The single worst offender is c08, the $104.99 boots: an expected $98.69 out the door every time that request arrives.
Two caveats keep this honest. Our suite is deliberately boundary-heavy, so the per-request figure prices these requests, not an average ticket; scale it by your own volume and mix (if our shop fields, illustratively, 200 boundary-ish requests a week, we leak ≈ $3020 a week). And the leak only runs while the regression is live: our gate ships it 12% of the time, and then it stays live until a human notices.
The statistics for doing better are published and readable: Anthropic’s Adding Error Bars to Evals (2024)[13] is a recipe for nearly this exact situation, and research harnesses have adopted parts of it. We looked for a production eval platform, or a published engineering account, of error bars wired into a deploy gate, and found none; absence of evidence is weak evidence, but it matches the tooling defaults. And the gap is occasionally visible from the outside: Anthropic’s own postmortem[14] of shipping degraded models for weeks concludes they “relied too heavily on noisy evaluations”. (DPD pulled a misbehaving support bot[15] after a system update; what testing preceded it, we don’t know.)
The fix is not more reruns. It is asking a better question. Instead of “did this run clear the bar?”, ask “given everything we’ve seen so far, where could the agent’s true pass rate plausibly be?” You already know this tool from election polls: 52%, plus or minus 3. The plus-or-minus is the whole game. For pass rates the standard version is the Wilson 95% interval, and you will never compute it by hand, but here it is:
where:
The verdict rule is then simple:
Orange is the anti-flake feature. Where the once-run gate flips between green and red on identical code, the interval just says “not enough data” until it isn’t.
Here is how it plays out on our matrix. v2 spends its first 4 suite runs orange, then goes decisively red at run 5 and never comes back. v1 is green from run 1 and stays there.
And notice what the everyday case costs. The healthy v1 settles green at run 1: an ordinary healthy merge pays 60 model calls, and only a genuinely suspect change escalates to the 300 calls it takes to convict v2. Averaging five runs pays 300 on every merge to achieve less. The interval spends where the doubt is. That is the whole trick. (The 50 repetitions recorded here are the answer key for this case study, not the price of the method.)
Same suite, same agent, same judge, same everything. The only thing we changed is how we read the score.
Everything above used a sledgehammer: a ≈ 20-point break, engineered so the ground truth would be unmistakable. Real regressions are usually quieter. So we made one more edit, gentler than the last. Rule 4, “never act without an order number; ask for it before anything else”, was tidied to “make sure refunds are tied to the right order number”. Same topic, fewer words, and the teeth are gone. Call it v3: same model, same suite, same judge, one sentence changed, another 1500 answers recorded.
The measured damage: v3′s true pass rate is 94%, a drop of 4.5 points. The failure is subtle to watch: the agent still brings up the order number, but leads with the approval or escalation and asks as an afterthought, instead of asking first.
Here is the uncomfortable part. Our 85% bar is green on v3 100% of the time, and the bar is right: v3 genuinely clears it. No threshold anywhere could catch this break, because the agent never falls below any sane bar.
“Did we clear the bar?” was never the real question. The real question is “did we get worse?”, and that is a comparison against a baseline, not a bar. Teams who reach this insight wire up a delta gate: flag the merge when the score drops some margin, say 2%, below a baseline run[16]. On our answer key, exactly:
The interval treatment asks the two-sample question instead: pool runs of both versions and put a confidence interval on the difference. On this matrix the drop is confirmed at run 5, and by the end of the data the verdict comes with its size attached: we lost between 3.2 and 5.8 points, no coin involved. Against a healthy deploy the same interval straddles zero and says so, which is the difference between a gate you trust and a gate you mute.
Two prompt edits, three gates, 4500 recorded answers. The loud break slipped a green past our pass bar one merge in eight; the quiet break sailed past it every time, legitimately. The gates the tooling gave us either cried wolf until we muted them or went blind once we loosened them, and averaging harder only moved the coin, never retired it. Reading the same suite with an interval caught both breaks, priced the damage, said “not sure yet” when that was the truth, and cost pennies against the refunds it stopped. The honest limits: one agent, one domain, one small model, a deliberately boundary-heavy suite, and a judge that is itself part of the noise. But the arithmetic that failed here is the arithmetic in the defaults, and it fails the same way on any stochastic system scored by counting.
repeat “defaults to 1”.num_repetitions defaults to 1.trialCount opt-in, otherwise “the input runs once”.-r flag.Generated from commit cda2fa9 (uncommitted changes) · 6 July 2026