It is 3am. The pager is screaming because your model just started routing Hindi voice notes about doctor appointments into the wrong tool, and your dashboard still says accuracy is 99%. The dashboard is lying. Public LLM benchmarks are theater, your golden set is the product, and if you do not own one you have no idea what you shipped.
The methodology below is what MemX executes before any model change reaches a single user, as of May 2026. 103 cases, 15 categories, 5 runs each, real token counts pulled from `usage_metadata` (not estimates), a non-negotiable flap-rate metric, and a feature-flag rollout that can be killed in 90 seconds. Regressions caught this way would have shipped silently otherwise. Copy the whole thing.
Quick takeaways. Public LLM benchmarks measure capability, not your product. The 2026 practitioner consensus (Hamel Husain, Shreya Shankar, Eugene Yan, Bryan Bischof, the O'Reilly Year-of-LLMs essay) converges on the same shape: hand-curated golden set, multi-run flap detection, per-category accuracy, latency percentiles, real-cost-per-case, CI-gated regressions, feature-flag rollout. This post is the seven-step version we actually use.
Why public benchmarks lie to your roadmap
MMLU is a roughly 16,000-question multiple-choice test set across 57 subjects, released in 2020. GPQA is 448 graduate-level science questions written by domain PhDs. SWE-bench Verified is 500 human-validated GitHub issues curated by OpenAI with the original authors. All useful. None of them are your product. None tell you whether the model will correctly route a Hindi voice note about a doctor appointment to `save_memory` instead of `search_memories` on the third retry while the load balancer is hot.
Public benchmarks measure capability. Your golden set measures product. Only one of those will ship.
Step 1: hand-curate 50 to 150 real traces
Pull from production logs. Do not synthesise yet. Hamel Husain's canonical 2024 essay states the rule plainly: "You must remove all friction from the process of looking at data." The first dataset you build will catch 80% of the regressions you see for the next year, so make it real, make it diverse, make it actually cover the categories your product serves.
Our golden set has 103 cases across 15 categories: `save_memory`, `search_memories`, `general_chat`, `edge_case`, `multilingual`, `mixed_intent`, `language_bias`, `follow_up_reminder`, `context_disambiguation`, `meta_conversation`, `modify_memory`, `multi_action`, `recurring`, `ambiguous_save`, `intent_based`. The category split matters more than the count. A 1,000-case eval that is 900 `save_memory` cases tells you nothing about ambiguity.
Step 2: schema every case for what an LLM cannot judge
Every case carries four required fields plus one that no public eval framework requires: `user_impact`.
- `input`: the actual user message.
- `expected_actions[]`: the array of correct actions (multi-action cases have more than one).
- `also_acceptable[]`: actions that are not the canonical answer but should not count as failures (a search that returns help text is degraded, not broken).
- `category`: which family this case belongs to (drives per-category accuracy).
- `user_impact`: one English sentence describing what happens to the user if this case fails. "Meaningless input saved as memory; clutters storage." "User query not answered; memory duplicated instead." If you cannot name the consequence, the test is not worth running.
Here is what most eval write-ups will not tell you. The `user_impact` column is the single most original beat in the MemX methodology, and none of the cited public eval frameworks require it. Adding it changes how failures get triaged. Every CI regression comment now includes the human consequence, which means PMs read it and engineers cannot dismiss it as noise.
Step 3: run every case 3 to 5 times, log a flap rate
Temperature 0 does not mean deterministic. Thinking Machines Lab sampled 1,000 identical completions from Qwen3-235B at temperature 0 in September 2025 and got 80 unique outputs; the runs diverged at the 103rd token. Horace He's team traced the root cause to batch-size variability on shared inference servers: changing batch composition changes reduction order in matmul and attention kernels, and floating-point addition is not associative.
The discipline. Run every case five times per candidate model. Flag a case as flapping if the five runs produce more than one distinct outcome. Report flap rate as a first-class metric alongside accuracy. In the MemX March 2026 benchmark, Gemini 3 Flash single-prompt flapped on 29 of 103 cases; the MemX production two-stage configuration flapped on 3. A model that is 99% accurate at single-shot but flaps every fifth run is not actually 99% accurate. It sits somewhere between 95% and 99% with a hidden tax.
Step 4: measure seven things on every run
| Metric | Why it matters | Our gate |
|---|---|---|
| Mean accuracy + variance range | A point estimate hides flapping. Report mean across 5 runs plus the high-low range. | Mean above 95% with stdev under 2pp |
| Per-category accuracy | Aggregate accuracy hides regressions in narrow categories (e.g., a 100% to 75% drop on multilingual). | No category below 90% |
| P50 / P90 / P99 latency | P50 is what users feel on average. P99 is what users feel under stress. Fallback paths win or lose at P99. | P99 under 5s for fallback |
| Schema validity % | If the output cannot be parsed, accuracy is zero regardless of the answer. | 100% required to ship |
| Real $/case from `usage_metadata` | Never estimate. Token counts from the API response are the only source of truth. | Within plus/minus 10% of baseline |
| Method distribution | % of cases routed to two-stage vs single-call vs fallback. Routing drift is the first signal of prompt degradation. | Watch for shift above 5pp |
| Flap rate | Cases where 5 runs produced more than one outcome. Hard floor on accuracy reliability. | Under 5 flapping per 100 |
Step 5: build a decision matrix per axis, not overall
There is no best model. There is only the best model for this surface. Our pipeline routes different stages to different models because the answer to "which model wins" is different per axis. Quality, latency, cost, consistency, and complexity each get their own column. The winner per column is sometimes the same. Usually it is not.
Anthropic's own "Choosing a model" doc says it cleanly: begin implementation with Claude Haiku 4.5, test the use case thoroughly, evaluate whether performance meets requirements, and upgrade only if a specific capability gap forces you to. That is the right default. Our extension: do not pick one model for the whole product. Pick a model per stage of the pipeline.
Step 6: when 4 models fail the same case, fix the prompt
The highest-ROI signal in the rig is the cross-model failure pattern. In our January 2026 benchmark, the input `12345` was misclassified as `save_memory` by all four tested models (gemini-2.0-flash, gemini-2.5-flash-lite, gpt-4o-mini, gpt-4.1-nano) on 3 of 3 runs each. That is not a bug in any one model. That is RLHF's instruction-following bias defaulting to action over abstention.
When you see a cross-model failure, do not blame the model. Add an explicit `unknown` class to your prompt. Stack an input-validation pre-filter in front of it. Drop a tiny FastText classifier ahead of the LLM. Tomani et al. (2024) found that uncertainty-based abstention improves correctness by 2 to 8% and avoids 50% of hallucinations on unanswerable questions, with safety gains of 70 to 99%. The win is architectural, not vendor-shaped.
Step 7: ship behind a feature flag, regress-test in CI
PostHog. Statsig. LaunchDarkly. Vercel AI Gateway. Pick one. The gradual rollout matters. 1% to 10% to 50% to 100%, with live regression detection at every step. MemX has killed three model rollouts in the past six months because production metrics drifted within the first hour at 10%. If you cannot kill a model deploy in under two minutes, you cannot deploy models safely.
The CI gate is the other half. A GitHub Action runs the golden set on every PR, posts a pass/fail comment with deltas vs baseline, and blocks merge on a regression in any category. The eval itself runs in the low tens of dollars per pass with a Haiku-class judge on a 100-case set; call it a few thousand dollars per year at weekly cadence. That is cheaper than one engineer-day of regression debugging.
What this rig has killed in production
Catches from the last six months, briefly. We caught a Gemini 3 Flash single-prompt eval scoring 73.4% on classification before it shipped (Flash-Lite two-stage was at 99.4%). We caught a prompt rewrite that lifted English accuracy from 96% to 98% but tanked Hindi from 100% to 75%; the per-category split surfaced it in four minutes. We caught a schema regression on GPT-4.1 nano that was returning 85.8% valid JSON before we normalised the response parser, then 100% after.
None of those catches needed PhD-level eval infrastructure. They needed a golden set, multi-run scoring, per-category breakdown, and a CI gate that comments on every PR. The whole rig fits in roughly 800 lines of Python.
Show your scars, not your slides. Anyone can publish marketing numbers from a single benchmark run. What earns engineering trust is publishing your methodology, your flap rates, and the failures you caught before they hurt anyone. The eval is the product.
Key takeaway. Stop trusting public benchmarks for production decisions. Hand-curate a golden set with user-impact annotations, run every case 5 times to measure flapping, gate model changes behind feature flags, and treat the eval suite as a first-class part of the product. Hamel and Shreya's Maven course is the canonical curriculum. This post is the 90-minute version.
01Why are LLMs not deterministic at temperature 0?
Temperature 0 means argmax sampling from the logit distribution; it does not make the logit distribution itself deterministic. Variable batch sizes on shared inference servers change reduction order in matmul and attention kernels, and floating-point addition is not associative. The Thinking Machines Lab 2025 study traced this in detail: 1,000 identical Qwen3-235B prompts at temperature 0 produced 80 unique completions.
02How big does a golden set need to be?
50 to 150 hand-curated cases beats 10,000 synthesised ones for catching production regressions. Hamel Husain's playbook and Shreya Shankar's research converge on this range. Category diversity matters more than count. A 100-case set across 15 categories outperforms a 1,000-case set across 3.
03What is the difference between LLM benchmarking and LLM evaluation?
Benchmarking measures capability on shared public datasets (MMLU, GPQA, SWE-bench). Evaluation measures your product on your data. Public benchmarks are useful for narrowing the candidate list. Production evaluation is the only thing that catches regressions in your actual workflow.
04How much does a production LLM eval suite cost to run?
A small-judge model evaluating a 100-case golden set runs in the low tens of dollars per pass; a few thousand dollars per year at weekly cadence. Total LLMOps cost is typically several times raw API spend once you add infra, observability, and human review. The eval itself is the cheapest insurance you can buy against regression debugging.
05Which LLM eval framework should I use in 2026?
Stanford HELM enters maintenance mode on June 1, 2026, so the academic baseline is freezing. For production, the practical shortlist is Promptfoo (YAML, red-team gate), DeepEval (Python, pytest-style, 50+ built-in metrics), Inspect AI (UK AISI, 200+ pre-built evaluations), or rolling your own in roughly 800 lines of Python. Most production teams pair two.
06How do you handle non-determinism in CI eval gates?
Run each case N=3 to 5 times, report mean accuracy with the high-low range, and track a separate "flap rate" metric for cases where the runs disagree. Block merges on regressions in mean accuracy or on flap-rate increases above your threshold. Treat the flap rate as the eval's signal-to-noise floor.
07Should I use LLM-as-judge or human review for grading?
Use both. LLM-as-judge (Haiku-class or GPT-4o-mini-class models) handles bulk grading at low cost and correlates well with humans on schema, format, and routing tasks. Human review stays in the loop for ambiguous cases, calibration drift checks, and any time the judge and the unit test disagree. Shreya Shankar's EvalGen work is the reference on aligning judges with human criteria.
