An LLM-as-a-judge, one model grading another model's output, does not score neutrally. The same model you use to grade outputs gives higher marks to responses that are longer, listed first, and written in a style it recognizes as its own. Zheng et al. named the three core failure modes in the paper that defined the field: position bias, verbosity bias, and self-enhancement bias. Across MT-Bench and Chatbot Arena, a strong judge agrees with human raters more than 80% of the time, the same rate at which two humans agree. That headline number is real. It also hides that reordering two answers or padding one with filler can flip the same judge. The 80% figure earns the judge real trust, which is exactly why the failure modes matter: a metric people lean on for production decisions can be steered by formatting that has nothing to do with answer quality.
Your judge can agree with humans 80% of the time and still be reversible by swapping which answer you paste first.
Each named bias below is paired with the one control that neutralizes it and the primary source that measured it. No eval expertise assumed.
Read the table. Copy the control. Ship a judge you can trust.
What LLM-as-a-judge is
BLEU and ROUGE measure overlap with a reference answer, not whether an open-ended response is helpful, grounded, or correct. LLM-as-a-judge means using one language model to score or compare the outputs of another, instead of relying on string-match metrics or human labelers. Reference overlap punishes a correct answer worded differently from the gold response and rewards a wrong answer that happens to share its vocabulary, which is why teams reach for a judge model in the first place. A judge model reads the question and the answer and returns a rating or a pairwise winner. The Hugging Face cookbook frames the workflow plainly: build a small human-labeled set first, then measure how closely your judge tracks it before you trust it on anything else.
The two common modes are pairwise (which of A or B is better) and pointwise (score this answer 1 to 5). Pairwise is more reliable because relative judgments are easier than absolute ones, but it is also where position bias lives. Pointwise avoids ordering effects and introduces scale-calibration problems instead. Neither mode is bias-free out of the box.
Bias 1: Position bias (the judge prefers whatever comes first)
Position bias is the tendency of a judge to favor the answer shown in a particular slot, usually the first one, regardless of content. In the MT-Bench study, even the strongest judge gave a consistent verdict only 65.0% of the time when the two answers were swapped: present A then B, present B then A, and it picked the same answer both ways only two times in three. Weaker judges were worse. The mechanism is simple to picture. The judge reads two answers in sequence, and the first one anchors its expectation of what a good answer looks like, so the second is graded against that anchor rather than on its own merits.
Swap the order of two answers and a leading judge changes its verdict roughly one time in three.
The fix: swap and require agreement
Run every pairwise comparison twice with the order flipped, and only declare a winner if the same answer wins in both orders. If the two runs disagree, call it a tie. This is the conservative protocol the original authors recommend, and it converts an order-sensitive guess into a stable result. The cost is one extra inference call per comparison, which is cheap next to the alternative: a leaderboard that reshuffles itself depending on which answer you happened to paste into slot one. For pointwise scoring, the equivalent control is to shuffle the order in which items are scored so no single answer always sits at the top of the batch.
Bias 2: Verbosity bias (longer wins, even when it says nothing new)
Verbosity bias is the judge rewarding length over quality. The MT-Bench authors tested it with a deliberately rigged setup they called the repetitive list attack: they took 23 model answers containing numbered lists, had GPT-4 rephrase each list without adding any new information, and prepended the padded copy to the original. Same content, more words. Two of the three judges fell for it almost every time, picking the bloated version in 91.3% of cases. Only the strongest judge resisted, failing just 8.7% of the time. The attack isolates the variable. The padded answer carries identical information to the original. The only thing that changed is word count, and word count alone was enough to reverse the verdict in the great majority of trials.
They had GPT-4 pad a list with zero new information, then prepended it. The padded copy won 91.3% of the time against two of three judges.
The fix: normalize for length
Make length a non-signal. Telling the judge to grade only on correctness and relevance helps but is not enough. Stronger controls: strip obvious padding and re-check whether the shorter answer now wins, cap or match lengths before comparison, and state outright that more words are not more value. The Hugging Face cookbook shows a related lever: switching from a wide float scale to a small integer scale with a written rationale field raised judge-to-human correlation from 0.567 to 0.843, because a tight rubric gives length less room to act as a proxy for quality.
Bias 3: Self-preference (the judge likes its own voice)
Self-preference bias, also called self-enhancement bias, is a judge scoring its own outputs higher than a neutral rater would. In the MT-Bench data, GPT-4 favored its own answers with about a 10% higher win rate, and Claude-v1 favored itself by about 25%. The authors were careful here. With limited data and small differences they could not prove the effect conclusively, so treat the exact percentages as directional rather than settled.
Later work pinned down a likely mechanism. Wataoka, Takahashi, and Ri found that judges assign higher scores to text with lower perplexity, meaning text that feels familiar to the model, whether or not the model wrote it. Their conclusion: the essence of the bias lies in perplexity, and self-preference shows up because a model prefers text that reads like its own. That reframes the problem usefully. You are not fighting vanity. You are fighting familiarity.
Judges do not reward their own answers out of ego. They reward low perplexity, the text that reads most like themselves.
The fix: mask identity and use a cross-model panel
Two controls stack here. First, strip any signal of authorship before judging, so the model cannot tell which answer is the one it generated. Second, never let a model be the sole judge of its own family. Use a different model as the judge, or better, a panel from different model families whose aggregated verdict dilutes any single model's familiarity pull. A panel also gives you a disagreement signal for free. When judges from different families split on the same pair, that split flags a genuinely close call that a single judge would have papered over with false confidence.
The bias-to-control map
| Bias | What it does | The control | Primary source |
|---|---|---|---|
| Position | Favors the answer shown first; a leading judge stays consistent only 65.0% of the time when answers are swapped | Run each pairwise comparison in both orders; declare a win only if it holds both ways, else a tie. Shuffle batch order for pointwise | Zheng et al., 2023 |
| Verbosity | Rewards length over substance; a padded list with no new content won 91.3% of the time against two of three judges | Instruct the judge to ignore length; strip padding, match or cap lengths, use a small integer rubric with a rationale field | Zheng et al., 2023; HF Cookbook |
| Self-preference | Scores familiar, low-perplexity text higher (about +10% to +25% self win rate in early data) | Mask answer identity; judge with a different model or a cross-model panel and aggregate | Wataoka et al., 2024 |
These three are not the whole list
Position, verbosity, and self-preference are the most measured biases, not the only ones. The Justice or Prejudice work catalogs 12 potential biases in LLM judges and introduces CALM, an automated framework that quantifies each one by systematically modifying inputs and watching the verdict move. CALM shows that even capable judges retain significant task-specific biases. That gap between three and twelve is the real takeaway. Solving position, verbosity, and self-preference does not certify a judge as unbiased; it removes the three you can measure most cleanly and leaves the rest for task-specific validation.
You just learned the three most-measured biases. One catalog counts 12.
A 2025 assessment found LLM judges are vulnerable to adversarial attacks and that judge behavior swings by up to 40% across prompt templates. The practical lesson: lock your judge prompt, version it, and re-validate it whenever it changes, because the wording of the rubric is itself a variable that moves scores.
Before trusting any judge, run it on a small human-labeled set and report its correlation to those labels. If you cannot state that number, you do not yet have a judge, you have a guess.
A practical checklist before you trust a judge
- Build a 30-plus example set with human scores and keep only the items where annotators agree, so your ground truth is clean.
- Report judge-to-human correlation on that set before using the judge anywhere else.
- Use a small integer scale with an explicit rubric and a written rationale field, not a wide float scale.
- Run pairwise comparisons in both orders and resolve disagreements as ties.
- Penalize or normalize length so padding cannot buy a higher score.
- Mask which answer the judge wrote, and prefer a cross-model panel over a single self-judge.
- Pin the judge prompt, version it, and re-validate after any wording change.
Switching from a wide float scale to a small integer scale with a written rationale moved judge-to-human correlation from 0.567 to 0.843.
Where an external memory layer fits
Most of the controls above depend on a stable, reusable evaluation set and a fixed judge prompt you can find again next month. A misplaced rubric or an unversioned prompt can cost more accuracy than any single bias you debiased for. MemX is an external AI memory layer, a personal RAG over your own notes, screenshots, docs, and saved references, which is a reasonable place to keep the labeled examples, the exact judge prompt, and the source citations behind each control so they are retrievable when you re-run an eval. It does not score answers for you and it will not remove a bias. These fixes are prompt-level and protocol-level; a memory layer's only job here is keeping the validated prompt and the human-labeled set you worked hard to build from drifting out of reach.
01How do I reduce bias in an LLM-as-a-judge?
Pair each measured bias with one control: run pairwise comparisons in both orders for position, normalize length for verbosity, and mask authorship plus use a cross-model panel for self-preference. Validate every judge against human labels before you trust its scores.
02What is LLM-as-a-judge?
It is using one language model to score or compare the outputs of another, instead of using string-match metrics or human labelers. It handles open-ended quality that BLEU and ROUGE cannot, but it needs validation against human labels before you trust it.
03Why does my LLM judge prefer longer answers?
That is verbosity bias. Judges treat length as a proxy for quality. In one test a padded list with no new content beat the original 91.3% of the time against two of three judges. Tell the judge to ignore length and normalize answer lengths before comparing.
04How do I fix position bias in pairwise evaluation?
Run each comparison twice with the answers in both orders. Only declare a winner if the same answer wins both ways; if the two runs disagree, score it a tie. Even strong judges are consistent only about 65% of the time without this swap.
05Is an LLM judge reliable for production evals?
Only after measuring its correlation to human labels and controlling for position, verbosity, and self-preference. Judges still shift up to 40% across prompt templates, so pin and version the prompt before relying on the scores.
