Almost every model decision made today runs through an LLM judge. Which checkpoint ships, which data mix wins, which reward signal a policy is trained against: in each case, a language model grades another language model because human annotation does not scale and static references do not exist for open-ended tasks.

Yet LLM judges are not neutral instruments without failure modes and biases and they can exhibit self-preference bias (SPB): judges systematically favor outputs produced by themselves, by models from their own family, or by models trained on data they generated. If your judge inflates the scores of its own relatives, you will pick the wrong model, publish a leaderboard that is partly an artifact of who did the grading, and, in a self-improvement loop, optimize a model toward its own preferences rather than the task.

Until now, essentially all of the evidence for SPB came from two evaluation paradigms: pairwise comparison, where a judge picks the better of two answers, and direct assessment, where a judge scores a single answer on a numeric scale. A third paradigm, rubric-based evaluation, replaces both with a set of binary yes/no criteria written for each specific instance. A medical answer about cardiac-arrest care might be checked against "correctly states that compression depth is 2 to 2.4 inches," with some criteria phrased negatively (things the answer should not do) and each carrying its own weight. The verdicts are interpretable, the paradigm correlates well with human judgment, and it now underpins benchmarks like HealthBench, WildBench, PaperBench and MultiChallenge, as well as a growing share of RL reward signals.

Figure 1. The three LLM-as-a-judge paradigms: pairwise comparison (pick the better of two answers), direct assessment (score one answer on a scale), and rubric-based evaluation (check one answer against binary criteria).

Because rubrics remove direct comparison and the numeric scale, it is tempting to assume they also remove the room a judge needs to flatter itself. After all, a criterion like "correctly states that compression depth is 2 to 2.4 inches" leaves no space for taste. Nobody had checked whether that intuition holds. Ours is, as far as we know, the first systematic study of SPB in rubric-based evaluation.

We find that:

• Judges favor their own outputs and their relatives' even when every rubric is a programmatically verifiable unit test. In the worst case, a judge is 20 times more likely to mark its own failed tests as passed than another model's.
• The bias survives controls for how hard the outputs are to judge.
• Rubric-based evaluation is significantly more robust than pairwise comparison, but no better than direct assessment.
• Ensembling judges reduces self-preference substantially but does not eliminate it, because judges err together above chance when generators fail.
• On HealthBench, a realistic benchmark with subjective rubrics, self-preference moves system-level scores by up to roughly 11 points, and by 4 to 5 points for frontier models, margins large enough to reorder a leaderboard.
• Negative rubrics and subjective themes like communication and emergency referrals concentrate the bias; accuracy and instruction-following rubrics mostly do not.

Measuring self-preference, not leniency

Notwithstanding, a judge that scores its own outputs generously is not necessarily biased, and it may be a lenient grader for everyone, or its outputs may genuinely be better.

We start from an overestimation rate: how often a judge rules more favorably for a generator than the ground truth warrants. At the rubric level, this is a false positive rate: among the rubrics that a generator objectively fails, how often does the judge wave them through as satisfied? Self-preference is then that rate on the judge's own outputs relative to the same judge's rate on unrelated models. We call this the Harmful Self-Preference Propensity Ratio (HSPP-R), extending the HSPP metric of Chen et al. (2025):

HSPP-R (self) = (overestimation rate on the judge's own outputs) ÷ (mean overestimation rate on generators outside the judge's family)

A ratio of 1 means no self-preference; 1.5 means the judge is 50% more likely to incorrectly pass its own failures than the failures of unrelated models. Dividing by the judge's behavior on others cancels general leniency, and excluding same-family generators from the denominator keeps family loyalty from hiding inside the baseline. We also report a family variant (the judge grading its relatives) and an instance-level variant based on rankings.

All of this needs a ground truth (which is exactly what LLM judges exist to approximate), and we solve it in two steps. For objective rubrics, we use IFEval (541 instruction-following prompts with programmatically verifiable instructions such as "no commas") and LiveCodeBench (177 post-2025 code problems, 5,685 private unit tests). Every instruction and every unit test becomes one binary rubric with a machine-checkable verdict, so rubric subjectivity is removed. For subjective rubrics, we use HealthBench, a medical chat benchmark with 5,000 instances and 48,562 human-written rubrics, and take as reference a majority vote of the largest judge from each family. This is deliberately conservative: every family is represented in the reference, which absorbs part of each judge's own bias.

As judges and generators, we use the same twelve models spanning the Gemma 3, Llama 4, Qwen 3, GPT, and Claude families. Every model grades every model, including itself, and all metrics come with bootstrap confidence intervals.

Objective rubrics do not fix the problem

Table 1. Rubric-level metrics per judge across IFEval, LiveCodeBench (LCB) and HealthBench (HB), with 95% bootstrap confidence intervals. MRA = Mean Rubric Accuracy; HSPP (Self) = rubric-level self-preference ratio.

On LiveCodeBench, most judges have a rubric-level HSPP ratio significantly above 1: they let their own failing code through more often than anyone else's. GPT-5 is the extreme case at 20.15 (95% CI 16.57 to 24.58). In absolute terms, it marks 38% of its own failed tests as passed, against 1.9% for unrelated generators. GPT-oss-120B sits at 5.97, and the Qwen 4B and Claude judges between 1.7 and 2.2. IFEval is milder, as you would expect from rubrics like "write in all lowercase": all but one judge sits above 1, though only two are individually significant. The family-level picture mirrors it: Gemma judges over-credit Gemma models, and GPT-5 reaches 11.91 at the family level on LiveCodeBench.

A judge that passes its own failed unit tests at a much higher rate than it does for an unrelated model's code is not exercising defensible judgment, despite fully objective rubrics. This is one of the most interesting findings: even carefully designed rubrics don't solve the problem, despite being more objective.

Rubric-based evaluation is still an improvement over alternative evaluation methods. Comparing all four paradigms on IFEval with the same judges and the same information (Figure 2), pairwise comparison is significantly more biased than everything else. This fits the mechanism proposed by Panickssery et al. (2024): SPB tracks a judge's ability to recognize its own output, and nothing makes recognition easier than putting your own answer next to somebody else's. Rubric-based evaluation, however, is roughly on par with direct assessment. Presenting rubrics one at a time elicits slightly more self-preference than presenting them all at once, possibly because a narrow scope gives the judge more room to rationalize away a single failure on its own output.

Figure 2. Accuracy versus self-preference on IFEval across judge paradigms. Left: instance level. Right: rubric level.

It is not just “my mistakes are harder to judge”

GPT-5 is the strongest generator in our pool for LiveCodeBench; perhaps its failures are subtle near-misses any judge would misgrade, while a weak model's failures are obvious.

To study the capability confound, we test every (generator, instance) pair by how accurately all twelve judges grade it, a judge-independent proxy for evaluation hardness. We confirm the premise (better generators' failures do concentrate in the hard bins), then recompute the ratio inside each bin. If the bias were an artifact of hardness, it should collapse to 1 everywhere. It does not: GPT-5 remains at 13.74 and 14.66 in the middle-hardness bins, and Claude Sonnet stays between 1.3 and 2.8 (Table 2). Two further controls in the paper, restricting comparators to the judge's own quality tertile and running one-versus-one comparisons on rubrics both models objectively fail, reach the same conclusion.

Table 2. LiveCodeBench self-preference stratified by evaluation hardness (mean rubric accuracy across all 12 judges). “Original” is the unstratified ratio.

Mechanistically, self-preference concentrates on the outputs a judge recognizes as its own: when we ask each judge to guess authorship, the LiveCodeBench ratio is 1.88 on outputs it identifies as its own and 0.72 on the rest. It also grows with output length, while perplexity, the natural "familiarity" explanation, does not account for it. And more thinking does not help: increasing the reasoning effort of GPT-5 and Claude Sonnet improves accuracy without reducing self-preference. All the artifacts behind these analyses are in the paper.

Ensembling helps but does not solve self-preference

On a positive note, the standard mitigation of replacing one judge with a panel of judges coming from different families partially works. A five-family majority-vote committee gives every member the same or better accuracy and the same or lower self-preference (Figure 3). The effect is strongest where the individual bias is worst: on LiveCodeBench, GPT-5's ratio drops from 20.15 to 4.18. Averaging over judges dilutes the bias but it does not delete it.

Figure 3. Impact of a five-family committee on IFEval, at the rubric level. Left: self-preference. Right: accuracy. Lines connect each member’s individual performance to its committee-aggregated performance.

Two follow-ups explain why. First, enumerating all 2,497 possible committees of two to six judges and regressing committee-level self-preference on committee properties, the dominant factor by far is member quality, not size or family diversity. A jury of weak, diverse judges is worse than a small panel of strong ones. Second, judges agree above chance even on the subset where all generators fail (Krippendorf alpha of 0.308 on IFEval and 0.154 on LiveCodeBench). Since ground truth is constant on this subset, positive agreement can only reflect shared errors: judges mark the same failed rubrics as passed. These correlated false positives explain part of the self-preference that survives majority voting.

The silver lining is that low agreement flags the rubrics where bias lives. Filtering rubrics by inter-judge agreement steadily reduces self-preference (Figure 4), and this filter needs no ground truth, only two or more judges.

Figure 4. Self-preference at the instance level (left) and rubric level (right) when filtering rubrics by inter-judge pairwise agreement threshold on IFEval, averaged over 12 judges.

What this costs on a real benchmark

Real rubric benchmarks are full of criteria like "acknowledges the patient's anxiety before giving advice," and that is where we expect the damage to be largest. HealthBench lets us measure it in benchmark points: for every judge-generator pair, we compute the judge's deviation from the reference score, centered by that judge's average bias across all generators.

Figure 5. Centered score delta matrix on HealthBench (single-rubric, weighted scoring, ×100). Orange = relative overestimation, blue = relative underestimation. Bordered cells are self-evaluation; dotted areas are within-family.

The diagonal of the matrix tells us the relative bonus in benchmark points a model gives to itself. GPT-5 gives itself a bonus of about 4.6 points on a 100-point scale; Gemma 3 4B gives itself nearly 11. The block structure shows family loyalty. Four to five points is not a rounding error in a world where frontier models are separated by two or three points on HealthBench, and where those points end up in launch posts and procurement decisions.

Llama 4 judges show below-1 self-preference (they seem, if anything, harder on themselves than on strangers), and Claude judges are close to neutral on HealthBench showing that not every family behaves this way. Whatever produces self-preference is not a universal law of LLM judging but a property of specific models, which suggests it is trainable and therefore fixable.

A couple of caveats: because the HealthBench reference includes every judge's family, it absorbs part of the bias we measure. Rebuilding it with leave-one-family-out voting moves GPT-5 from 1.54 to about 1.68. The numbers above are the conservative ones. Furthermore, the ensemble reference is a proxy, not the ground truth, but it is still informative: we have shown ensembles are generally better evaluators than single models and partly mitigate self-preference.

Which rubrics are dangerous

One remaining question is if there are types of rubrics that pose more danger than others. To study this, we fit a Bayesian mixed-effects logistic regression on every HealthBench rubric where overestimation is possible, with interaction terms capturing whether a rubric property amplifies overestimation specifically when the judge grades itself (Figure 6). Three patterns come out of it.

Negative rubrics are more vulnerable than positive ones. Criteria phrased as things the model should not do (“is overly verbose”) reliably increase the odds that a judge lets its own output off the hook. Negation is harder to verify and easier to argue with, and judges argue in their own favor.

Subjective themes concentrate the bias. Rubrics about communication, emergency referrals, global health and complex responses show the largest amplification, while the accuracy and instruction-following axes show essentially none.

Rubric length does not matter. Long criteria are harder to satisfy, but not differentially harder to grade fairly when the output is your own.

Figure 6. Interaction coefficients (is_self × property) from the HealthBench mixed-effects regression. Positive values reinforce self-preference. Left to right: rubric properties (negative polarity, length), rubric axes (reference: accuracy), themes (reference: untagged).

Practical guidance

Everything above adds up to a short list of things you can do now:

• Avoid letting a model be the sole judge of its own outputs, or its family’s.
• Prefer a small panel of strong judges over a large panel of diverse ones, and do not expect the panel to be enough.
• Write rubrics positively. “States X” is safer than “fails to state X.”
• Filter on inter-judge agreement. Low-agreement rubrics are where the bias concentrates, and you can find them without any ground truth.
• Be most careful where verification is subjective.
• Do not assume reasoning saves you. Higher reasoning effort improved accuracy in our experiments without reducing self-preference.

Open questions

Our reference on HealthBench is an ensemble of judges, not humans. It is a reasonable and deliberately conservative proxy, but not ground truth. Our notion of “family” is also coarse: most models share some relatedness through distillation and overlapping training data, and how to define and measure relatedness properly is an open problem. Finally, we measure the bias but do not remove it.

The direction we care about most is reward modelling. A biased judge in an evaluation loop misinforms a decision; a biased judge in a training loop is a gradient. We expect self-preference to accelerate reward hacking, and to be worst exactly where rubric-based RL is most attractive: tasks that only one system can currently solve, and that therefore only that system can plausibly grade. Beyond that, the fact that some families show little or no self-preference tells us this may be a property of models. That points at interventions on the judge itself, such as activation steering (Roytburg et al., 2025) or training-time debiasing.

Acknowledgements

This work is supported by the project DECOLLAGE (ERC-2022-CoG 101088763) and by FCT/MECI through national funds and when applicable cofunded EU funds under UID/50008: Instituto de Telecomunicações. We also thank the COLM reviewers, whose feedback made the paper considerably stronger.