How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them?

Question#

How much signal do public LLM benchmarks still carry, and what replaces them? (Synthesizing the 2026 eval-science cluster: BenchPress rank-2 redundancy, CORE-Bench life-after-saturation, UBD contamination correction, and the judge-bias audits.)

Short answer#

Far less independent signal than the number of benchmarks implies — a 133-benchmark public scorecard is effectively two numbers (Benchmark Score Redundancy) — and the signal that remains is corrupted through four distinct channels: an unnamed test-time-compute budget (Compute-Controlled Benchmarking), training-data contamination (Benchmark Contamination and Decontamination), vendor-optimistic self-reporting (Benchmark Score Redundancy), and an unvalidated grading layer (LLM-Judge Validation, Reference-Free Judge Over-Crediting). What survives best is ordinal signal under a verified invariance: rankings, not absolute scores, and only across the axis you have actually checked.

But the convergent 2026 answer is not that benchmarks get replaced. Every paper in the cluster rejects retire-and-replace; each instead adds an instrument that recovers signal the headline number hides. The "replacement" is a portfolio of five moves — predict-don't-run, re-instrument what saturated, put compute on the x-axis, refresh tasks from production, validate the judge — plus three jobs (failure-mode discovery, contamination monitoring, incentive shaping) that only running a real benchmark can do.

Part 1 — How much signal is left#

The count of benchmarks wildly overstates independent signal#

Three results at three granularities say the same thing:

• Matrix level: Zeng & Papailiopoulos's 84-model × 133-benchmark public score matrix is effectively rank-2 — held-out Soft-Impute completion bottoms at rank 2, and the top-2 SVD components explain >90% of cross-model variance in every fully-observed submatrix. Five probe benchmarks ({GPQA-Diamond, HLE, Codeforces, MMLU-Pro, ARC-AGI-1}) recover a model's full 133-benchmark scorecard to 3.93 points (Benchmark Score Redundancy). This confirms, on a heterogeneous frontier-era matrix, the earlier g-factor findings (85% of variance across 12 leaderboard benchmarks; "general capability + provider residual").
• Benchmark level: accuracy saturates — and stays saturated even after the benchmark is repaired. On CORE-Bench v1.1, after fixing 15 task-level errors and 20 exploitable shortcuts, the top agent hits 100% and the next four tie at ~97.4%, statistically indistinguishable (Measuring Beyond Accuracy Saturation). Task Time-Horizon Scaling logs the same dynamic across suites (SWE-bench, CORE-Bench saturating within ~15 months) as capability doubles every ~4 months.
• Item level: ~27% of standard benchmark problems are non-discriminative (ceiling/floor) (Scale-Dependent Prompt Sensitivity, as quantified in Benchmark Score Redundancy's item-level counterpart framing).

These are the same fact at different zoom levels: saturation is near-zero score spread, near-zero spread is what makes a score trivially predictable, and predictability is what makes the matrix low-rank (Benchmark Score Redundancy ↔ Measuring Beyond Accuracy Saturation connection).

The signal that remains is corrupted through four channels#

The cluster jointly builds a taxonomy of ways the headline number lies (an extension of the Reward Hacking taxonomy):

1. Unnamed compute budget. If capability is a function of inference budget (Large-Scale Test-Time Compute), a score without its budget is undefined. The grid hid GPT-5.5's efficiency jump over 5.4; Gemma 4's headline table benchmarks a thinking model against a non-thinking predecessor, confounding generation gain with inference spend — while controlling correctly in its own long-context table (Compute-Controlled Benchmarking). Benchmark-maxxing (best-of-N, judge-pick scaffolds) inflates the grid without any capability gain once compute is equalized.
2. Contamination. Test samples leaking into training make the score measure memorization, not capability — and the standard fix is itself under-measured: paraphrase+permutation halves dataset-level residual contamination (17.2→8.4) while per-sample D_KL to a clean model rises >13%, so decontamination that looks successful at the aggregate level can worsen the underlying distortion (Benchmark Contamination and Decontamination).
3. Vendor optimism. Roughly four in five scores in the public grid come from the model provider's own materials, under heterogeneous harnesses (same model shifts 1–3 points across runs, 5+ across harnesses). The rank-2 paper itself flags that shared reporting bias may manufacture part of the cross-benchmark correlation it exploits (Benchmark Score Redundancy).
4. Unvalidated grading. Where the metric is an LLM judge, the validation layer is systematically under-rigorous: exact-match agreement overstates chance-corrected κ by 33–41pp on MT-Bench (a judge reporting "85% agreement" has κ ≈ 0.48); judge rankings shift up to 14 positions across benchmarks; and perfectly reproducible judges hide severe bias — the consistency–bias paradox (LLM-Judge Validation). A second, orthogonal invalidity: with no reference answer in the prompt, judges systematically over-credit wrong answers — adding the gold answer flips up to 85% of verdicts, and human annotation confirms the stricter verdicts are the correct ones (Reference-Free Judge Over-Crediting).

What survives: ordinal signal under a verified invariance#

Two results bound what can still be trusted:

• BenchPress-completed scores preserve 92.1% of same-benchmark pairwise model orderings when the true gap is ≥5 points (Benchmark Score Redundancy) — prediction noise rarely flips a meaningful ranking.
• DRACO finds system-under-test rankings stable across judge models while absolute magnitudes vary; Norman et al. find judge rankings fragile across benchmarks. The reconciliation is the operating rule: a ranking is trustworthy only across the axis you have actually verified it stable on (LLM-Judge Validation).

So: relative comparisons on a shared harness at a stated budget retain real signal; absolute scores, cross-paper comparisons, and un-budgeted grids mostly do not.

Part 2 — What replaces them: a portfolio, not a successor#

No paper in the cluster proposes abandoning benchmarks. Each contributes one instrument; together they form a division of labor:

Move	Mechanism	What it buys	Source
Predict, don't run	Rank-2 logit-space ALS matrix completion; 5 probes → full scorecard (3.93 MedAE); per-cell reliability layer (top-20% trusted predictions: 1.83 MedAE)	Cuts eval cost on the benchmark-count axis; a new model needs only 5 seed scores	Benchmark Score Redundancy
Re-instrument what saturated	Keep the saturated benchmark, measure six non-accuracy axes: reliability (93% pass vs 32.1% self-confidence; discrimination ≈ random), efficiency (60% cheaper at equal accuracy; tokens vs dollars rank differently), model-vs-scaffold (44pp scaffold swing; 31% task-level disagreement at equal accuracy; oracle router → 100%), OOD transfer, construct validity, human uplift (2.11× faster reproduction)	A saturated leaderboard still discriminates agents — just not on accuracy	Measuring Beyond Accuracy Saturation
Put compute on the x-axis	Report capability curves against tokens/cost/time; fix a budget and compare within it; UK AISI's "minimum informative budgets" as adopted government practice	Un-confounds capability from inference spend; reveals efficiency gains the grid structurally cannot show	Compute-Controlled Benchmarking
Refresh tasks from production	Mine de-identified real usage, difficulty-proxied (thumbs-down sampling), PII-stripped, augmented, human-gated; continuously regenerable	Representativeness + contamination prevention (fresh tasks are hard to pre-memorize); the correction-side complement is UBD, which repairs an already-contaminated model without a clean reference (>40–60% relative D_KL reduction)	Production-Sourced Evaluation, Benchmark Contamination and Decontamination
Validate the judge	Norman's Minimum Viable Validation Protocol (chance-correct, position-swap, replicate, cross-validate on ≥2 benchmarks, audit the paradox) + Kranti & Vajjala's calibration/sensitivity probes before reference-free deployment	Makes the grading layer trustworthy; a representative task graded by an unvalidated judge is still an unreliable eval	LLM-Judge Validation, Reference-Free Judge Over-Crediting

What benchmarks alone still do#

The rank-2 paper's own scope caveat is the keystone: scores are inferable, not benchmarks unnecessary. Three functions no prediction, curve, or judge-audit replaces (Benchmark Score Redundancy):

• Failure-mode discovery — a perfectly predictable benchmark can still catch the next regression; saturation itself is what surfaced CORE-Bench's 15 task errors and 20 shortcuts, invisible to weaker agents (Measuring Beyond Accuracy Saturation).
• Contamination and distribution-shift monitoring — the integrity checks that keep the rest of the portfolio honest (Benchmark Contamination and Decontamination).
• Incentive shaping — benchmarks steer what labs optimize; retiring them doesn't remove the pressure, it just relocates it (Compute-Controlled Benchmarking's bad-equilibrium framing).

Residual risks the portfolio doesn't close#

• Goodhart concentrates. If "run 5 probes and infer the rest" becomes practice, the probe set is a small, public, high-leverage optimization target — the same eval-report Goodhart pressure, now focused on five benchmarks (Benchmark Score Redundancy open question; Reward Hacking).
• The fixes are unintegrated. BenchPress runs on top of the uncontrolled public grid Brown/AISI critique; combining "control compute per eval" with "predict across evals" is unaddressed (Benchmark Score Redundancy ↔ Compute-Controlled Benchmarking).
• The redundancy itself may be partly artifact. Whether a fully standardized re-evaluation would still be rank-2 — or whether vendor reporting bias inflates the correlation — is open (Benchmark Score Redundancy).
• Judge validation is a snapshot. English-only, thinking-suppressed, five-week window; hosted judges drift silently, and calibration proper (ECE/Brier) is still unmeasured (LLM-Judge Validation).
• Living benchmarks need living maintenance. Log-analysis-driven re-instrumentation is non-exhaustive and may itself become a Goodhart target once developers know the rubrics (Measuring Beyond Accuracy Saturation).

Sources#

Concept articles: Benchmark Score Redundancy (Zeng & Papailiopoulos, arXiv 2606.24020), Measuring Beyond Accuracy Saturation (Nadgir et al., arXiv 2606.26158), Benchmark Contamination and Decontamination (Sun, Zhan & Gales, arXiv 2606.23313), LLM-Judge Validation (Norman et al., arXiv 2606.19544), Reference-Free Judge Over-Crediting (Kranti & Vajjala, arXiv 2607.12885), Compute-Controlled Benchmarking (Brown No Priors 2026-06-26; Gemma 4 report; UK AISI 2026-07-02), Production-Sourced Evaluation (DRACO; Google agent-quality flywheel), plus Task Time-Horizon Scaling, Scale-Dependent Prompt Sensitivity, Large-Scale Test-Time Compute, Reward Hacking, DRACO Benchmark, LLM-as-a-Judge.

Date: 2026-07-16.