LLM Harness Sensitivity: How Benchmark Choices Skew AI Model Rankings

In the race to crown the best large language model, subtle differences in how benchmarks are run can dramatically alter outcomes. A new preprint on arXiv investigates the sensitivity of LLM evaluation harnesses — the codebases used to administer benchmarks like MMLU and HumanEval — and finds that choices such as prompt formatting, few-shot examples, and inference parameters can shift pass rates enough to reorder models on leaderboards. The authors argue that this sensitivity undermines the validity of current comparisons, which often treat harness outputs as ground truth.

The study systematically varies components of popular harnesses — including prompt templates, option ordering in multiple-choice questions, and decoding temperature — across a suite of open models. Results show that some models are more robust to these perturbations than others, meaning that the 'best' model can depend on the harness version used. For example, a simple change like reordering answer choices can swing accuracy by several points, enough to move a model from first to fourth on a typical leaderboard.

These findings have immediate implications for developers who rely on benchmarks to choose models for production. A model that excels on one harness may falter on another, suggesting that the competitive benchmark landscape says more about evaluation engineering than about model quality. The authors recommend that the community standardize harness configurations, report sensitivity intervals, or move toward more robust evaluation methods that average over multiple setups.

For practitioners, the practical takeaway is to evaluate models using the same harness and configuration that matches their deployment context — not just the one that tops a public leaderboard. Running your own evaluation with realistic prompts and settings will give you a better sense of how a model will perform in your specific use case. As the field matures, evaluations that account for sensitivity will become as important as the models they test.