LLM benchmark rankings and evaluation setup

Three bodies of research show how evaluation methodology drives benchmark rankings independently of model differences. A study by Rohan Paul kept 3,679 questions and 12 models fixed and varied only evaluation parameters, finding Gemma 4-31B's accuracy ranged from 31% to 89%, and four of twelve models each reached rank 1 under at least one valid configuration. The study reports that 95.7% of the average gap between neighboring models stems from questions whose answers change with evaluation setup, and that the scoring method is the single biggest driver of ranking instability. A separate mechanism appears in data quality: removing incorrect samples from MMLU to produce MMLU-Redux caused Llama-3.1-405B to rise from 16th to 1st in Virology and Qwen-2-72B-Instruct to drop from 1st to 8th in College Chemistry. Cameron Wolfe's overview also describes Fluid Benchmarking, which dynamically selects the most informative evaluation examples for a specific model rather than applying a single fixed subset. A study of 160 commonsense problems found that questions taking humans longer also caused reasoning models to use more tokens, and that both humans and models tend to fail on the same questions; for GPT-OSS-20B, the human-model difficulty correlation rose from 0.41 to 0.55 when averaged across multiple reasoning paths rather than a single run. On the ARC-AGI-2 benchmark, two sources report different pictures. BenchLM's September 2, 2026 snapshot shows GPT-5.6 Sol leading at 92.5%, Claude Opus 5 at 90.4%, and Claude Fable 5.1 at 90% among 22 ranked models, with the top models clustered within 2.5 percentage points. A separate source reports Grok 4 leading ARC-AGI-2 at 16%, DeepSeek v3.2 at 4%, Qwen 3 at 1%, Llama 4 Maverick at 0%, and Claude Opus 4.6 at 69% on the same benchmark.