Cheating LLM Benchmarks is a research initiative aimed at exploring cheating behaviors in automated language model (LLM) benchmarking by constructing what are known as 'null models.' The project’s experiments have revealed that even simple null models can achieve high win rates in these benchmarks, challenging the validity and reliability of current benchmarking practices. This research is crucial for understanding the limitations of current language models and improving benchmarking methodologies.