Recent Advances in Large Language Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation
A survey repository accompanying a paper on how large language model benchmarking has moved from static test sets, which risk data contamination from training corpora, to dynamic evaluation methods designed to avoid it.
This repository accompanies a survey that examines methods for strengthening static LLM benchmarks against contamination, identifies their limits, and highlights the lack of standardized criteria for evaluating dynamic benchmarks, which regenerate or vary their test data instead of relying on a fixed set.
Based on that analysis, the authors propose design principles for dynamic benchmarking and discuss the limitations of existing dynamic benchmarks, aiming to give researchers a guide to recent work on data contamination in LLM evaluation and directions for future benchmarks.
The repository is maintained as a living bibliography: contributions of missed papers or updates to a paper's publication venue are accepted as pull requests in a fixed Markdown citation format.