When the test
becomes familiar.

Contamination is exposure to evaluation material.

In language-model evaluation, benchmark contamination means that test examples, answers, or close variants may have entered a model’s pretraining or fine-tuning data. If the same material later appears in an evaluation, a high score may partly reflect prior exposure rather than transfer to unfamiliar tasks.

This is different from a model knowing ordinary facts about the topic. It is also not automatically evidence of deliberate cheating: public datasets can be copied, discussed, or included in large collections without a model developer knowing which examples are present.

Contamination narrows what a result can support. It does not make every score false or every public benchmark useless. For the wider reading checklist, see how to read an AI benchmark ↗.

A copied question is only one way information can overlap.

Detection methods find clues, not a complete training history.

When training data is available, researchers can search for exact or approximate overlap. With proprietary models, a test may instead probe whether the model recognizes unusual benchmark details. Deng and colleagues described a retrieval-based search and a “Testset Slot Guessing” method that asks a model to recover masked parts of test examples.

Overlap signals do not reveal exactly when, where, or how a model encountered material. A detector can also miss paraphrased examples or flag similarities that do not establish memorization. Read the method, the test cases, and the authors’ stated limits before treating a contamination score as a verdict.

A benchmark result is still evidence about a defined test.

If overlap is suspected, the measured accuracy can overstate performance on unseen examples. The size of that effect depends on which items were exposed, whether answers or just topics overlapped, and how closely the test resembles the real task. A suspected overlap does not tell you the exact score a clean test would produce.

Keep the claim tied to the benchmark version, model version, prompt, tools, and scoring rule that were actually tested. To judge transfer, add fresh tasks that check the same capability under different examples and conditions. A benchmark can be useful for comparison while still being a poor stand-in for every real-world use.

Newer work explores stricter contamination controls. Zhang and colleagues’ 2026 FTD method proposes a user-set false-discovery-rate bound for contaminated samples mistakenly retained as clean. That is a guarantee claimed for their method under its assumptions—not a universal guarantee that an opaque model has a contamination-free training history.

Protect the final test, then test transfer.

1. Version the data. Record the benchmark release, collection dates, split, and any changes made to test examples.
2. Separate development from the final test. Tune prompts and settings on development data; limit repeated decisions based on the final holdout.
3. Check for more than copied text. Look for near-duplicates, answer leakage, and possible semantic overlap where the data and evaluation methods allow it.
4. Add fresh tasks. Use private or newly collected examples where appropriate, and test meaningful variations instead of relying only on surface paraphrases.
5. Report the limits. State what was checked, what training information was unavailable, and what a score cannot establish.

C²LEVA is one research example: its authors describe a 22-task bilingual benchmark with automated test-data renewal and data-protection measures during release. Those are design choices in a specific benchmark, not a guarantee that every evaluation can eliminate exposure.

For the broader test setup, continue with the benchmark-reading guide ↗ or the guide to evaluating AI agents ↗.