A benchmark is a question.
Not a verdict.

What did the system actually have to do?

Benchmarks turn an open question into tasks with inputs, tools, environments, and success criteria. SWE-bench, for example, gives a model a real GitHub issue and repository, then asks it to produce a change that resolves the issue. Its original release assembled 2,294 issues from 12 Python repositories.

That is a useful test of repository-level engineering. It is not a universal measure of software judgment, product quality, or how well an agent works with a particular team. The task boundary matters: issue resolution is one kind of engineering work, with its own mix of code reading, editing, and verification.

Read the SWE-bench paper ↗

Tools and surroundings change the test.

An agent that edits a repository, operates a browser, or controls a desktop faces different kinds of uncertainty. OSWorld makes that distinction concrete: its original benchmark contains 369 tasks in real computer environments, with setup configurations and execution-based checks.

Before comparing results, find out whether the systems received the same model version, prompt, tools, starting state, time or interaction budget, and chance to recover from a mistake. When those conditions differ, the comparison is between two complete setups—not just between two models.

A result should also tell you what was allowed to change. A larger context window, extra retries, a stronger tool, or a longer run may be a reasonable product choice; it simply needs to be visible in the report.

Read the OSWorld paper ↗

How computer-use agents interact with those environments ↗

“Passed” only means what the check can see.

Some tasks have exact answers. Others need a rubric, expert review, or a model-based judge. Each method sees a different slice of quality. A test suite can verify behavior it covers; it cannot certify every property a user cares about. A fluent response can satisfy a style rubric while missing a source or an important exception.

Look for the scoring rule, the evaluator’s version, and examples of how borderline cases are treated. If a second model judges the answer, its prompt and model are part of the experiment too. Human review can add context, but the review process and disagreement rule should be described rather than hidden behind a single label. Read how model-based judges make and miss a call ↗.

Quality is not one column.

Completion is one signal in a larger story. Useful evaluations may also report error types, retries, recovery, latency, compute or API cost, and how much review a person needed. Those are questions for the evaluator to measure; a benchmark score does not answer them automatically.

HELM offers a useful example of broader coverage. It organizes evaluation around scenarios and multiple metrics—including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency—and makes gaps in its coverage visible. That is a stronger map than one leaderboard column, but no finite set of scenarios describes every real use.

For tasks with randomness, one run can conceal variation. Repeated runs and a spread or interval make it easier to tell a stable difference from a lucky result. For practical decisions, put quality beside the resources and human effort needed to get it.

Read the HELM paper ↗

Could the test already be familiar?

A public benchmark’s examples may later appear in training data or tuning workflows. Researchers call this data contamination. It can make a score look stronger than performance on genuinely unseen tasks, but exposure is not something you can conclude from a high score alone.

A position paper on NLP evaluation argues for benchmark-specific contamination checks and notes that measuring exposure is difficult. Read a result for what it establishes: whether the authors checked, what evidence they used, and what remains unknown. When a benchmark is public, a private or newly collected follow-up set can help test whether the result transfers.

For a focused guide to overlap types, detection limits, and stronger test design, read what benchmark contamination means ↗.

Read the paper on benchmark data contamination ↗

Keep the evidence next to the headline.

When you read a benchmark report—or design one for your own product—look for enough detail to reproduce the comparison and understand its limits.

A useful benchmark narrows uncertainty. It does not end the conversation: it gives you a precise question to test next, in the context where the system will actually be used.