Confidence is cheap.
Evidence is engineered.

Editing a repository raises the standard of proof.

A code suggestion in a conversation is one thing. Reading a repository, changing files, running tools, and deciding whether a task is complete is another. Each step raises a question: did the agent understand the request, stay inside its scope, and produce the intended behavior?

SWE-bench studies this work using real GitHub issues and repositories. Its original benchmark contains 2,294 tasks from 12 Python projects; an agent receives a codebase and issue description, then edits the project. The benchmark makes repository-level work measurable, while its result still describes those tasks and that evaluation setup—not every language, codebase, or release decision.

Read the SWE-bench paper ↗

Context → plan → change → verify → review.

Start with the repository state and the acceptance criteria. Identify what the agent may read or change. After it acts, review the diff against the request, run relevant checks, and inspect the final project state. The explanation can point you toward important choices; the files and results let you examine them directly.

Keep the final diff small enough to understand. Note the exact checks and environment; distinguish a check that passed from one that was not run. If a test failed, include the failure and any repair rather than reporting only the last green result.

A passing suite covers its cases—not every behavior.

Tests check selected inputs and expected outcomes. A relevant suite can catch regressions and demonstrate the changed behavior, but it may not cover an edge case, an interaction elsewhere in the repository, or a requirement the test author never encoded. A patch can pass while still solving the wrong problem.

A 2026 SWE-Proof preprint examines this gap on 500 coding tasks derived from SWE-bench Verified. In its evaluation of Claude Opus 4.8, the authors report that about a quarter of patches that passed every hidden test still admitted counterexamples under their formal checks. This finding belongs to their tasks, model, specifications, and verification pipeline; it is not a general failure rate for software tests or coding agents.

Use tests alongside the diff, the task’s acceptance criteria, and checks for relevant boundaries. If the change affects permissions, data handling, dependencies, or external behavior, review those paths explicitly. For a practical checklist, read how to review AI-generated code ↗.

A proof is only as relevant as the behavior it specifies.

Formal verification can establish a stronger claim than a finite test suite: that an implementation satisfies a specified property over the domain represented by the proof. It does not, by itself, establish that the specification captures the original request, that assumptions about unchanged functions are correct, or that a separate repository patch matches the verified implementation.

The SWE-Proof study makes that distinction concrete. Supplying a correct formal specification raised Claude Opus 4.8’s issue-resolution result from 85.0% to 95.1% in the reported setup. But asking the agent to write its own specification did not improve resolution over the baseline, and only 56% of those specifications passed the authors’ audit. The paper identifies faithfulness—whether the specification covers the full required behavior—as a central difficulty.

So a proof should travel with its specification, assumptions, and correspondence to the delivered patch. “Verified” is useful only when a reviewer can see what was proved and what remained outside the proof.

Read the SWE-Proof preprint ↗

Judge whether the work succeeded and what it cost to review.

A useful evaluation asks whether the task met its criteria, whether the patch introduced unrelated changes, whether failures were recovered from safely, and whether the required permissions held. It also measures the time a person spent understanding or correcting the result. These outcomes say more than whether the agent sounded certain.

Keep the conditions visible: repository version, task wording, model, tools, permissions, test suite, and any proof assumptions. Repeat tasks when the goal is to estimate reliability, inspect failure cases, and preserve a baseline when any of those conditions change.

That is the direction behind Grablex Code: an AI engineering workspace where project context, deliberate actions, and verification belong in the same workflow. Explore Grablex Code ↗

For a repeatable evaluation plan, see how to evaluate an AI agent ↗. For the effect on developers’ work, read what the productivity research measures ↗.