Measure the work.
Not the demo.

Measure work that is finished and accepted.

Time-to-completion is useful only when the task has a clear finish line. Define what “done” means before comparing workflows: required behavior, review criteria, tests, and any conditions for accepting the change.

Then count the whole route. Include setup, prompts, tool runs, waiting, reviewing suggestions, debugging, testing, revisions, and handoff. A short generation step does not make the task faster if the review and rework take longer.

Speed is only one dimension. Keep correctness, maintainability, security, and the effort needed to understand the final patch visible too. A faster result that fails the acceptance criteria is not a completed task.

Different experiments answer different questions.

Research on coding tools includes short, controlled exercises, assistants used during ordinary company work, and developers changing mature projects they already know. Those settings measure different tasks with different outcomes. Their percentages are not a single league table.

These findings can all be true in their own settings. A short, bounded exercise, a weekly workplace task count, and a change to a familiar codebase are not interchangeable measures. Tool versions and usage habits also change over time.

Test the tasks your team actually ships.

1. Choose representative work. Include routine changes, unfamiliar tasks, maintenance in a known repository, and cases where the assistant is likely to struggle.
2. Write the acceptance criteria first. Keep the task definition and quality bar the same whether a person uses AI or not.
3. Compare repeated cases. Where practical, randomly assign comparable tasks or alternate conditions. Record important differences if random assignment is not possible.
4. Measure until acceptance. Include the person’s time spent prompting, waiting, reviewing, correcting, testing, and handing off the result.
5. Record quality and recovery. Note whether the change passed review, what had to be reworked, whether tests caught issues, and how much reviewer effort it required.
6. Keep the setup visible. Record tool and model versions, agent settings, repository familiarity, task type, and who did the work.

Use enough repeated tasks to see whether an apparent gain is consistent. Preserve unfinished or abandoned tasks in the record; excluding failures makes the tool look better than the workflow actually is.

Separate speed, quality, and the cost of review.

Lines of code, commits, accepted suggestions, and self-reported time savings can describe activity. By themselves, they do not show whether more valuable work was completed or whether the result became harder to maintain.

Keep the measurements tied to a decision: which tasks benefit, where review needs to change, and which work should stay with a person. If an average hides very different results, separate the task groups instead of smoothing them into one number.

Adopt a tool where the evidence supports it.

Start with a bounded workflow and decide in advance what would count as a useful improvement. Review the result with the people doing and checking the work. Expand to a different task class only after measuring that class too.

Keep the baseline and revisit the comparison when models, tool access, agent autonomy, or team practice changes. An improvement on a draft or a small patch does not automatically transfer to security-sensitive work, difficult maintenance, or decisions with a different cost of error.

Measure the quality and the full cost.

For an engineering review checklist, read how to review AI-generated code ↗. For a broader systems view, see how to measure the full cost of intelligence ↗.