How to Compare Coding Agents on Your Repository

An agent finishes a change in four minutes. Another takes twelve. Which one saved more work?

You cannot tell until someone reviews both patches. The faster attempt might have removed an important validation check, introduced an unnecessary dependency, or passed only the test it wrote itself. The slower attempt might be ready to merge. Timing the conversation stops the measurement before the expensive part is over.

If you are choosing between Cursor, Claude Code, or another coding agent, evaluate the work you would actually assign next week. This guide gives you a repeatable trial, a task brief you can reuse, and a blank scorecard. It measures whether a change is correct, how much human attention it needs, and what remains outside the requested scope.

There is no product ranking here. The Git setup was checked in a disposable repository; no model runs were performed for this article. The outcome of your trial belongs in the scorecard.

Pick a task that can actually fail

“Improve the codebase” is a poor trial. A large cleanup creates plenty to admire and very little agreement about what success means. Use a bounded problem with an observable result: an input accepted incorrectly, a loading state that never clears, or a command that reports the wrong exit status. Prefer a real backlog item whose surrounding code you understand.

Before launching an agent, write the reproduction, expected behavior, and compatibility requirements. State the permitted module and the check that owns the behavior. This preparation makes review possible even if the agent produces a convincing explanation of a different task.

For example, this is a task shape, not a claim about a particular repository:

The page-size parser accepts 0 and negative integers.
It should accept integers from 1 through 100, inclusive.
Reject other inputs using the existing validation error type.
An omitted value must keep the existing default of 20.
 
Work in the parser and its existing unit-test file.
Do not change dependencies or the public command syntax.
Use the repository's canonical parser test command.

That brief supplies boundaries on both sides of the change. An implementation that fixes negative inputs while changing the default has failed a stated requirement. An implementation that rejects everything except integers still needs examination for its treatment of empty strings, whitespace, or values arriving from the real caller.

Keep acceptance criteria visible to every candidate. You can reserve independently chosen examples for review, but do not add new requirements after seeing the patches. A surprise test should expose an error in the stated behavior, not reward a private interpretation of the task.

For an initial pilot, choose a few different task classes: a bug fix, a small feature in a familiar module, and an explanation of unfamiliar code. Keep results separate. A good explanation does not establish that the same setup will make safe changes.

Freeze the repository before comparing attempts

A second agent must not inherit the first agent's solution. Resolve the starting revision once, then create a separate working tree for each attempt. Run the following from the repository you want to evaluate; the temporary directory holds only the trial checkouts:

baseline=$(git rev-parse --verify HEAD)
trial_dir=$(mktemp -d "${TMPDIR:-/tmp}/coding-agent-trial.XXXXXX")
 
git worktree add --detach "$trial_dir/candidate-a" "$baseline"
git worktree add --detach "$trial_dir/candidate-b" "$baseline"
 
git -C "$trial_dir/candidate-a" rev-parse HEAD
git -C "$trial_dir/candidate-b" rev-parse HEAD

The two printed commit IDs must equal the recorded baseline. Detached working trees are useful for disposable attempts because no evaluation branch name is needed. They contain the committed snapshot, so any uncommitted local fix or ignored configuration from your original directory must be accounted for separately. Git documents the behavior in its worktree reference.

Run the project's normal preparation and focused checks in both directories before the trial. Record pre-existing failures. If one attempt starts with installed dependencies and another spends ten minutes downloading them, decide whether installation is part of the experience you are evaluating. Either choice is reasonable when recorded; silently mixing them is not.

Preserve the completed trial directories until you have saved the patches and evidence. To repeat an attempt, create a fresh directory from the original baseline. Reusing a repaired attempt makes it harder to know what the next agent received.

Separate file isolation from runtime isolation

A working tree gives each attempt its own checked-out files and index. It does not give each process a separate machine. Git's object store and many repository settings are shared, and ordinary processes can still reach other host resources. A worktree is not a sandbox.

This distinction matters when the task launches an application. Both copies may default to the same database path, port, browser profile, or provider account. Two terminals can show different working directories while their applications modify the same state.

Record the relevant resources before running the trial:

If you want to evaluate an agent's reasoning, reduce differences in the surrounding setup. If you want to evaluate the whole product experience, include setup and approval friction in the record. A desktop environment, a model, and an execution service contribute different things. Your result should identify the combination that was tested.

Give both attempts the same contract

If your task is still a vague request, use the worked pagination brief to turn it into observable acceptance criteria before sharing it with the candidates.

Download the evaluation brief, fill it out, and keep an unchanged copy with the results. Record the agent version, selected model, effective permission policy, tools, and time limit. When an interface chooses a model automatically and does not expose the final choice, record that field as unknown.

Do not force different products to use similarly named settings and assume equivalence. A permission label is not evidence of identical authority. Record what each attempt could read, write, execute, and access over the network. A configuration mismatch can explain a result without being a defect in either agent.

Set an intervention rule before starting. For example, allow one clarification that repeats an existing requirement, with every follow-up copied into the notes. If you debug candidate A's patch yourself while candidate B receives no help, the outputs measure different amounts of human work. The help may be worthwhile; it must be counted.

Use a stopping rule as well. A time limit, a completed response, or a blocked tool call should leave an identifiable original result. Save that state before giving repair advice. Otherwise a successful assisted recovery can erase evidence that the initial attempt failed.

Review correctness before assigning a score

Start with the acceptance criteria, then inspect the diff. Run the canonical checks yourself and retain their output. An agent's statement that tests passed is a lead to verify, not the verification record.

A practical review sequence is:

1. Reproduce the original failure against the baseline, where possible.
2. Exercise the required behavior against the candidate's unchanged final output.
3. Read the affected callers and the patch for compatibility problems.
4. Check that the change stayed within scope and that tests were not weakened.
5. Record any repair separately, including the time it required.

Passing the relevant tests and reading the patch answer different questions. A test can miss an unrequested change in another file. A plausible implementation can miss a boundary case that an independent check exposes. Neither activity can be replaced by the candidate's confidence.

For an explanation task, review the cited files and specific claims instead. Count a fabricated entry point or an incorrect dependency relationship as an error. A polished repository map with false details creates work for the next person who trusts it.

For an evidence-gathering task, the Python repository briefing collector offers a bounded input you can inspect before evaluating an agent's interpretation of it.

Keep blocked attempts visible. “Could not authenticate,” “required service unavailable,” and “implemented incorrectly” need different labels. Infrastructure failures affect usefulness, but mixing them with code defects makes the next decision harder.

For an interrupted attempt, record what survived before deciding how to continue. The session recovery guide separates retained conversation, workspace edits, and process state; its CompozyOS commands apply when that is your trial environment.

Count the human minutes

The scorecard deliberately separates agent wall time from human intervention, review, and repair. Wall time measures elapsed waiting. Human time measures attention that could have gone elsewhere. They can overlap, so do not add them together and call the result elapsed delivery time.

For each attempt, record:

The cost field may remain unknown when an interface reports only account-wide subscription usage. Do not turn that missing value into zero, or divide a monthly bill into a precise per-task price without explaining the allocation.

Avoid a weighted “agent score” in the first pilot. A short review time should not compensate numerically for a wrong implementation. First ask whether each candidate produced acceptable work. Then compare the attention and waiting needed to get there.

Make a decision the evidence can support

One attempt is useful for finding setup problems and obvious failure modes. It cannot establish that a product is generally better. Repeat representative tasks from fresh sessions, alternate which candidate goes first, and retain every result. Repeating the same easy task many times still tells you little about the rest of your backlog.

When reporting results, include denominators and conditions: how many attempts, which task classes, which configurations, and how blocked runs were handled. Keep the individual rows available even if you summarize them. A single average can conceal a candidate that finishes quickly most of the time and occasionally needs a complete rewrite.

A useful decision might be to use one setup for repository exploration while keeping a different one for small fixes in a well-tested module. It might also be to keep your current tool because the alternative adds review work. Neither conclusion needs a universal winner.

Leave the trial with a completed brief, original patches, check output, and a scorecard whose unknowns are visible. The next comparison can then reuse your method instead of starting with another impressive conversation.