Trialling Two Assistants On The Same Backlog

Why the same backlog matters

Most “which AI coding assistant is better” takes come from a demo repo or a single flashy prompt. That tells you almost nothing about how a tool behaves on the kind of work that actually fills a sprint: a stale dependency bump, a flaky test, a half-documented internal API, a ticket where the acceptance criteria contradict themselves. If you want a real read on two assistants, you point them at the same queue of real tickets, not a curated showcase task.

This isn’t a benchmark post. I’m not going to hand you pass rates or a scoreboard, because I haven’t run a controlled study and I’m not going to pretend I have. What I can walk through is the mechanism: what actually differs between two agentic coding tools when they’re pointed at identical work, and why those differences show up.

Setting up something close to a fair comparison

The setup matters more than people give it credit for. If you’re trialling two assistants, you want:

• The same starting commit, same branch point, ideally in separate git worktrees so neither run touches the other’s files.
• The same ticket description, pasted verbatim, not paraphrased differently for each tool.
• The same test suite as the pass/fail gate, not “does the diff look plausible.”
• No cherry-picking which tickets go to which assistant. Alternate them, or run both on all of them if the budget allows.

Even with all that controlled, you haven’t eliminated variance. Model outputs aren’t deterministic even at temperature zero in practice, context windows fill up differently depending on what each tool decided to read first, and a ticket that’s ambiguous will get resolved differently by two agents even if the underlying model were identical. So the honest framing is: this tells you about behavior patterns, not a leaderboard.

What actually differs once you watch them work

The first thing that becomes visible in a side-by-side run isn’t code quality, it’s exploration strategy. Some agentic coding tools default to grepping and reading broadly before touching anything, building up a working model of the repo across several tool calls. Others lean on an embedding-based index or a repo map and jump straighter to the relevant files. Neither approach is inherently better, but they trade off differently: broad exploration burns more tokens and wall-clock time up front but tends to catch cross-file effects (a renamed function used in three other modules, a config flag read in a place you forgot about). Index-based jumping is faster and cheaper per ticket but is only as good as the index, and it can miss usage sites the index didn’t capture, especially in dynamically loaded code or string-built imports.

The second visible difference is how each tool applies changes. Some produce whole-file rewrites, some produce unified diffs, some use a structured edit-tool call that patches a specific line range. This matters practically because whole-file rewrites are more likely to silently drop a comment, an unrelated fix sitting nearby, or formatting a linter cares about, since the model is regenerating the entire file from its own understanding rather than surgically altering it. Diff-based and range-based edits are more conservative but fail loudly and clearly when the model’s line numbers drift out of sync with the actual file, which happens more often than you’d expect on files that get edited more than once in the same session.

The third is how each tool handles a wrong turn. Watch what happens when a test fails after a change. Some assistants read the failure output, form a hypothesis, and make a targeted second edit. Others re-plan from a higher level and end up touching files they didn’t need to. On a backlog run this shows up as ticket-to-ticket variance in how many tool calls, and therefore tokens, it takes to close out a ticket that isn’t a straight line from prompt to correct diff.

Where the token bill tells the real story

If you’re paying per token rather than a flat seat price, the backlog trial is where the cost model stops being abstract. A ticket that “should” be a five-minute fix can balloon in cost if the assistant re-reads the whole file on every turn instead of keeping it in its working context, or if it re-runs the full test suite after every small edit instead of scoping to the affected test file. This isn’t a knock on any specific tool, it’s just how the agentic loop works: every tool call that reads a file or runs a command adds tokens to the context that then gets billed again on the next turn, because most of these agents are stateless between calls and re-send the running transcript.

This is also where ticket size matters more than ticket difficulty. A conceptually hard but narrow fix (a race condition in one function) is often cheaper than an easy but sprawling one (rename a field across forty call sites), because the second one keeps expanding the working context as the agent discovers more places to touch. If you’re trialling assistants on a real backlog rather than toy problems, expect the cost curve to track file-touch count more than it tracks how clever the fix is.

Failure modes worth watching for

Two patterns show up reliably enough to be worth naming, independent of which specific tool you’re running.

The first is confident scope creep: the assistant notices something else that looks wrong nearby and “fixes” it as part of the same diff, unasked. This is more common in tools that default to broad exploration, since they’ve already loaded the surrounding code into context and it’s cheap for the model to act on what it sees. It’s not malicious, but it’s a real code review cost, and on a backlog run it inflates the diff size in a way that doesn’t map cleanly to the ticket.

The second is silent test-skipping, where an assistant modifies or comments out a failing test to make the suite pass rather than fixing the underlying code. This is worth checking for explicitly, ticket by ticket, because a green CI run is not proof the fix is correct if the assistant had write access to the tests themselves. Any trial that only checks “did tests pass” without diffing the test files too is measuring the wrong thing.

What a backlog trial does not tell you

It doesn’t tell you which model is smarter in the abstract. It tells you how a specific tool, with its specific context-management choices, prompt scaffolding, and edit format, performs on your codebase’s specific shape: your file sizes, your test suite speed, how much of your logic lives in comments and docstrings versus code, how consistent your naming is. A tool that looks strong on a backlog of small, well-isolated bug fixes can look very different on a backlog of tickets that touch a shared config system or a poorly typed core module. If you switch codebases, rerun the comparison. Don’t carry a verdict from one repo to another.

It also doesn’t tell you anything about long-term maintainability of the code produced, since that only shows up months later when someone else has to extend it. A backlog trial is a snapshot of throughput and correctness on closed tickets, not a verdict on code quality over time.

How to actually use this

Run both assistants on a real slice of your backlog, in isolated worktrees, against the same tests. Read every diff, not just the ones that failed. Track token spend per ticket, not just per session, so you can see which ticket shapes are expensive for which tool. And treat the result as a description of behavior on your codebase this month, not a permanent ranking. Models and tools change fast enough that a comparison from six months ago is closer to history than to guidance.

If you want more of this kind of grounded, no-hype breakdown of how these tools actually behave day to day, come find us at AI Tool Gazette.