For three years the coding-agent leaderboard has been a single column. SWE-bench hands a model a bug report and a repository, lets it work, and asks one question: did the final patch pass the hidden tests? Every headline you have read — "72% on SWE-bench Verified," "a new state of the art" — collapses an agent's entire performance into that one resolved-or-not bit. It is a clean number. It is also a lie of omission, because it grades the agent as a vending machine: task in, patch out, no one in the room.

That is not how anyone actually uses these things. Real coding assistance is a conversation. You ask for something half-specified, the agent guesses, you say "no, not the auth module, the session store," it tries again, you add a constraint you forgot, and eventually the tests go green. The final patch passing tells you nothing about whether that took one turn or nine.

SWE-Together, a new benchmark out of a group publishing under the Togetherbench banner, is built to measure the turns.

A second axis, not a harder test#

The construction is the interesting part. The authors started with 11,260 recorded real user–agent coding sessions and curated them down to 109 replayable tasks — keeping only sessions with recoverable repository state, a legible user goal, and an observable outcome. Each task begins the way real work does: a first user message and a scripted interaction, not a tidy specification.

To replay that interaction against a different agent than the one that originally ran it, they built a reactive LLM user simulator — a stand-in human that preserves the original user's intent, answers clarifying questions, and pushes back when the agent drifts. Then they score two things that a static benchmark cannot separate:

Correctness asks whether the agent got there. User Correction asks how many times you had to grab the wheel. They are different measurements, and the second one has been invisible.

This is the whole thesis in one line. A single pass rate silently averages over an enormous variance in how much babysitting each success required. Two agents can both land at 60% resolved while one of them needed you to intervene three times as often to get there. On a static benchmark they tie. On SWE-Together they do not, and the tie-breaker is the thing you actually feel every day.

Where the models land#

On the reported results, Claude Opus 4.8 leads on both axes at once: the highest correctness at roughly 63% pass@1, and the fewest corrective turns of any model measured. That "at once" is the finding worth sitting with. It would have been more theoretically interesting if the axes traded off — if the most capable model were also the most stubborn, the one that confidently marched to a wrong answer you had to keep yanking back. Instead, at least at the frontier, capability and steerability currently move together: the model that solves the most is also the one that argues with you least.

But do not over-read a single leaderboard. The more durable point is structural. Once you have a correction axis, you can imagine models that game one and not the other — an agent tuned to solve benchmark tasks in one heroic shot but that ignores mid-course feedback, or a sycophantic one that accepts every correction gracefully while never getting the job done. A single number can't catch either. Two orthogonal numbers can. It is the same suspicion that produced error-recovery benchmarks: the aggregate pass rate hides how an agent behaves once something has already gone wrong.

The ruler is made of the same material#

Here is the part a numbers desk is obligated to flag. The human in SWE-Together is not human. It is an LLM playing a user, and every correction it issues is a model's judgment that the agent went off-track. So the benchmark is partly measuring how well the agent negotiates with another model's idea of what a user wants — a hall of mirrors that a static test suite, for all its narrowness, never enters.

The authors clearly know this, which is why the benchmark carries a second guard rail: Intent Coverage, a check on whether the simulator consistently conveys the original user's underlying intents across different runs. That is the right instinct. A benchmark that grades collaboration is only as trustworthy as its fake collaborator, and Intent Coverage is the audit on the ruler itself. Treat any User Correction score as a measurement of a pair — the agent and the simulator — not of the agent alone.

SWE-Together is not arriving alone, either. SWE-INTERACT reimagines SWE-bench tasks as user-driven long-horizon sessions; a parallel line of work compiles a user's repeated corrections into runtime enforcement so the agent stops making the same mistake twice. The static, one-shot benchmark had a good run. But the field has quietly noticed that the interesting failure is no longer the wrong answer — it is the right answer that cost you five corrections to reach. The leaderboard is finally growing a second column, and it measures patience.