July 7, 2026. Most AI agent benchmarks ask one question: did the agent finish the task? A new benchmark from Meta researchers, published June 29 on arXiv and circulating widely in AI coverage this week, asks the question that actually decides whether agents pay off in a business: how much human correction did it take to get there? SWE-Together replays real multi turn working sessions between users and coding agents, and it scores every model on both final correctness and the corrective steering it demanded along the way. That second number is the one most operators have never measured, and it maps directly to what supervised AI automation costs in practice.
What SWE-Together measures
- Real sessions, not puzzle prompts. The team curated 109 repository level tasks from 11,260 recorded user and agent coding sessions, keeping only sessions with recoverable repo states, clear goals, and observable outcomes. A reactive LLM based user simulator replays each session against any agent, preserving the original user's intent and pushing back when progress goes off course.
- Two scores per agent. Each model gets correctness metrics, including pass at 1, and a User Correction score counting how many corrective messages the simulated user had to send per task. Fewer corrections means less babysitting.
- Claude Opus 4.8 tops the board. It leads pass at 1 at 63 percent and needs the fewest corrections per task at 1.38, but it is also the heaviest runner at about 74,000 output and reasoning tokens and 23.3 minutes per task.
- GPT-5.5 is the efficiency pick. It ranks second on correctness at 58 percent pass at 1 with 1.59 corrections, while using the fewest tokens, about 29,900, and the least wall clock time, 10.7 minutes per task, roughly twice as fast as the leader.
- Capability and babysitting move together. Across the seven evaluated models, corrective steering is strongly inversely correlated with pass at 1, a Pearson correlation of minus 0.92. The weakest model needed 2.17 corrections per task. And even the best agent sits about 15 points below the human reference patches, so no model is unsupervised yet.
What it means for operators
Supervision is the hidden line item in AI automation. A model that looks cheap per token but needs constant redirection burns the most expensive resource you have, a skilled person's attention. SWE-Together is the first widely shared benchmark to put a number on that, and the practical takeaway generalizes far beyond coding: when you pilot any agent, count corrections per task, not just completions. That one metric tells you whether the agent is compounding your team's time or consuming it. The results also frame a real routing decision, since the most autonomous model was also the slowest and heaviest while the runner up was twice as fast at lower cost, the same effort versus economy tradeoff we flagged when Claude Sonnet 5 launched. Route long unsupervised work to the model that needs the least steering, and quick supervised tasks to the fast efficient one. It is more evidence that testing agents against real recorded work before production, with a human on irreversible steps, is what separates the agent projects that survive from the 40 percent Gartner expects to be canceled. If you want that discipline applied to your own workflows, from model selection to measured pilots, that is the core of a serious AI automation build, and exactly what you get when you hire an AI engineer who measures steering cost, not just demo success.
Frequently Asked Questions
SWE-Together is a benchmark from Meta researchers, published June 29, 2026 on arXiv, that evaluates AI coding agents in interactive multi turn sessions. It rebuilds 109 repository level tasks from 11,260 real user and agent sessions and replays them with an LLM based user simulator, scoring both final correctness and how many corrective messages each agent required.
Claude Opus 4.8 led with 63 percent pass at 1 and the fewest corrections per task at 1.38, though it used the most tokens and time. GPT-5.5 ranked second on correctness while being the fastest and most token efficient of the seven models evaluated.
Every correction an agent needs consumes skilled human time, which is usually the most expensive part of an automation. Two agents with similar success rates can have very different real costs once you count supervision, so corrections per task is a better predictor of ROI than benchmark scores alone.
Measure corrections per task in your own pilot before rolling an agent into production, route autonomous long running work to models that need the least steering and quick supervised tasks to faster cheaper ones, and keep a human approval step on anything irreversible.