A new benchmark puts coding agents at 23.9% where expert-built agents reach 82.2%
Tau-tau-Bench asks agents to build customer-service systems rather than to pass unit tests. The gap between the best model and a human-engineered agent is the whole result.
A benchmark published on arXiv on 8 September puts the best AI systems at 23.9 percent on the task of building customer-service agents, against 82.2 percent for agents built by expert humans.
The benchmark, τ^τ-Bench, is recursive by design: it evaluates coding agents on their ability to construct other agents. That framing is what makes the numbers interesting, because it measures the work that the industry is currently claiming is nearly automated.
Why the gap is so much larger than on other benchmarks
Coding benchmarks have been saturating. Models routinely clear high percentages on SWE-bench-style tasks, and vendors now report software engineering as a headline capability — OpenAI claims Astra is the best model yet for bug-finding and code analysis, and Nvidia's Nemotron-3-Ultra-CC reportedly scored 535.4 of 600 at the 2026 International Olympiad in Informatics, ahead of the top human at 498.27.
So a 23.9 percent result needs explaining, and the explanation is in what the task requires.
A typical coding benchmark supplies a specification and a test suite. Success is defined in advance and verified mechanically. Building a working customer-service agent is not that: it requires deciding what the system should do in situations the specification does not cover, handling users who are unclear or hostile, choosing where to fail safely, and making judgement calls about scope that no test encodes.
The 82.2 percent human figure establishes the task is achievable. The 23.9 percent figure suggests what current systems are missing is not code generation.
The result to hold alongside it
BAAI published work on 3 September running in the opposite direction. Its DisCo framework converted 1,000 GitHub repositories into 5,000 reusable agent skills at roughly $40 per repository, and reported an MLE-bench improvement from 31.1 percent to 72.9 percent — with a publicly inspectable skill library.
Those two findings are compatible and together they say something useful. Agents improve substantially when given well-structured, reusable components built from existing work. They remain weak when asked to make the design decisions that produce those components in the first place.
That is a description of the current frontier that fits the deployment evidence better than any benchmark score. Cognition is at $900 million of run-rate revenue selling an autonomous coding agent; Meta's AI-native workforce push produced 220 percent more code changes, 36 percent more delivered features and 40 percent more incidents. More code, more shipped, more broken.
What the benchmark does not settle
A single benchmark from one group, newly published and not independently replicated, is weak evidence about a capability frontier. The scoring of what constitutes a working customer-service agent is a judgement call that the paper's authors have made, and different judgements would move the number.
The expert baseline is also not a like-for-like comparison: experts building agents had unlimited time and their own tooling, which flatters the human figure in ways the paper would need to control for.
What is useful is the shape of the task. Benchmarks that supply a specification and a test suite measure the part of software engineering that was already the easiest part. This measures the part that consumes most of a senior engineer's day, and on that part the systems are not close.
Runs the newsroom. Rename this profile in the studio to your own byline.
Related
Every weekday, the AI stories that moved money or shipped code.
No cross-posting, unsubscribe anytime. See all newsletters