Why another SWE benchmark
Specific Labs released Real-SWE, a benchmark that runs frontier AI coding agents against private, real-world enterprise codebases. The pitch is simple and hard to argue with: public benchmarks leak. SWE-bench tasks live in public repositories, and their solutions are on the internet. Any model trained after the benchmark published has, at minimum, an unclear advantage. Real-SWE sidesteps that by licensing tasks from real companies whose code is private, so the solutions are not anywhere the models could have seen them.
The tasks are not synthetic puzzles either. They are billing logic, tax calculations, customer migrations, multi-service changes with real business consequences. Each one comes with the context and conventions of an existing product, which is precisely the part public benchmarks strip out.
The numbers
Eight model and harness combinations were evaluated, each averaged over eight independent runs per task, which is a more careful protocol than most. The results are sobering for anyone who has watched demo videos and concluded coding is solved.
Anthropic’s Fable 5.1 running in Claude Code leads at a 38.8 percent resolution rate. OpenAI’s GPT-6 Astra in Codex CLI follows at 33.8 percent, then Google’s Gemini 3.8 Flash at 31.2 percent and Z.ai’s GLM 5.3 at 28.8 percent. At the bottom, OpenAI’s GPT-5.6 Sol manages 16.2 percent. Read those numbers again: the best frontier model, in its own native harness, completes fewer than four in ten tasks from real production codebases.
The spread matters as much as the top line. GPT-5.6 Sol is roughly two and a half times behind Fable 5.1 on identical tasks, which suggests the difference between model generations is still enormous on genuinely unfamiliar code. There is no plateau here.
What makes enterprise tasks hard
Real-SWE highlights two axes that synthetic benchmarks underweight. The first is the codebase itself: proprietary systems whose structure, history, and constraints the agent has never seen. The second is instruction specificity. Real tickets say things like fix invoice billing so each business charges the right tax and exempt customers are not taxed. That requires discovering business rules and how they interact with external services, not pattern-matching a known bug shape.
The evaluation setup runs each agent in an isolated sandbox with tasks in Harbor format and verifiers injected at grading time, inspired by or taken directly from existing test suites in the codebases. That last part is a quiet strength: grading against the company’s own tests is about as close to ground truth as benchmark design gets.
Token usage tells its own story. The published per-task figures show agents burning anywhere from a few thousand tokens on tasks they handle cleanly to well over a hundred thousand on the hardest ones, sometimes without resolving them. GLM 5.3 burned the most overall at around 117k tokens per task average, yet still finished behind models that used a fraction of the compute. Efficiency is not the same as capability, but at enterprise prices the gap between a model that solves a task in 13k tokens and one that needs 90k while failing is a line item, not a curiosity. Any team paying per million tokens should look at the cost-per-resolution column before the leaderboard position.
Reading the results honestly
A few cautions are worth holding onto. One benchmark vendor choosing its own tasks, with models evaluated through specific harnesses, is a data point, not a verdict. Harness choice clearly influences results, which the leaderboard itself acknowledges by naming the harness for every entry. And eight base tasks is a small sample, even with eight runs each, so the confidence intervals will be wide at the task level.
What this means for procurement
For teams choosing between these tools, the benchmark suggests three practical questions to ask vendors. First, what resolution rate do you see on codebases like ours, and can you prove it on a pilot of our own tasks rather than a public leaderboard? Second, what does the token cost per resolved task look like, since the raw scores hide an order-of-magnitude spread in compute burned? Third, which harness is the number from, and does your recommended workflow match the harness that produced it?
It also matters how you deploy. The resolution rates here are for autonomous task completion, where the agent gets a ticket and comes back with a change. Most teams use these models interactively, for completion, explanation, and review, where the effective hit rate is higher because a human steers. The 38.8 percent number is not a ceiling on usefulness; it is a measure of unsupervised end-to-end capability, which is a different product than the one most developers actually run.
Still, the direction is credible, and the headline finding matches what most teams discover when they bring coding agents to their own repos: performance on private, messy, convention-heavy code is meaningfully worse than the public benchmark numbers suggest. If your team is making tooling decisions based on SWE-bench scores, Real-SWE is a useful corrective. The models are improving quickly, but on real work with real stakes, four in ten is where the frontier sits today.