Every Benchmark Chart Is a Bar Chart Where Someone’s Bar Is Tallest

This is satire, which is fortunate, because the Department of Maximum Rectangle has issued its annual finding: every coding tool is now the best coding tool, provided it is measured inside the correct terrarium. The terrarium must contain one repository species, a carefully approved shell, a 30-minute sunlamp, and no product manager asking whether the patch can be deployed before lunch. Under these controlled conditions, one bar will be tallest. Science has asked that you look at it.

The chart is not lying. It is merely wearing business casual.

A benchmark result can be perfectly real and still be a poor purchasing decision. That is not a scandal; it is the ordinary property of measurements. A coding-agent score represents a model plus an agent harness plus tools plus prompts plus budgets plus retries plus environment plus the particular definition of “done.” Change any one of those and the rectangle may experience a growth event.

The Official Institute for Contextual Victory recommends the following announcement format: “Tool Pelican resolves 73.2% of the tasks, a number that rises to 104% when adjusted for moral clarity.” The first number is intentionally plausible-looking; the second is there so nobody mistakes this for reporting. The accompanying footnote should explain that Pelican received the full repository, a bespoke search tool, three retries, an environment constructed by twelve monks, and permission to ask the benchmark politely for hints.

The interesting detail is usually in that footnote. OpenAI’s guidance on third-party evaluations makes the unglamorous point that capability claims depend on elicitation and that evaluators should use a harness suited to the task. This is not an accusation against bars. It is an instruction to read the harness before appointing a bar to the architecture council.

Run the benchmark once, preferably on a machine you do not need today

If a benchmark influences a tool decision, run enough of it to discover what it measures. SWE-bench’s own repository describes an evaluation that applies generated patches to real issue tasks and tests them in Docker-based environments. Its quick-start command is admirably free of marketing:

python -m swebench.harness.run_evaluation \
  --predictions_path gold \
  --max_workers 1

Then meet the benchmark’s operational personality. The SWE-bench README recommends an x86_64 machine with at least 120 GB of free storage, 16 GB RAM, and 8 CPU cores for resource-intensive evaluation; ARM support is described as experimental. This is useful information. It tells you that a score is not a tiny pure number found under a leaf. It is the end product of a substantial execution environment, one that can fail because Docker is sad or because your laptop has elected to become warm enough to host sourdough.

The correct response is not “benchmarks are fake.” The correct response is “what did we hold constant?” Did each agent get identical model access? The same timeout? The same command permissions? The same test command? The same retry policy? The same checkout? If Agent A can read the whole repository and Agent B receives a zip file through a mail slot, the graph is not comparing agents. It is comparing office layouts.

The benchmark selection meeting

At Acme Synergy Foundry, a fictional organization with a 900-person Benchmark Alignment Guild, tool selection begins with a 46-slide deck. Slide 31 contains the decisive evidence: a chart cropped so precisely that the y-axis has become a rumor. The winning tool is approved for all developers, including the team whose production work is Kotlin, the team whose failures are Terraform plans, and the person maintaining a 14-year-old Perl integration called “please_dont.”

A working developer can conduct a cheaper meeting. Take ten or twenty closed tasks from your own backlog: flaky tests, dependency upgrades, an API migration, a production bug with a reproduction, a documentation change that touches generated code, and one task where the best answer is “do not change anything.” Record the repository state, command permissions, model, spend cap, wall-clock timeout, test command, and whether a human had to redirect the agent. Keep the failed runs. Especially keep the run that produced a beautiful refactor of the wrong subsystem in 11 minutes and 42 seconds. That is product information.

• Set a budget before the run, not after the best result. “Unlimited until demo day” is not a budget.
• Use the same task description and acceptance tests for every tool. A tool that receives a better brief has not become more intelligent; it has acquired a better manager.
• Record first-pass success separately from success after intervention. These are different workflows with different staffing costs.
• Track wall-clock time and human review time. A cheap patch that requires forty minutes of archaeological review is an expensive patch wearing a mustache.
• Read the diff. Passing tests are necessary evidence, not a notarized certificate that the change belongs in your codebase.

What the tall bar is bad at

The tall bar is bad at communicating uncertainty, setup burden, and the distinction between “can solve” and “will solve on Tuesday while your CI credentials are expired.” It is bad at showing whether the agent made six harmless edits or changed a lockfile, a deployment script, and the spiritual orientation of the monorepo. It is bad at telling you whether the tool helps in the editor, in a terminal, in a code review, or only after a research engineer has installed a harness with an acronym.

It is also bad at predicting the task that currently has your name on it. That task has private conventions, stale docs, partial observability, a test suite with one ceremonial failure, and a teammate who remembers why a function is named normalize2. No public benchmark has yet captured the performance impact of being able to ask that teammate a question.

A modest policy for people who must buy tools

Treat public benchmarks as a shortlist generator, not a verdict. Use them to find candidates, inspect their harnesses, and reject claims that omit the operational details you would need to reproduce the result. Then run the candidates on representative work with a defined spend and a test command your CI would recognize. This may result in fewer dramatic launch graphics, but it has the compensating advantage of selecting a tool your developers will open after their first coffee.

One true observation remains standing after the rectangles have gone home: a benchmark is most valuable when it makes you write down what good work means for your team.