A Pull Request Is Not a Product Metric

The good news is that we have a number: developers in Microsoft’s early-2026 rollout of Claude Code and GitHub Copilot CLI merged roughly 24% more pull requests than the study’s counterfactual estimated they otherwise would have. The better news is that the paper says, in the same breath, that a merged pull request is a proxy for output, not the value delivered. This is the kind of caveat that should arrive with a small brass plaque, because it is about to be run over by a dashboard.

The study is worth taking seriously. It covers tens of thousands of engineers, observes a four-month window, and does not present agent use as a magical personality trait bestowed on people who enjoy terminals. The authors also point out that adoption spread socially and that retention correlated more with coding activity than demographics. In plain English: your staff engineer using an agent visibly is more consequential than an all-hands slide containing an approved logo.

What the 24% means, and what it very much does not

It means a particular group of developers, using particular command-line agents in one company’s workflow, merged more PRs than a modeled no-agent baseline. It does not mean that your team will ship 24% more customer value, have 24% fewer incidents, or discover that the payments migration was merely waiting for a subscription plan and the courage to type “continue.” The paper itself is careful here; the dangerous interpretation is the frictionless journey from research finding to quarterly OKR.

PRs are a reasonable operational signal because they are observable, dated, and already sitting in the database where everyone is tempted to ask for a chart. They are also a unit whose size is set by local habits. One team’s excellent PR is a 40-line bug fix with a focused regression test. Another team’s excellent PR is a 1,200-line migration split into twelve mergeable steps. A third team has discovered that a documentation typo and a database index can both be called “small.” The metric politely records all of this without understanding any of it.

There is an important practical distinction here. Agents are plausibly good at producing work that is easy to package: narrow refactors, test cases, mechanical API migrations, dependency chores, and the kind of repeated edit that makes a human briefly consider a quieter career in forestry. More mergeable units may be a genuine workflow improvement. But an agent can also make it cheaper to split work into more units. If the denominator changes because the definition of “one PR” quietly changed, congratulations: your graph improved and your measurement program became performance art.

Measure the workflow, not the confetti

If you are evaluating an agent rollout, take the PR lift as a hypothesis, not an outcome. Run a six-week trial on a few teams that already have stable CI and a deploy history. Do not start with the repository where every test takes 47 minutes and two approvals are required from a person currently hiking in Patagonia. That is a test of organizational geology, not an AI tool.

Before enabling anything, save a baseline. For each participating repository, track five things weekly: merged PRs; median time from PR creation to merge; reverts or production hotfixes within 14 days; CI failure or retry rate; and one delivery or product measure the team already trusts, such as successful deployment completion, checkout latency, or a support-contact rate. The fifth measure is deliberately local. “Value” lives in the system you operate, not in an unusually confident spreadsheet tab named Productivity.

gh pr list --repo ORG/REPO --state merged --limit 500 \
  --json number,createdAt,mergedAt,additions,deletions,changedFiles \
  --jq '.[] | [.number, .createdAt, .mergedAt, .additions, .deletions, .changedFiles] | @tsv'

That command gives you a quick export for a before-and-after check. GitHub CLI supports merged-state filtering and the JSON fields used above. Treat mergedAt - createdAt as queue time, not engineering time: it includes reviewers being in meetings, CI contemplating the void, and the occasional Friday afternoon when nobody wishes to own a migration. Diff size is useful context, too, but it is not a quality score unless your architecture has become unusually fond of haiku.

The rule that keeps this from becoming a scoreboard

Do not set a PR-count target for individual developers. The fastest route to a 24% gain is to create smaller PRs, and the second-fastest is to route work toward people whose tasks happen to divide well. Neither tells you whether the agent helped. Individual targets also make reviewers the unpaid cleanup crew for an incentive system they did not ask to join, which is how a tooling pilot becomes a culture initiative with an incident channel.

Instead, ask whether the team’s throughput rose while its downstream signals stayed flat or improved. If merged PRs rise 24%, median merge time rises 30%, reverts rise, and the release cadence does not move, the agent may still be useful for developers. But the organization has not found a productivity multiplier; it has found a more efficient way to move work into review. That can be worth fixing, but call it what it is.

The most credible use of this research is modest: CLI agents appear capable of increasing merged output in a large real-world rollout, so they deserve a trial in work with clear tests, bounded scope, and an owner who can judge the result. They are bad at making a vague success criterion less vague. They are also bad at explaining why the “simple cleanup” deleted the condition guarding a decade-old enterprise customer. In fairness, so are people, but people at least look embarrassed in the incident review.

Try the tool. Let it draft the boring parts. Measure what happens after merge. And when somebody asks for the 24% dashboard, give them one with a second line underneath: “Did anything get better?” It is not a perfect metric. That is why it is useful.