Measuring AI Coding ROI: Cost per Merged PR Beats Lines of Code Generated

Measure AI coding ROI as fully loaded cost per merged PR, not lines of code generated. It’s not a perfect measure of business value, but it is much harder to game and much closer to the thing your team actually has to review, test, merge, deploy, and support.

A coding agent can generate 20,000 lines overnight and still create no useful change: the branch may be abandoned, the diff may be split into smaller human-written PRs, or reviewers may spend an afternoon deleting most of it. Generated LOC mostly tells you that a model emitted tokens. A merged PR tells you that a change crossed at least one meaningful human and CI boundary.

Start with a denominator your team recognizes

Don’t begin by asking the agent vendor for a dashboard. Start with GitHub, GitLab, or whatever system is the source of truth for merging. Pick a 28-day window, count merged PRs, and calculate the cost associated with the cohort of people and repositories using the tool during that same window.

For a GitHub repository, this gives you a quick export to inspect. Replace the date and repository, then save the output before you start trying to explain it in a spreadsheet:

gh pr list \
  --repo acme/payments-api \
  --state merged \
  --search "merged:>=2026-08-27" \
  --limit 1000 \
  --json number,title,mergedAt,additions,deletions,changedFiles,labels,author \
  > merged-prs.json

GitHub CLI exposes merged time, changed-file count, additions, deletions, labels, author, and review data as JSON fields, so you don’t need to scrape the PR page or invent a parallel reporting system. Do not use the additions and deletions fields as the score. Keep them for diagnosis: a sudden rise in giant PRs can explain why cost per merged PR improved while review time became miserable.

The basic calculation is deliberately boring:

Fully loaded AI cost for the period ÷ eligible merged PRs for the period = cost per merged PR.

“Eligible” needs a written rule. A practical first rule is: count merged PRs authored by people in the trial cohort, excluding dependency bots, automated version bumps, reverts, and PRs in repositories the cohort did not touch. Don’t count only PRs carrying an ai-assisted label unless that label is enforced; voluntary labels become incomplete the moment someone forgets to add one.

Put the real costs in the numerator

The license price is rarely the whole numerator. Include the per-seat subscription or usage charge, model/API overages, and incremental CI or sandbox spend caused by agent runs. If the tool runs jobs in your CI, include those jobs even if they still fit inside a bundled allowance. An allowance is not proof that the work was free; it is a bill you have prepaid.

For GitHub organizations on enhanced billing, the billing usage API can return usage records with product, repository, quantity, unit type, price, and net amount. That makes it possible to allocate GitHub Actions spend by repository instead of dividing the monthly organization invoice evenly. GitHub also notes that private-repository jobs on hosted runners can be billed by runner type and that displayed job minutes may not include the final pricing multipliers, so use the billing report—not a stopwatch on the Actions page—for the dollar amount.

Be consistent about human time. I would not try to monetize every minute a developer spends prompting; that turns this into a timesheet project and invites bad data. Do include a separately tracked one-time implementation cost when it is material: setting up policy, writing repository instructions, connecting an MCP server, or building a custom evaluation harness. Amortize that cost over a stated period, such as six months, rather than quietly burying it in month one.

Use a baseline, not a before-and-after story

Compare the 28-day trial against the preceding 28 days for the same repositories and, where possible, the same developers. Record total merged PRs, total engineering days available, and the fully loaded tool cost. If half the team was on incident rotation, a release freeze started, or a major migration ended, write that down next to the number. The spreadsheet will not discover confounders for you.

A useful reporting row looks like this: “Payments API, 8 engineers, 28 days, $1,240 AI and incremental CI spend, 62 eligible merged PRs, $20 per merged PR; baseline: 49 merged PRs with 8 engineers.” That is actionable. It supports a follow-up question: did the 13 additional merges represent useful throughput, or did the agent merely fragment work into smaller reviews?

This is also why cost per merged PR should be a team or repository metric, not a developer leaderboard. One engineer may own gnarly schema migrations, another may ship small UI fixes, and a third may spend the month reviewing. Ranking people by this number is a fast route to PR splitting, shallow changes, and resentment.

Add guardrails so the number can’t lie alone

A merged PR is an acceptance signal, not evidence that the change was good. Pair the cost metric with a small set of guardrails and review them in the same window:

• Median time from PR creation to merge. If agent-assisted work increases merge count but pushes review time from hours into days, you moved work into the approval queue.
• Revert or hotfix rate for changes from the cohort. Define the lookup window in advance—14 or 30 days—and do not change it after seeing results.
• Change lead time and deployment frequency for the affected service. DORA defines lead time as commit-to-production time and explicitly warns that broad metric targets can be gamed.
• Median changed files and additions plus deletions per PR. This is a smoke alarm for review-unfriendly agent output, not a productivity score.

If the merged-PR count rises but the revert rate rises with it, you did not find ROI. If cost per merged PR improves while lead time gets worse, inspect queueing: perhaps the agent produces code faster than reviewers can validate it. The fix might be narrower agent tasks, better tests, required plans for cross-cutting changes, or simply fewer parallel runs—not a larger model budget.

Run the first measurement before buying another dashboard

Create the cohort as a GitHub team or a short, fixed list of users. Choose two or three active repositories. Set a 28-day start and end date. Export merged PRs on the same day each week, pull the billing line items, and write down exclusions in a README beside the spreadsheet. After one cycle, compare against the prior 28 days; after two cycles, decide whether the signal is stable enough to expand.

Cost per merged PR is bad at valuing rare, high-leverage work. A one-line production fix, a security review that prevents an incident, and a difficult architecture decision may each look terrible under this metric. Keep an exception field for work that plainly should not be judged by PR volume. But for the ordinary stream of maintenance, features, tests, refactors, and integration work where AI coding tools claim to help, it is a much more honest starting point than a counter of generated lines.