The Information Machine
ConcludedRan 10 days6 sourcesupdated 17 Oct 2026

Harvard AI coding agents study

The gist

Latest: Harvard Study: AI Coding Lifts Output 30%, Review Time 49%

The Harvard study's finding that increased code generation did not translate to more shipped software points to code review as the binding constraint on AI-assisted development productivity. The CMU and Microsoft benchmark result that AI reviewers miss roughly one in four bugs even under favorable conditions, and perform worse in private codebases, bears directly on whether AI review can close that bottleneck.

When stories like this move, I write about them in an email each morning.

One free email each morning, around 8 AM US Eastern.

The full picture

A Harvard economics paper by Fiona Chen and James Stratton, drawing on 300 million work events across 718 firms and roughly 700,000 workers, found that AI coding agents raised lines of code written by 30%. That gain did not produce a meaningful increase in completed tasks or shipped software at the firm level. Code review times rose 49%, the share of pull requests requiring changes nearly doubled, and reviewer comments per pull request rose 35%. Firm employment levels did not change following AI coding tool adoption. The researchers identified agent use through internal system traces including bot accounts and 'Co-authored-by' metadata in commit messages and pull request descriptions; the underlying firm data are proprietary and individual AI tool licenses went unrecorded.

A separate Carnegie Mellon and Microsoft study using a benchmark called CRBugBench, covering 1,800 code review tasks each containing a confirmed, human-fixed bug, found that the best-performing AI code review agent caught 71.9% of known bugs in its top 5 comments. Performance dropped by as much as 25.5% on one company's private repositories, where bugs more often require project-specific context to identify.

How it developed
16 October 2026added 17 Oct

A Harvard economics paper by Fiona Chen and James Stratton found AI coding agents raised lines of code written by 30% across 718 firms, but the gain did not translate to more shipped software at the firm level, with code review times rising 49% and the share of pull requests requiring changes nearly doubling.

Firm employment did not change, and the study, covering roughly 700,000 workers across 700 firms, noted AI authorship metadata in commit histories may need attention under retention schedules and legal holds.

12 October 2026

JetBrains Research paper published arguing coding-agent benchmarks don't match real developer usage patterns

8 October 2026

Harvard study finding published: AI coding agents raised code output 30% across 718 firms but did not increase shipped software; code review times rose 49%

Sources
1 more source

Get the daily email

When stories like this move, I write about them in an email each morning.

One free email each morning, around 8 AM US Eastern.

See the latest edition · Privacy