Developer productivity and AI · Peer-reviewed empirical study or research synthesis
The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers
Peng, S. & Demirer, M. (2026). The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science (Articles in Advance).
Plain-English take
Pooling the experiments, AI access increased completed tasks by 26.08% (SE 10.3%); less-experienced developers adopted more and had larger gains, while individual experiments were noisy and heterogeneous. Founder-use interpretation: Strong evidence for testing AI-assisted engineering productivity, not a promise of proportional headcount or SaaS growth; run a controlled rollout with quality, security, rework, and customer outcomes.
How much weight to give this
4.5/5 Useful empirical or synthesis evidence with context-specific transfer limits
The 1–5 evidence-check score is a practical reading aid, not a formal peer review or a probability that a conclusion is true. It combines design strength (2 points), sample reporting (1), comparison/control clarity (1), and how much of the paper could be checked (1). Access affects how confidently this directory can assess a study, not the study's underlying quality. Blinding is reported separately because it is not relevant to every design.
Scored from the described study design; an analytical or review design is not treated as a randomized field estimate.
The setting and reported sample detail are stated explicitly; size alone does not establish representativeness.
The reported comparison or synthesis method is described from the retrieved article record.
A publisher page or abstract was directly retrieved; full methods access was not established.
Double-blind? Blinding is not stated in the publisher record reviewed; that does not establish whether blinding was or was not used.
Research question
What the authors investigated
What effect did access to generative AI have on software developers’ task completion in three company field experiments?
Design
Three randomized field experiments run in ordinary business operations at Microsoft, Accenture, and an anonymous Fortune 100 company; 4,867 software developers received randomized access or no access to an AI coding assistant.
Sample and setting
Three randomized field experiments run in ordinary business operations at Microsoft, Accenture, and an anonymous Fortune 100 company; 4,867 software developers received randomized access or no access to an AI coding assistant.
Reported result
What the study found
Pooling the experiments, AI access increased completed tasks by 26.08% (SE 10.3%); less-experienced developers adopted more and had larger gains, while individual experiments were noisy and heterogeneous.
This is a paraphrase of the source. Check the original paper for definitions, estimates, statistical uncertainty and analysis details.
Scope and transfer
Limitations to keep in view
Company-specific coding assistant, firms, tasks, and outcome metric limit external validity; pooled estimate masks variation and does not establish effects on product quality, revenue, or long-run retention.
Possible founder application
Strong evidence for testing AI-assisted engineering productivity, not a promise of proportional headcount or SaaS growth; run a controlled rollout with quality, security, rework, and customer outcomes.
Evidence access
Publisher abstract or article landing page captured; full-text access not established.
Publisher article page directly retrieved with Firecrawl (reference 01a122bc-9e6a-738d-9df6-6070598d013e). The annotation follows the retrieved title/abstract or landing-page record; full methods and correction notices were not independently reviewed.
If only an abstract or metadata was available, the profile is limited to claims supported there. A paper's paywall does not change its evidentiary status, but it limits what we can substantiate from accessible text.