Product adoption and customer behavior · Mixed-methods case study of an IBM AI code assistant under development: surveys of two user cohorts and unmoderated usability testing.
Examining the Use and Impact of an AI Code Assistant on Developer Productivity
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025). Examining the Use and Impact of an AI Code Assistant on Developer Productivity. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems.
Plain-English take
The assistant generally produced net perceived productivity increases, but benefits were not experienced equally; the study also surfaced motivations, expectations about speed/quality, and questions about ownership and responsibility for generated code. For a SaaS founder: For developer-tool founders, measure perceived and behavioral productivity together, segment by user, and make provenance, ownership, responsibility, and output-quality workflows explicit. Avoid assuming that aggregate positive sentiment means every team or task benefits.
How much weight to give this
3.5/5 Context-specific or model-based evidence; use as a hypothesis
The 1–5 evidence-check score is a practical reading aid, not a formal peer review or a probability that a conclusion is true. It combines design strength (2 points), sample reporting (1), comparison/control clarity (1), and how much of the paper could be checked (1). Access affects how confidently this directory can assess a study, not the study's underlying quality. Blinding is reported separately because it is not relevant to every design.
The design type and its limits are described from the directly fetched article.
Reported sample details are included; when the source does not give a count, that is stated.
The experimental, review, observational, or model comparison is identified without converting it into a causal claim.
The publisher abstract or record was directly retrieved.
Double-blind? Blinding is not reported in the source reviewed; that does not establish whether blinding was or was not used.
Research question
What the authors investigated
How does use of an AI code assistant affect developers' perceived productivity, usage choices, and concerns about generated-code responsibility?
Design
Mixed-methods case study of an IBM AI code assistant under development: surveys of two user cohorts and unmoderated usability testing.
Sample and setting
IBM developer users of a general-purpose assistant supporting common programming languages; publisher abstract reports survey cohorts totaling N=669 and unmoderated usability testing N=15.
Reported result
What the study found
The assistant generally produced net perceived productivity increases, but benefits were not experienced equally; the study also surfaced motivations, expectations about speed/quality, and questions about ownership and responsibility for generated code.
This is a paraphrase of the source. Check the original paper for definitions, estimates, statistical uncertainty and analysis details.
Scope and transfer
Limitations to keep in view
This is an organizational case study using self-reported perceptions plus usability testing, not a randomized productivity trial; it concerns one assistant and IBM context, and the abstract cautions that gains vary across users.
Possible founder application
For developer-tool founders, measure perceived and behavioral productivity together, segment by user, and make provenance, ownership, responsibility, and output-quality workflows explicit. Avoid assuming that aggregate positive sentiment means every team or task benefits.
Evidence access
Publisher abstract directly retrieved; full-text access not established
Direct publisher scrape verified title/record and available text; Firecrawl source ID 01a1228d-e249-7029-8a2f-e072d63d2379. Correction notices were not separately checked.
If only an abstract or metadata was available, the profile is limited to claims supported there. A paper's paywall does not change its evidentiary status, but it limits what we can substantiate from accessible text.