Customer behavior and adoption · Full-text observational qualitative empirical study
Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Ranim Khojah, Mazen Mohamad, Philipp Leitner, Francisco Gomes de Oliveira Neto (2024). ACM Transactions on Software Engineering and Methodology.
Plain-English take
Practitioners more often sought guidance or learning than ready-to-use software artifacts; simple artifact generation and brainstorming were useful, complex tasks were harder; about two-thirds expressed high or moderate trust but none trusted outputs entirely, and lack of sources reduced trust. For a founder, use this as a question to test in your own context, not a universal playbook.
How much weight to give this
3.5/5 Useful evidence with meaningful caveats
The 1–5 evidence-check score is a practical reading aid, not a formal peer review or a probability that a conclusion is true. It combines design strength (2 points), sample reporting (1), comparison/control clarity (1), and how much of the paper could be checked (1). Access affects how confidently this directory can assess a study, not the study's underlying quality. Blinding is reported separately because it is not relevant to every design.
Synthesis or analytical design addresses the question, but conclusions depend on included literature or model assumptions.
The reviewed source reports a sample or corpus size (24); size alone does not establish representativeness.
The comparison structure is limited or depends on the paper's model/context.
Firecrawl retrieved a full paper or full-text publisher page for this annotation.
Double-blind? Blinding was not established from the captured source; it may not apply to this study design.
Research question
What the authors investigated
How do professional software engineers use ChatGPT in daily work, and what factors shape perceived usefulness and trust?
Design
One-week observational study; qualitative analysis of ChatGPT dialogue logs and exit-survey reflections; framework organized around interaction purpose, internal factors and external factors.
Sample and setting
24 professional software engineers at 10 companies in Europe, using GPT-3.5 over five business days.
Reported result
What the study found
Practitioners more often sought guidance or learning than ready-to-use software artifacts; simple artifact generation and brainstorming were useful, complex tasks were harder; about two-thirds expressed high or moderate trust but none trusted outputs entirely, and lack of sources reduced trust.
This is a paraphrase of the source. Check the original paper for definitions, estimates, statistical uncertainty and analysis details.
Scope and transfer
Limitations to keep in view
Limited, Europe-centered nonrepresentative sample; inductive dialogue classification has subjectivity; conducted before ChatGPT Enterprise, so privacy/security effects of that service were not assessed.
Possible founder application
Treat this as evidence for the specific customer behavior and adoption question studied. Compare the setting and sample to your company before changing a growth decision; use a local experiment where the paper does not match your context.
Evidence access
Full text retrieved from publisher or linked open repository
Publisher DOI record and accessible article/abstract directly support the bibliographic record and the summary fields; editorial review remains appropriate for duplicate checking against the unseen 300-study library. Correction notices were not separately verified in this collection; consult the current publisher record before applying the findings.
If only an abstract or metadata was available, the profile is limited to claims supported there. A paper's paywall does not change its evidentiary status, but it limits what we can substantiate from accessible text.