Developer experience · Quasi-experimental repository study

Quality gatekeepers: investigating the effects of code review bots on pull request activities

Empirical Software Engineering · 2022

Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, Marco A. Gerosa

Original paper ↗

Summary

Investigates how code-review bots affect pull-request activity.

For your marketing

Possible founder application

Pilot review automation with guardrails and measure review time, rework, and developer burden.

Reported result

What the study found

Investigates how code-review bots affect pull-request activity.

Research question

What the authors investigated

How does the study’s stated design explain its reported outcome in this setting?

Design

Quasi-experimental repository study

Sample and setting

Open-source pull requests using code-review bots; abstract-only

Scope and transfer

Limitations to keep in view

Bot adoption may be confounded by project maturity and maintainer choices.

Research sources

abstract_only

Study strength

3.5/5 Abstract-level evidence is useful for framing questions; verify full methods before transfer.

How this was assessed

The 1–5 evidence-check score is a practical reading aid, not a formal peer review or a probability that a conclusion is true. It combines design strength (2 points), sample reporting (1), comparison/control clarity (1), and how much of the paper could be checked (1). Access affects how confidently this directory can assess a study, not the study's underlying quality. Blinding is reported separately because it is not relevant to every design.

Design transparency · 2/2

Based on the reported study design in the retrieved source.

Sample and setting · 1/1

The sample context is stated; a numerical count is preserved only when reported.

Outcome clarity · 1/1

The finding is summarized from publisher or article metadata.

Access for checking · 0.5/1

Only abstract or bibliographic access was confirmed.

Double-blind? The retrieved publisher record does not establish whether participants or outcome assessors were blinded.