Paid media and measurement · Large-scale randomized advertising field experiment and statistical analysis
When Less Is More: Data and Power in Advertising Experiments
Johnson, G. A., Lewis, R. A., & Reiley, D. H. (2017). Marketing Science, 36(1), 43–53.
Plain-English take
The paper's main point is that more granular data do not necessarily make advertising effects easier to estimate and examines tradeoffs between measurement precision and experimentation resources. For your marketing: Plan experiments around outcome variance and expected effect size.
How much weight to give this
4.0/5 Useful evidence with meaningful caveats
The 1–5 evidence-check score is a practical reading aid, not a formal peer review or a probability that a conclusion is true. It combines design strength (2 points), sample reporting (1), comparison/control clarity (1), and how much of the paper could be checked (1). Access affects how confidently this directory can assess a study, not the study's underlying quality. Blinding is reported separately because it is not relevant to every design.
Random assignment provides a strong comparison for effects in the tested setting.
A usable sample size is not stated in the profile's accessible source.
Randomized comparison; confirm the control condition and analysis in the paper.
This profile could be checked against an abstract or metadata, not the full paper.
Double-blind? Blinding is not reported in the source reviewed; that does not establish whether blinding was or was not used.
Research question
What the authors investigated
How do the amount and structure of available data affect the power and cost of online advertising experiments?
Design
The study uses a randomized field experiment involving approximately three million Yahoo! users and examines the relationship between data availability and experimental measurement of ad effects.
Sample and setting
Yahoo! advertising experiment at large scale; the publisher search result reports evidence that retailer ads increased sales by 3.6% in the studied campaign.
Reported result
What the study found
The paper highlights that more granular data do not necessarily make advertising effects easier to estimate and examines tradeoffs between measurement precision and experimentation resources.
This is a paraphrase of the source. Check the original paper for definitions, estimates, statistical uncertainty and analysis details.
Scope and transfer
Limitations to keep in view
One platform and retailer campaign context; the 3.6% result should not be treated as a general ad-lift benchmark. The paper's title concerns statistical power and design, which require reading the methods before choosing an experiment size.
Possible founder application
Plan experiments around outcome variance and expected effect size. Large audiences alone do not guarantee useful estimates; choose an outcome and design that can detect a decision-relevant change.
Evidence access
Publisher record and abstract; full text access may vary
INFORMS publisher metadata and publisher-indexed summary; details are labeled conservatively where the abstract preview is limited.
If only an abstract or metadata was available, the profile is limited to claims supported there. A paper's paywall does not change its evidentiary status, but it limits what we can substantiate from accessible text.