Advertising · Large-scale randomized advertising field experiment and statistical analysis
When Less Is More: Data and Power in Advertising Experiments
Marketing Science · 2017
Summary
The paper's main point is that more granular data do not necessarily make advertising effects easier to estimate and examines tradeoffs between measurement precision and experimentation resources.
For your marketing
Possible founder application
Plan experiments around outcome variance and expected effect size. Large audiences alone do not guarantee useful estimates; choose an outcome and design that can detect a decision-relevant change.
Reported result
What the study found
The paper highlights that more granular data do not necessarily make advertising effects easier to estimate and examines tradeoffs between measurement precision and experimentation resources.
Research question
What the authors investigated
How do the amount and structure of available data affect the power and cost of online advertising experiments?
Design
The study uses a randomized field experiment involving approximately three million Yahoo! users and examines the relationship between data availability and experimental measurement of ad effects.
Sample and setting
Yahoo! advertising experiment at large scale; the publisher search result reports evidence that retailer ads increased sales by 3.6% in the studied campaign.
Scope and transfer
Limitations to keep in view
One platform and retailer campaign context; the 3.6% result should not be treated as a general ad-lift benchmark. The paper's title concerns statistical power and design, which require reading the methods before choosing an experiment size.
Research sources
INFORMS publisher metadata and publisher-indexed summary; details are labeled conservatively where the abstract preview is limited.