Paid media and measurement · Comparison of observational estimators against randomized field experiments
A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook
Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). Marketing Science, 38(2), 193–225.
Plain-English take
Commonly used observational approaches based on industry-available data often failed to accurately measure the true advertising effect in these comparisons. For your marketing: Treat attribution models as hypotheses and validate them against holdouts where possible.
How much weight to give this
4.5/5 Strong design for its specific question
The 1–5 evidence-check score is a practical reading aid, not a formal peer review or a probability that a conclusion is true. It combines design strength (2 points), sample reporting (1), comparison/control clarity (1), and how much of the paper could be checked (1). Access affects how confidently this directory can assess a study, not the study's underlying quality. Blinding is reported separately because it is not relevant to every design.
Random assignment provides a strong comparison for effects in the tested setting.
A usable sample size is not stated in the profile's accessible source.
Randomized comparison; confirm the control condition and analysis in the paper.
An accessible full paper or manuscript is linked, making methods easier to check.
Double-blind? Not applicable in the same way as a participant-treatment experiment.
Research question
What the authors investigated
Can commonly used observational methods recover the causal effect of online ads as measured by randomized experiments?
Design
The authors compare observational measurement approaches with randomized Facebook ad experiments across 15 campaigns; the campaigns met minimum sample and conversion-tracking criteria.
Sample and setting
Fifteen Facebook campaigns, each with at least one million users and business-relevant conversion tracking, selected to compare measurement methods.
Reported result
What the study found
Commonly used observational approaches based on industry-available data often failed to accurately measure the true advertising effect in these comparisons.
This is a paraphrase of the source. Check the original paper for definitions, estimates, statistical uncertainty and analysis details.
Scope and transfer
Limitations to keep in view
The analysis concerns a selected set of Facebook campaigns and methods available at the time. Results do not imply every observational model fails equally or that randomized tests are feasible for every startup campaign.
Possible founder application
Treat attribution models as hypotheses and validate them against holdouts where possible. When experiments are infeasible, document assumptions and do not present platform-attributed conversions as incremental impact.
Evidence access
Publisher abstract and author-hosted paper
INFORMS publisher abstract and author-hosted paper; the 15-campaign design and minimum sample criteria are specified in the manuscript.
If only an abstract or metadata was available, the profile is limited to claims supported there. A paper's paywall does not change its evidentiary status, but it limits what we can substantiate from accessible text.