Translator Disclaimer
March 2021 An evaluation of statistical methods for aggregate patterns of replication failure
Jacob M. Schauer, Kaitlyn G. Fitzgerald, Sarah Peko-Spicer, Mena C. R. Whalen, Rrita Zejnullahi, Larry V. Hedges
Author Affiliations +
Ann. Appl. Stat. 15(1): 208-229 (March 2021). DOI: 10.1214/20-AOAS1387


Several programs of research have sought to assess the replicability of scientific findings in different fields, including economics and psychology. These programs attempt to replicate several findings and use the results to say something about large-scale patterns of replicability in a field. However, little work has been done to understand the analytic methods used to do this, including what they are assessing and what their statistical properties are. This article examines several methods that have been used to study patterns of replicability in the social sciences. We describe in concrete terms how each method operationalizes the idea of “replication” and examine various statistical properties, including bias, precision and statistical power. We find that some analytic methods rely on an operational definition of replication that can be misleading. Other methods involve more sound definitions of replication, but most of these have limitations, such as large bias and uncertainty or low power. The findings suggest that we should use caution interpreting the results of such analyses and that work on more accurate methods may be useful to future replication research efforts.


The authors would like to thank the referees and Editors for their constructive comments that improved the quality of this paper.

This work was supported by Institute of Education Sciences (IES) Grant R305B140042 as well as National Science Foundation (NSF) Grant DRL-1841075.


Download Citation

Jacob M. Schauer. Kaitlyn G. Fitzgerald. Sarah Peko-Spicer. Mena C. R. Whalen. Rrita Zejnullahi. Larry V. Hedges. "An evaluation of statistical methods for aggregate patterns of replication failure." Ann. Appl. Stat. 15 (1) 208 - 229, March 2021.


Received: 1 March 2020; Revised: 1 August 2020; Published: March 2021
First available in Project Euclid: 18 March 2021

Digital Object Identifier: 10.1214/20-AOAS1387

Keywords: bias , Meta-analysis , power , replication

Rights: Copyright © 2021 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.15 • No. 1 • March 2021
Back to Top