Open Access
Translator Disclaimer
June 2022 The assessment of replication success based on relative effect size
Leonhard Held, Charlotte Micheloud, Samuel Pawel
Author Affiliations +
Ann. Appl. Stat. 16(2): 706-720 (June 2022). DOI: 10.1214/21-AOAS1502

Abstract

Replication studies are increasingly conducted in order to confirm original findings. However, there is no established standard how to assess replication success, and, in practice, many different approaches are used. The purpose of this paper is to refine and extend a recently proposed reverse-Bayes approach for the analysis of replication studies. We show how this method is directly related to the relative effect size, the ratio of the replication to the original effect estimate. This perspective leads to a new proposal to recalibrate the assessment of replication success, the golden level. The recalibration ensures that, for borderline significant original studies, replication success can only be achieved if the replication effect estimate is larger than the original one. Conditional power for replication success can then take any desired value if the original study is significant and the replication sample size is large enough. Compared to the standard approach to require statistical significance of both the original and replication study, replication success at the golden level offers uniform gains in project power and controls the type-I error rate if the replication sample size is not smaller than the original one. An application to data from four large replication projects shows that the new approach leads to more appropriate inferences, as it penalizes shrinkage of the replication estimate, compared to the original one, while ensuring that both effect estimates are sufficiently convincing on their own.

Funding Statement

Support by the Swiss National Science Foundation (Project #189295) is gratefully acknowledged.

Acknowledgments

We acknowledge helpful and constructive comments by the Editor and a referee on an earlier version of this article.

Citation

Download Citation

Leonhard Held. Charlotte Micheloud. Samuel Pawel. "The assessment of replication success based on relative effect size." Ann. Appl. Stat. 16 (2) 706 - 720, June 2022. https://doi.org/10.1214/21-AOAS1502

Information

Received: 1 September 2020; Revised: 1 April 2021; Published: June 2022
First available in Project Euclid: 13 June 2022

Digital Object Identifier: 10.1214/21-AOAS1502

Keywords: power , replication studies , sceptical p-value , shrinkage , two-trials rule , type-I error rate

Rights: Copyright © 2022 Institute of Mathematical Statistics

JOURNAL ARTICLE
15 PAGES


SHARE
Vol.16 • No. 2 • June 2022
Back to Top