Open Access
November 2023 Replication Success Under Questionable Research Practices—a Simulation Study
Francesca Freuli, Leonhard Held, Rachel Heyard
Author Affiliations +
Statist. Sci. 38(4): 621-639 (November 2023). DOI: 10.1214/23-STS904


Increasing evidence suggests that the reproducibility and replicability of scientific findings is threatened by researchers employing questionable research practices (QRPs) in order to achieve statistically significant results. Numerous metrics have been developed to determine replication success but it has not yet been investigated how well those metrics perform in the presence of QRPs. This paper aims to compare the performance of different metrics quantifying replication success in the presence of four types of QRPs: cherry picking of outcomes, questionable interim analyses, questionable inclusion of covariates, and questionable subgroup analyses. Our results show that the metric based on the version of the sceptical p-value that is recalibrated in terms of effect size performs better in maintaining low values of overall type-I error rate, but often requires larger replication sample sizes compared to metrics based on significance, the controlled version of the sceptical p-value, meta-analysis or Bayes factors, especially when severe QRPs are employed.


We thank Samuel Pawel for helpful comments on our manuscript.


Download Citation

Francesca Freuli. Leonhard Held. Rachel Heyard. "Replication Success Under Questionable Research Practices—a Simulation Study." Statist. Sci. 38 (4) 621 - 639, November 2023.


Published: November 2023
First available in Project Euclid: 6 November 2023

Digital Object Identifier: 10.1214/23-STS904

Keywords: power , Questionable research practices , rejection ratio , replication success , simulation study , type-I error rate

Rights: Copyright © 2023 Institute of Mathematical Statistics

Vol.38 • No. 4 • November 2023
Back to Top