Annals of Applied Statistics
- Ann. Appl. Stat.
- Volume 14, Number 3 (2020), 1063-1087.
Statistical methods for replicability assessment
Large-scale replication studies like the Reproducibility Project: Psychology (RP:P) provide invaluable systematic data on scientific replicability, but most analyses and interpretations of the data fail to agree on the definition of “replicability” and disentangle the inexorable consequences of known selection bias from competing explanations. We discuss three concrete definitions of replicability based on: (1) whether published findings about the signs of effects are mostly correct, (2) how effective replication studies are in reproducing whatever true effect size was present in the original experiment and (3) whether true effect sizes tend to diminish in replication. We apply techniques from multiple testing and postselection inference to develop new methods that answer these questions while explicitly accounting for selection bias. Our analyses suggest that the RP:P dataset is largely consistent with publication bias due to selection of significant effects. The methods in this paper make no distributional assumptions about the true effect sizes.
Ann. Appl. Stat., Volume 14, Number 3 (2020), 1063-1087.
Received: April 2019
Revised: February 2020
First available in Project Euclid: 18 September 2020
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Hung, Kenneth; Fithian, William. Statistical methods for replicability assessment. Ann. Appl. Stat. 14 (2020), no. 3, 1063--1087. doi:10.1214/20-AOAS1336. https://projecteuclid.org/euclid.aoas/1600454857
- Supplement A: Supplement to “Statistical methods for replicability assessment”. We detail considerations made for individual studies, as well as evaluate our approximation of $t$-distributions by normal distributions.
- Supplement B: Code for “Statistical methods for replicability assessment”. We generated the figures, performed the analyses on the RP:P dataset and ran relevant simulations in R.