Modeling Publication Selection Effects in Meta-Analysis

Abstract

Publication selection effects arise in meta-analysis when the effect magnitude estimates are observed in (available from) only a subset of the studies that were actually conducted and the probability that an estimate is observed is related to the size of that estimate. Such selection effects can lead to substantial bias in estimates of effect magnitude. Research on the selection process suggests that much of the selection occurs because researchers, reviewers and editors view the results of studies as more conclusive when they are more highly statistically significant. This suggests a model of the selection process that depends on effect magnitude via the p-value or significance level. A model of the selection process involving a step function relating the p-value to the probability of selection is introduced in the context of a random effects model for meta-analysis. The model permits estimation of a weight function representing selection along the mean and variance of effects. Some ideas for graphical procedures and a test for publication selection are also introduced. The method is then applied to a meta-analysis of test validity studies.