Interpreting p-Values and Confidence Intervals Using Well-Calibrated Null Preference Priors

Abstract

We propose well-calibrated null preference priors for use with one-sided hypothesis tests, such that resulting Bayesian and frequentist inferences agree. Null preference priors mean that they have nearly $100\mathrm{\%}$ of their prior belief in the null hypothesis, and well-calibrated priors mean that the resulting posterior beliefs in the alternative hypothesis are not overconfident. This formulation expands the class of problems giving Bayes-frequentist agreement to include problems involving discrete distributions such as binomial and negative binomial one- and two-sample exact (i.e., valid) tests. When applicable, these priors give posterior belief in the null hypothesis that is a valid p-value, and the null preference prior emphasizes that large p-values may simply represent insufficient data to overturn prior belief. This formulation gives a Bayesian interpretation of some common frequentist tests, as well as more intuitively explaining lesser known and less straightforward confidence intervals for two-sample tests.