Statistical Science

Bayes, Reproducibility and the Quest for Truth

D. A. S. Fraser, M. Bédard, A. Wong, Wei Lin, and A. M. Fraser

Full-text: Open access

Abstract

We consider the use of default priors in the Bayes methodology for seeking information concerning the true value of a parameter. By default prior, we mean the mathematical prior as initiated by Bayes [Philos. Trans. R. Soc. Lond. 53 (1763) 370–418] and pursued by Laplace [Théorie Analytique des Probabilités (1812) Courcier], Jeffreys [Theory of Probability (1961) Clarendon Press], Bernardo [J. Roy. Statist. Soc. Ser. B 41 (1979) 113–147] and many more, and then recently viewed as “potentially dangerous” [Science 340 (2013) 1177–1178] and “potentially useful” [Science 341 (2013) 1452]. We do not mean, however, the genuine prior [Science 340 (2013) 1177–1178] that has an empirical reference and would invoke standard frequency modelling. And we do not mean the subjective or opinion prior that an individual might have and would be viewed as specific to that individual. A mathematical prior has no referenced frequency information, but on occasion is known otherwise to lead to repetition properties called confidence. We investigate the presence of such supportive property, and ask can Bayes give reliability for other than the particular parameter weightings chosen for the conditional calculation. Thus, does the methodology have reproducibility? Or is it a leap of faith.

For sample-space analysis, recent higher-order likelihood methods with regular models show that third-order accuracy is widely available using profile contours [In Past, Present and Future of Statistical Science (2014) 237–252 CRC Press].

But for parameter-space analysis, accuracy is widely limited to first order. An exception arises with a scalar full parameter and the use of the scalar Jeffreys [J. Roy. Statist. Soc. Ser. B 25 (1963) 318–329]. But for vector full parameter even with a scalar interest parameter, difficulties have long been known [J. Roy. Statist. Soc. Ser. B 35 (1973) 189–233] and with parameter curvature, accuracy beyond first order can be unavailable [Statist. Sci. 26 (2011) 299–316]. We show, however, that calculations on the parameter space can give full second-order information for a chosen scalar interest parameter; these calculations, however, require a Jeffreys prior that is used fully restricted to the one-dimensional profile for that interest parameter. Such a prior is effectively data-dependent and parameter-dependent and is focally restricted to the one-dimensional contour; these priors fall outside the usual Bayes approach and yet with substantial calculations can still give less than frequency analysis.

We provide simple examples using discrete extensions of Jeffreys prior. These serve as counter-examples to general claims that Bayes can offer accuracy for statistical inference. To obtain this accuracy with Bayes, more effort is required compared to recent likelihood methods, which still remain more accurate. And with vector full parameters, accuracy beyond first order is routinely not available, as a change in parameter curvature causes Bayes and frequentist values to change in opposite direction, yet frequentist has full reproducibility.

An alternative is to view default Bayes as an exploratory technique and then ask does it do as it overtly claims? Is it reproducible as understood in contemporary science? The posterior gives a distribution for an interest parameter and, thereby, a quantile for the interest parameter; an oracle could record whether it was left or right of the true value. If the average split in evaluative repetitions is in accord with the nominal level, then the approach is providing accuracy. And if not, then what is up, other than performance specific to the parameter frequencies in the prior. No one has answers although speculative claims abound.

Article information

Source
Statist. Sci. Volume 31, Number 4 (2016), 578-590.

Dates
First available in Project Euclid: 19 January 2017

Permanent link to this document
https://projecteuclid.org/euclid.ss/1484816587

Digital Object Identifier
doi:10.1214/16-STS573

Keywords
Confidence curved parameter exponential model gamma mean genuine prior Jeffreys L’Aquila linear parameter opinion prior regular model reproducibility risks rotating parameter two theories Vioxx Welch–Peers

Citation

Fraser, D. A. S.; Bédard, M.; Wong, A.; Lin, Wei; Fraser, A. M. Bayes, Reproducibility and the Quest for Truth. Statist. Sci. 31 (2016), no. 4, 578--590. doi:10.1214/16-STS573. https://projecteuclid.org/euclid.ss/1484816587.


Export citation

References

  • Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philos. Trans. R. Soc. Lond. 53 370–418.
  • Bernardo, J.-M. (1979). Reference posterior distributions for Bayesian inference. J. Roy. Statist. Soc. Ser. B 41 113–147.
  • Brazzale, A. R., Davison, A. C. and Reid, N. (2007). Applied Asymptotics: Case Studies in Small-Sample Statistics. Cambridge Series in Statistical and Probabilistic Mathematics 23. Cambridge Univ. Press, Cambridge.
  • Cakmak, S., Fraser, D. A. S., McDunnough, P., Reid, N. and Yuan, X. (1998). Likelihood centered asymptotic model exponential and location model versions. J. Statist. Plann. Inference 66 211–222.
  • Daniels, H. E. (1954). Saddlepoint approximations in statistics. Ann. Math. Statist. 25 631–650.
  • Dawid, A. P., Stone, M. and Zidek, J. V. (1973). Marginalization paradoxes in Bayesian and structural inference. J. Roy. Statist. Soc. Ser. B 35 189–233.
  • Efron, B. (2013). Bayes’ theorem in the 21st century. Science 340 1177–1178.
  • Fisher, R. (1930). Inverse probability. Proc. Camb. Philos. Soc. 26 528–535.
  • Fisher, R. A. (1956). Statistical Methods and Scientific Inference. Oliver and Boyd, Edinburgh.
  • Fraser, D. A. S. (2003). Likelihood for component parameters. Biometrika 90 327–339.
  • Fraser, D. A. S. (2011). Is Bayes posterior just quick and dirty confidence? Statist. Sci. 26 299–316.
  • Fraser, D. A. S. (2013). Bayes’ confidence. Science 341 1452.
  • Fraser, D. A. S. (2014). Why does statistics have two theories? In Past, Present and Future of Statistical Science (X. Lin, C. Genest, D. L. Banks, G. Molenberghs, D. W. Scott and J.-L. Wang, eds.) 237–252. CRC Press, Boca Raton, FL.
  • Fraser, D. A. S. (2016). Definitive testing of an interest parameter: Using parameter continuity. J. Statist. Res. 47 193–169.
  • Fraser, A. M., Fraser, D. A. S. and Staicu, A.-M. (2010). Second order ancillary: A differential view from continuity. Bernoulli 16 1208–1223.
  • Fraser, D. A. S. and Reid, N. (1995). Ancillaries and third order significance. Util. Math. 47 33–53.
  • Fraser, D. A. S. and Reid, N. (2002). Strong matching of frequentist and Bayesian parametric inference. J. Statist. Plann. Inference 103 263–285.
  • Fraser, D. A. S., Reid, N., Marras, E. and Yi, G. Y. (2010). Default priors for Bayesian and frequentist inference. J. R. Stat. Soc. Ser. B Stat. Methodol. 72 631–654.
  • Ghosh, M. (2011). Objective priors: An introduction for frequentists. Statist. Sci. 26 187–202.
  • Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. Proc. Roy. Soc. London, Ser. A. 186 453–461.
  • Jeffreys, H. (1961). Theory of Probability, 3rd ed. Clarendon Press, Oxford.
  • Laplace, P. S. (1812). Théorie Analytique des Probabilités. Courcier, Paris.
  • McNutt, M. (2014). Reproducibility. Science 343 229.
  • McNutt, M. (2015). Editorial retraction. Science 348 1100.
  • Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Phil. Trans. Roy. Soc. A 237 333–380.
  • Reid, N. and Fraser, D. A. S. (2010). Mean loglikelihood and higher-order approximations. Biometrika 97 159–170.
  • Savage, L. J. (1972). The Foundations of Statistics, revised ed. Dover Publications, New York.
  • Welch, B. L. and Peers, H. W. (1963). On formulae for confidence points based on integrals of weighted likelihoods. J. Roy. Statist. Soc. Ser. B 25 318–329.
  • Yang, R. and Berger, J. O. (1996). A catalog of noninformative priors. Institute of Statistics and Decision Sciences, Duke Univ., Durham, DC.