Statistical Science

Objective Priors: An Introduction for Frequentists

Malay Ghosh

Full-text: Open access

Abstract

Bayesian methods are increasingly applied in these days in the theory and practice of statistics. Any Bayesian inference depends on a likelihood and a prior. Ideally one would like to elicit a prior from related sources of information or past data. However, in its absence, Bayesian methods need to rely on some “objective” or “default” priors, and the resulting posterior inference can still be quite valuable.

Not surprisingly, over the years, the catalog of objective priors also has become prohibitively large, and one has to set some specific criteria for the selection of such priors. Our aim is to review some of these criteria, compare their performance, and illustrate them with some simple examples. While for very large sample sizes, it does not possibly matter what objective prior one uses, the selection of such a prior does influence inference for small or moderate samples. For regular models where asymptotic normality holds, Jeffreys’ general rule prior, the positive square root of the determinant of the Fisher information matrix, enjoys many optimality properties in the absence of nuisance parameters. In the presence of nuisance parameters, however, there are many other priors which emerge as optimal depending on the criterion selected. One new feature in this article is that a prior different from Jeffreys’ is shown to be optimal under the chi-square divergence criterion even in the absence of nuisance parameters. The latter is also invariant under one-to-one reparameterization.

Article information

Source
Statist. Sci. Volume 26, Number 2 (2011), 187-202.

Dates
First available in Project Euclid: 1 August 2011

Permanent link to this document
http://projecteuclid.org/euclid.ss/1312204006

Digital Object Identifier
doi:10.1214/10-STS338

Mathematical Reviews number (MathSciNet)
MR2858380

Zentralblatt MATH identifier
06075154

Citation

Ghosh, Malay. Objective Priors: An Introduction for Frequentists. Statist. Sci. 26 (2011), no. 2, 187--202. doi:10.1214/10-STS338. http://projecteuclid.org/euclid.ss/1312204006.


Export citation

References

  • Amari, S. (1982). Differential geometry of curved exponential families—Curvatures and information loss. Ann. Statist. 10 357–387.
  • Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 53 370–418.
  • Berger, J. O. (1985). Statistical Decision Theory and Related Topics, 2nd ed. Springer, New York.
  • Berger, J. O. and Bernardo, J. M. (1989). Estimating a product of means. Bayesian analysis with reference priors. J. Amer. Statist. Assoc. 84 200–207.
  • Berger, J. O. and Bernardo, J. M. (1992a). On the development of reference priors (with discussion). In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 35–60. Oxford Univ. Press, New York.
  • Berger, J. O. and Bernardo, J. M. (1992b). Reference priors in a variance components problem. In Bayesian Analysis in Statistics and Econometrics (P. K. Goel and N. S. Iyengar, eds.) 177–194. Springer, New York.
  • Berger, J. O., Bernardo, J. M. and Sun, D. (2009). The formal definition of reference priors. Ann. Statist. 37 905–938.
  • Berger, J. O., de Oliveira, V. and Sanso, B. (2001). Objective Bayesian analysis of spatially correlated data. J. Amer. Statist. Assoc. 96 1361–1374.
  • Berger, J. O. and Yang, R. (1994). Noninformative priors and Bayesian testing for the AR(1) model. Econometric Theory 10 461–482.
  • Bernardo, J. M. (1979). Reference posterior distributions for Bayesian inference (with discussion). J. R. Stat. Soc. Ser. B Stat. Methodol. 41 113–147.
  • Bernardo, J. M. (2005). Reference analysis. In Handbook of Bayesian Statisics 25 (D. K. Dey and C. R. Rao, eds.). North-Holland, Amsterdam.
  • Bhattacharyya, A. K. (1943). On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math. Soc. 35 99–109.
  • Bickel, P. J. and Ghosh, J. K. (1990). A decomposition for the likelihood ratio statistic and the Bartlett correction—A Bayesian argument. Ann. Statist. 18 1070–1090.
  • Clarke, B. and Barron, A. (1990). Information-theoretic asymptotics of Bayes methods. IEEE Trans. Inform. Theory 36 453–471.
  • Clarke, B. and Barron, A. (1994). Jeffreys’ prior is asymptotically least favorable under entropy risk. J. Statist. Plann. Inference 41 37–60.
  • Clarke, B. and Sun, D. (1997). Reference priors under the chi-square distance. Sankhyā A 59 215–231.
  • Clarke, B. and Sun, D. (1999). Asymptotics of the expected posterior. Ann. Inst. Statist. Math. 51 163–185.
  • Cressie, N. and Read, T. R. C. (1984). Multinomial goodness-of-fit tests. J. Roy. Statist. Soc. Ser. B 46 440–464.
  • Cox, D. R. and Reid, N. (1987). Parameter orthogonality and approximate conditional inference (with discussion). J. Roy Statist. Soc. Ser. B 53 79–109.
  • Datta, G. S. (1996). On priors providing frequentist validity of Bayesian inference for multiple parametric functions. Biometrika 83 287–298.
  • Datta, G. S. and Ghosh, J. K. (1995a). On priors providing frequentist validity for Bayesian inference. Biometrika 82 37–45.
  • Datta, G. S. and Ghosh, J. K. (1995b). Noninformative priors for maximal invariant in group models. Test 4 95–114.
  • Datta, G. S. and Ghosh, M. (1995c). Some remarks on noninformative priors. J. Amer. Statist. Assoc. 90 1357–1363.
  • Datta, G. S. and Ghosh, M. (1995d). Hierarchical Bayes estimators of the error variance in one-way ANOVA models. J. Statist. Plann. Inference 45 399–411.
  • Datta, G. S. and Ghosh, M. (1996). On the invariance of noninformative priors. Ann. Statist. 24 141–159.
  • Datta, G. S., Ghosh, M. and Mukerjee, R. (2000). Some new results on probability matching priors. Calcutta Statist. Assoc. Bull. 50 179–192.
  • Datta, G. S., Ghosh, M. and Kim, Y. (2002). Probability matching priors for one-way unbalanced rendom effects models. Statist. Decisions 20 29–51.
  • Datta, G. S. and Mukerjee, R. (2004). Probability Matching Priors: Higher Order Asymptotics. Springer, New York.
  • Datta, G. S., Mukerjee, R., Ghosh, M. and Sweeting, T. J. (2000). Bayesian prediction with approximate frequentist validity. Ann. Statist. 28 1414–1426.
  • Datta, G. S. and Sweeting, T. J. (2005). Probability matching priors. In Bayesian Thinking, Modeling and Computation. Handbook of Statistics 25 (D. K. Dey and C. R. Rao, eds.). North-Holland, Amsterdam.
  • Ferguson, T. (1996). A Course in Large Sample Theory. Chapman & Hall/CRC Press, Boca Raton, FL.
  • Fraser, D. A. S. and Reid, N. (2002). Strong matching of frequentist and Bayesian parametric inference. J. Statist. Plann. Inference 103 263–285.
  • Ghosal, S. (1997). Reference priors in multiparameter nonregular cases. Test 6 159–186.
  • Ghosal, S. (1999). Probability matching priors for nonregular cases. Biometrika 86 956–964.
  • Ghosh, J. K., Delampady, M. and Samanta, T. (2006). An Introduction to Bayesian Analysis. Springer, New York.
  • Ghosh, M. and Liu, R. (2011). Moment matching priors. Sankhyā A. To appear.
  • Ghosh, J. K. and Mukerjee, R. (1992). Non-informative priors (with discussion). In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 195–210. Oxford Univ. Press, New York.
  • Ghosh, J. K., Sinha, B. K. and Joshi, S. N. (1982). Expansion for posterior probability and integrated Bayes risk. In Statistical Decision Theory and Related Topics III 1 403–456. Academic Press, New York.
  • Ghosh, M., Carlin, B. P. and Srivastava, M. S. (1994). Probability matching priors for linear calibration. Test 4 333–357.
  • Ghosh, M. and Heo, J. (2003). Default Bayesian priors for regression models with second-order autogressive residuals. J. Time Ser. Anal. 24 269–282.
  • Ghosh, M., Mergel, V. and Liu, R. (2011). A general divergence criterion for prior selection. Ann. Inst. Statist. Math. 63 43–58.
  • Ghosh, M. and Mukerjee, R. (1998). Recent developments on probability matching priors. In Applied Statistical Science III (S. E. Ahmed, M. Ahsanullah and B. K. Sinha, eds.) 227–252. Nova Science Publishers, New York.
  • Ghosh, M., Yin, M. and Kim, Y.-H. (2003). Objective Bayesian inference for ratios of regression coefficients in linear models. Statist. Sinica 13 409–422.
  • Halmos, P. (1950). Measure Theory. Van Nostrand, New York.
  • Hartigan, J. A. (1964). Invariant prior densities. Ann. Math. Statist. 35 836–845.
  • Hartigan, J. A. (1998). The maximum likelihood prior. Ann. Statist. 26 2083–2103.
  • Hellinger, E. (1909). Neue Begründung der Theorie quadratischen Formen von unendlichen vielen Veränderlichen. J. Reine Angew. Math. 136 210–271.
  • Huzurbazar, V. S. (1950). Probability distributions and orthogonal parameters. Proc. Camb. Phil. Soc. 46 281–284.
  • Jeffreys, H. (1961). Theory of Probability and Inference, 3rd ed. Cambridge Univ. Press, London.
  • Johnson, R. A. (1970). Asymptotic expansions associated with posterior distribution. Ann. Math. Statist. 41 851–864.
  • Kass, R. E. and Wasserman, L. (1996). The selection of prior distributions by formal rules. J. Amer. Statist. Assoc. 91 1343–1370.
  • Laplace, P. S. (1812). Theorie Analytique des Probabilities. Courcier, Paris.
  • Lindley, D. V. (1956). On the measure of the information provided by an experiment. Ann. Math. Statist. 27 986–1005.
  • Liu, R. (2009). On some new contributions towards objective priors. Unpublished Ph.D. dissertation. Dept. Statistics, Univ. Florida, Gainesville, FL.
  • Morris, C. N. and Normand, S. L. (1992). Hierarchical models for combining information and for meta-analysis. In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 321–344. Oxford. Univ. Press, New York.
  • Mukerjee, R. and Dey, D. K. (1993). Frequentist validity of posterior quantiles in the presence of a nuisance parameter: Higher-order asymptotics. Biometrika 80 499–505.
  • Mukerjee, R. and Ghosh, M. (1997). Second-order probability matching priors. Biometrika 84 970–975.
  • Nachbin, L. (1965). The Haar Integral. Van Nostrand, New York.
  • Reid, N. (1996). Likelihood and Bayesian approximation methods (with discussion). In Bayesian Statistics 5 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 351–368. Oxford Univ. Press, New York.
  • Severini, T. A., Mukerjee, R. and Ghosh, M. (2002). On an exact probability matching property of right-invariant priors. Biometrika 89 952–957.
  • Staicu, A.-M. and Reid, N. (2008). On probability matching priors. Canad. J. Statist. 36 613–622.
  • Tibshirani, R. J. (1989). Noninformative priors for one parameter of many. Biometrika 76 604–608.
  • Welch, B. L. and Peers, H. W. (1963). On formulae for confidence points based on integrals of weighted likelihoods. J. Roy Statist. Soc. Ser. B 35 318–329.
  • Ye, K. (1994). Bayesian reference prior analysis on the ratio of variances for the balanced one-way random effect model. J. Statist. Plann. Inference 41 267–280.

See also

  • Discussion of: Objective Priors: An Introduction for Frequentists by M. Ghosh.
  • Discussion of: Objective Priors: An Introduction for Frequentists by M. Ghosh.
  • Rejoinder: Objective Priors: An Introduction for Frequentists.