Bayesian Analysis

Combining Expert Opinions in Prior Elicitation

Isabelle Albert, Sophie Donnet, Chantal Guihenneuc-Jouyaux, Samantha Low-Choy, Kerrie Mengersen, and Judith Rousseau

Full-text: Open access


We consider the problem of combining opinions from different experts in an explicitly model-based way to construct a valid subjective prior in a Bayesian statistical approach. We propose a generic approach by considering a hierarchical model accounting for various sources of variation as well as accounting for potential dependence between experts. We apply this approach to two problems. The first problem deals with a food risk assessment problem involving modelling dose-response for Listeria monocytogenes contamination of mice. Two hierarchical levels of variation are considered (between and within experts) with a complex mathematical situation due to the use of an indirect probit regression. The second concerns the time taken by PhD students to submit their thesis in a particular school. It illustrates a complex situation where three hierarchical levels of variation are modelled but with a simpler underlying probability distribution (log-Normal).

Article information

Bayesian Anal., Volume 7, Number 3 (2012), 503-532.

First available in Project Euclid: 28 August 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Bayesian statistics Hierarchical model Random effects Risk assessment


Albert, Isabelle; Donnet, Sophie; Guihenneuc-Jouyaux, Chantal; Low-Choy, Samantha; Mengersen, Kerrie; Rousseau, Judith. Combining Expert Opinions in Prior Elicitation. Bayesian Anal. 7 (2012), no. 3, 503--532. doi:10.1214/12-BA717.

Export citation


  • Albert, I., Grenier, E., Denis, J.-B., and Rousseau, J. (2008). “Quantitative risk assessment from farm to fork and beyond: a global Bayesian approach concerning food-borne diseases.” Risk Analysis, 28: 557–571.
  • Bayarri, M. J. and Berger, J. O. (2004). “The interplay of Bayesian and Frequentist Analysis.” Statistical Science, 19: 58–80.
  • Bedrick, E. J., Christensen, R., and Johnson, W. (1996). “A new perspective on priors for generalized linear models.” Journal of the American Statistical Association, 91(436): 1450–1460.
  • Berger, J. (2006). “The Case for Objective Bayesian Analysis.” Bayesian Analysis, 1: 385–402.
  • Burgman, M. A., McBride, M., Ashton, R., Speirs-Bridge, A., Flander, L., Wintle, B., Fidler, F., Rumpff, L., and Twardy, C. (2011). “Expert Status and Performance.” PLoS ONE, 6(7): e22998: 1–7.
  • Clemen, R. T. (2008). “Comment on Cooke’s classical method.” Reliability Engineering and System Safety, 93: 760–765.
  • Cooke, R. M. and Goossens, L. L. H. J. (2008). “TU Delft expert judgment data base.” Reliability Engineering and System Safety, 93: 657–674.
  • Denham, R. and Mengersen, K. (2007). “Geographically Assisted Elicitation of Expert Opinion for Regression Models.” Bayesian Analysis, 2(1): 99–136.
  • Fisher, R., O’Leary, R. A., Low-Choy, S., Mengersen, K., and Caley, M. J. (2012). “A software tool for elicitation of expert knowledge about species richness or similar counts.” Environmental Modelling and Software, 30.
  • French, S. (1985). “Group consensus probability distributions: a critical survey.” In Berger, J., Bernardo, J., Dawid, A., and Smith, A. (eds.), Bayesian Statistics 2, 183–201. North-Holland, Amsterdam: Oxford University Press.
  • — (2007). “Web-enabled strategic GDSS, e-democracy and Arrow’s theorem: A Bayesian perspective.” Decision Support Systems, 43: 1476–1484.
  • — (2008). “Comments by Prof. French.” Reliability Engineering and System Safety, 93: 766–768.
  • — (2011). “Aggregating expert judgement.” Revista de la real academia de ciencias exactas, fisicas y naturales. Serie A Matematicas, 105: 181–206.
  • Gelfand, A., Mallick, B., and Dey, D. (1995). “Modelling expert opinion arising as a partial probabilistic specification.” Journal of the American Statististical Association, 90: 598–604.
  • Gelman, A. (2006). “Prior distributions for variance parameters in hierarchical models.” Bayesian Analysis, 1(3): 515–533.
  • Genest, C. and Zidek, J. V. (1986). “Combining probability distributions. A critique and annoted bibliography.” Statistical Science, 1: 114–148.
  • Gill, J. and Walker, L. D. (2005). “Elicited Priors for Bayesian Model Specifications in Political Science Research.” The Journal of Politics, 67(3): 841–872.
  • Haas, C. N., Rose, J. B., and Gerba, C. P. (1999). Quantitative microbial risk assessment. Wiley.
  • James, A., Low Choy, S., and Mengersen, K. (2010). “Elicitator: An Expert Elicitation Tool for Regression in Ecology.” Environmental Modelling & Software, 25(1): 129–145.
  • Kadane, J. B., Dickey, J. M., Winkler, R. L., Smith, W. S., and Peters, S. C. (1980). “Interactive elicitation of opinion for a normal linear model.” Journal of the American Statististical Association, 75: 845–854.
  • Kuhnert, P., Martin, T. G., and Griffiths, S. P. (2010). “A guide to eliciting and using expert knowledge in Bayesian ecological models.” Ecology Letters, 13: 900–914.
  • Kynn, M. (2005). “Eliciting expert opinion for a Bayesian logistic regression model in natural resources.” PhD thesis, School of Mathematical Sciences, Faculty of Science, Queensland University of Technology. URL 50830.084943
  • — (2008). “The ‘ heuristics and biases’ bias in expert elicitation.” Journal of the Royal Statistical Society, Series A, 171: 239–264.
  • Lin, S.-W. and Bier, V. M. (2008). “A study of expert overconfidence (with discussion).” Reliability Engineering and System Safety, 93: 711–721.
  • Lindley, D. V. (1983). “Reconciliation of probability distributions.” Operations Research, 31: 866–880.
  • Lindley, D. V., Tversky, A., and Brown, R. V. (1979). “On the reconciliation of probability assessments (with discussion).” Journal of the Royal Statistical Society A, 142: 146–180.
  • Lipscomb, J., Parmigiani, G., and Hasselblad, V. (1998). “Combining expert judgment by hierarchical modeling: an application to physician staffing.” Management Science, 44: 149–161.
  • Low-Choy, S. (2012). “Priors: Silent or Active Partners in Bayesian inference?” In C., A., Mengersen, K., and Pettitt, A. N. (eds.), Case Studies in Bayesian Statistical Modelling and Analysis. John Wiley & Sons, Inc, London.
  • Low-Choy, S., Mengersen, K., and Rousseau, J. (2008). “Encoding Expert Opinion on Skewed Non-Negative Distributions.” Journal of Applied Probability and Statistics, 3: 1– 21.
  • Low-Choy, S., Murray, J., James, A., and Mengersen, K. (2010). “Indirect elicitation from ecological experts: from methods and software to habitat modelling and rock-wallabies.” In O’Hagan, A. and West, M. (eds.), Oxford Handbook of Applied Bayesian Analysis. Oxford University Press, UK.
  • Low Choy, S., O’Leary, R., and Mengersen, K. (2009). “Elicitation by Design for Ecology: using expert opinion to inform priors for Bayesian statistical models.” Ecology, 90: 265–277.
  • Martin, T., Burgman, M., Fidler, F., Kuhnert, P., Low-Choy, S., MacBride, M., and Mengersen, K. (2012). “Elicitation of Expert Knowledge in Conservation Biology.” Conservation Biology, 26(1): 29–38.
  • Merrick, J. R. W. (2008). “Getting the Right Mix of Experts.” Decision Analysis, 5(1): 43–52.
  • Oakley, J. E. and O’Hagan, A. (2007). “Uncertainty in prior elicitations: a nonparametric approach.” Biometrika, 94(2): 427–441.
  • O’Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, R., Garthwaite, P., Jenkinson, D., Oakley, J., and Rakow, T. (2006). Uncertain Judgements: Eliciting Experts’ Probabilities. Wiley.
  • Perera, A. H., Drew, C. A., and Johnson, C. J. (eds.) (2011). Expert Knowledge and Its Applications in Landscape Ecology. Springer, NY.
  • Roback, P. J. and Givens, G. H. (2001). “Supra-Bayesian pooling of priors linked by a deterministic simulation model.” Communications in Statistics - Simulation and Computation, 30(3): 447–476.
  • Spetzler, C. S. and Staël von Holstein, C.-A. S. (1975). “Probability encoding in decision analysis.” Management Science, 22(3): 340–358.
  • Spiegelhalter, D. J., Thomas, A., Best, N. G., and Lunn, D. (2003). “WinBUGS version 1.4 user manual.” Technical report, MRC Biostatistics Unit, Cambridge.
  • West, M. (1988). “Modelling Expert Opinion.” In Bernardo, J., Degroot, M., Lindley, D., and Smith, A. (eds.), Bayesian Statistics 3, 493–508. Clarendon Press.
  • Winkler, R. L. (1968). “The consensus of subjective probability distributions.” Management Science, 15: 361–375.

See also

  • Related item: Simon French. Comment on Article by Albert et al. Bayesian Anal., Vol. 7, Iss. 3 (2012), 533–536.
  • Related item: John Paul Gosling. Comment on Article by Albert et al. Bayesian Anal., Vol. 7, Iss. 3 (2012), 537–540.
  • Related item: Isabelle Albert, Sophie Donnet, Chantal Guihenneuc-Jouyaux, Samantha Low-Choy, Kerrie Mengersen, Judith Rousseau. Rejoinder. Bayesian Anal., Vol. 7, Iss. 3 (2012), 541–546.