Bayesian Analysis

Bayes Factor Testing of Multiple Intraclass Correlations

Joris Mulder and Jean-Paul Fox

Full-text: Open access


The intraclass correlation plays a central role in modeling hierarchically structured data, such as educational data, panel data, or group-randomized trial data. It represents relevant information concerning the between-group and within-group variation. Methods for Bayesian hypothesis tests concerning the intraclass correlation are proposed to improve decision making in hierarchical data analysis and to assess the grouping effect across different group categories. Estimation and testing methods for the intraclass correlation coefficient are proposed under a marginal modeling framework where the random effects are integrated out. A class of stretched beta priors is proposed on the intraclass correlations, which is equivalent to shifted F priors for the between groups variances. Through a parameter expansion it is shown that this prior is conditionally conjugate under the marginal model yielding efficient posterior computation. A special improper case results in accurate coverage rates of the credible intervals even for minimal sample size and when the true intraclass correlation equals zero. Bayes factor tests are proposed for testing multiple precise and order hypotheses on intraclass correlations. These tests can be used when prior information about the intraclass correlations is available or absent. For the noninformative case, a generalized fractional Bayes approach is developed. The method enables testing the presence and strength of grouped data structures without introducing random effects. The methodology is applied to a large-scale survey study on international mathematics achievement at fourth grade to test the heterogeneity in the clustering of students in schools across countries and assessment cycles.

Article information

Bayesian Anal., Volume 14, Number 2 (2019), 521-552.

First available in Project Euclid: 10 August 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Intraclass correlations Bayes factors stretched beta priors shifted F priors hierarchical models

Creative Commons Attribution 4.0 International License.


Mulder, Joris; Fox, Jean-Paul. Bayes Factor Testing of Multiple Intraclass Correlations. Bayesian Anal. 14 (2019), no. 2, 521--552. doi:10.1214/18-BA1115.

Export citation


  • Albert, J. H. and Chib, S. (1993). “Bayesian Analysis of Binary and Polychotomous Response Data.” Journal of the American Statistical Association, 88(422): 669–679. URL
  • Armagan, A., Dunson, D. B., and Clyde, M. (2011). “Generalized Beta Mixtures of Gaussians.” In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems 24, 523–531.
  • Bayarri, M. J., Berger, J. O., Forte, A., and García-Donato, G. (2012). “Criteria for Bayesian model choice with application to variable selection.” The Annals of Statistics, 40: 1550–1577.
  • Berger, J. O. (1994). “An overview of robust Bayesian analysis (with discussion).” Test, 3: 5–124.
  • Berger, J. O. (2006). “The case for objective Bbayesian analysis.” Bayesian Analysis, 1: 385–402.
  • Berger, J. O. and Bernardo, J. (1992). Reference priors in a variance component’s problem, 323–340. London: Oxford University Press.
  • Berger, J. O., De Oliveira, V., and Sansó, B. (2006). “Objective Bayesian analysis of spatially correlated data.” Journal of the American Statistical Association, 96: 1361–1374.
  • Berger, J. O., Liseo, B., and Wolpert, R. L. (1999). “Integrated likelihood methods for eliminating nuisance parameters.” Statistical Science, 14(1): 1–22. URL
  • Berger, J. O. and Mortera, J. (1995). “Discussion to fractional Bayes factors for model comparison (by O’Hagan).” Journal of the Royal Statistical Society Series B, 56: 130.
  • Berger, J. O. and Pericchi, L. (2001). “Objective Bayesian methods for model selection: Introduction and comparison (with discussion).” In Lahiri, P. (ed.), Model Selection, volume 38 of Monograph Series, 135–207. Beachwood Ohio, institute of mathematical statistics lecture notes edition.
  • Berger, J. O. and Pericchi, L. R. (1996). “The intrinsic Bayes factor for model selection and prediction.” Journal of the American Statistical Association, 91: 109–122.
  • Berger, J. O. and Sun, D. (2008). “Objective priors for the bivariate normal model.” The Annals of Statistics, 36: 963–982.
  • Böing-Messing, F., van Assen, M. A. L. M., Hofman, A., Hoijtink, H., and Mulder, J. “Bayesian evaluation of constrained hypotheses on variances of multiple independent groups.” Psychological Methods, 22: 262–287.
  • Box, G. E. P. and Tiao, G. C. (1973). Bayesian Inference in Statistical Snalysis. Reading, MA: Addison-Wesley.
  • Braeken, J., Mulder, J., and Wood, S. (2015). “Relative effects at work: Bayes factors for order hypotheses.” Journal of Management, 41: 544–573.
  • Chung, Y. and Dey, D. K. (1998). “Bayesian approach to estimation of intraclass correlation using reference prior.” Communications in Statistics, 26: 2241–2255.
  • Conigliani, C. and O’Hagan, A. (2000). “Sensitivity of the fractional Bayes factor to prior distributions.” The Canadian Journal of Statistics, 28: 343–352.
  • Consonni, G. and Paroli, R. (2017). “Objective Bayesian Comparison of Constrained Analysis of Variance Models.” Psychometrika, 82: 589–609.
  • De Santis, F. and Spezzaferri, F. (2001). “Consistent fractional Bayes factor for nested normal linear models.” Journal of Statistical Planning and Inference, 97: 305–321.
  • Donner, A. and Zou, G. (2002). “Testing the equality of dependent intraclass correlation coefficients.” Journal of the Royal Statistical Society Series D, 51: 379–379.
  • Fox, J.-P. (2010). Bayesian Item Response Modeling: Theory and Applications. New York: Springer.
  • Fox, J.-P., Mulder, J., and Sinharay, S. (2017). “Bayes factor covariance testing in item response models.” Psychometrika, 82: 979–1006.
  • García-Donato, G. and Sun, D. (2007). “Objective priors for hypothesis testing in one-way random effects models.” Canadian Journal of Statistics, 35: 302–320.
  • Gelman, A. (2006). “Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper).” Bayesian Analysis, 1: 515–534.
  • Ghosh, J. K. and Mukerjee, R. (1992). Non-informative priors, 195–210. Oxford University Press.
  • Gilks, W. R. (1995). “Discussion to fractional Bayes factors for model comparison (by O’Hagan).” Journal of the Royal Statistical Society Series B, 56: 118–120.
  • Griffin, J. and Brown, P. (2005). “Alternative prior distributions for variable selection with very many more variables than observations.” Technical report, University of Warwick.
  • Gu, X., Mulder, J., and Hoijtink, H. (2014). “Bayesian evaluation of inequality constrained hypotheses.” Psychological Methods, 9(4): 511-527.
  • Gu, X., Mulder, J., Decovíc, M., and Hoijtink, H. (2017). “Approximated adjusted fractional Bayes factors: A general method for testing informative hypotheses.” British Journal of Mathematical and Statistical Psychology.
  • Haldane, J. B. S. (1932). “A note on inverse probability.” Mathematical Proceedings of the Cambridge Philosophical Society, 28: 55–61.
  • Harris, J. A. (1913). “On the calculation of intra-class and inter-class coefficients of correlation from class moments when the number of possible combinations is large.” Biometrika, 9: 446–472.
  • Hedges, L. V. and Hedberg, E. C. (2007). “Intraclass Correlation Values for Planning Group-Randomized Trials in Education.” Educational Evaluation and Policy Analysis, 29(1): 60–87. URL
  • Hoijtink, H. (2011). Informative Hypotheses: Theory and Practice for Behavioral and Social Scientists. New York: Chapman & Hall/CRC.
  • Hoijtink, H., Gu, X., and Mulder, J. (2018). “Bayesian evaluation of informative hypotheses for multiple populations.” British Journal of Mathematical and Statistical Psychology.
  • Jeffreys, H. (1961). Theory of Probability-3rd ed. New York: Oxford University Press.
  • Johnson, V. E. and Rossell, D. (2010). “On the use of non-local prior densities in Bayesian hypothesis tests.” Journal of the Royal Statistical Society Series B, 72: 143–170.
  • Kass, R. E. and Vaidyanathan, S. K. (1992). “Approximate Bayes Factors and Orthogonal Parameters, with Application to Testing Equality of Two Binomial Proportions.” Journal of the Royal Statistical Society, Series B, 54: 129–144.
  • Kass, R. E. and Wasserman, L. (1995). “A Reference Bayesian Test for Nested Hypotheses and Its Relationship to the Schwarz Criterion.” Journal of the American Statistical Association, 90: 928–934.
  • Kinney, S. and Dunson, D. B. (2007). “Fixed and Random Effects Selection in Linear and Logistic Models.” Biometrics, 63: 690–698.
  • Kish, L. (1965). Survey sampling. Chichester: Wiley.
  • Konishi, S. and Gupta, K. A. (1989). “Testing the equality of several intraclass correlation coefficients.” Journal of Statistical Planning and Inference, (1): 93–105.
  • Lancaster, H. O. (1965). “The Helmert matrices.” The American Mathematical Monthly, 72: 4–12.
  • Liang, F., Paulo, R., Molina, G., Clyde, M. A., and Berger, J. O. (2008). “Mixtures of $g$ priors for Bayesian variable selection.” Journal of American Statistical Association, 103(481): 410–423.
  • Liang, K.-Y. and Zeger, S. L. (1986). “Longitudinal data analysis using generalized linear models.” Biometrika, 73(1): 13.
  • Ly, A., Verhagen, J., and Wagenmakers, E.-J. (2016). “An evaluation of alternative methods for testing hypotheses, from the perspective of Harold Jeffreys.” Journal of Mathematical Psychology, 72: 43–55.
  • Mulder, J. (2014). “Prior adjusted default Bayes factors for testing (in)equality constrained hypotheses.” Computational Statistics and Data Analysis, 71: 448–463.
  • Mulder, J. (2016). “Bayes factors for testing order-constrained hypotheses on correlations.” Journal of Mathematical Psychology, 72: 104–115.
  • Mulder, J. and Fox, J.-P. (2013). “Bayesian tests on components of the compound symmetry covariance matrix.” Statistics and Computing, 23: 109–122.
  • Mulder, J. and Fox, J.-P. (2018). “Supplemetary material for “Bayes factor testing of multiple intraclass correlations”.” Bayesian Analysis.
  • Mulder, J., Klugkist, I., van de Schoot, A., Meeus, W., Selfhout, M., and Hoijtink, H. (2009). “Bayesian Model Selection of Informative Hypotheses for Repeated Measurements.” Journal of Mathematical Psychology, 53: 530–546.
  • Mulder, J. and Pericchi, L. R. (2018). “The matrix-$F$ prior for estimating and testing covariance matrices.” Bayesian Analysis.
  • Mullis, I., Martin, M. O., Foy, P., and Arora, A. (2011). TIMSS 2011 International Results in Mathematics. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College Chestnut Hill, MA, USA and International Association for the Evaluation of Educational Achievement (IEA) IEA Secretariat, Amsterdam, the Netherlands, first edition.
  • O’Hagan, A. (1995). “Fractional Bayes Factors for Model Comparison (with discussion).” Journal of the Royal Statistical Society Series B, 57: 99–138.
  • O’Hagan, A. (1997). “Properties of intrinsic and fractional Bayes factors.” Test, 6: 101–118.
  • Olson, J. F., Martin, M. O., and Mullis, I. V. S. (2008). TIMSS 2007 Technical Report. Chestnut Hill, MA: TIMSS and PIRLS International Study Center, Boston College, first edition.
  • Pauler, D. K., Wakefield, J. C., and Kass, R. E. (1999). “Bayes Factors and Approximations for Variance Component Models.” Journal of the American Statistical Association, 94(448): 1242–1253.
  • Pérez, M. E., Pericchi, L. R., and Ramirez, I. C. (2017). “The Scaled Beta2 Distribution as a Robust Prior for Scales.” Bayesian Analysis, 12: 615.
  • Pericchi, L. R. (2005). “Model selection and hypothesis testing based on objective probabilities and Bayes Factors.” In D. K. Dey, C. R. Rao (eds.), Bayesian Thinking: Modeling and Computation, 115–149. Elsevier, Amsterdam, The Netherlands.
  • Pericchi, L. R., Liu, G., and Torres, D. (2008). “Objective Bayes factors for informative hypotheses: “Completing” the informative hypothesis and “splitting” the Bayes factors.” In H. Hoijtink, I. Klugkist, and P. Boelen (eds.), Bayesian Evaluation of Informative Hypotheses, 131–154. New York: Springer.
  • Polson, N. G. and Scott, J. G. (2012). “On the Half-Cauchy Prior for a Global Scale Parameter.” Bayesian Analysis, 7.
  • Raftery, A. E. (1995). “Bayesian model selection in social research.” Sociological Methodology, 25: 111–163.
  • Raudenbush, S. W. (1997). “Statistical analysis and optimal design for cluster randomized trials.” Psychological Methods, (2): 173–185. URL
  • Schwarz, G. E. (1978). “Estimating the dimension of a model.” Annals of Statistics, 6: 461–464.
  • Scott, J. G. and Berger, J. O. (2006). “An exploration of aspects of Bayesian multiple testing.” Journal of Statistical Planning and Inference, 136: 2144–2162.
  • Searle, S. R., Casella, G., and McCulloch, C. E. (1992). Variance Components. John Wiley & Sons, Inc., Hoboken, NJ, USA, first edition.
  • Sellke, T., Bayarri, M. J., and Berger, J. O. (2001). “Calibration of $p$ Values for Testing Precise Null Hypotheses.” The American Statistician, 55: 62–71.
  • Severini, T. A., Mukerjee, R., and Ghosh, M. (2002). “On an exact probability matching property of right-invariant priors.” Biometrika, 89: 952–957.
  • Silvapulle, M. J. and Sen, P. K. (2004). Constrained Statistical Inference: Inequality, Order, and Shape Restrictions. Hoboken, NJ: John Wiley, second edition.
  • Spiegelhalter, D. J. (2001). “Bayesian methods for cluster randomized trials with continuous responses.” Statistics in Medicine, 20(3): 435–452. URL<435::AID-SIM804>3.0.CO;2-E
  • Stein, C. (1985). On the coverage probability of confidence sets based on a prior distribution, 485–514. PWN-Polish Scientific Publishers.
  • Van Geel, M., Keuning, T., Visscher, A., and Fox, J.-P. (2017). “Changes in educators’ data literacy during a data-based decision making intervention.” Teaching and Teacher Education, 64: 187–198.
  • Welch, B. and Peers, H. W. (1963). “On formulae for confidence points based on integrals of weighted likelihoods.” Journal of the Royal Statistical Society Series B, 25: 318–29.
  • Westfall, P. and Gönen, M. (1996). “Asymptotic properties of ANOVA Bayes factors.” Communications in Statistics: Theory and Methods, 25: 3101–3123.
  • Ye, K. (1994). “Bayesian reference prior analysis on the ratio of variances for the balanced one-way random effect model.” Journal of Statistical Planning and Inference, 41: 267–280.
  • Zellner, A. (1986). “On assessing prior distributions and Bayesian regression analysis with $g$-prior distributions.” In Goel, P. K. and Zellner, A. (eds.), Bayesian inference and decision techniques: Essays in honor of Bruno de Finetti, 233–243. Amsterdam: Elsevier.
  • Zellner, A. and Siow, A. (1980). “Posterior odds ratios for selected regression hypotheses.” In J. M. Bernardo, D. V. Lindley, A. F. M. Smith, M. H. DeGroot (ed.), Bayesian statistics: proceedings of the first international meeting held in Valencia, 585–603. Spain: University of Valencia.

Supplemental materials

  • The supplementary material for “Bayes Factor Testing of Multiple Intraclass Correlations”. The supplementary material for “Bayes factor testing of multiple intraclass correlations” contains the proof of Lemma 1, the proof of Lemma 2, the conditional posterior distributions for the Gibbs sampler, the proof of Lemma 3, the Gibbs sampler under a constrained model, the analytic expression of the marginal likelihood (with derivation) for a standard random intercept model using fractional Bayes methodology, and a simulation study when testing interval hypotheses.