Institute of Mathematical Statistics Lecture Notes - Monograph Series

Mixed Models, Posterior Means and Penalized Least-Squares

Yolanda Muñoz Maldonado

Full-text: Open access


This paper reviews the connections between estimators that derive from three different modeling methodologies: Mixed-effects models, Bayesian models and Penalized Least-squares. Extension of classical results on the equivalence for smoothing spline estimators and best linear unbiased prediction and/or posterior analysis of certain Gaussian signal-plus-noise models is examined in a more general setting. These connections allow for the application of an efficient, linear time algorithm, to estimate parameters, compute random effects predictions and evaluate likelihoods in a large class of model scenarios. We also show that the methods of generalized cross-validation, restricted maximum likelihood and unbiased risk prediction can be used to estimate the variance components or adaptively select the smoothing parameters in any of the three settings.

Chapter information

Javier Rojo, ed., Optimality: The Third Erich L. Lehmann Symposium (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2009), 216-236

First available in Project Euclid: 3 August 2009

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62J05: Linear regression
Secondary: 65C20: Models, numerical methods [See also 68U20]

smoothing splines p-splines ridge regression varying coefficient models Bayesian prediction Kalman fiter adaptive selection

Copyright © 2009, Institute of Mathematical Statistics


Rojo, Javier. Mixed Models, Posterior Means and Penalized Least-Squares. Optimality, 216--236, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2009. doi:10.1214/09-LNMS5713.

Export citation


  • [1] Aitkin, M. (1999). A general maximum likelihood analysis of variance components in generalized linear models. Statist. Meth. Med. Res. 55 117–128.
  • [2] Anderson, B. D. O. and Moore, J. B. (1979). Optimal Filtering. Prentice Hall, Englewood Cliffs, NJ.
  • [3] Ansley, C. F. and Kohn, R. (1985). Estimation, filtering, and smoothing in state space models with incompletely specified initial conditions. Ann. Statist. 13 1286–1316.
  • [4] Brumback, B. A. and Rice, J. A. (1998). Smoothing spline models for the analysis of nested and crossed samples of curves. J. Amer. Statist. Assoc. 93 961–993.
  • [5] Crainiceanu, C. M., Ruppert, D. and Wand, M. P. (2005). Bayesian analysis for penalized spline regression using WinBugs. Journal of Statistical Software 14 1–24.
  • [6] Craven, P. and Wahba, G. (1979). Smoothing noisy data with spline functions. Numer. Math. 31 377–403.
  • [7] Cressie, N. (1990). Reply: Letters to the editor. Amer. Statist. 44 256–258.
  • [8] Dempster, A. P., Laird, N. M. and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Statist. Soc. Ser. B 39 1–22.
  • [9] Eilers, P. H. C. and Marx, B. D. (1996). Flexible smoothing with B-splines and penalties. Statist. Sci. 11 89–102.
  • [10] Eppright, E., Fox, H., Fryer, B., Lamkin, G., Vivian, V. and Fuller, E. (1972). Nutrition of infants and preschool children in the North Central Region of the United States of America. World Review of Nutrition and Dietetics 14 269–332.
  • [11] Eubank, R. L., Huang, C., Muñoz Maldonado, Y., Wang, N., Wang, S. and Buchanan, R. J. (2004). Smoothing spline estimation in varying coefficient models. J. Roy. Statist. Soc. Ser. B 66 653–667.
  • [12] Eubank, R. L. (1988). Spline Smoothing and Nonparametric Regression, 1st ed. Marcel Dekker, New York.
  • [13] Eubank, R. L., Huang, C. and Wang, S. (2003). Adaptive order selection for spline smoothing. J. Comput. Graph. Statist. 12 2546–2559.
  • [14] Eubank, R. L. (2006). A Kalman Filter Primer. Chapman & Hall/CRC, Boca Raton, FL.
  • [15] de Finetti, B. (1964). Foresight: Its logical laws, its subjective sources. In Studies in Subjective Probability. Wiley, New York.
  • [16] Gantmakher, F. R. (1959). The Theory of Matrices. Chelsea Pub. Co., New York.
  • [17] Golub, G. H., Heath, M. and Wahba, G. (1979). Generalized-cross validation as a method for choosing a good ridge parameter. Technometrics 58 215–223.
  • [18] Graybill, F. A. (1976). The Theory and Application of the Linear Model. Duxbury, North Scituate, MA.
  • [19] Guo, W. (2002). Functional mixed effects models. Biometrics 58 121–128.
  • [20] Guo, W. (2003). Functional data analysis in longitudinal settings using smoothing splines. Statist. Meth. Med. Res. 13 1–24.
  • [21] Hastie, T. and Tibshirani, R. (1993). Varying-coefficient models. J. Roy. Statist. Soc. Ser. B 4 757–796.
  • [22] Heckman, N. (1997). The theory and application of penalized least squares methods or reproducing kernel Hilbert spaces made easy. Technical Report 216, The Univ. British Columbia.
  • [23] Henderson, C. R., Kempthorne, O., Searle, S. R. and Krosigk, C. M. (1959). The estimation of environmental and genetic trends from records subject to culling. Biometrics 15 192–218.
  • [24] Householder, A. (1964). The Theory aof Matrices in Numerical Analysis. Dover, New York.
  • [25] Kohn, R. and Ansley, C. F. (1987). A new algorithm for spline smoothing based on smoothing a stochastic process. SIAM J. Sci. Statist. Comput. 8 33–48.
  • [26] Kohn, R. and Ansley, C. F. (1989). A fast algorithm for signal extraction, influence and cross-validation in state-space models. Biometrika 76 65–79.
  • [27] Kohn, R., Ansley, C. F. and Tharm, D. (1993). Performance of cross-validation and maximum likelihood estimators of spline smoothing parameters. J. Amer. Statist. Assoc. 86 1042–1050.
  • [28] Koopman, S. J. and Durbin, J. (1998). Fast filtering and smoothing for multivariate state space models. J. Times Ser. Anal. 21 281–296.
  • [29] Lindley, D. V. and Smith, A. F. M. (1972). Bayes estimates for the linear model. J. Roy. Statist. Soc. Ser. B 34 1–41.
  • [30] Piehpo, H. P. and Ogutu, J. O. (2007). Simple state-space models in a mixed-model framework. Amer. Statist. 61 224–232.
  • [31] Nychka, D. (1988). Bayesian confidence intervals for smoothing splines. J. Amer. Statist. Assoc. 83 1134–1143.
  • [32] Nychka, D. (2000). Spatial process estimates as smoothers. In Smoothing and Regression. Approaches, Computation and Application. Wiley, New York.
  • [33] Patterson, H. D. and Thompson, R. (1971). Recovery of interblock information when cell sizes are unequal. Biometrika 58 545–554.
  • [34] Riesby, N., Gram, L.F., Bech, P., Nagy, A., Petersen, G. O., Ortmann, J., Ibsen, I., Dencker, S. J., Jacobsen, O., Krautwald, O., Søndergaard, I. and Chirstiansen, J. (1977). Imipramine: Clinical effects and pharmacokinetic variability. Psychopharmacology 54 263–272.
  • [35] Robinson, G. K. (1991). That BLUP is a good thing: The estimation of random effects. Statist. Sci. 6 15–32.
  • [36] Ruppert, D., Wand, M. P. and Carroll, R. J. (2003). Semiparametric Regression. Cambridge Univ. Press, New York.
  • [37] Ruppert, D. (2002). Selecting the number of knots for penalized splines. J. Comput. Graph. Statist. 11 735–757.
  • [38] Sallas, W. M. and Harville, D. A. (1981). Best linear recursive estimation for mixed linear models. J. Amer. Statist. Assoc. 76 860–869.
  • [39] Schafer, J. L. and Recai, M. Y. (2002). Computational strategies for multivariate linear mixed-effects models with missing values. J. Comput. Graph. Statist. 11 437–457.
  • [40] Speed, T. (1991). That BLUP is a good Thing: The estimation of random effects: Comment. Statist. Sci. 6 42–44.
  • [41] Stein, M. L. (2000). A comparison of generalized cross validation and modified maximum likelihood for estimating the parameters of a stochastic process. Ann. Statist. 18 1139–1157.
  • [42] Strandén, I. and Lidauer, M. (1999). Solving large mixed linear models using preconditioned conjugate gradient iteration. Journal of Diary Science 82 2779–2787.
  • [43] Wahba, G. (1978). Improper priors, spline smoothing, and the problem of guarding against model errors in regression. J. Roy. Statist. Soc. Ser. B 40 364–372.
  • [44] Wahba, G. (1983). Bayesian “confidence intervals” for the cross-validated smoothing spline. J. Roy. Statist. Soc. Ser. B 40 364–372.
  • [45] Wahba, G. (1990). Spline models for observational data. SIAM 59. Philadelphia, Pennsylvania.
  • [46] Wahba, G. (1985). A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem. Ann. Statist. 13 1378–1402.
  • [47] Wahba, G. (1990). Comment on Cressie: Letters to the editor. Amer. Statist. 44 255–256.
  • [48] Wang, Y. (1998). Smoothing spline models with correlated random errors. J. Amer. Statist. Assoc. 93 341–348.
  • [49] Wang, Y. (1998). Mixed effects smoothing spline analysis of variance. J. Roy. Statist. Soc. Ser. B 60 159–174.