## Institute of Mathematical Statistics Lecture Notes - Monograph Series

- Optimality
- 2009, 216-236

### Mixed Models, Posterior Means and Penalized Least-Squares

#### Abstract

This paper reviews the connections between estimators that derive from three different modeling methodologies: Mixed-effects models, Bayesian models and Penalized Least-squares. Extension of classical results on the equivalence for smoothing spline estimators and best linear unbiased prediction and/or posterior analysis of certain Gaussian signal-plus-noise models is examined in a more general setting. These connections allow for the application of an efficient, linear time algorithm, to estimate parameters, compute random effects predictions and evaluate likelihoods in a large class of model scenarios. We also show that the methods of generalized cross-validation, restricted maximum likelihood and unbiased risk prediction can be used to estimate the variance components or adaptively select the smoothing parameters in any of the three settings.

#### Chapter information

**Source***Optimality: The Third Erich L. Lehmann Symposium* (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2009)

**Dates**

First available in Project Euclid: 3 August 2009

**Permanent link to this document**

https://projecteuclid.org/euclid.lnms/1249305331

**Digital Object Identifier**

doi:10.1214/09-LNMS5713

**Mathematical Reviews number (MathSciNet)**

MR2681665

**Zentralblatt MATH identifier**

1271.62088

**Subjects**

Primary: 62J05: Linear regression

Secondary: 65C20: Models, numerical methods [See also 68U20]

**Keywords**

smoothing splines p-splines ridge regression varying coefficient models Bayesian prediction Kalman fiter adaptive selection

**Rights**

Copyright © 2009, Institute of Mathematical Statistics

#### Citation

Rojo, Javier. Mixed Models, Posterior Means and Penalized Least-Squares. Optimality, 216--236, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2009. doi:10.1214/09-LNMS5713. https://projecteuclid.org/euclid.lnms/1249305331

#### References

- [1] Aitkin, M. (1999). A general maximum likelihood analysis of variance components in generalized linear models.
*Statist. Meth. Med. Res.***55**117–128. - [2] Anderson, B. D. O. and Moore, J. B. (1979).
*Optimal Filtering*. Prentice Hall, Englewood Cliffs, NJ. - [3] Ansley, C. F. and Kohn, R. (1985). Estimation, filtering, and smoothing in state space models with incompletely specified initial conditions.
*Ann. Statist.***13**1286–1316. - [4] Brumback, B. A. and Rice, J. A. (1998). Smoothing spline models for the analysis of nested and crossed samples of curves.
*J. Amer. Statist. Assoc.***93**961–993. - [5] Crainiceanu, C. M., Ruppert, D. and Wand, M. P. (2005). Bayesian analysis for penalized spline regression using WinBugs.
*Journal of Statistical Software***14**1–24. - [6] Craven, P. and Wahba, G. (1979). Smoothing noisy data with spline functions.
*Numer. Math.***31**377–403. - [7] Cressie, N. (1990). Reply: Letters to the editor.
*Amer. Statist.***44**256–258. - [8] Dempster, A. P., Laird, N. M. and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm.
*J. Roy. Statist. Soc. Ser. B***39**1–22. - [9] Eilers, P. H. C. and Marx, B. D. (1996). Flexible smoothing with
*B*-splines and penalties.*Statist. Sci.***11**89–102. - [10] Eppright, E., Fox, H., Fryer, B., Lamkin, G., Vivian, V. and Fuller, E. (1972). Nutrition of infants and preschool children in the North Central Region of the United States of America.
*World Review of Nutrition and Dietetics***14**269–332. - [11] Eubank, R. L., Huang, C., Muñoz Maldonado, Y., Wang, N., Wang, S. and Buchanan, R. J. (2004). Smoothing spline estimation in varying coefficient models.
*J. Roy. Statist. Soc. Ser. B***66**653–667. - [12] Eubank, R. L. (1988).
*Spline Smoothing and Nonparametric Regression*, 1st ed. Marcel Dekker, New York. - [13] Eubank, R. L., Huang, C. and Wang, S. (2003). Adaptive order selection for spline smoothing.
*J. Comput. Graph. Statist.***12**2546–2559. - [14] Eubank, R. L. (2006).
*A Kalman Filter Primer*. Chapman & Hall/CRC, Boca Raton, FL. - [15] de Finetti, B. (1964). Foresight: Its logical laws, its subjective sources. In
*Studies in Subjective Probability*. Wiley, New York. - [16] Gantmakher, F. R. (1959).
*The Theory of Matrices*. Chelsea Pub. Co., New York. - [17] Golub, G. H., Heath, M. and Wahba, G. (1979). Generalized-cross validation as a method for choosing a good ridge parameter.
*Technometrics***58**215–223. - [18] Graybill, F. A. (1976).
*The Theory and Application of the Linear Model*. Duxbury, North Scituate, MA. - [19] Guo, W. (2002). Functional mixed effects models.
*Biometrics***58**121–128. - [20] Guo, W. (2003). Functional data analysis in longitudinal settings using smoothing splines.
*Statist. Meth. Med. Res.***13**1–24. - [21] Hastie, T. and Tibshirani, R. (1993). Varying-coefficient models.
*J. Roy. Statist. Soc. Ser. B***4**757–796. - [22] Heckman, N. (1997). The theory and application of penalized least squares methods or reproducing kernel Hilbert spaces made easy. Technical Report 216, The Univ. British Columbia.
- [23] Henderson, C. R., Kempthorne, O., Searle, S. R. and Krosigk, C. M. (1959). The estimation of environmental and genetic trends from records subject to culling.
*Biometrics***15**192–218. - [24] Householder, A. (1964).
*The Theory aof Matrices in Numerical Analysis*. Dover, New York. - [25] Kohn, R. and Ansley, C. F. (1987). A new algorithm for spline smoothing based on smoothing a stochastic process.
*SIAM J. Sci. Statist. Comput.***8**33–48. - [26] Kohn, R. and Ansley, C. F. (1989). A fast algorithm for signal extraction, influence and cross-validation in state-space models.
*Biometrika***76**65–79. - [27] Kohn, R., Ansley, C. F. and Tharm, D. (1993). Performance of cross-validation and maximum likelihood estimators of spline smoothing parameters.
*J. Amer. Statist. Assoc.***86**1042–1050. - [28] Koopman, S. J. and Durbin, J. (1998). Fast filtering and smoothing for multivariate state space models.
*J. Times Ser. Anal.***21**281–296. - [29] Lindley, D. V. and Smith, A. F. M. (1972). Bayes estimates for the linear model.
*J. Roy. Statist. Soc. Ser. B***34**1–41. - [30] Piehpo, H. P. and Ogutu, J. O. (2007). Simple state-space models in a mixed-model framework.
*Amer. Statist.***61**224–232. - [31] Nychka, D. (1988). Bayesian confidence intervals for smoothing splines.
*J. Amer. Statist. Assoc.***83**1134–1143. - [32] Nychka, D. (2000). Spatial process estimates as smoothers. In
*Smoothing and Regression. Approaches, Computation and Application*. Wiley, New York. - [33] Patterson, H. D. and Thompson, R. (1971). Recovery of interblock information when cell sizes are unequal.
*Biometrika***58**545–554. - [34] Riesby, N., Gram, L.F., Bech, P., Nagy, A., Petersen, G. O., Ortmann, J., Ibsen, I., Dencker, S. J., Jacobsen, O., Krautwald, O., Søndergaard, I. and Chirstiansen, J. (1977). Imipramine: Clinical effects and pharmacokinetic variability.
*Psychopharmacology***54**263–272. - [35] Robinson, G. K. (1991). That BLUP is a good thing: The estimation of random effects.
*Statist. Sci.***6**15–32. - [36] Ruppert, D., Wand, M. P. and Carroll, R. J. (2003).
*Semiparametric Regression*. Cambridge Univ. Press, New York. - [37] Ruppert, D. (2002). Selecting the number of knots for penalized splines.
*J. Comput. Graph. Statist.***11**735–757. - [38] Sallas, W. M. and Harville, D. A. (1981). Best linear recursive estimation for mixed linear models.
*J. Amer. Statist. Assoc.***76**860–869. - [39] Schafer, J. L. and Recai, M. Y. (2002). Computational strategies for multivariate linear mixed-effects models with missing values.
*J. Comput. Graph. Statist.***11**437–457. - [40] Speed, T. (1991). That BLUP is a good Thing: The estimation of random effects: Comment.
*Statist. Sci.***6**42–44. - [41] Stein, M. L. (2000). A comparison of generalized cross validation and modified maximum likelihood for estimating the parameters of a stochastic process.
*Ann. Statist.***18**1139–1157. - [42] Strandén, I. and Lidauer, M. (1999). Solving large mixed linear models using preconditioned conjugate gradient iteration.
*Journal of Diary Science***82**2779–2787. - [43] Wahba, G. (1978). Improper priors, spline smoothing, and the problem of guarding against model errors in regression.
*J. Roy. Statist. Soc. Ser. B***40**364–372. - [44] Wahba, G. (1983). Bayesian “confidence intervals” for the cross-validated smoothing spline.
*J. Roy. Statist. Soc. Ser. B***40**364–372. - [45] Wahba, G. (1990). Spline models for observational data.
*SIAM***59**. Philadelphia, Pennsylvania. - [46] Wahba, G. (1985). A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem.
*Ann. Statist.***13**1378–1402. - [47] Wahba, G. (1990). Comment on Cressie: Letters to the editor.
*Amer. Statist.***44**255–256. - [48] Wang, Y. (1998). Smoothing spline models with correlated random errors.
*J. Amer. Statist. Assoc.***93**341–348. - [49] Wang, Y. (1998). Mixed effects smoothing spline analysis of variance.
*J. Roy. Statist. Soc. Ser. B***60**159–174.