Electronic Journal of Statistics

Functional models for longitudinal data with covariate dependent smoothness

David K. Mensah, David J. Nott, Linda S. L. Tan, and Lucy Marshall

Full-text: Open access


This paper considers functional models for longitudinal data with subject and group specific trends modelled using Gaussian processes. Fitting Gaussian process regression models is a computationally challenging task, and various sparse approximations to Gaussian processes have been considered in the literature to ease the computational burden. This manuscript builds on a fast non-standard variational approximation which uses a sparse spectral representation and is able to treat uncertainty in the covariance function hyperparameters. This allows fast variational computational methods to be extended to models where there are many functions to be estimated and where there is a hierarchical model involving the covariance function parameters. The main goal of this paper is to implement this idea in the context of functional models for longitudinal data by allowing individual specific smoothness related to covariates for different subjects. Understanding the relationship of smoothness to individual specific covariates is of great interest in some applications. The methods are illustrated with simulated data and a dataset of streamflow curves generated by a rainfall runoff model, and compared with MCMC. It is also shown how these methods can be used to obtain good proposal distributions for MCMC analyses.

Article information

Electron. J. Statist., Volume 10, Number 1 (2016), 527-549.

Received: July 2014
First available in Project Euclid: 4 March 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Functional data Gaussian processes longitudinal data variational Bayes


Mensah, David K.; Nott, David J.; Tan, Linda S. L.; Marshall, Lucy. Functional models for longitudinal data with covariate dependent smoothness. Electron. J. Statist. 10 (2016), no. 1, 527--549. doi:10.1214/16-EJS1113. https://projecteuclid.org/euclid.ejs/1457123505

Export citation


  • Andrieu, C. and Thoms, J. (2008). A tutorial on adaptive MCMC., Statistics and Computing 18 343–373.
  • Attias, H. (2000). A variational Bayesian framework for graphical models. In, Advances in Neural Information Processing Systems 12 209–215. MIT Press.
  • Bishop, C. M. (2006)., Pattern Recognition and Machine Learning. Springer.
  • de Freitas, N., Højen-Sørensen, P., Jordan, M. I. and Russell, S. (2001). Variational MCMC. In, Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence. UAI’01 120–127. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
  • Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A. and Rubin, D. B. (2013)., Bayesian Data Analysis, Third Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis.
  • Ghahramani, Z. and Beal, M. J. (2001). Propagation algorithms for variational Bayesian learning. In, Advances in Neural Information Processing Systems 13 (T. K. Leen, T. G. Dietterich and V. Tresp, eds.) 507–513. MIT Press.
  • Goldsmith, J., Wand, M. P. and Crainiceanu, C. (2011). Functional regression via variational Bayes., Electronic Journal of Statistics 5 572–602.
  • Honkela, A., Valpola, H. and Karhunen, J. (2003). Accelerating cyclic update algorithms for parameter estimation by pattern searches., Neural Processing Letters 17 191–203.
  • Kavetski, D., Kuczera, G. and Franks, S. W. (2006). Bayesian analysis of input uncertainty in hydrological modeling: 1.Theory., Water Resources Research 42 1–9.
  • Knowles, D. A. and Minka, T. P. (2011). Non-conjugate variational message passing for multinomial and binary regression. In, Advances in Neural Information Processing Systems 24 (J. Shawe-Taylor, R. S. Zemel, P. Bartlett, F. Pereira and K. Q. Weinberger, eds.) 1701–1709.
  • Lázaro-Gredilla, M., Quiñonero-Candela, J., Rasmussen, C. E. and Figueiras-Vidal, A. R. (2010). Sparse spectrum Gaussian process regression., Journal of Machine Learning Research 11 1865–1881.
  • Mensah, D. K., Tan, L. S. L., Marshall, L. and Nott, D. J. (2016). Supplementary material for “Functional models for longitudinal data with covariate dependent smoothness”. DOI:, 10.1214/16-EJS1113SUPPA.
  • Müller, H. G. and Yao, F. (2010). Empirical dynamics for longitudinal data., The Annals of Statistics 38 3458–3486.
  • Ormerod, J. T. and Wand, M. P. (2010). Explaining variational approximations., The American Statistician 64 140–153.
  • Rasmussen, C. E. and Williams, C. (2006)., Gaussian Processes for Machine Learning. The MIT Press.
  • Reithinger, F., Jank, W., Tutz, G. and Shmueli, G. (2008). Smoothing sparse and unevenly sampled curves using semiparametric mixed models: An application to online auctions., Journal of the Royal Statistical Society, Series C 57 127–148.
  • Renard, B., Kavetski, D., Kuczera, G., Thyer, M. and Franks, S. W. (2010). Understanding predictive uncertainty in hydrologic modeling: The challenge of identifying input and structural errors., Water Resources Research 46. W05521.
  • Rohde, D. and Wand, M. P. (2015). Semiparametric mean field variational Bayes: General principles and numerical issues. Available at, http://matt-wand.utsacademics.info/RohdeWand.pdf.
  • Salakhutdinov, R. and Roweis, S. (2003). Adaptive overrelaxed bound optimization methods. In, Proceedings of the 20th International Conference on Machine Learning (T. Fawcett and N. Mishra, eds.) 664–671. AAAI Press.
  • Sawicz, K., Wagener, T., Sivapalan, M., Troch, P. and Carrillo, G. (2011). Catchment classification: Empirical analysis of hydrologic similarity based on catchment function in the eastern USA., Hydrology and Earth System Sciences 15 2895–2911.
  • Shi, J. Q. and Choi, T. (2011)., Gaussian process regression analysis for functional data. Chapman and Hall/CRC.
  • Shi, J. Q., Murray-Smith, R. and Titterington, D. M. (2005). Hierarchical Gaussian process mixtures for regression., Statistics and Computing 15 31–41.
  • Shi, J. Q., Wang, B., Will, E. J. and West, R. M. (2012). Mixed-effects Gaussian process functional regression models with application to dose-response curve prediction., Statistics in Medicine 31 3165–3177.
  • Sivapalan, M., Takeuchi, K., Franks, S., Gupta, V., Karambiri, H., Lakshmi, V., Liang, X., McDonnell, J., Mendiondo, E., O’connell, P. et al. (2003). IAHS Decade on Predictions in Ungauged Basins (PUB), 2003–2012: Shaping an exciting future for the hydrological sciences., Hydrological Sciences Journal 48 857–880.
  • Smith, T. and Marshall, L. (2009). Exploring uncertainty and model predictive performance concepts via a modular snowmelt-runoff modeling framework., Environmental Modelling & Software 26 691–701.
  • Tan, L. S. L. and Nott, D. J. (2014). A Stochastic Variational Framework for Fitting and Diagnosing Generalized Linear Mixed Models., Bayesian Anal. 9 963–1004.
  • Tan, L. S. L., Ong, V. M. H., Nott, D. J. and Jasra, A. (2015). Variational inference for sparse spectrum Gaussian process regression., Statistics and Computing, to appear.
  • Wagener, T. and Wheater, H. S. (2006). Parameter estimation and regionalization for continuous rainfall-runoff models including uncertainty., Journal of Hydrology 320 132–154.
  • Wagener, T., Sivapalan, M., Troch, P. and Woods, R. (2007). Catchment classification and Hydrologic Similarity., Geography Compass 1 901–931.
  • Wand, M. P., Ormerod, J. T., Padoan, S. A., Fuhrwirth, R., et al. (2011). Mean field variational Bayes for elaborate distributions., Bayesian Analysis 6 847–900.
  • Wand, M. P. (2014). Fully simplified multivariate normal updates in non-conjugate variational message passing., Journal of Machine Learning Research 15 1351–1369.
  • Wang, Y. and Khardon, R. (2012). Nonparametric Bayesian Mixed-effect model: A sparse Gaussian process approach., http://arxiv.org/abs/1211.6653.
  • Wang, B., Titterington, D. et al. (2006). Convergence properties of a general algorithm for calculating variational Bayesian estimates for a normal mixture model., Bayesian Analysis 1 625–650.
  • Wang, S., Jank, W., Shmueli, G. and Smith, P. (2008). Modeling price dynamics in eBay auctions using principal differential analysis., Journal of the American Statistical Association 103 1100–1118.
  • Waterhouse, S., Mackay, D. and Robinson, T. (1996). Bayesian methods for mixture of experts. In, Advances in Neural Information Processing Systems 8 351–357. MIT Press.
  • Winn, J. and Bishop, C. M. (2005). Variational message passing., Journal of Machine Learning Research 6 661–694.
  • Zeger, S. L. and Diggle, P. J. (1994). Semiparametric models for longitudinal data with application to CD4 cell numbers in HIV seroconverters., Biometrics 50 689–699.
  • Zhu, B. and Dunson, D. B. (2012). Stochastic volatility regression for functional data dynamics., http://arxiv.org/abs/1212.0181.
  • Zhu, B., Taylor, J. M. G. and Song, P. X. K. (2011). Semiparametric stochastic modeling of the rate function in longitudinal studies., Journal of the American Statistical Association 106 1485–1495.

Supplemental materials