Annals of Statistics

Regularized estimation in sparse high-dimensional time series models

Sumanta Basu and George Michailidis

Full-text: Open access

Enhanced PDF (518 KB)

Abstract

Many scientific and economic problems involve the analysis of high-dimensional time series datasets. However, theoretical studies in high-dimensional statistics to date rely primarily on the assumption of independent and identically distributed (i.i.d.) samples. In this work, we focus on stable Gaussian processes and investigate the theoretical properties of $\ell_{1}$-regularized estimates in two important statistical problems in the context of high-dimensional time series: (a) stochastic regression with serially correlated errors and (b) transition matrix estimation in vector autoregressive (VAR) models. We derive nonasymptotic upper bounds on the estimation errors of the regularized estimates and establish that consistent estimation under high-dimensional scaling is possible via $\ell_{1}$-regularization for a large class of stable processes under sparsity constraints. A key technical contribution of the work is to introduce a measure of stability for stationary processes using their spectral properties that provides insight into the effect of dependence on the accuracy of the regularized estimates. With this proposed stability measure, we establish some useful deviation bounds for dependent data, which can be used to study several important regularized estimates in a time series setting.

Article information

Source
Ann. Statist., Volume 43, Number 4 (2015), 1535-1567.

Dates
Received: February 2014
Revised: January 2015
First available in Project Euclid: 17 June 2015

Permanent link to this document
https://projecteuclid.org/euclid.aos/1434546214

Digital Object Identifier
doi:10.1214/15-AOS1315

Mathematical Reviews number (MathSciNet)
MR3357870

Zentralblatt MATH identifier
1317.62067

Subjects
Primary: 62M10: Time series, auto-correlation, regression, etc. [See also 91B84] 62J99: None of the above, but in this section
Secondary: 2M15

Keywords
High-dimensional time series stochastic regression vector autoregression covariance estimation lasso

Citation

Basu, Sumanta; Michailidis, George. Regularized estimation in sparse high-dimensional time series models. Ann. Statist. 43 (2015), no. 4, 1535--1567. doi:10.1214/15-AOS1315. https://projecteuclid.org/euclid.aos/1434546214


Export citation

References

  • Bańbura, M., Giannone, D. and Reichlin, L. (2010). Large Bayesian vector auto regressions. J. Appl. Econometrics 25 71–92.
    Mathematical Reviews (MathSciNet): MR2751790
    Digital Object Identifier: doi:10.1002/jae.1137
  • Basu, S. (2014). Modeling and estimation of high-dimensional vector autoregressions. Ph.D. thesis, Univ. Michigan, Ann Arbor, MI.
  • Basu, S. and Michailidis, G. (2015). Supplement to “Regularized estimation in sparse high-dimensional time series models.” DOI:10.1214/15-AOS1315SUPP.
  • Bernanke, B. S., Boivin, J. and Eliasz, P. (2005). Measuring the effects of monetary policy: A factor-augmented vector autoregressive (FAVAR) approach. Q. J. Econ. 120 387–422.
  • Bickel, P. J. and Levina, E. (2008). Covariance regularization by thresholding. Ann. Statist. 36 2577–2604.
    Mathematical Reviews (MathSciNet): MR2485008
    Zentralblatt MATH: 1196.62062
    Digital Object Identifier: doi:10.1214/08-AOS600
    Project Euclid: euclid.aos/1231165180
  • Bickel, P. J., Ritov, Y. and Tsybakov, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705–1732.
    Mathematical Reviews (MathSciNet): MR2533469
    Zentralblatt MATH: 1173.62022
    Digital Object Identifier: doi:10.1214/08-AOS620
    Project Euclid: euclid.aos/1245332830
  • Chen, X., Xu, M. and Wu, W. B. (2013). Covariance and precision matrix estimation for high-dimensional time series. Ann. Statist. 41 2994–3021.
    Mathematical Reviews (MathSciNet): MR3161455
    Zentralblatt MATH: 1294.62123
    Digital Object Identifier: doi:10.1214/13-AOS1182
    Project Euclid: euclid.aos/1388545676
  • Chudik, A. and Pesaran, M. H. (2011). Infinite-dimensional VARs and factor models. J. Econometrics 163 4–22.
    Mathematical Reviews (MathSciNet): MR2803662
    Digital Object Identifier: doi:10.1016/j.jeconom.2010.11.002
  • Davis, R. A., Zang, P. and Zheng, T. (2012). Sparse vector autoregressive modeling. Preprint. Available at arXiv:1207.0520.
    arXiv: 1207.0520
  • De Mol, C., Giannone, D. and Reichlin, L. (2008). Forecasting using a large number of predictors: Is Bayesian shrinkage a valid alternative to principal components? J. Econometrics 146 318–328.
    Mathematical Reviews (MathSciNet): MR2465176
    Digital Object Identifier: doi:10.1016/j.jeconom.2008.08.011
  • Dobriban, E. and Fan, J. (2013). Regularity properties of high-dimensional covariate matrices. Preprint. Available at arXiv:1305.5198.
    arXiv: 1305.5198
  • Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
    Mathematical Reviews (MathSciNet): MR1946581
    Zentralblatt MATH: 1073.62547
    Digital Object Identifier: doi:10.1198/016214501753382273
  • Fan, Y. and Lv, J. (2013). Asymptotic equivalence of regularization methods in thresholded parameter space. J. Amer. Statist. Assoc. 108 1044–1061.
    Mathematical Reviews (MathSciNet): MR3174683
    Zentralblatt MATH: 06224986
    Digital Object Identifier: doi:10.1080/01621459.2013.803972
  • Fan, J., Lv, J. and Qi, L. (2011). Sparse high-dimensional models in economics. Annual Review of Economics 3 291–317.
  • Friston, K. (2009). Causal modelling and brain connectivity in functional magnetic resonance imaging. PLoS Biol. 7 e1000033.
  • Giraitis, L., Koul, H. L. and Surgailis, D. (2012). Large Sample Inference for Long Memory Processes. Imperial College Press, London.
    Mathematical Reviews (MathSciNet): MR2977317
    Zentralblatt MATH: 1279.62016
  • Grenander, U. and Szegö, G. (1958). Toeplitz Forms and Their Applications. Univ. California Press, Berkeley.
    Mathematical Reviews (MathSciNet): MR94840
  • Hamilton, J. D. (1994). Time Series Analysis. Princeton Univ. Press, Princeton, NJ.
    Mathematical Reviews (MathSciNet): MR1278033
  • Han, F. and Liu, H. (2013). Transition matrix estimation in high dimensional time series. Proceedings of the 30th International Conference on Machine Learning (ICML-13) 28 172–180.
  • Kumar, P. R. and Varaiya, P. (1986). Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice Hall, New York.
  • Liebscher, E. (2005). Towards a unified approach for proving geometric ergodicity and mixing properties of nonlinear autoregressive processes. J. Time Series Anal. 26 669–689.
    Mathematical Reviews (MathSciNet): MR2188304
    Zentralblatt MATH: 1092.62091
    Digital Object Identifier: doi:10.1111/j.1467-9892.2005.00412.x
  • Loh, P.-L. and Wainwright, M. J. (2012). High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity. Ann. Statist. 40 1637–1664.
    Mathematical Reviews (MathSciNet): MR3015038
    Zentralblatt MATH: 1257.62063
    Digital Object Identifier: doi:10.1214/12-AOS1018
    Project Euclid: euclid.aos/1346850068
  • Loh, P.-L. and Wainwright, M. J. (2013). Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima. Preprint. Available at arXiv:1305.2436.
    arXiv: 1305.2436
    Mathematical Reviews (MathSciNet): MR3161456
    Zentralblatt MATH: 1288.62081
    Digital Object Identifier: doi:10.1214/13-AOS1162
    Project Euclid: euclid.aos/1388545677
  • Lütkepohl, H. (2005). New Introduction to Multiple Time Series Analysis. Springer, Berlin.
    Mathematical Reviews (MathSciNet): MR2172368
  • Michailidis, G. and d’Alché-Buc, F. (2013). Autoregressive models for gene regulatory network inference: Sparsity, stability and causality issues. Math. Biosci. 246 326–334.
    Mathematical Reviews (MathSciNet): MR3132054
    Zentralblatt MATH: 06269513
    Digital Object Identifier: doi:10.1016/j.mbs.2013.10.003
  • Negahban, S. and Wainwright, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069–1097.
    Mathematical Reviews (MathSciNet): MR2816348
    Zentralblatt MATH: 1216.62090
    Digital Object Identifier: doi:10.1214/10-AOS850
    Project Euclid: euclid.aos/1304947044
  • Negahban, S. N., Ravikumar, P., Wainwright, M. J. and Yu, B. (2012). A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers. Statist. Sci. 27 538–557.
    Mathematical Reviews (MathSciNet): MR3025133
    Digital Object Identifier: doi:10.1214/12-STS400
    Project Euclid: euclid.ss/1356098555
  • Parter, S. V. (1961). Extreme eigenvalues of Toeplitz forms and applications to elliptic difference equations. Trans. Amer. Math. Soc. 99 153–192.
    Mathematical Reviews (MathSciNet): MR120492
    Zentralblatt MATH: 0099.32403
    Digital Object Identifier: doi:10.1090/S0002-9947-1961-0120492-5
  • Priestley, M. B. (1981). Spectral Analysis and Time Series. Vol. 2. Multivariate Series, Prediction and Control, Probability and Mathematical Statistics. Academic Press, London.
    Mathematical Reviews (MathSciNet): MR628736
  • Raskutti, G., Wainwright, M. J. and Yu, B. (2010). Restricted eigenvalue properties for correlated Gaussian designs. J. Mach. Learn. Res. 11 2241–2259.
    Mathematical Reviews (MathSciNet): MR2719855
    Zentralblatt MATH: 1242.62071
  • Rosenblatt, M. (1985). Stationary Sequences and Random Fields. Springer, Boston, MA.
    Mathematical Reviews (MathSciNet): MR885090
    Zentralblatt MATH: 0597.62095
  • Rudelson, M. and Vershynin, R. (2013). Hanson–Wright inequality and sub-Gaussian concentration. Electron. Commun. Probab. 18 no. 82, 9.
    Mathematical Reviews (MathSciNet): MR3125258
    Zentralblatt MATH: 06346931
    Digital Object Identifier: doi:10.1214/ECP.v18-2865
  • Rudelson, M. and Zhou, S. (2013). Reconstruction from anisotropic random measurements. IEEE Trans. Inform. Theory 59 3434–3447.
    Mathematical Reviews (MathSciNet): MR3061256
    Digital Object Identifier: doi:10.1109/TIT.2013.2243201
  • Seth, A. K., Chorley, P. and Barnett, L. C. (2013). Granger causality analysis of fMRI BOLD signals is invariant to hemodynamic convolution but not downsampling. NeuroImage 65 540–555.
  • Shojaie, A. and Michailidis, G. (2010). Discovering graphical Granger causality using the truncating lasso penalty. Bioinformatics 26 i517–i523.
  • Sims, C. A. (1980). Macroeconomics and reality. Econometrica 48 1–48.
  • Smith, S. M. (2012). The future of FMRI connectivity. NeuroImage 62 1257–1266.
  • Song, S. and Bickel, P. J. (2011). Large vector auto regressions. Preprint. Available at arXiv:1106.3915v1.
    arXiv: 1106.3915v1
  • Stock, J. H. and Watson, M. W. (2005). Implications of dynamic factor models for VAR analysis. Working Paper No. 11467, National Bureau of Economic Research, Cambridge, MA.
  • van de Geer, S. A. and Bühlmann, P. (2009). On the conditions used to prove oracle results for the Lasso. Electron. J. Stat. 3 1360–1392.
    Mathematical Reviews (MathSciNet): MR2576316
    Zentralblatt MATH: 06166484
    Digital Object Identifier: doi:10.1214/09-EJS506
    Project Euclid: euclid.ejs/1260801227
  • van de Geer, S., Bühlmann, P. and Zhou, S. (2011). The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso). Electron. J. Stat. 5 688–749.
    Mathematical Reviews (MathSciNet): MR2820636
    Zentralblatt MATH: 1274.62471
    Digital Object Identifier: doi:10.1214/11-EJS624
    Project Euclid: euclid.ejs/1311600467
  • Vershynin, R. (2010). Introduction to the non-asymptotic analysis of random matrices. Preprint. Available at arXiv:1011.3027.
    arXiv: 1011.3027
    Mathematical Reviews (MathSciNet): MR2963170
    Digital Object Identifier: doi:10.1017/CBO9780511794308.006
  • Wu, W. B. (2005). Nonlinear system theory: Another look at dependence. Proc. Natl. Acad. Sci. USA 102 14150–14154 (electronic).
    Mathematical Reviews (MathSciNet): MR2172215
    Zentralblatt MATH: 1135.62075
    Digital Object Identifier: doi:10.1073/pnas.0506715102
  • Wu, W.-B. and Wu, Y. N. (2014). High-dimensional linear models with dependent observations. Preprint.
  • Xiao, H. and Wu, W. B. (2012). Covariance matrix estimation for stationary time series. Ann. Statist. 40 466–493.
  • Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894–942.
    Mathematical Reviews (MathSciNet): MR2604701
    Zentralblatt MATH: 1183.62120
    Digital Object Identifier: doi:10.1214/09-AOS729
    Project Euclid: euclid.aos/1266586618
  • Zhou, S. (2010). Thresholded Lasso for high dimensional variable selection and statistical estimation. Technical Report 511, Dept. Statistics, Univ. Michigan, Ann Arbor, MI. Available at arXiv:1002.1583.
    arXiv: 1002.1583

Supplemental materials

  • Supplement to “Regularized estimation in sparse high-dimensional time series models”. For the sake of brevity, we moved the appendices containing many of the technical proofs and detailed discussions to the supplementary document [Basu and Michailidis (2015)].
    Digital Object Identifier: doi:10.1214/15-AOS1315SUPP
    Supplemental PDF (336 KB)

The Institute of Mathematical Statistics

Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more