Electronic Journal of Statistics

Performance bounds for parameter estimates of high-dimensional linear models with correlated errors

Wei-Biao Wu and Ying Nian Wu

Full-text: Open access

Enhanced PDF (725 KB)

Abstract

This paper develops a systematic theory for high-dimensional linear models with dependent errors and/or dependent covariates. To study properties of estimates of the regression parameters, we adopt the framework of functional dependence measures ([43]). For the covariates two schemes are addressed: the random design and the deterministic design. For the former we apply the constrained $\ell_{1}$ minimization approach, while for the latter the Lasso estimation procedure is used. We provide a detailed characterization on how the error rates of the estimates depend on the moment conditions that control the tail behaviors, the dependencies of the underlying processes that generate the errors and the covariates, the dimension and the sample size. Our theory substantially extends earlier ones by allowing dependent and/or heavy-tailed errors and the covariates. As our main tools, we derive exponential tail probability inequalities for dependent sub-Gaussian errors and Nagaev-type inequalities for dependent non-sub-Gaussian errors that arise from linear or non-linear processes.

Article information

Source
Electron. J. Statist., Volume 10, Number 1 (2016), 352-379.

Dates
Received: March 2015
First available in Project Euclid: 17 February 2016

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1455715966

Digital Object Identifier
doi:10.1214/16-EJS1108

Mathematical Reviews number (MathSciNet)
MR3466186

Zentralblatt MATH identifier
1333.62172

Keywords
Consistency dependence-adjusted norm exponential inequality functional and predictive dependence measures high-dimensional time series impulse response function Nagaev inequality predictive persistence support recovery

Citation

Wu, Wei-Biao; Wu, Ying Nian. Performance bounds for parameter estimates of high-dimensional linear models with correlated errors. Electron. J. Statist. 10 (2016), no. 1, 352--379. doi:10.1214/16-EJS1108. https://projecteuclid.org/euclid.ejs/1455715966


Export citation

References

  • [1] Azuma, K. (1967) Weighted sums of certain dependent random variables., Tohoku Mathematical Journal, 19, 357–367.
    Mathematical Reviews (MathSciNet): MR221571
    Digital Object Identifier: doi:10.2748/tmj/1178243286
    Project Euclid: euclid.tmj/1178243286
  • [2] Barnett, W. A., Chae, U. and Keating, J. (2012) Forecast design in monetary capital stock measurement., Global Journal of Economics, 1, 1250005.
  • [3] Basu, S. and Michailidis, G. (2015) Regularized estimation in sparse high-dimensional time series models., Annals of Statistics, 43, 1535–1567.
    Mathematical Reviews (MathSciNet): MR3357870
    Digital Object Identifier: doi:10.1214/15-AOS1315
    Project Euclid: euclid.aos/1434546214
  • [4] Bickel, P., Ritov, Y. and Tsybakov, A. (2009) Simultaneous analysis of Lasso and Dantzig selector., Annals of Statistics, 37, 1705–1732.
    Mathematical Reviews (MathSciNet): MR2533469
    Digital Object Identifier: doi:10.1214/08-AOS620
    Project Euclid: euclid.aos/1245332830
  • [5] Bühlmann, P. and van de Geer, S. (2011) Statistics for High-Dimensional Data: Methods, Theory and Applications., Springer.
    Mathematical Reviews (MathSciNet): MR2807761
  • [6] Bunea, F., Tsybakov, A., and Wegkamp, M. (2007) Sparsity oracle inequalities for the lasso., Electronic Journal of Statistics, 1, 169–194.
    Mathematical Reviews (MathSciNet): MR2312149
    Digital Object Identifier: doi:10.1214/07-EJS008
    Project Euclid: euclid.ejs/1179759718
  • [7] Burkholder, D. L. (1973) Distribution function inequalities for martingales., Annals of Probability, 1, 19–42.
    Mathematical Reviews (MathSciNet): MR365692
    Digital Object Identifier: doi:10.1214/aop/1176997023
  • [8] Cai, T., Liu, W. and Luo, X. (2011) A constrained $\ell_1$ minimization approach to sparse precision matrix estimation., Journal of American Statistical Association, 106, 594–607.
    Mathematical Reviews (MathSciNet): MR2847973
    Digital Object Identifier: doi:10.1198/jasa.2011.tm10155
  • [9] Candes E. and Tao, T. (2007) The Dantzig selector: statistical estimation when p is much larger than n (with discussion)., Annals of Statistics, 35, 2313–2404.
    Mathematical Reviews (MathSciNet): MR2382644
    Digital Object Identifier: doi:10.1214/009053606000001523
    Project Euclid: euclid.aos/1201012958
  • [10] Chernozhukov, V., Chetverikov, D. and Kato, K. (2014) Testing many moment inequalities., http://arxiv.org/abs/1312.7614
  • [11] Chow, Y. S. and Teicher, H. (1988)., Probability Theory, 2nd ed. Springer, New York.
    Mathematical Reviews (MathSciNet): MR953964
  • [12] Davis, R. A., Zang, P., and Zheng, T. (2015) Sparse Vector Autoregressive Modeling., Journal of Computational and Graphical Statistics.
  • [13] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties., Journal of American Statistical Association, 96, 1348–1360.
    Mathematical Reviews (MathSciNet): MR1946581
    Digital Object Identifier: doi:10.1198/016214501753382273
  • [14] Friedman, J. H., Hastie, T., and Tibshirani, R. (2010) Regularization paths for generalized linear models via coordinate descent., Journal of Statistical Software, 33, 1–22.
  • [15] Gupta, S. (2012) A note on the asymptotic distribution of LASSO estimator for correlated data., Sankhya A, 74, 10–28.
    Mathematical Reviews (MathSciNet): MR3010290
    Digital Object Identifier: doi:10.1007/s13171-012-0006-8
  • [16] Han F. and Liu, H. (2013) A direct estimation of high dimensional stationary vector autoregressions., http://arxiv.org/abs/1307.0293
    Mathematical Reviews (MathSciNet): MR3450535
  • [17] Hsiao, C. (1979) Autoregressive modeling of canadian money and income data., Journal of the American Statistical Association, 74, 553–560.
  • [18] Kaul, A. (2014) Lasso with long memory regression errors., Journal of Statistical Planning and Inference, 153, 11–26.
    Mathematical Reviews (MathSciNet): MR3229019
    Digital Object Identifier: doi:10.1016/j.jspi.2014.05.003
  • [19] Kock, A. and Callot, L. (2012) Oracle inequalities for high dimensional vector autoregressions, Research Paper 12, CREATES, Aarhus, University.
  • [20] Krolzig, H. M. (2003) General-to-specific model selection procedures for structural vector autoregressions., Oxford Bulletin of Economics and Statistics, 65, 769–801.
  • [21] Lesigne, E. and Volný, D. Large deviations for martingales., Stochastic Processes and their Applications, 96, 143–159, 2001.
    Mathematical Reviews (MathSciNet): MR1856684
    Digital Object Identifier: doi:10.1016/S0304-4149(01)00112-0
  • [22] Liu, H. and Wang, L. (2012) TIGER: A Tuning-Insensitive Approach for Optimally Estimating Gaussian Graphical Models., http://arxiv.org/ abs/1209.2437
  • [23] Liu, W., Xiao, H., and Wu, W. B. (2013) Probability and moment inequalities under dependence., Statistica Sinica, 23, 1257–1272. doi:10.5705/ss.2011.287.
    Mathematical Reviews (MathSciNet): MR3114713
  • [24] Loh, P.-L. (2015) Statistical consistency and asymptotic normality for high-dimensional robust $M$-estimators., http://arxiv.org/abs/1501.00312
  • [25] Loh, P.-L. and Wainwright, M. J. (2012), High-dimensional regression with noisy and missing data: provable guarantees with nonconvexity., Ann. Stat., 40, 1637–1664.
    Mathematical Reviews (MathSciNet): MR3015038
    Digital Object Identifier: doi:10.1214/12-AOS1018
    Project Euclid: euclid.aos/1346850068
  • [26] Lütkepohl, H. (2005), New introduction to multiple time series analysis, Springer.
    Mathematical Reviews (MathSciNet): MR2172368
  • [27] Meinshausen, N. and Bühlmann, P. (2006) High dimensional graphs and variable selection with the Lasso., Annals of Statistics, 34, 1436–1462.
    Mathematical Reviews (MathSciNet): MR2278363
    Digital Object Identifier: doi:10.1214/009053606000000281
    Project Euclid: euclid.aos/1152540754
  • [28] Meinshausen, N. and Yu, B. (2009) Lasso–type recovery of sparse representations for high-dimensional data., Annals of Statistics, 37, 246–270.
    Mathematical Reviews (MathSciNet): MR2488351
    Digital Object Identifier: doi:10.1214/07-AOS582
    Project Euclid: euclid.aos/1232115934
  • [29] Merlevede, F., Peligrad, M., and Rio, E. (2011) A Bernstein type inequality and moderate deviations for weakly dependent sequences., Probability Theory and Related Fields, 141, 435–474.
    Mathematical Reviews (MathSciNet): MR2851689
    Digital Object Identifier: doi:10.1007/s00440-010-0304-9
  • [30] Nagaev, S. V. (1979). Large deviations of sums of independent random variables., Annals of Probability, 7, 745–789.
    Mathematical Reviews (MathSciNet): MR542129
    Digital Object Identifier: doi:10.1214/aop/1176994938
    Project Euclid: euclid.aop/1176994938
  • [31] Nardi, Y. and Rinaldo, A. (2011) Autoregressive process modeling via the lasso procedure., Journal of Multivariate Analysis, 102, 528–549.
    Mathematical Reviews (MathSciNet): MR2755014
    Digital Object Identifier: doi:10.1016/j.jmva.2010.10.012
  • [32] Peligrad, M., Sang, H., Zhong, Y., and Wu, W. B. (2014) Exact moderate and large deviations for linear processes, Statistica Sinica, 24, 957–1969.
    Mathematical Reviews (MathSciNet): MR3235407
  • [33] Ravikumar, P., Wainwright, M. J., Raskutti, G., and Yu, B. (2011) High-dimensional covariance estimation by minimizing l1-penalized log-determinant divergence., Electronic Journal of Statistics, 5, 935–980.
    Mathematical Reviews (MathSciNet): MR2836766
    Digital Object Identifier: doi:10.1214/11-EJS631
    Project Euclid: euclid.ejs/1316092865
  • [34] Sims, C. (1980) Macroeconomics and reality., Econometrica, 48, 1–48.
    Mathematical Reviews (MathSciNet): MR559536
    Digital Object Identifier: doi:10.1080/00031305.1980.10482711
  • [35] Rosenthal, H. P. (1970) On the subspaces of $L_p$ ($p>2$) spanned by sequences of independent random variables., Israel Journal of Mathematics, 8, 273–303.
    Mathematical Reviews (MathSciNet): MR271721
    Digital Object Identifier: doi:10.1007/BF02771562
  • [36] Song, S. and Bickel, P. J. (2011) Large vector autoregressions., http://arxiv.org/abs/1106.3915
  • [37] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso., Journal of the Royal Statistical Society: B, 58, 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
  • [38] Tong, H. (1990), Nonlinear time series analysis: A dynamic approach, Oxford University Press, Oxford.
    Mathematical Reviews (MathSciNet): MR1079320
  • [39] Vershynin, R. (2012), Introduction to the non-asymptotic analysis of random matrices. In Chapter 5 of: Compressed Sensing, Theory and Applications. pp. 210–268. Edited by Y. Eldar and G. Kutyniok. Cambridge University Press.
    Mathematical Reviews (MathSciNet): MR2963170
    Digital Object Identifier: doi:10.1017/CBO9780511794308.006
  • [40] Wainwright, M. J. (2009). Sharp thresholds for noisy and high-dimensional recovery of sparsity using constrained quadratic programming (Lasso)., IEEE Transactions on Information Theory, 55, 2183–2202.
    Mathematical Reviews (MathSciNet): MR2729873
    Digital Object Identifier: doi:10.1109/TIT.2009.2016018
  • [41] Wang, H., Li, G., and Tsai, C.-L. (2007) Regression coefficient and autoregressive order shrinkage and selection via the lasso., Journal of Royal Statistical Society, B, 69, 63–78.
    Mathematical Reviews (MathSciNet): MR2301500
  • [42] Wiener, N. (1958), Nonlinear Problems in Random Theory. MIT Press.
    Mathematical Reviews (MathSciNet): MR100912
  • [43] Wu, W. B. (2005) Nonlinear system theory: another look at dependence., Proceedings of National Academy of Science, 102, 14150–14154.
    Mathematical Reviews (MathSciNet): MR2172215
    Digital Object Identifier: doi:10.1073/pnas.0506715102
  • [44] Wu, W. B. (2011) Asymptotic theory for stationary processes, Statistics and Its Interface, 4, 207–226.
    Mathematical Reviews (MathSciNet): MR2812816
    Digital Object Identifier: doi:10.4310/SII.2011.v4.n2.a15
  • [45] Wu, W. B. and Shao, X. (2004) Limit theorems for iterated random functions., Journal of Applied Probability, 41, 425–436.
    Mathematical Reviews (MathSciNet): MR2052582
    Digital Object Identifier: doi:10.1017/S0021900200014406
    Project Euclid: euclid.jap/1082999076
  • [46] Zhao, P. and Yu, B. (2006) On model selection consistency of Lasso., Journal of Machine Learning Research, 7, 2541–2567.
    Mathematical Reviews (MathSciNet): MR2274449
  • [47] Zhang, C. H. (2010) Nearly unbiased variable selection under minimax concave penalty., Ann. Statist., 38, 894–942.
    Mathematical Reviews (MathSciNet): MR2604701
    Digital Object Identifier: doi:10.1214/09-AOS729
    Project Euclid: euclid.aos/1266586618
  • [48] Zhang, C. H. and Huang, J. (2008) The sparsity and bias of the Lasso selection in high dimensional linear regression., Annals of Statistics, 36, 1567–1594.
    Mathematical Reviews (MathSciNet): MR2435448
    Digital Object Identifier: doi:10.1214/07-AOS520
    Project Euclid: euclid.aos/1216237292
  • [49] Zhang, T. (2009) Some sharp performance bounds for least squares regression with l1 regularization., Annals of Statistics, 37, 2109–2144.
    Mathematical Reviews (MathSciNet): MR2543687
    Digital Object Identifier: doi:10.1214/08-AOS659
    Project Euclid: euclid.aos/1247663750

The Institute of Mathematical Statistics and the Bernoulli Society

Electronic Journal of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more