## The Annals of Statistics

### Gaussian and bootstrap approximations for high-dimensional U-statistics and their applications

Xiaohui Chen

#### Abstract

This paper studies the Gaussian and bootstrap approximations for the probabilities of a nondegenerate U-statistic belonging to the hyperrectangles in $\mathbb{R}^{d}$ when the dimension $d$ is large. A two-step Gaussian approximation procedure that does not impose structural assumptions on the data distribution is proposed. Subject to mild moment conditions on the kernel, we establish the explicit rate of convergence uniformly in the class of all hyperrectangles in $\mathbb{R}^{d}$ that decays polynomially in sample size for a high-dimensional scaling limit, where the dimension can be much larger than the sample size. We also provide computable approximation methods for the quantiles of the maxima of centered U-statistics. Specifically, we provide a unified perspective for the empirical bootstrap, the randomly reweighted bootstrap and the Gaussian multiplier bootstrap with the jackknife estimator of covariance matrix as randomly reweighted quadratic forms and we establish their validity. We show that all three methods are inferentially first-order equivalent for high-dimensional U-statistics in the sense that they achieve the same uniform rate of convergence over all $d$-dimensional hyperrectangles. In particular, they are asymptotically valid when the dimension $d$ can be as large as $O(e^{n^{c}})$ for some constant $c\in(0,1/7)$.

The bootstrap methods are applied to statistical applications for high-dimensional non-Gaussian data including: (i) principled and data-dependent tuning parameter selection for regularized estimation of the covariance matrix and its related functionals; (ii) simultaneous inference for the covariance and rank correlation matrices. In particular, for the thresholded covariance matrix estimator with the bootstrap selected tuning parameter, we show that for a class of sub-Gaussian data, error bounds of the bootstrapped thresholded covariance matrix estimator can be much tighter than those of the minimax estimator with a universal threshold. In addition, we also show that the Gaussian-like convergence rates can be achieved for heavy-tailed data, which are less conservative than those obtained by the Bonferroni technique that ignores the dependency in the underlying data distribution.

#### Article information

Source
Ann. Statist., Volume 46, Number 2 (2018), 642-678.

Dates
Revised: February 2017
First available in Project Euclid: 3 April 2018

https://projecteuclid.org/euclid.aos/1522742432

Digital Object Identifier
doi:10.1214/17-AOS1563

Mathematical Reviews number (MathSciNet)
MR3782380

Zentralblatt MATH identifier
06870275

#### Citation

Chen, Xiaohui. Gaussian and bootstrap approximations for high-dimensional U-statistics and their applications. Ann. Statist. 46 (2018), no. 2, 642--678. doi:10.1214/17-AOS1563. https://projecteuclid.org/euclid.aos/1522742432

#### References

• [1] Adamczak, R. (2008). A tail inequality for suprema of unbounded empirical processes with applications to Markov chains. Electron. J. Probab. 13 1000–1034.
• [2] Arcones, M. A. and Giné, E. (1992). On the bootstrap of $U$ and $V$ statistics. Ann. Statist. 20 655–674.
• [3] Arcones, M. A. and Giné, E. (1993). Limit theorems for $U$-processes. Ann. Probab. 21 1494–1542.
• [4] Bentkus, V. (2003). On the dependence of the Berry–Esseen bound on dimension. J. Statist. Plann. Inference 113 385–402.
• [5] Bentkus, V., Götze, F. and van Zwet, W. R. (1997). An Edgeworth expansion for symmetric statistics. Ann. Statist. 25 851–896.
• [6] Bentkus, V. Y. (1985). Lower bounds for the rate of convergence in the central limit theorem in Banach spaces. Litovsk. Mat. Sb. 25 10–21.
• [7] Bickel, P. J. and Freedman, D. A. (1981). Some asymptotic theory for the bootstrap. Ann. Statist. 9 1196–1217.
• [8] Bickel, P. J., Götze, F. and van Zwet, W. R. (1986). The Edgeworth expansion for $U$-statistics of degree two. Ann. Statist. 14 1463–1484.
• [9] Bickel, P. J. and Levina, E. (2008). Covariance regularization by thresholding. Ann. Statist. 36 2577–2604.
• [10] Bickel, P. J. and Levina, E. (2008). Regularized estimation of large covariance matrices. Ann. Statist. 36 199–227.
• [11] Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer, Heidelberg.
• [12] Cai, T., Liu, W. and Luo, X. (2011). A constrained $\ell_{1}$ minimization approach to sparse precision matrix estimation. J. Amer. Statist. Assoc. 106 594–607.
• [13] Cai, T. T. and Zhou, H. H. (2012). Optimal rates of convergence for sparse covariance matrix estimation. Ann. Statist. 40 2389–2420.
• [14] Callaert, H. and Veraverbeke, N. (1981). The order of the normal approximation for a studentized $U$-statistic. Ann. Statist. 9 194–200.
• [15] Chang, J., Zhou, W., Zhou, W.-X. and Wang, L. (2017). Comparing large covariance matrices under weak conditions on the dependence structure and its application to gene clustering. Biometrics 73 31–41.
• [16] Chen, L. H. Y., Fang, X. and Shao, Q.-M. (2013). From Stein identities to moderate deviations. Ann. Probab. 41 262–293.
• [17] Chen, S. X., Zhang, L.-X. and Zhong, P.-S. (2010). Tests for high-dimensional covariance matrices. J. Amer. Statist. Assoc. 105 810–819.
• [18] Chen, X. (2016). Gaussian approximation for the sup-norm of high-dimensional matrix-variate U-statistics and its applications. Preprint. Available at arXiv:1602.00199.
• [19] Chen, X. (2018). Supplement to “Gaussian and bootstrap approximations for high-dimensional U-statistics and their applications.” DOI:10.1214/17-AOS1563SUPP.
• [20] Chen, X., Xu, M. and Wu, W. B. (2013). Covariance and precision matrix estimation for high-dimensional time series. Ann. Statist. 41 2994–3021.
• [21] Chen, X., Xu, M. and Wu, W. B. (2016). Regularized estimation of linear functionals of precision matrices for high-dimensional time series. IEEE Trans. Signal Process. 64 6459–6470.
• [22] Chernozhukov, V., Chetverikov, D. and Kato, K. (2013). Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors. Ann. Statist. 41 2786–2819.
• [23] Chernozhukov, V., Chetverikov, D. and Kato, K. (2015). Comparison and anti-concentration bounds for maxima of Gaussian random vectors. Probab. Theory Related Fields 162 47–70.
• [24] Chernozhukov, V., Chetverikov, D. and Kato, K. (2017). Central limit theorems and bootstrap in high dimensions. Ann. Probab. 45 2309–2352.
• [25] DasGupta, A., Lahiri, S. N. and Stoyanov, J. (2014). Sharp fixed $n$ bounds and asymptotic expansions for the mean and the median of a Gaussian sample maximum, and applications to the Donoho–Jin model. Stat. Methodol. 20 40–62.
• [26] Dehling, H. and Mikosch, T. (1994). Random quadratic forms and the bootstrap for $U$-statistics. J. Multivariate Anal. 51 392–413.
• [27] Dempster, A. P. (1972). Covariance selection. Biometrics 28 157–175.
• [28] de la Peña, V. c. H. and Giné, E. (1999). Decoupling: From Dependence to Independence, Randomly Stopped Processes. $U$-Statistics and Processes. Martingales and Beyond. Springer, New York.
• [29] Einmahl, U. and Li, D. (2008). Characterization of LIL behavior in Banach space. Trans. Amer. Math. Soc. 360 6677–6693.
• [30] El Karoui, N. (2008). Operator norm consistent estimation of large-dimensional sparse covariance matrices. Ann. Statist. 36 2717–2756.
• [31] Fan, J., Liao, Y. and Mincheva, M. (2011). High-dimensional covariance matrix estimation in approximate factor models. Ann. Statist. 39 3320–3356.
• [32] Giné, E., Latała, R. and Zinn, J. (2000). Exponential and moment inequalities for $U$-statistics. In High Dimensional Probability, II (Seattle, WA, 1999). Progress in Probability 47 13–38. Birkhäuser, Boston, MA.
• [33] Götze, F. (1987). Approximations for multivariate $U$-statistics. J. Multivariate Anal. 22 212–229.
• [34] Gregory, G. G. (1977). Large sample theory for $U$-statistics and tests of fit. Ann. Statist. 5 110–123.
• [35] Hoeffding, W. (1948). A class of statistics with asymptotically normal distribution. Ann. Math. Stat. 19 293–325.
• [36] Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. J. Amer. Statist. Assoc. 58 13–30.
• [37] Houdré, C. and Reynaud-Bouret, P. (2003). Exponential inequalities, with constants, for U-statistics of order two. In Stochastic Inequalities and Applications. Progress in Probability 56 55–69. Birkhäuser, Basel.
• [38] Hsing, T. and Wu, W. B. (2004). On weighted $U$-statistics for stationary processes. Ann. Probab. 32 1600–1631.
• [39] Hušková, M. and Janssen, P. (1993). Consistency of the generalized bootstrap for degenerate $U$-statistics. Ann. Statist. 21 1811–1823.
• [40] Hušková, M. and Janssen, P. (1993). Generalized bootstrap for studentized $U$-statistics: A rank statistic approach. Statist. Probab. Lett. 16 225–233.
• [41] Janssen, P. (1994). Weighted bootstrapping of $U$-statistics. J. Statist. Plann. Inference 38 31–41.
• [42] Klein, T. and Rio, E. (2005). Concentration around the mean for maxima of empirical processes. Ann. Probab. 33 1060–1077.
• [43] Lam, C. and Fan, J. (2009). Sparsistency and rates of convergence in large covariance matrix estimation. Ann. Statist. 37 4254–4278.
• [44] Lam, C. and Yao, Q. (2012). Factor modeling for high-dimensional time series: Inference for the number of factors. Ann. Statist. 40 694–726.
• [45] Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces: Isoperimetry and Processes. Ergebnisse der Mathematik und Ihrer Grenzgebiete (3) [Results in Mathematics and Related Areas (3)] 23. Springer, Berlin.
• [46] Lehmann, E. L. (1999). Elements of Large-Sample Theory. Springer, New York.
• [47] Lo, A. Y. (1987). A large sample study of the Bayesian bootstrap. Ann. Statist. 15 360–375.
• [48] Mai, Q., Zou, H. and Yuan, M. (2012). A direct approach to sparse discriminant analysis in ultra-high dimensions. Biometrika 99 29–42.
• [49] Mason, D. M. and Newton, M. A. (1992). A rank statistics approach to the consistency of a general bootstrap. Ann. Statist. 20 1611–1624.
• [50] Massart, P. (2000). About the constants in Talagrand’s concentration inequalities for empirical processes. Ann. Probab. 28 863–884.
• [51] Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the lasso. Ann. Statist. 34 1436–1462.
• [52] Muirhead, R. J. (1982). Aspects of Multivariate Statistical Theory. Wiley, New York.
• [53] Nagaev, S. V. (1979). Large deviations of sums of independent random variables. Ann. Probab. 7 745–789.
• [54] Peng, J., Wang, P., Zhou, N. and Zhu, J. (2009). Partial correlation estimation by joint sparse regression models. J. Amer. Statist. Assoc. 104 735–746.
• [55] Petrov, V. V. (1975). Sums of Independent Random Variables. Springer, New York.
• [56] Portnoy, S. (1986). On the central limit theorem in $\mathbf{R}^{p}$ when $p\to\infty$. Probab. Theory Related Fields 73 571–583.
• [57] Præstgaard, J. and Wellner, J. A. (1993). Exchangeably weighted bootstraps of the general empirical process. Ann. Probab. 21 2053–2086.
• [58] Rothman, A. J., Bickel, P. J., Levina, E. and Zhu, J. (2008). Sparse permutation invariant covariance estimation. Electron. J. Stat. 2 494–515.
• [59] Rubin, D. B. (1981). The Bayesian bootstrap. Ann. Statist. 9 130–134.
• [60] Serfling, R. J. (1980). Approximation Theorems of Mathematical Statistics. Wiley, New York.
• [61] Shao, Q.-M. and Wang, Q. (2013). Self-normalized limit theorems: A survey. Probab. Surv. 10 69–93.
• [62] Talagrand, M. (1996). New concentration inequalities in product spaces. Invent. Math. 126 505–563.
• [63] van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York.
• [64] Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing 210–268. Cambridge Univ. Press, Cambridge.
• [65] Wang, Q. and Jing, B.-Y. (2004). Weighted bootstrap for $U$-statistics. J. Multivariate Anal. 91 177–198.
• [66] Yuan, M. (2010). High dimensional inverse covariance matrix estimation via linear programming. J. Mach. Learn. Res. 11 2261–2286.
• [67] Yuan, M. and Lin, Y. (2007). Model selection and estimation in the Gaussian graphical model. Biometrika 94 19–35.
• [68] Zhang, C.-H. (1999). Sub-Bernoulli functions, moment inequalities and strong laws for nonnegative and symmetrized $U$-statistics. Ann. Probab. 27 432–453.
• [69] Zhang, D. and Wu, W. B. (2017). Gaussian approximation for high-dimensional time series. Ann. Statist. 45 1895–1919.
• [70] Zhang, X. and Cheng, G. (2014). Bootstrapping high dimensional time series. Available at arXiv:1406.1037.

#### Supplemental materials

• Supplement to “Gaussian and bootstrap approximations for high- dimensional U-statistics and their applications”. This supplemental file contains additional proofs, technical lemmas and simulation results.