Annals of Statistics

Inference in adaptive regression via the Kac–Rice formula

Jonathan E. Taylor, Joshua R. Loftus, and Ryan J. Tibshirani

Full-text: Open access

Enhanced PDF (504 KB)

Abstract

We derive an exact $p$-value for testing a global null hypothesis in a general adaptive regression setting. Our approach uses the Kac–Rice formula [as described in Random Fields and Geometry (2007) Springer, New York] applied to the problem of maximizing a Gaussian process. The resulting test statistic has a known distribution in finite samples, assuming Gaussian errors. We examine this test statistic in the case of the lasso, group lasso, principal components and matrix completion problems. For the lasso problem, our test relates closely to the recently proposed covariance test of Lockhart et al. [Ann. Statist. (2004) 42 413–468].

In a few specific settings, our proposed tests will be less powerful than other previously known (and well-established) tests. However, it should be noted that the real strength of our proposal here is its generality. We provide a framework for constructing valid tests across a wide class of regularized regression problems, and as far as we can tell, such a unified view was not possible before this work.

Article information

Source
Ann. Statist., Volume 44, Number 2 (2016), 743-770.

Dates
Received: February 2015
Revised: September 2015
First available in Project Euclid: 17 March 2016

Permanent link to this document
https://projecteuclid.org/euclid.aos/1458245734

Digital Object Identifier
doi:10.1214/15-AOS1386

Mathematical Reviews number (MathSciNet)
MR3476616

Zentralblatt MATH identifier
1337.62304

Subjects
Primary: 62M40: Random fields; image analysis
Secondary: 62J05: Linear regression

Keywords
Gaussian processes convex analysis regularized regression

Citation

Taylor, Jonathan E.; Loftus, Joshua R.; Tibshirani, Ryan J. Inference in adaptive regression via the Kac–Rice formula. Ann. Statist. 44 (2016), no. 2, 743--770. doi:10.1214/15-AOS1386. https://projecteuclid.org/euclid.aos/1458245734


Export citation

References

  • [1] Adler, R. J. and Taylor, J. E. (2007). Random Fields and Geometry. Springer, New York.
    Mathematical Reviews (MathSciNet): MR2319516
    Zentralblatt MATH: 1149.60003
  • [2] Arias-Castro, E., Candès, E. J. and Plan, Y. (2011). Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism. Ann. Statist. 39 2533–2556.
    Mathematical Reviews (MathSciNet): MR2906877
    Zentralblatt MATH: 1231.62136
    Digital Object Identifier: doi:10.1214/11-AOS910
    Project Euclid: euclid.aos/1322663467
  • [3] Azaïs, J.-M. and Wschebor, M. (2008). A general expression for the distribution of the maximum of a Gaussian field and the approximation of the tail. Stochastic Process. Appl. 118 1190–1218.
    Mathematical Reviews (MathSciNet): MR2428714
    Digital Object Identifier: doi:10.1016/j.spa.2007.07.016
  • [4] Brillinger, D. R. (1972). On the number of solutions of systems of random equations. Ann. Math. Statist. 43 534–540.
    Mathematical Reviews (MathSciNet): MR300329
    Zentralblatt MATH: 0238.60040
    Digital Object Identifier: doi:10.1214/aoms/1177692634
    Project Euclid: euclid.aoms/1177692634
  • [5] Candès, E. J. and Recht, B. (2009). Exact matrix completion via convex optimization. Found. Comput. Math. 9 717–772.
    Mathematical Reviews (MathSciNet): MR2565240
    Digital Object Identifier: doi:10.1007/s10208-009-9045-5
  • [6] Choi, Y., Taylor, J. and Tibshirani, R. (2014). Selecting the number of principal components: Estimation of the true rank of a noisy matrix. Preprint. Available at arXiv:1410.8260.
    arXiv: 1410.8260
  • [7] Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist. 32 407–499.
    Mathematical Reviews (MathSciNet): MR2060166
    Zentralblatt MATH: 1091.62054
    Digital Object Identifier: doi:10.1214/009053604000000067
    Project Euclid: euclid.aos/1083178935
  • [8] Federer, H. (1959). Curvature measures. Trans. Amer. Math. Soc. 93 418–491.
    Mathematical Reviews (MathSciNet): MR110078
    Zentralblatt MATH: 0089.38402
    Digital Object Identifier: doi:10.1090/S0002-9947-1959-0110078-1
  • [9] Fithian, W., Sun, D. and Taylor, J. (2014). Optimal inference after model selection. Preprint. Available at arXiv:1410.2597.
    arXiv: 1410.2597
  • [10] Lockhart, R., Taylor, J., Tibshirani, R. J. and Tibshirani, R. (2014). A significance test for the lasso. Ann. Statist. 42 413–468.
    Mathematical Reviews (MathSciNet): MR3210970
    Digital Object Identifier: doi:10.1214/13-AOS1175
    Project Euclid: euclid.aos/1400592161
  • [11] Loftus, J. and Taylor, J. (2014). A significance test for forward stepwise model selection. Preprint. Available at arXiv:1405.3920.
    arXiv: 1405.3920
  • [12] Mazumder, R., Hastie, T. and Tibshirani, R. (2010). Spectral regularization algorithms for learning large incomplete matrices. J. Mach. Learn. Res. 11 2287–2322.
    Mathematical Reviews (MathSciNet): MR2719857
    Zentralblatt MATH: 1242.68237
  • [13] Mukherjee, A., Chen, K., Wang, N. and Zhu, J. (2015). On the degrees of freedom of reduced-rank estimators in multivariate regression. Biometrika 102 457–477.
    Mathematical Reviews (MathSciNet): MR3371016
    Digital Object Identifier: doi:10.1093/biomet/asu067
  • [14] Takemura, A. and Kuriki, S. (2002). On the equivalence of the tube and Euler characteristic methods for the distribution of the maximum of Gaussian fields over piecewise smooth domains. Ann. Appl. Probab. 12 768–796.
    Mathematical Reviews (MathSciNet): MR1910648
    Zentralblatt MATH: 1016.60042
    Digital Object Identifier: doi:10.1214/aoap/1026915624
    Project Euclid: euclid.aoap/1026915624
  • [15] Taylor, J. (2013). The geometry of least squares in the 21st century. Bernoulli 19 1449–1464.
    Mathematical Reviews (MathSciNet): MR3102558
    Digital Object Identifier: doi:10.3150/12-BEJSP15
    Project Euclid: euclid.bj/1377612860
  • [16] Taylor, J., Lockhart, R., Tibshirani, R. J. and Tibshirani, R. (2014). Exact post-selection inference for sequential regression procedures. Preprint. Available at arXiv:1401.3889.
    arXiv: 1401.3889
  • [17] Taylor, J., Loftus, J. and Tibshirani, R. J. (2015). Supplement to “Inference in adaptive regression via the Kac–Rice formula.” DOI:10.1214/15-AOS1386SUPP.
  • [18] Taylor, J., Takemura, A. and Adler, R. J. (2005). Validity of the expected Euler characteristic heuristic. Ann. Probab. 33 1362–1396.
    Mathematical Reviews (MathSciNet): MR2150192
    Zentralblatt MATH: 1083.60031
    Digital Object Identifier: doi:10.1214/009117905000000099
    Project Euclid: euclid.aop/1120224584
  • [19] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288.
    Mathematical Reviews (MathSciNet): MR1379242
  • [20] Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005). Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. Ser. B. Stat. Methodol. 67 91–108.
    Mathematical Reviews (MathSciNet): MR2136641
    Zentralblatt MATH: 1060.62049
    Digital Object Identifier: doi:10.1111/j.1467-9868.2005.00490.x
  • [21] Tibshirani, R. J. (2013). The lasso problem and uniqueness. Electron. J. Stat. 7 1456–1490.
    Mathematical Reviews (MathSciNet): MR3066375
    Zentralblatt MATH: 06167979
    Digital Object Identifier: doi:10.1214/13-EJS815
    Project Euclid: euclid.ejs/1369148600
  • [22] Tibshirani, R. J. and Taylor, J. (2011). The solution path of the generalized lasso. Ann. Statist. 39 1335–1371.
    Mathematical Reviews (MathSciNet): MR2850205
    Zentralblatt MATH: 1234.62107
    Digital Object Identifier: doi:10.1214/11-AOS878
    Project Euclid: euclid.aos/1304514656
  • [23] Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B. Stat. Methodol. 68 49–67.
    Mathematical Reviews (MathSciNet): MR2212574
    Zentralblatt MATH: 1141.62030
    Digital Object Identifier: doi:10.1111/j.1467-9868.2005.00532.x

Supplemental materials

The Institute of Mathematical Statistics

Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more