The Annals of Statistics

High-dimensional influence measure

Junlong Zhao, Chenlei Leng, Lexin Li, and Hansheng Wang

Full-text: Open access

Abstract

Influence diagnosis is important since presence of influential observations could lead to distorted analysis and misleading interpretations. For high-dimensional data, it is particularly so, as the increased dimensionality and complexity may amplify both the chance of an observation being influential, and its potential impact on the analysis. In this article, we propose a novel high-dimensional influence measure for regressions with the number of predictors far exceeding the sample size. Our proposal can be viewed as a high-dimensional counterpart to the classical Cook’s distance. However, whereas the Cook’s distance quantifies the individual observation’s influence on the least squares regression coefficient estimate, our new diagnosis measure captures the influence on the marginal correlations, which in turn exerts serious influence on downstream analysis including coefficient estimation, variable selection and screening. Moreover, we establish the asymptotic distribution of the proposed influence measure by letting the predictor dimension go to infinity. Availability of this asymptotic distribution leads to a principled rule to determine the critical value for influential observation detection. Both simulations and real data analysis demonstrate usefulness of the new influence diagnosis measure.

Article information

Source
Ann. Statist., Volume 41, Number 5 (2013), 2639-2667.

Dates
First available in Project Euclid: 19 November 2013

Permanent link to this document
https://projecteuclid.org/euclid.aos/1384871348

Digital Object Identifier
doi:10.1214/13-AOS1165

Mathematical Reviews number (MathSciNet)
MR3161440

Zentralblatt MATH identifier
1360.62411

Subjects
Primary: 62J20: Diagnostics
Secondary: 62E20: Asymptotic distribution theory

Keywords
Cook’s distance high-dimensional diagnosis influential observation LASSO marginal correlations variable screening

Citation

Zhao, Junlong; Leng, Chenlei; Li, Lexin; Wang, Hansheng. High-dimensional influence measure. Ann. Statist. 41 (2013), no. 5, 2639--2667. doi:10.1214/13-AOS1165. https://projecteuclid.org/euclid.aos/1384871348


Export citation

References

  • [1] Anderson, E. B. (1992). Diagnostics in categorical data analysis. J. R. Stat. Soc. Ser. B Stat. Methodol. 54 781–791.
  • [2] Banerjee, M. (1998). Cook’s distance in linear longitudinal models. Comm. Statist. Theory Methods 27 2973–2983.
  • [3] Banerjee, M. and Frees, E. W. (1997). Influence diagnostics for linear longitudinal models. J. Amer. Statist. Assoc. 92 999–1005.
  • [4] Belloni, A. and Chernozhukov, V. (2011). $\ell _{1}$-penalized quantile regression in high-dimensional sparse models. Ann. Statist. 39 82–130.
  • [5] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B Stat. Methodol. 57 289–300.
  • [6] Benjamini, Y. and Hochberg, Y. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. J. Educ. Behav. Stat. 25 60–83.
  • [7] Bickel, P. J. and Levina, E. (2008). Covariance regularization by thresholding. Ann. Statist. 36 2577–2604.
  • [8] Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 2313–2351.
  • [9] Chatterjee, S. and Hadi, A. S. (1988). Sensitivity Analysis in Linear Regression. Wiley, New York.
  • [10] Chiang, A. P., Beck, J. S., Yen, H. J., Tayeh, M. K., Scheetz, T. E., Swiderski, R., Nishimura, D., Braun, T. A., Kim, K. Y., Huang, J., Elbedour, K., Carmi, R., Slusarski, D. C., Casavant, T. L., Stone, E. M. and Sheffield, V. C. (2006). Homozygosity mapping with SNP arrays identifies a novel gene for Bardet–Biedl syndrome (BBS11). Proc. Natl. Acad. Sci. USA 103 6287–6292.
  • [11] Christensen, R., Pearson, L. M. and Johnson, W. (1992). Case-deletion diagnostics for mixed models. Technometrics 34 38–45.
  • [12] Cook, R. D. (1977). Detection of influential observation in linear regression. Technometrics 19 15–18.
  • [13] Cook, R. D. (1979). Influential observations in linear regression. J. Amer. Statist. Assoc. 74 169–174.
  • [14] Cook, R. D. and Weisberg, S. (1982). Residuals and Influence in Regression. Chapman & Hall, London.
  • [15] Critchley, F., Atkinson, R. A., Lu, G. and Biazi, E. (2001). Influence analysis based on the case sensitivity function. J. R. Stat. Soc. Ser. B Stat. Methodol. 63 307–323.
  • [16] Davison, A. C. and Tsai, C. L. (1992). Regression model diagnostics. Int. Stat. Rev. 55 337–353.
  • [17] Donoho, D. L. (2000). High-dimensional data analysis: The curses and blessings of dimensionality. Technical report, Stanford Univ.
  • [18] Draper, N. R. and Smith, H. (1998). Applied Regression Analysis, 3rd ed. Wiley, New York.
  • [19] Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist. 32 407–499.
  • [20] Efron, B., Tibshirani, R., Storey, J. D. and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment. J. Amer. Statist. Assoc. 96 1151–1160.
  • [21] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348–1360.
  • [22] Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space. J. R. Stat. Soc. Ser. B Stat. Methodol. 70 849–911.
  • [23] Fan, J. and Song, R. (2010). Sure independence screening in generalized linear models with NP-dimensionality. Ann. Statist. 38 3567–3604.
  • [24] Fu, W. J. (1998). Penalized regressions: The bridge versus the Lasso. J. Comput. Graph. Statist. 7 397–416.
  • [25] Fung, W.-K., Zhu, Z.-Y., Wei, B.-C. and He, X. (2002). Influence diagnostics and outlier tests for semiparametric mixed models. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 565–579.
  • [26] Huang, J., Horowitz, J. L. and Ma, S. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Statist. 36 587–613.
  • [27] Huang, J., Ma, S. and Zhang, C.-H. (2008). Adaptive Lasso for sparse high-dimensional regression models. Statist. Sinica 18 1603–1618.
  • [28] Johnstone, I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. Ann. Statist. 29 295–327.
  • [29] Pan, J.-X. and Fang, K.-T. (2002). Growth Curve Models and Statistical Diagnostics. Springer, New York.
  • [30] Preisser, J. S. and Qaqish, B. F. (1996). Deletion diagnostics for generalised estimating equations. Biometrika 83 551–562.
  • [31] Scheetz, T., Kim, K., Swiderski, R., Philp, A., Braun, T., Knudtson, K., Dorrance, A., DiBona, G., Huang, J., Casavant, T. et al. (2006). Regulation of gene expression in the mammalian eye and its relevance to eye disease. Proc. Natl. Acad. Sci. USA 103 14429–14434.
  • [32] Storey, J. D. (2002). A direct approach to false discovery rates. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 479–498.
  • [33] Thomas, W. and Cook, R. D. (1989). Assessing influence on regression coefficients in generalized linear models. Biometrika 76 741–749.
  • [34] Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267–288.
  • [35] Wang, H. (2009). Forward regression for ultra-high dimensional variable screening. J. Amer. Statist. Assoc. 104 1512–1524.
  • [36] Wang, H. and Leng, C. (2007). Unified Lasso estimation via least squares approximation. J. Amer. Statist. Assoc. 101 1418–1429.
  • [37] Wang, H., Li, G. and Jiang, G. (2007). Robust regression shrinkage and consistent variable selection through the LAD–Lasso. J. Bus. Econom. Statist. 25 347–355.
  • [38] Williams, D. A. (1987). Generalized linear model diagnostics using the deviance and single case deletions. J. R. Stat. Soc. Ser. C. Appl. Stat. 36 181–191.
  • [39] Xiang, L., Tse, S.-K. and Lee, A. H. (2002). Influence diagnostics for generalized linear mixed models: Applications to clustered data. Comput. Statist. Data Anal. 40 759–774.
  • [40] Zhang, H. H. and Lu, W. (2007). Adaptive Lasso for Cox’s proportional hazards model. Biometrika 94 691–703.
  • [41] Zhao, J., Leng, C., Li, L. and Wang, H. (2013). Supplement to “High-dimensional influence measure.” DOI:10.1214/13-AOS1165SUPP.
  • [42] Zhu, H., Ibrahim, J. G. and Cho, H. (2012). Perturbation and scaled Cook’s distance. Ann. Statist. 40 785–811.
  • [43] Zhu, H., Ibrahim, J. G., Lee, S. and Zhang, H. (2007). Perturbation selection and influence measures in local influence analysis. Ann. Statist. 35 2565–2588.
  • [44] Zhu, H., Lee, S.-Y., Wei, B.-C. and Zhou, J. (2001). Case-deletion measures for models with incomplete data. Biometrika 88 727–737.
  • [45] Zou, H. (2006). The adaptive LASSO and its oracle properties. J. Amer. Statist. Assoc. 101 1418–1429.

Supplemental materials