## Bernoulli

• Bernoulli
• Volume 15, Number 3 (2009), 799-828.

### The Dantzig selector and sparsity oracle inequalities

#### Abstract

Let $$Y_j=f_{\ast}(X_j)+\xi_j,\qquad j=1,\dots, n,$$ where $X, X_1,\dots, X_n$ are i.i.d. random variables in a measurable space $(S,\mathcal{A})$ with distribution $\Pi$ and $\xi, \xi_1,\dots ,\xi_n$ are i.i.d. random variables with ${\mathbb E}\xi=0$ independent of $(X_1,\dots, X_n).$ Given a dictionary $h_1,\dots, h_{N}: S\mapsto{\mathbb R},$ let $f_{\lambda}:=\sum_{j=1}^N \lambda_j h_j$, $\lambda=(\lambda_1,\dots, \lambda_N)\in{\mathbb R}^N.$ Given $\varepsilon>0,$ define $$\hat\Lambda_{\varepsilon}:=\Biggl\{\lambda\in{\mathbb R}^N: \max_{1\leq k\leq N} \Biggl|n^{-1}\sum_{j=1}^n \bigl(f_{\lambda}(X_j)-Y_j\bigr)h_k(X_j)\Biggr| \leq\varepsilon \Biggr\}$$ and $$\hat\lambda:=\hat\lambda^{\varepsilon}\in \operatorname{Argmin}_{\lambda\in\hat\Lambda_{\varepsilon}}\|\lambda\| _{\ell_1}.$$ In the case where $f_{\ast}:=f_{\lambda^{\ast}}, \lambda^{\ast}\in {\mathbb R}^N,$ Candes and Tao Ann. Statist. 35 (2007) 2313-2351] suggested using $\hat\lambda$ as an estimator of $\lambda^{\ast}.$ They called this estimator “the Dantzig selector”. We study the properties of $f_{\hat\lambda}$ as an estimator of $f_{\ast}$ for regression models with random design, extending some of the results of Candes and Tao (and providing alternative proofs of these results).

#### Article information

Source
Bernoulli, Volume 15, Number 3 (2009), 799-828.

Dates
First available in Project Euclid: 28 August 2009

https://projecteuclid.org/euclid.bj/1251463282

Digital Object Identifier
doi:10.3150/09-BEJ187

Mathematical Reviews number (MathSciNet)
MR2555200

Zentralblatt MATH identifier
05815956

#### Citation

Koltchinskii, Vladimir. The Dantzig selector and sparsity oracle inequalities. Bernoulli 15 (2009), no. 3, 799--828. doi:10.3150/09-BEJ187. https://projecteuclid.org/euclid.bj/1251463282

#### References

• Bickel, P., Ritov, Y. and Tsybakov, A. (2009). Simultaneous analysis of LASSO and Dantzig selector. Ann. Statist. To appear.
• Bobkov, S. and Houdré, C. (1997). Isoperimetric constants for product probability measures. Ann. Probab. 25 184–205.
• Bunea, F., Tsybakov, A. and Wegkamp, M. (2007). Sparsity oracle inequalities for the LASSO. Electron. J. Statist. 1 169–194.
• Candes, E., Romberg, J. and Tao, T. (2006). Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52 489–509.
• Candes, E. and Tao, T. (2005). Decoding by linear programming. IEEE Trans. Inform. Theory 51 4203–4215.
• Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313–2351.
• de la Pena, V. and Giné, E. (1998). Decoupling: From Dependence to Independence. New York: Springer.
• Donoho, D.L. (2006a). For most large underdetermined systems of linear equations the minimal 1-norm solution is also the sparsest solution. Commun. Pure Appl. Math. 59 797–829.
• Donoho, D.L. (2006b). For most large underdetermined systems of equations the minimal 1-norm near-solution approximates the sparsest near-solution. Commun. Pure Appl. Math. 59 907–934.
• Koltchinskii, V. (2005). Model selection and aggregation in sparse classification problems. In Oberwolfach Reports: Meeting on Statistical and Probabilistic Methods of Model Selection, October 2005. European Mathematical Society Publishing House.
• Koltchinskii, V. (2009). Sparsity in penalized empirical risk minimization. Ann. Inst. H. Poincaré Probab. Statist. 45 7–57.
• Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. New York: Springer.
• Mendelson, S., Pajor, A. and Tomczak-Jaegermann, N. (2007). Reconstruction and subgaussian operators in asymptotic geometric analysis. Geom. Funct. Anal. 17 1248–1282.
• Rudelson, M. and Vershynin, R. (2005). Geometric approach to error correcting codes and reconstruction of signals. Int. Math. Res. Not. 64 4019–4041.
• van de Geer, S. (2008). High-dimensional generalized linear models and the Lasso. Ann. Statist. 36 614–645.
• van der Vaart, A. and Wellner, J. (1996). Weak Convergence and Empirical Processes. New York: Springer.