• Bernoulli
  • Volume 15, Number 3 (2009), 799-828.

The Dantzig selector and sparsity oracle inequalities

Vladimir Koltchinskii

Full-text: Open access


Let $$ Y_j=f_{\ast}(X_j)+\xi_j,\qquad j=1,\dots, n, $$ where $X, X_1,\dots, X_n$ are i.i.d. random variables in a measurable space $(S,\mathcal{A})$ with distribution $\Pi$ and $\xi, \xi_1,\dots ,\xi_n$ are i.i.d. random variables with ${\mathbb E}\xi=0$ independent of $(X_1,\dots, X_n).$ Given a dictionary $h_1,\dots, h_{N}: S\mapsto{\mathbb R},$ let $ f_{\lambda}:=\sum_{j=1}^N \lambda_j h_j$, $ \lambda=(\lambda_1,\dots, \lambda_N)\in{\mathbb R}^N. $ Given $\varepsilon>0,$ define $$ \hat\Lambda_{\varepsilon}:=\Biggl\{\lambda\in{\mathbb R}^N: \max_{1\leq k\leq N} \Biggl|n^{-1}\sum_{j=1}^n \bigl(f_{\lambda}(X_j)-Y_j\bigr)h_k(X_j)\Biggr| \leq\varepsilon \Biggr\} $$ and $$\hat\lambda:=\hat\lambda^{\varepsilon}\in \operatorname{Argmin}_{\lambda\in\hat\Lambda_{\varepsilon}}\|\lambda\| _{\ell_1}. $$ In the case where $f_{\ast}:=f_{\lambda^{\ast}}, \lambda^{\ast}\in {\mathbb R}^N,$ Candes and Tao Ann. Statist. 35 (2007) 2313-2351] suggested using $\hat\lambda$ as an estimator of $\lambda^{\ast}.$ They called this estimator “the Dantzig selector”. We study the properties of $f_{\hat\lambda}$ as an estimator of $f_{\ast}$ for regression models with random design, extending some of the results of Candes and Tao (and providing alternative proofs of these results).

Article information

Bernoulli, Volume 15, Number 3 (2009), 799-828.

First available in Project Euclid: 28 August 2009

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Dantzig selector oracle inequalities regression sparsity


Koltchinskii, Vladimir. The Dantzig selector and sparsity oracle inequalities. Bernoulli 15 (2009), no. 3, 799--828. doi:10.3150/09-BEJ187.

Export citation


  • Bickel, P., Ritov, Y. and Tsybakov, A. (2009). Simultaneous analysis of LASSO and Dantzig selector. Ann. Statist. To appear.
  • Bobkov, S. and Houdré, C. (1997). Isoperimetric constants for product probability measures. Ann. Probab. 25 184–205.
  • Bunea, F., Tsybakov, A. and Wegkamp, M. (2007). Sparsity oracle inequalities for the LASSO. Electron. J. Statist. 1 169–194.
  • Candes, E., Romberg, J. and Tao, T. (2006). Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52 489–509.
  • Candes, E. and Tao, T. (2005). Decoding by linear programming. IEEE Trans. Inform. Theory 51 4203–4215.
  • Candes, E. and Tao, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313–2351.
  • de la Pena, V. and Giné, E. (1998). Decoupling: From Dependence to Independence. New York: Springer.
  • Donoho, D.L. (2006a). For most large underdetermined systems of linear equations the minimal 1-norm solution is also the sparsest solution. Commun. Pure Appl. Math. 59 797–829.
  • Donoho, D.L. (2006b). For most large underdetermined systems of equations the minimal 1-norm near-solution approximates the sparsest near-solution. Commun. Pure Appl. Math. 59 907–934.
  • Koltchinskii, V. (2005). Model selection and aggregation in sparse classification problems. In Oberwolfach Reports: Meeting on Statistical and Probabilistic Methods of Model Selection, October 2005. European Mathematical Society Publishing House.
  • Koltchinskii, V. (2009). Sparsity in penalized empirical risk minimization. Ann. Inst. H. Poincaré Probab. Statist. 45 7–57.
  • Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces. New York: Springer.
  • Mendelson, S., Pajor, A. and Tomczak-Jaegermann, N. (2007). Reconstruction and subgaussian operators in asymptotic geometric analysis. Geom. Funct. Anal. 17 1248–1282.
  • Rudelson, M. and Vershynin, R. (2005). Geometric approach to error correcting codes and reconstruction of signals. Int. Math. Res. Not. 64 4019–4041.
  • van de Geer, S. (2008). High-dimensional generalized linear models and the Lasso. Ann. Statist. 36 614–645.
  • van der Vaart, A. and Wellner, J. (1996). Weak Convergence and Empirical Processes. New York: Springer.