The Dantzig selector and sparsity oracle inequalities

Abstract

Let

Y_j=f_*(X_j)+ξ_j,  j=1, …, n,

where X, X₁, …, X_n are i.i.d. random variables in a measurable space with distribution Π and ξ, ξ₁, …, ξ_n are i.i.d. random variables with independent of (X₁, …, X_n). Given a dictionary h₁, …, h_N: S↦ℝ, let f_λ:=∑_j=1^Nλ_jh_j, λ=(λ₁, …, λ_N)∈ℝ^N. Given ɛ>0, define

̂Λ_ɛ:={λ∈ℝ^N: max_1≤k≤N|n⁻¹∑_j=1ⁿ(f_λ(X_j)−Y_j)h_k(X_j)|≤ɛ}

and

̂λ:=̂λ^ɛ∈Argmin_λ∈̂Λɛ‖λ‖_ℓ1.

In the case where f_*:=f_λ^*, λ^*∈ℝ^N, Candes and Tao [Ann. Statist. 35 (2007) 2313–2351] suggested using ̂λ as an estimator of λ^*. They called this estimator “the Dantzig selector”. We study the properties of f_̂λ as an estimator of f_* for regression models with random design, extending some of the results of Candes and Tao (and providing alternative proofs of these results).