Open Access
April 2009 On surrogate loss functions and f-divergences
XuanLong Nguyen, Martin J. Wainwright, Michael I. Jordan
Ann. Statist. 37(2): 876-904 (April 2009). DOI: 10.1214/08-AOS595


The goal of binary classification is to estimate a discriminant function γ from observations of covariate vectors and corresponding binary labels. We consider an elaboration of this problem in which the covariates are not available directly but are transformed by a dimensionality-reducing quantizer Q. We present conditions on loss functions such that empirical risk minimization yields Bayes consistency when both the discriminant function and the quantizer are estimated. These conditions are stated in terms of a general correspondence between loss functions and a class of functionals known as Ali-Silvey or f-divergence functionals. Whereas this correspondence was established by Blackwell [Proc. 2nd Berkeley Symp. Probab. Statist. 1 (1951) 93–102. Univ. California Press, Berkeley] for the 0–1 loss, we extend the correspondence to the broader class of surrogate loss functions that play a key role in the general theory of Bayes consistency for binary classification. Our result makes it possible to pick out the (strict) subset of surrogate loss functions that yield Bayes consistency for joint estimation of the discriminant function and the quantizer.


Download Citation

XuanLong Nguyen. Martin J. Wainwright. Michael I. Jordan. "On surrogate loss functions and f-divergences." Ann. Statist. 37 (2) 876 - 904, April 2009.


Published: April 2009
First available in Project Euclid: 10 March 2009

zbMATH: 1162.62060
MathSciNet: MR2502654
Digital Object Identifier: 10.1214/08-AOS595

Primary: 62G10 , 62K05 , 68Q32

Keywords: Ali-Silvey divergences , Bayes consistency , Binary classification , discriminant analysis , f-divergences , nonparametric decentralized detection , quantizer design , statistical machine learning , surrogate losses

Rights: Copyright © 2009 Institute of Mathematical Statistics

Vol.37 • No. 2 • April 2009
Back to Top