Translator Disclaimer
August 2021 Universal Bayes consistency in metric spaces
Steve Hanneke, Aryeh Kontorovich, Sivan Sabato, Roi Weiss
Author Affiliations +
Ann. Statist. 49(4): 2129-2150 (August 2021). DOI: 10.1214/20-AOS2029


We extend a recently proposed 1-nearest-neighbor based multiclass learning algorithm and prove that our modification is universally strongly Bayes consistent in all metric spaces admitting any such learner, making it an “optimistically universal” Bayes-consistent learner. This is the first learning algorithm known to enjoy this property; by comparison, the k-NN classifier and its variants are not generally universally Bayes consistent, except under additional structural assumptions, such as an inner product, a norm, finite dimension or a Besicovitch-type property.

The metric spaces in which universal Bayes consistency is possible are the “essentially separable” ones—a notion that we define, which is more general than standard separability. The existence of metric spaces that are not essentially separable is widely believed to be independent of the ZFC axioms of set theory. We prove that essential separability exactly characterizes the existence of a universal Bayes-consistent learner for the given metric space. In particular, this yields the first impossibility result for universal Bayes consistency.

Taken together, our results completely characterize strong and weak universal Bayes consistency in metric spaces.

Funding Statement

Aryeh Kontorovich was supported in part by the Israel Science Foundation (Grant No. 755/15), Paypal and IBM. Sivan Sabato was supported in part by the Israel Science Foundation Grant No. 555/15.


We thank Vladimir Pestov for sharing with us his proof of the existence of a measurable total order. We also thank Robert Furber, Iosif Pinelis, Menachem Kojman, and Roberto Colomboni for helpful discussions.


Download Citation

Steve Hanneke. Aryeh Kontorovich. Sivan Sabato. Roi Weiss. "Universal Bayes consistency in metric spaces." Ann. Statist. 49 (4) 2129 - 2150, August 2021.


Received: 1 June 2019; Revised: 1 October 2020; Published: August 2021
First available in Project Euclid: 29 September 2021

Digital Object Identifier: 10.1214/20-AOS2029

Primary: 54E70 , 62C12 , 97K80
Secondary: 03E17 , 03E55

Keywords: Bayes consistency , ‎classification‎ , metric space , nearest neighbor

Rights: Copyright © 2021 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.49 • No. 4 • August 2021
Back to Top