Translator Disclaimer
December 2020 Clustering in Block Markov Chains
Jaron Sanders, Alexandre Proutière, Se-Young Yun
Ann. Statist. 48(6): 3488-3512 (December 2020). DOI: 10.1214/19-AOS1939

Abstract

This paper considers cluster detection in Block Markov Chains (BMCs). These Markov chains are characterized by a block structure in their transition matrix. More precisely, the $n$ possible states are divided into a finite number of $K$ groups or clusters, such that states in the same cluster exhibit the same transition rates to other states. One observes a trajectory of the Markov chain, and the objective is to recover, from this observation only, the (initially unknown) clusters. In this paper, we devise a clustering procedure that accurately, efficiently and provably detects the clusters. We first derive a fundamental information-theoretical lower bound on the detection error rate satisfied under any clustering algorithm. This bound identifies the parameters of the BMC, and trajectory lengths, for which it is possible to accurately detect the clusters. We next develop two clustering algorithms that can together accurately recover the cluster structure from the shortest possible trajectories, whenever the parameters allow detection. These algorithms thus reach the fundamental detectability limit, and are optimal in that sense.

Citation

Download Citation

Jaron Sanders. Alexandre Proutière. Se-Young Yun. "Clustering in Block Markov Chains." Ann. Statist. 48 (6) 3488 - 3512, December 2020. https://doi.org/10.1214/19-AOS1939

Information

Received: 1 February 2018; Revised: 1 July 2019; Published: December 2020
First available in Project Euclid: 11 December 2020

MathSciNet: MR4185817
Digital Object Identifier: 10.1214/19-AOS1939

Subjects:
Primary: 60J10, 60J20, 62H30

Rights: Copyright © 2020 Institute of Mathematical Statistics

JOURNAL ARTICLE
25 PAGES

This article is only available to subscribers.
It is not available for individual sale.
+ SAVE TO MY LIBRARY

SHARE
Vol.48 • No. 6 • December 2020
Back to Top