Open Access
June 2018 Unique entity estimation with application to the Syrian conflict
Beidi Chen, Anshumali Shrivastava, Rebecca C. Steorts
Ann. Appl. Stat. 12(2): 1039-1067 (June 2018). DOI: 10.1214/18-AOAS1163


Entity resolution identifies and removes duplicate entities in large, noisy databases and has grown in both usage and new developments as a result of increased data availability. Nevertheless, entity resolution has tradeoffs regarding assumptions of the data generation process, error rates, and computational scalability that make it a difficult task for real applications. In this paper, we focus on a related problem of unique entity estimation, which is the task of estimating the unique number of entities and associated standard errors in a data set with duplicate entities. Unique entity estimation shares many fundamental challenges of entity resolution, namely, that the computational cost of all-to-all entity comparisons is intractable for large databases. To circumvent this computational barrier, we propose an efficient (near-linear time) estimation algorithm based on locality sensitive hashing. Our estimator, under realistic assumptions, is unbiased and has provably low variance compared to existing random sampling based approaches. In addition, we empirically show its superiority over the state-of-the-art estimators on three real applications. The motivation for our work is to derive an accurate estimate of the documented, identifiable deaths in the ongoing Syrian conflict. Our methodology, when applied to the Syrian data set, provides an estimate of $191\text{,}874\pm 1\text{,}772$ documented, identifiable deaths, which is very close to the Human Rights Data Analysis Group (HRDAG) estimate of 191,369. Our work provides an example of challenges and efforts involved in solving a real, noisy challenging problem where modeling assumptions may not hold.


Download Citation

Beidi Chen. Anshumali Shrivastava. Rebecca C. Steorts. "Unique entity estimation with application to the Syrian conflict." Ann. Appl. Stat. 12 (2) 1039 - 1067, June 2018.


Received: 1 October 2017; Revised: 1 March 2018; Published: June 2018
First available in Project Euclid: 28 July 2018

zbMATH: 06980484
MathSciNet: MR3834294
Digital Object Identifier: 10.1214/18-AOAS1163

Keywords: clustering , entity resolution , hashing , Syrian conflict

Rights: Copyright © 2018 Institute of Mathematical Statistics

Vol.12 • No. 2 • June 2018
Back to Top