Translator Disclaimer
December 2015 Entity Resolution with Empirically Motivated Priors
Rebecca C. Steorts
Bayesian Anal. 10(4): 849-875 (December 2015). DOI: 10.1214/15-BA965SI

Abstract

Databases often contain corrupted, degraded, and noisy data with duplicate entries across and within each database. Such problems arise in citations, medical databases, genetics, human rights databases, and a variety of other applied settings. The target of statistical inference can be viewed as an unsupervised problem of determining the edges of a bipartite graph that links the observed records to unobserved latent entities. Bayesian approaches provide attractive benefits, naturally providing uncertainty quantification via posterior probabilities. We propose a novel record linkage approach based on empirical Bayesian principles. Specifically, the empirical Bayesian-type step consists of taking the empirical distribution function of the data as the prior for the latent entities. This approach improves on the earlier HB approach not only by avoiding the prior specification problem but also by allowing both categorical and string-valued variables. Our extension to string-valued variables also involves the proposal of a new probabilistic mechanism by which observed record values for string fields can deviate from the values of their associated latent entities. Categorical fields that deviate from their corresponding true value are simply drawn from the empirical distribution function. We apply our proposed methodology to a simulated data set of German names and an Italian household survey on income and wealth, showing our method performs favorably compared to several standard methods in the literature. We also consider the robustness of our methods to changes in the hyper-parameters.

Citation

Download Citation

Rebecca C. Steorts. "Entity Resolution with Empirically Motivated Priors." Bayesian Anal. 10 (4) 849 - 875, December 2015. https://doi.org/10.1214/15-BA965SI

Information

Published: December 2015
First available in Project Euclid: 9 September 2015

zbMATH: 1335.62023
MathSciNet: MR3432242
Digital Object Identifier: 10.1214/15-BA965SI

Rights: Copyright © 2015 International Society for Bayesian Analysis

JOURNAL ARTICLE
27 PAGES


SHARE
Vol.10 • No. 4 • December 2015
Back to Top