Open Access
2009 Dynamics of Bayesian updating with dependent data and misspecified models
Cosma Rohilla Shalizi
Electron. J. Statist. 3: 1039-1074 (2009). DOI: 10.1214/09-EJS485


Much is now known about the consistency of Bayesian updating on infinite-dimensional parameter spaces with independent or Markovian data. Necessary conditions for consistency include the prior putting enough weight on the correct neighborhoods of the data-generating distribution; various sufficient conditions further restrict the prior in ways analogous to capacity control in frequentist nonparametrics. The asymptotics of Bayesian updating with mis-specified models or priors, or non-Markovian data, are far less well explored. Here I establish sufficient conditions for posterior convergence when all hypotheses are wrong, and the data have complex dependencies. The main dynamical assumption is the asymptotic equipartition (Shannon-McMillan-Breiman) property of information theory. This, along with Egorov’s Theorem on uniform convergence, lets me build a sieve-like structure for the prior. The main statistical assumption, also a form of capacity control, concerns the compatibility of the prior and the data-generating process, controlling the fluctuations in the log-likelihood when averaged over the sieve-like sets. In addition to posterior convergence, I derive a kind of large deviations principle for the posterior measure, extending in some cases to rates of convergence, and discuss the advantages of predicting using a combination of models known to be wrong. An appendix sketches connections between these results and the replicator dynamics of evolutionary theory.


Download Citation

Cosma Rohilla Shalizi. "Dynamics of Bayesian updating with dependent data and misspecified models." Electron. J. Statist. 3 1039 - 1074, 2009.


Published: 2009
First available in Project Euclid: 29 October 2009

zbMATH: 1326.62017
MathSciNet: MR2557128
Digital Object Identifier: 10.1214/09-EJS485

Primary: 62C10 , 62G20 , 62M09
Secondary: 60F10 , 62M05 , 92D15 , 94A17

Keywords: Asymptotic equipartition , Bayesian consistency , Bayesian nonparametrics , Egorov’s theorem , large deviations , posterior convergence , replicator dynamics , sofic systems

Rights: Copyright © 2009 The Institute of Mathematical Statistics and the Bernoulli Society

Back to Top