Institute of Mathematical Statistics Collections
Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems
It has been widely realized that Monte Carlo methods (approximation via a sample ensemble) may fail in large scale systems. This work offers some theoretical insight into this phenomenon in the context of the particle filter. We demonstrate that the maximum of the weights associated with the sample ensemble converges to one as both the sample size and the system dimension tends to infinity. Specifically, under fairly weak assumptions, if the ensemble size grows sub-exponentially in the cube root of the system dimension, the convergence holds for a single update step in state-space models with independent and identically distributed kernels. Further, in an important special case, more refined arguments show (and our simulations suggest) that the convergence to unity occurs unless the ensemble grows super-exponentially in the system dimension. The weight singularity is also established in models with more general multivariate likelihoods, e.g. Gaussian and Cauchy. Although presented in the context of atmospheric data assimilation for numerical weather prediction, our results are generally valid for high-dimensional particle filters.
First available in Project Euclid: 7 April 2008
Permanent link to this document
Digital Object Identifier
Primary: 93E11: Filtering [See also 60G35] 62L12: Sequential estimation 86A22: Inverse problems [See also 35R30] 60G50: Sums of independent random variables; random walks 86A32: Geostatistics 86A10: Meteorology and atmospheric physics [See also 76Bxx, 76E20, 76N15, 76Q05, 76Rxx, 76U05]
Copyright © 2008, Institute of Mathematical Statistics
Bengtsson, Thomas; Bickel, Peter; Li, Bo. Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems. Probability and Statistics: Essays in Honor of David A. Freedman, 316--334, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2008. doi:10.1214/193940307000000518. https://projecteuclid.org/euclid.imsc/1207580091
-  Anderson, J. and Anderson, S. (1999). A monte-carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Monthly Weather Review 127 2741–2758.
-  Anderson, T. (2003). An Introduction to Multivariate Statistical Analysis, 3rd ed. Wiley, New York.
-  Bengtsson, T., Snyder, C. and Nychka, D. (2003). Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophysical Research-Atmospheres 108 8775.
-  Berliner, L. M. (2001). Monte Carlo based ensemble forecasting. Stat. Comput. 11 269–275.
-  Berliner, M. and Wikle, C. (2006). Approximate importance sampling Monte Carlo for data assimilation. In review.
-  Burgers, G., P. J., van Leeuwen, P. and Evensen, G. (1998). Analysis scheme in the ensemble Kalman filter. Monthly Weather Review 126 1719–1724.
-  Cramér, H. (1946). Mathematical Methods of Statistics. Princeton Univ. Press.
-  Donoho, D. (2000). High-dimensional data analysis: The curses and blessings of dimensionality. Aide-Memoire of a Lecture at AMS conference on Math Challenges of 21st Centuary. Available at http://www-stat.stanford.edu/~donoho/Lectures/AMS2000/AMS2000.html.
-  Doucet, A., Freitas, N. and Gordon, N., eds. (2001). Sequential Monte Carlo Methods in Practice. Springer, New York.
-  Evensen, G. (1994). Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophysical Research 99 143–162.
-  Evensen, G. and van Leeuwen, P. J. (1996). Assimilation of geostat altimeter data for the Agulhas Current using the ensemble Kalman filter. Monthly Weather Review 124 85–96.
-  Furrer, R. and Bengtsson, T. (2007). Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. J. Multivariate Analysis-Revised 98 227–255.
-  Gilks, W. and Berzuini, C. (2001). Following a moving target—Monte Carlo inference for dynamic Bayesian models. J. R. Stat. Soc. Ser. B Stat. Methodol. 63 127–146.
-  Gordon, N., Salmon, D. and Smith, A. (1993). Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proceedings-F 140 107–113.
-  Hamill, T. M., J. S. W. and Snyder, C. (2001). Distance-dependent filtering of background error covariance estimates in an ensemble kalman filter. Monthly Weather Review 129 2776–2790.
-  Hoar, T., Milliff, R., Nychka, D., Wikle, C. and Berliner, L. (2003). Winds from a Bayesian hierarchical model: Computation for atmosphere-ocean research. J. Comput. Graph. Statist. 4 781–807.
-  Houtekamer, P. and Mitchell, H. (2001). A sequential ensemble Kalman filter for atmospheric data assimilation. Monthly Weather Review 129 123–137.
-  Houtekamer, P. L. and Mitchell, H. L. (1998). Data assimilation using an ensemble Kalman filter technique. Monthly Weather Review 126 796–811.
-  Liu, J. (2001). Monte Carlo Strategies in Scientific Computing. Springer, New York.
-  Liu, J. and Chen, R. (1998). Sequential Monte Carlo methods for dynamic systems. J. Amer. Statist. Assoc. 93 1032–1044.
-  Molteni, F., Buizza, R., Palmer, T. and Petroliagis, T. (1996). The ECMWF ensemble prediction system: Methodology and validation. Quarterly J. Roy. Meteorological Society 122 73–119.
-  Pitt, M. and Shepard, N. (1999). Filtering via simulation: Auxilliary particle filters. J. Amer. Statist. Assoc. 94 590–599.
-  Saulis, L. and Statulevicius, V. (2000). Limit Theorems of Probability Theory. Springer, New York.
-  Storvik, G. (2002). Particle filters for state-space models with the presence of unknown static parameters. IEEE Transactions on Signal Processing 50 281–289.
-  Tippett, M. K., Anderson, J. L., Bishop, C. H., Hamill, T. M. and Whitaker, J. S. (2003). Ensemble square-root filters. Monthly Weather Review 131 1485–1490.
-  Toth, Z. and Kalnay, E. (1997). Ensemble forecasting at NCEP and the breeding method. Monthly Weather Review 125 3297–3319.
-  van Leeuwen, P. (2003). A variance minimizing filter for large-scale applications. Monthly Weather Review 131 2071–2084.
-  Wikle, C., Milliff, R., Nychka, D. and Berliner, L. (2001). Spatiotemporal hierarchical bayesian modeling: Tropical ocean surface winds. J. Amer. Statist. Assoc. 96 382–397.
-  Wilson, S. and Stefanou, G. (2005). Bayesian approaches to content-based image retrieval. Proceedings of the International Workshop/Conference on Bayesian Statistics and Its Applications, Varanasi, India (January 2005).