Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems
Thomas Bengtsson, Peter Bickel, Bo Li
Abstract
It has been widely realized that Monte Carlo methods (approximation via a sample ensemble) may fail in large scale systems. This work offers some theoretical insight into this phenomenon in the context of the particle filter. We demonstrate that the maximum of the weights associated with the sample ensemble converges to one as both the sample size and the system dimension tends to infinity. Specifically, under fairly weak assumptions, if the ensemble size grows sub-exponentially in the cube root of the system dimension, the convergence holds for a single update step in state-space models with independent and identically distributed kernels. Further, in an important special case, more refined arguments show (and our simulations suggest) that the convergence to unity occurs unless the ensemble grows super-exponentially in the system dimension. The weight singularity is also established in models with more general multivariate likelihoods, e.g. Gaussian and Cauchy. Although presented in the context of atmospheric data assimilation for numerical weather prediction, our results are generally valid for high-dimensional particle filters.
First Page:
Show
Hide
Primary Subjects: 93E11, 62L12, 86A22, 60G50, 86A32, 86A10
Keywords: ensemble forecast; inverse problem; Monte Carlo; multivariate Cauchy; multivariate likelihood; numerical weather prediction; sample ensemble; state-space model
Full-text: Open access
Links and Identifiers
Permanent link to this document: http://projecteuclid.org/euclid.imsc/1207580091
Digital Object Identifier: doi:10.1214/193940307000000518
References
[1] Anderson, J. and Anderson, S. (1999). A monte-carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Monthly Weather Review 127 2741–2758.
[2] Anderson, T. (2003). An Introduction to Multivariate Statistical Analysis, 3rd ed. Wiley, New York.
Mathematical Reviews (MathSciNet):
MR1990662
[3] Bengtsson, T., Snyder, C. and Nychka, D. (2003). Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophysical Research-Atmospheres 108 8775.
[4] Berliner, L. M. (2001). Monte Carlo based ensemble forecasting. Stat. Comput. 11 269–275.
Mathematical Reviews (MathSciNet):
MR1842976
Digital Object Identifier: doi:10.1023/A:1016656422040
[5] Berliner, M. and Wikle, C. (2006). Approximate importance sampling Monte Carlo for data assimilation. In review.
[6] Burgers, G., P. J., van Leeuwen, P. and Evensen, G. (1998). Analysis scheme in the ensemble Kalman filter. Monthly Weather Review 126 1719–1724.
[7] Cramér, H. (1946). Mathematical Methods of Statistics. Princeton Univ. Press.
Mathematical Reviews (MathSciNet):
MR16588
[8] Donoho, D. (2000). High-dimensional data analysis: The curses and blessings of dimensionality. Aide-Memoire of a Lecture at AMS conference on Math Challenges of 21st Centuary. Available at http://www-stat.stanford.edu/~donoho/Lectures/AMS2000/AMS2000.html.
[9] Doucet, A., Freitas, N. and Gordon, N., eds. (2001). Sequential Monte Carlo Methods in Practice. Springer, New York.
Mathematical Reviews (MathSciNet):
MR1847783
[10] Evensen, G. (1994). Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophysical Research 99 143–162.
[11] Evensen, G. and van Leeuwen, P. J. (1996). Assimilation of geostat altimeter data for the Agulhas Current using the ensemble Kalman filter. Monthly Weather Review 124 85–96.
[12] Furrer, R. and Bengtsson, T. (2007). Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. J. Multivariate Analysis-Revised 98 227–255.
Mathematical Reviews (MathSciNet):
MR2301751
Digital Object Identifier: doi:10.1016/j.jmva.2006.08.003
[13] Gilks, W. and Berzuini, C. (2001). Following a moving target—Monte Carlo inference for dynamic Bayesian models. J. R. Stat. Soc. Ser. B Stat. Methodol. 63 127–146.
Mathematical Reviews (MathSciNet):
MR1811995
Zentralblatt MATH:
0976.62021
Digital Object Identifier: doi:10.1111/1467-9868.00280
JSTOR: links.jstor.org
[14] Gordon, N., Salmon, D. and Smith, A. (1993). Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proceedings-F 140 107–113.
[15] Hamill, T. M., J. S. W. and Snyder, C. (2001). Distance-dependent filtering of background error covariance estimates in an ensemble kalman filter. Monthly Weather Review 129 2776–2790.
[16] Hoar, T., Milliff, R., Nychka, D., Wikle, C. and Berliner, L. (2003). Winds from a Bayesian hierarchical model: Computation for atmosphere-ocean research. J. Comput. Graph. Statist. 4 781–807.
Mathematical Reviews (MathSciNet):
MR2037932
Digital Object Identifier: doi:10.1198/1061860032616
JSTOR: links.jstor.org
[17] Houtekamer, P. and Mitchell, H. (2001). A sequential ensemble Kalman filter for atmospheric data assimilation. Monthly Weather Review 129 123–137.
[18] Houtekamer, P. L. and Mitchell, H. L. (1998). Data assimilation using an ensemble Kalman filter technique. Monthly Weather Review 126 796–811.
[19] Liu, J. (2001). Monte Carlo Strategies in Scientific Computing. Springer, New York.
Mathematical Reviews (MathSciNet):
MR1842342
[20] Liu, J. and Chen, R. (1998). Sequential Monte Carlo methods for dynamic systems. J. Amer. Statist. Assoc. 93 1032–1044.
Mathematical Reviews (MathSciNet):
MR1649198
Zentralblatt MATH:
1064.65500
Digital Object Identifier: doi:10.2307/2669847
JSTOR: links.jstor.org
[21] Molteni, F., Buizza, R., Palmer, T. and Petroliagis, T. (1996). The ECMWF ensemble prediction system: Methodology and validation. Quarterly J. Roy. Meteorological Society 122 73–119.
[22] Pitt, M. and Shepard, N. (1999). Filtering via simulation: Auxilliary particle filters. J. Amer. Statist. Assoc. 94 590–599.
Mathematical Reviews (MathSciNet):
MR1702328
Zentralblatt MATH:
1072.62639
Digital Object Identifier: doi:10.2307/2670179
JSTOR: links.jstor.org
[23] Saulis, L. and Statulevicius, V. (2000). Limit Theorems of Probability Theory. Springer, New York.
Mathematical Reviews (MathSciNet):
MR1798811
[24] Storvik, G. (2002). Particle filters for state-space models with the presence of unknown static parameters. IEEE Transactions on Signal Processing 50 281–289.
[25] Tippett, M. K., Anderson, J. L., Bishop, C. H., Hamill, T. M. and Whitaker, J. S. (2003). Ensemble square-root filters. Monthly Weather Review 131 1485–1490.
[26] Toth, Z. and Kalnay, E. (1997). Ensemble forecasting at NCEP and the breeding method. Monthly Weather Review 125 3297–3319.
[27] van Leeuwen, P. (2003). A variance minimizing filter for large-scale applications. Monthly Weather Review 131 2071–2084.
[28] Wikle, C., Milliff, R., Nychka, D. and Berliner, L. (2001). Spatiotemporal hierarchical bayesian modeling: Tropical ocean surface winds. J. Amer. Statist. Assoc. 96 382–397.
Mathematical Reviews (MathSciNet):
MR1939342
Zentralblatt MATH:
1022.62117
Digital Object Identifier: doi:10.1198/016214501753168109
JSTOR: links.jstor.org
[29] Wilson, S. and Stefanou, G. (2005). Bayesian approaches to content-based image retrieval. Proceedings of the International Workshop/Conference on Bayesian Statistics and Its Applications, Varanasi, India (January 2005).
Institute of Mathematical Statistics Collections
