Electronic Journal of Statistics

Importance sampling the union of rare events with an application to power systems analysis

Art B. Owen, Yury Maximov, and Michael Chertkov

Full-text: Open access


We consider importance sampling to estimate the probability $\mu$ of a union of $J$ rare events $H_{j}$ defined by a random variable $\boldsymbol{x}$. The sampler we study has been used in spatial statistics, genomics and combinatorics going back at least to Karp and Luby (1983). It works by sampling one event at random, then sampling $\boldsymbol{x}$ conditionally on that event happening and it constructs an unbiased estimate of $\mu$ by multiplying an inverse moment of the number of occuring events by the union bound. We prove some variance bounds for this sampler. For a sample size of $n$, it has a variance no larger than $\mu(\bar{\mu}-\mu)/n$ where $\bar{\mu}$ is the union bound. It also has a coefficient of variation no larger than $\sqrt{(J+J^{-1}-2)/(4n)}$ regardless of the overlap pattern among the $J$ events. Our motivating problem comes from power system reliability, where the phase differences between connected nodes have a joint Gaussian distribution and the $J$ rare events arise from unacceptably large phase differences. In the grid reliability problems even some events defined by $5772$ constraints in $326$ dimensions, with probability below $10^{-22}$, are estimated with a coefficient of variation of about $0.0024$ with only $n=10{,}000$ sample values.

Article information

Electron. J. Statist., Volume 13, Number 1 (2019), 231-254.

Received: February 2018
First available in Project Euclid: 30 January 2019

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Creative Commons Attribution 4.0 International License.


Owen, Art B.; Maximov, Yury; Chertkov, Michael. Importance sampling the union of rare events with an application to power systems analysis. Electron. J. Statist. 13 (2019), no. 1, 231--254. doi:10.1214/18-EJS1527. https://projecteuclid.org/euclid.ejs/1548817590

Export citation


  • Adler, R. J., J. Blanchet, and J. Liu (2008). Efficient simulation for tail probabilities of Gaussian random fields. In, Winter Simulation Conference, 2008, pp. 328–336. IEEE.
  • Adler, R. J., J. H. Blanchet, and J. Liu (2012). Efficient Monte Carlo for high excursions of Gaussian random fields., The Annals of Applied Probability 22(3), 1167–1214.
  • Ahn, D. and K.-K. Kim (2018). Efficient simulation for expectations over the union of half-spaces., ACM Transactions on Modeling and Computer Simulation (TOMACS) 28(3), 23.
  • Asmussen, S. and P. W. Glynn (2007)., Stochastic simulation: algorithms and analysis, Volume 57. Springer Science & Business Media.
  • Botev, Z. I. and P. L’Ecuyer (2015). Efficient probability estimation and simulation of the truncated multivariate student-t distribution. In, Winter Simulation Conference (WSC), 2015, pp. 380–391. IEEE.
  • Botev, Z. I., M. Mandjes, and A. Ridder (2015). Tail distribution of the maximum of correlated gaussian random variables. In, Proceedings of the 2015 Winter Simulation Conference, pp. 633–642. IEEE Press.
  • Chertkov, M., F. Pan, and M. G. Stepanov (2011). Predicting failures in power grids: The case of static overloads., IEEE Transactions on Smart Grid 2(1), 162–172.
  • Chertkov, M., M. Stepanov, F. Pan, and R. Baldick (2011). Exact and efficient algorithm to discover extreme stochastic events in wind generation over transmission power grids. In, 2011 50th IEEE Conference on Decision and Control and European Control Conference, pp. 2174– 2180.
  • Cornuet, J., J.-M. Marin, A. Mira, and C. P. Robert (2012). Adaptive multiple importance sampling., Scandinavian Journal of Statistics 39(4), 798–812.
  • Cranley, R. and T. N. L. Patterson (1976). Randomization of number theoretic methods for multiple integration., SIAM Journal of Numerical Analysis 13(6), 904–914.
  • Cunningham, S. W. (1969). Algorithm AS 24: From normal integral to deviate., Journal of the Royal Statistical Society. Series C 18(3), 290–293.
  • Doucet, A. (2010). A note on efficient conditional simulation of Gaussian distributions. Technical report, University of British, Columbia.
  • Elvira, V., L. Martino, D. Luengo, and M. F. Bugallo (2015a). Efficient multiple importance sampling estimators., IEEE Signal Processing Letters 22(10), 1757–1761.
  • Elvira, V., L. Martino, D. Luengo, and M. F. Bugallo (2015b). Generalized multiple importance sampling., arXiv preprint arXiv:1511.03095.
  • Fliscounakis, S., P. Panciatici, F. Capitanescu, and L. Wehenkel (2013). Contingency ranking with respect to overloads in very large power systems taking into account uncertainty, preventive, and corrective actions., IEEE Transactions on Power Systems 28(4), 4909–4917.
  • Frigessi, A. and C. Vercellis (1985). An analysis of Monte Carlo algorithms for counting problems., Calcolo 22(4), 413–428.
  • Genz, A. (2004). Numerical computation of rectangular bivariate and trivariate normal and t probabilities., Statistics and Computing 14(3), 251–260.
  • Genz, A. and F. Bretz (2009)., Computation of Multivariate Normal and $t$ Probabilities. Berlin: Springer-Verlag.
  • Genz, A., F. Bretz, T. Miwa, X. Mi, F. Leisch, F. Scheipl, and T. Hothorn (2017)., mvtnorm: Multivariate Normal and t Distributions. R package version 1.0-6.
  • Hesterberg, T. C. (1988)., Advances in importance sampling. Ph. D. thesis, Stanford University.
  • Kahn, H. and A. Marshall (1953). Methods of reducing sample size in Monte Carlo computations., Journal of the Operations Research Society of America 1(5), 263–278.
  • Karp, R. M. and M. Luby (1983). Monte-Carlo algorithms for enumeration and reliability problems. In, Foundations of Computer Science, 1983, 24th Annual Symposium on, pp. 56–64. IEEE.
  • Kersulis, J., I. Hiskens, M. Chertkov, S. Backhaus, and D. Bienstock (2015, June). Temperature-based instanton analysis: Identifying vulnerability in transmission networks. In, 2015 IEEE Eindhoven PowerTech, pp. 1–6.
  • Lafortune, E. P. and Y. D. Willems (1993). Bidirectional path tracing. In, Proceedings of CompuGraphics, pp. 95–104.
  • L’Ecuyer, P., M. Mandjes, and B. Tuffin (2009). Importance sampling and rare event simulation. In G. Rubino and B. Tuffin (Eds.), Rare event simulation using Monte Carlo methods, pp. 17–38. Chichester, UK: John Wiley & Sons.
  • Liu, J. S. (2001)., Monte Carlo strategies in scientific computing. New York: Springer.
  • Miwa, T., A. Hayter, and S. Kuriki (2003). The evaluation of general non-centred orthant probabilities., Journal of the Royal Statistical Society, Series B 65(1), 223–234.
  • Naiman, D. Q. and C. E. Priebe (2001). Computing scan statistic p values using importance sampling, with applications to genetics and medical image analysis., Journal of Computational and Graphical Statistics 10(2), 296–328.
  • Niederreiter, H. (1972). On a number-theoretical integration method., Aequationes Math 8(3), 304–311.
  • Owen, A. B. (2013)., Monte Carlo theory, methods and examples.
  • Owen, A. B. and Y. Zhou (2000). Safe and effective importance sampling., Journal of the American Statistical Association 95(449), 135–143.
  • Priebe, C. E., D. Q. Naiman, and L. M. Cope (2001). Importance sampling for spatial scan analysis: computing scan statistic p-values for marked point processes., Computational statistics & data analysis 35(4), 475–485.
  • R Core Team (2015)., R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.
  • Sauer, P. W. and J. P. Christensen (1984). Active linear DC circuit models for power system analysis., Electric machines and power systems 9(2–3), 103–112.
  • Shi, J., D. O. Siegmund, and B. Yakir (2007). Importance sampling for estimating p values in linkage analysis., Journal of the American Statistical Association 102(479), 929–937.
  • Stott, B., J. Jardim, and O. Alsaç (2009). DC power flow revisited., IEEE Transactions on Power Systems 24(3), 1290–1300.
  • Van den Bergh, K., E. Delarue, and W. D’haeseleer (2014). DC power flow in unit commitment models. Technical Report WP EN2014-12, KU, Leuven.
  • Veach, E. and L. Guibas (1994, June 13–15). Bidirectional estimators for light transport. In, 5th Annual Eurographics Workshop on Rendering, pp. 147–162.
  • Yang, J., F. Alajaji, and G. Takahara (2014). A short survey on bounding the union probability using partial information. Technical report, University of, Toronto.
  • Zimmerman, R. D., C. E. Murillo-Sánchez, and R. J. Thomas (2011). MATPOWER: steady-state operations, planning, and analysis tools for power systems research and education., IEEE Transactions on power systems 26(1), 12–19.