Bayesian Analysis

Fast Simulation of Hyperplane-Truncated Multivariate Normal Distributions

Yulai Cong, Bo Chen, and Mingyuan Zhou

Full-text: Open access


We introduce a fast and easy-to-implement simulation algorithm for a multivariate normal distribution truncated on the intersection of a set of hyperplanes, and further generalize it to efficiently simulate random variables from a multivariate normal distribution whose covariance (precision) matrix can be decomposed as a positive-definite matrix minus (plus) a low-rank symmetric matrix. Example results illustrate the correctness and efficiency of the proposed simulation algorithms.

Article information

Bayesian Anal. Volume 12, Number 4 (2017), 1017-1037.

First available in Project Euclid: 1 March 2017

Permanent link to this document

Digital Object Identifier

Cholesky decomposition conditional distribution equality constraints high-dimensional regression structured covariance/precision matrix

Creative Commons Attribution 4.0 International License.


Cong, Yulai; Chen, Bo; Zhou, Mingyuan. Fast Simulation of Hyperplane-Truncated Multivariate Normal Distributions. Bayesian Anal. 12 (2017), no. 4, 1017--1037. doi:10.1214/17-BA1052.

Export citation


  • Albert, J. H. and Chib, S. (1993). “Bayesian analysis of binary and polychotomous response data.” Journal of the American Statistical Association, 88(422): 669–679.
  • Altmann, Y., McLaughlin, S., and Dobigeon, N. (2014). “Sampling from a multivariate Gaussian distribution truncated on a simplex: a review.” In 2014 IEEE Workshop on Statistical Signal Processing (SSP), 113–116. IEEE.
  • Bazot, C., Dobigeon, N., Tourneret, J.-Y., Zaas, A. K., Ginsburg, G. S., and Hero III, A. O. (2013). “Unsupervised Bayesian linear unmixing of gene expression microarrays.” BMC Bioinformatics, 14(1): 1.
  • Bhattacharya, A., Chakraborty, A., and Mallick, B. K. (2016). “Fast sampling with Gaussian scale mixture priors in high-dimensional regression.” Biometrika, 103(4): 985.
  • Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). “Latent Dirichlet allocation.” Journal of Machine Learning Research, 3: 993–1022.
  • Botev, Z. (2016). “The normal law under linear restrictions: simulation and estimation via minimax tilting.” Journal of the Royal Statistical Society: Series B (Statistical Methodology).
  • Caron, F. and Doucet, A. (2008). “Sparse Bayesian nonparametric regression.” In ICML, 88–95. ACM.
  • Carvalho, C. M., Polson, N. G., and Scott, J. G. (2010). “The horseshoe estimator for sparse signals.” Biometrika, 97(2): 465–480.
  • Chopin, N. (2011). “Fast simulation of truncated Gaussian distributions.” Statistics and Computing, 21(2): 275–288.
  • Cong, Y., Chen, B., and Zhou, M. (2017). “Fast Simulation of Hyperplane-Truncated Multivariate Normal Distributions: Supplementary Material.” Bayesian Analysis.
  • Cong, Y., Chen, B., and Zhou, M. (2017). “Deep latent Dirichlet allocation with topic-layer-adaptive stochastic gradient Riemannian (TLASGR) MCMC.” Preprint.
  • Damien, P. and Walker, S. G. (2001). “Sampling truncated normal, beta, and gamma densities.” Journal of Computational and Graphical Statistics, 10(2): 206–215.
  • Dobigeon, N., Moussaoui, S., Coulon, M., Tourneret, J.-Y., and Hero, A. O. (2009a). “Joint Bayesian endmember extraction and linear unmixing for hyperspectral imagery.” IEEE Transactions on Signal Processing, 57(11): 4355–4368.
  • Dobigeon, N., Moussaoui, S., Tourneret, J.-Y., and Carteret, C. (2009b). “Bayesian separation of spectral sources under non-negativity and full additivity constraints.” Signal Processing, 89(12): 2657–2669.
  • Doucet, A. (2010). “A note on efficient conditional simulation of Gaussian distributions.” Departments of Computer Science and Statistics, University of British Columbia.
  • Gelfand, A. E., Smith, A. F., and Lee, T.-M. (1992). “Bayesian analysis of constrained parameter and truncated data problems using Gibbs sampling.” Journal of the American Statistical Association, 87(418): 523–532.
  • Gelfand, A. E. and Smith, A. F. M. (1990). “Sampling-based approaches to calculating marginal densities.” Journal of the American Statistical Association, 85(410): 398–409.
  • Geman, S. and Geman, D. (1984). “Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 721–741.
  • Geweke, J. (1991). “Efficient simulation from the multivariate normal and student-t distributions subject to linear constraints and the evaluation of constraint probabilities.” In Computing science and statistics: Proceedings of the 23rd symposium on the interface, 571–578.
  • Geweke, J. F. (1996). “Bayesian inference for linear models subject to linear inequality constraints.” In Modelling and Prediction Honoring Seymour Geisser, 248–263. Springer.
  • Golub, G. H. and Van Loan, C. F. (2012). Matrix Computations, volume 3. JHU Press.
  • Heckerman, D. (1998). “A tutorial on learning with Bayesian networks.” In Learning in graphical models, 301–354. Springer.
  • Hoffman, M., Blei, D., and Bach, F. (2010). “Online learning for latent Dirichlet allocation.” In NIPS.
  • Hoffman, Y. and Ribak, E. (1991). “Constrained realizations of Gaussian fields-A simple algorithm.” The Astrophysical Journal, 380: L5–L8.
  • Holmes, C. C. and Held, L. (2006). “Bayesian auxiliary variable models for binary and multinomial regression.” Bayesian Analysis, 1(1): 145–168.
  • Imai, K. and van Dyk, D. A. (2005). “A Bayesian analysis of the multinomial probit model using marginal data augmentation.” Journal of Econometrics, 124(2): 311–334.
  • Johndrow, J., Dunson, D., and Lum, K. (2013). “Diagonal orthant multinomial probit models.” In AISTATS, 29–38.
  • Lan, S., Zhou, B., and Shahbaba, B. (2014). “Spherical Hamiltonian Monte Carlo for constrained target distributions.” In ICML, 629–637.
  • Ma, Y., Chen, T., and Fox, E. (2015). “A complete recipe for stochastic gradient MCMC.” In NIPS, 2899–2907.
  • McCulloch, R. E., Polson, N. G., and Rossi, P. E. (2000). “A Bayesian analysis of the multinomial probit model with fully identified parameters.” Journal of Econometrics, 99(1): 173–193.
  • Neelon, B. and Dunson, D. B. (2004). “Bayesian isotonic regression and trend analysis.” Biometrics, 60(2): 398–406.
  • Pakman, A. and Paninski, L. (2014). “Exact Hamiltonian Monte Carlo for truncated multivariate Gaussians.” Journal of Computational and Graphical Statistics, 23(2): 518–542.
  • Polson, N. G., Scott, J. G., and Windle, J. (2014). “The Bayesian bridge.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(4): 713–733.
  • Pritchard, J. K., Stephens, M., and Donnelly, P. (2000). “Inference of population structure using multilocus genotype data.” Genetics, 155(2): 945–959.
  • Robert, C. P. (1995). “Simulation of truncated normal variables.” Statistics and Computing, 5(2): 121–125.
  • Rodriguez-Yam, G., Davis, R. A., and Scharf, L. L. (2004). “Efficient Gibbs sampling of truncated multivariate normal with application to constrained linear regression.” Technical report.
  • Rue, H. (2001). “Fast sampling of Gaussian Markov random fields.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(2): 325–338.
  • Schmidt, M. (2009). “Linearly constrained Bayesian matrix factorization for blind source separation.” In NIPS, 1624–1632.
  • Tong, Y. L. (2012). The Multivariate Normal Distribution. Springer Science & Business Media.
  • Train, K. E. (2009). Discrete Choice Methods With Simulation. Cambridge University Press.
  • Zhou, M., Cong, Y., and Chen, B. (2016). “Augmentable gamma belief networks.” Journal of Machine Learning Research, 17(163): 1–44.
  • Zhou, M., Hannah, L., Dunson, D. B., and Carin, L. (2012). “Beta-negative binomial process and Poisson factor analysis.” In AISTATS, 1462–1471.

Supplemental materials