Electronic Journal of Statistics

Rate-adaptive Bayesian independent component analysis

Weining Shen, Jing Ning, and Ying Yuan

Full-text: Open access


We consider independent component analysis (ICA) using a Bayesian approach. The latent sources are allowed to be block-wise independent while the underlying block structure is unknown. We consider prior distributions on the block structure, the mixing matrix and the marginal density functions of latent sources using a Dirichlet mixture and random series priors. We obtain a minimax-optimal posterior contraction rate of the joint density of the latent sources. This finding reveals that Bayesian ICA adaptively achieves the optimal rate of convergence according to the unknown smoothness level of the true marginal density functions and the unknown block structure. We evaluate the empirical performance of the proposed method by simulation studies.

Article information

Electron. J. Statist., Volume 10, Number 2 (2016), 3247-3264.

Received: December 2015
First available in Project Euclid: 16 November 2016

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Adaptive estimation Dirichlet mixture prior independent component analysis nonparametric Bayes posterior contraction rate


Shen, Weining; Ning, Jing; Yuan, Ying. Rate-adaptive Bayesian independent component analysis. Electron. J. Statist. 10 (2016), no. 2, 3247--3264. doi:10.1214/16-EJS1183. https://projecteuclid.org/euclid.ejs/1479287220

Export citation


  • Arbel, J., Gayraud, G. and Rousseau, J. (2013). Bayesian optimal adaptive estimation using a sieve prior., Scand. J. Stat. 40 549–570.
  • Bach, F. R. and Jordan, M. I. (2002). Kernel Independent Component Analysis., J. Mach. Learn. Res. 3 1–48.
  • Belitser, E. and Serra, P. (2014). Adaptive Priors Based on Splines with Random Knots., Bayesian Anal 9 859–882.
  • Bell, A. J. and Sejnowski, T. J. (1995). An information-maximization approach to blind separation and blind deconvolution., Neural Computation 7 1129–1159.
  • Bell, A. J. and Sejnowski, T. J. (1997). The “independent components” of natural scenes are edge filters., Vision Research 37 3327–3338.
  • Bhatacharya, A., Pati, D. and Dunson, D. B. (2014). Anisotropic function estimation with multi-bandwidth Gaussian process., Ann. Statist. 32 352–381.
  • Cardoso, J. F. (1998). Multidimensional independent component analysis. In, Proc. of ICASSP ’98.
  • Cardoso, J. F. (1999). High-order contrasts for independent component analysis., Neural Computation 11 157–192.
  • Castillo, I. (2014). On Bayesian supremum norm contraction rates., Ann. Statist. 42 2058–2091.
  • Chen, A. and Bickel, P. (2006). Efficient Independent component analysis., Ann. Statist. 34 2825–2855.
  • Comon, P. (1994). Independent component analysis. A new concept?, Signal Processing 36 287–314.
  • de Boor, C. (2001)., A Practical Guide to Splines. Springer.
  • de Jonge, R. and van Zanten, H. (2012). Adaptive estimation of multivariate functions using conditionally Gaussian tensor-product spline priors., Electron. J. Stat. 6 1984–2001.
  • Donnet, S., Rivoirard, V., Rousseau, J. and Scricciolo, C. (2015). Posterior concentration rates for empirical Bayes procedures, with applications to Dirichlet Process mixtures. Technical Report, arXiv:1406.4406.
  • Eloyan, A. and Ghosh, S. K. (2013). A semiparametric approach to source separation using independent component analysis., Comput. Stat. Data. Anal. 58 383–396.
  • Eriksson, J. and Koivunen, V. (2004). Identifiability, Separability, and Uniqueness of Linear ICA Models., IEEE Signal Processing Letters 11 601–604.
  • Ghosal, S., Ghosh, J. K. and van der Vaart, A. (2000). Convergence Rates of Posterior Distributions., Ann. Statist. 28 500–531.
  • Ghosal, S. and van der Vaart, A. (2001). Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities., Ann. Statist. 29 1233–1263.
  • Ghosal, S. and van der Vaart, A. (2007). Posterior convergence rates of Dirichlet mixtures at smooth densities., Ann. Statist. 35 697–723.
  • Green, P. J. (1995). Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination., Biometrika 82 711–732.
  • Hastie, T. and Tibshirani, R. (2003). Independent component analysis through product density estimation. In, Advances in Neural Information Processing Systems 15 649–656.
  • Højen-Sørensen, P. A., Winther, O. and Hansen, L. K. (2002). Mean-field approaches to independent component analysis., Neural Computation 14 889–918.
  • Hyvärinen, A. (1999). Fast and robust fixed-point algorithms for independent component analysis., IEEE Transactions on Neural Networks 10 626–634.
  • Hyvärinen, A., Karhunen, J. and Oja, E. (2001)., Independent Component Analysis. Wiley.
  • Hyvärinen, A. and Karthikesh, R. (2000). Sparse Priors On The Mixing Matrix In Independent Component Analysis. In, Proc. Int. Workshop on ICA2000 477–452.
  • Juditsky, A. B., Lepski, O. V. and Tsybakov, A. B. (2009). Nonparametric Estimation of Composite Functions., Ann. Statist. 37 1360–1404.
  • Lee, T. W., Girolami, M. and Sejnowski, T. J. (1999). Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources., Neural Computation 11 417–441.
  • Olshausen, B. A. and Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images., Nature 381 607–609.
  • Roberts, S. and Choudrey, R. (2003). Data decomposition using independent component analysis with prior constraints., Pattern Recognition 36 1813–1825.
  • Roberts, S. and Choudrey, R. (2005). Bayesian independent component analysis with prior constraints: an application in biosignal analysis. In, First international conference on Deterministic and Statistical Methods in Machine Learning 159–179.
  • Roberts, S. and Everson, R. (2001)., Independent Component Analysis: Principles and Practice. Cambridge University Press.
  • Samarov, A. and Tsybakov, A. (2004). Nonparametric independent component analysis., Bernoulli 10 565–582.
  • Samworth, R. J. and Yuan, M. (2012). Independent Component Analysis via Nonparametric Maximum Likelihood Estimation., Ann. Statist. 40 2973–3002.
  • Sarkar, A., Mallick, B. K., Staudenmayer, J., Pati, D. and Carroll, R. J. (2014). Bayesian Semiparametric Density Deconvolution in the Presence of Conditionally Heteroscedastic Measurement Errors., J. Comp. Graph. Stat. 24 1101–1125.
  • Scricciolo, C. (2014). Adaptive Bayesian Density Estimation in Lp-metrics with Pitman-Yor or Normalized Inverse-Gaussian Process Kernel Mixtures., Bayesian Anal. 9 475–520.
  • Shen, W. and Ghosal, S. (2015). Adaptive Bayesian procedures using random series priors., Scand. J. Stat. 42 1194–1213.
  • Shen, W. and Ghosal, S. (2016). Adaptive Bayesian density regression for high dimesnional data., Bernoulli 22 396–420.
  • Shen, W., Tokdar, S. T. and Ghosal, S. (2013). Adaptive Bayesian multivariate density estimation with Dirichlet mixtures., Biometrika 100 623–640.
  • Theis, F. J. (2004). Uniqueness of complex and multidimensional independent component analysis., Signal Processing 84 951–956.
  • Theis, F. J. (2005). Multidimensional independent component analysis using characteristic functions. In, Proc. of EUSIPCO.
  • Winther, O. and Petersen, K. B. (2007). Bayesian independent component analysis: Variational methods and non-negative decompositions., Digital Signal Processing 17 858–872.