The Annals of Applied Statistics

The shuffle estimator for explainable variance in fMRI experiments

Yuval Benjamini and Bin Yu

Full-text: Open access


In computational neuroscience, it is important to estimate well the proportion of signal variance in the total variance of neural activity measurements. This explainable variance measure helps neuroscientists assess the adequacy of predictive models that describe how images are encoded in the brain. Complicating the estimation problem are strong noise correlations, which may confound the neural responses corresponding to the stimuli. If not properly taken into account, the correlations could inflate the explainable variance estimates and suggest false possible prediction accuracies.

We propose a novel method to estimate the explainable variance in functional MRI (fMRI) brain activity measurements when there are strong correlations in the noise. Our shuffle estimator is nonparametric, unbiased, and built upon the random effect model reflecting the randomization in the fMRI data collection process. Leveraging symmetries in the measurements, our estimator is obtained by appropriately permuting the measurement vector in such a way that the noise covariance structure is intact but the explainable variance is changed after the permutation. This difference is then used to estimate the explainable variance. We validate the properties of the proposed method in simulation experiments. For the image-fMRI data, we show that the shuffle estimates can explain the variation in prediction accuracy for voxels within the primary visual cortex (V1) better than alternative parametric methods.

Article information

Ann. Appl. Stat., Volume 7, Number 4 (2013), 2007-2033.

First available in Project Euclid: 23 December 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Analysis-of-variance encoding models explainable variance fMRI permutations prediction random-effects shuffle estimator vision


Benjamini, Yuval; Yu, Bin. The shuffle estimator for explainable variance in fMRI experiments. Ann. Appl. Stat. 7 (2013), no. 4, 2007--2033. doi:10.1214/13-AOAS681.

Export citation


  • Benjamini, Y. and Yu, B. (2013). Supplement to “The shuffle estimator for explainable variance in fMRI experiments.” DOI:10.1214/13-AOAS681SUPP.
  • Buracas, G. T. and Boynton, G. M. (2002). Efficient design of event-related fMRI experiments using M-sequences. NeuroImage 16 801–813.
  • Carandini, M., Demb, J. B., Mante, V., Tolhurst, D. J., Dan, Y., Olshausen, B. A., Gallant, J. L. and Rust, N. C. (2005). Do we know what the early visual system does? J. Neurosci. 25 10577–10597.
  • Dayan, P. and Abbott, L. F. (2001). Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. MIT Press, Cambridge, MA.
  • Fox, M. D. and Raichle, M. E. (2007). Spontaneous fluctuations in brain activity observed with functional magnetic resonance imaging. Nat. Rev. Neurosci. 8 700–711.
  • Haefner, R. and Cumming, B. (2008). An improved estimator of variance explained in the presence of noise. Adv. Neural Inf. Process. Syst. 21 585–592.
  • Hsu, A., Borst, A. and Theunissen, F. E. (2004). Quantifying variability in neural responses and its application for the validation of model predictions. Network: Comput. Neural Syst. 15 91–109.
  • Huettel, S. A. (2012). Event-related fMRI in cognition. NeuroImage 62 1152–1156.
  • Josephs, O., Turner, R. and Friston, K. (1997). Event-related fMRI. Hum. Brain Mapp. 5 243–248.
  • Kay, K. N., Naselaris, T., Prenger, R. J. and Gallant, J. L. (2008). Identifying natural images from human brain activity. Nature 452 352–355.
  • Laird, N., Lange, N. and Stram, D. (1987). Maximum likelihood computations with repeated measures: Application of the EM algorithm. J. Amer. Statist. Assoc. 82 97–105.
  • Lyons, R. (1988). Strong laws of large numbers for weakly correlated random variables. Michigan Math. J. 35 353–359.
  • Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M. and Gallant, J. L. (2009). Bayesian reconstruction of natural images from human brain activity. Neuron 63 902–915.
  • Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B. and Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Curr. Biol. 21 1641–1646.
  • Owen, A. B. (2007). The pigeonhole bootstrap. Ann. Appl. Stat. 1 386–411.
  • Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T. and Chang, E. F. (2012). Reconstructing speech from human auditory cortex. PLoS Comput. Biol. 10 e1001251.
  • Pasupathy, A. and Connor, C. E. (1999). Responses to contour features in macaque area V4. J. Neurophysiol. 82 2490–2502.
  • Pereira, F., Detre, G. and Botvinick, M. (2011). Generating text from functional brain images. Front. Human Neurosci. 5.
  • Roddey, J. C., Girish, B. and Miller, J. P. (2000). Assessing the performance of neural encoding models in the presence of noise. J. Comput. Neurosci. 8 95–112.
  • Sahani, M. and Linden, J. F. (2003). How linear are auditory cortical responses? Adv. Neural Inf. Process. Syst. 15 125–132.
  • Scheffé, H. (1959). The Analysis of Variance. Wiley, New York.
  • Searle, S. R. (1971). Linear Models. Wiley, New York.
  • Shoham, S., Paninski, L. M., Fellows, M. R., Hatsopoulos, N. G., Donoghue, J. P. and Normann, R. A. (2005). Statistical encoding model for a primary motor cortical brain-machine interface. IEEE Trans. Biomed. Eng. 52 1312–1322.
  • Townsend, E. C. and Searle, S. R. (1971). Best quadratic unbiased estimation of variance components from unbalanced data in the 1-way classification. Biometrics 27 643–657.
  • Vu, V. Q., Ravikumar, P., Naselaris, T., Kay, K. N., Gallant, J. L. and Yu, B. (2011). Encoding and decoding V1 fMRI responses to natural images with sparse nonparametric models. Ann. Appl. Stat. 5 1159–1182.
  • Vul, E., Harris, C., Winkielman, P. and Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science 4 274–290.
  • Williams, R. M. (1952). Experimental designs for serially correlated observations. Biometrika 39 151–167.
  • Woolrich, M. W., Ripley, B. D., Brady, M. and Smith, S. M. (2001). Temporal autocorrelation in univariate linear modeling of fMRI data. Neuroimage 14 1370–1386.

Supplemental materials

  • Supplementary material: Supplementary material. We provide proofs for Lemmas 1 and 2 in the supplementary, as well as describe and prove conditions for consistency of the shuffle estimator.