The Annals of Applied Statistics

Encoding and decoding V1 fMRI responses to natural images with sparse nonparametric models

Vincent Q. Vu, Pradeep Ravikumar, Thomas Naselaris, Kendrick N. Kay, Jack L. Gallant, and Bin Yu

Full-text: Open access


Functional MRI (fMRI) has become the most common method for investigating the human brain. However, fMRI data present some complications for statistical analysis and modeling. One recently developed approach to these data focuses on estimation of computational encoding models that describe how stimuli are transformed into brain activity measured in individual voxels. Here we aim at building encoding models for fMRI signals recorded in the primary visual cortex of the human brain. We use residual analyses to reveal systematic nonlinearity across voxels not taken into account by previous models. We then show how a sparse nonparametric method [J. Roy. Statist. Soc. Ser. B 71 (2009b) 1009–1030] can be used together with correlation screening to estimate nonlinear encoding models effectively. Our approach produces encoding models that predict about 25% more accurately than models estimated using other methods [Nature 452 (2008a) 352–355]. The estimated nonlinearity impacts the inferred properties of individual voxels, and it has a plausible biological interpretation. One benefit of quantitative encoding models is that estimated models can be used to decode brain activity, in order to identify which specific image was seen by an observer. Encoding models estimated by our approach also improve such image identification by about 12% when the correct image is one of 11,500 possible images.

Article information

Ann. Appl. Stat. Volume 5, Number 2B (2011), 1159-1182.

First available in Project Euclid: 13 July 2011

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Neuroscience vision fMRI nonparametric prediction


Vu, Vincent Q.; Ravikumar, Pradeep; Naselaris, Thomas; Kay, Kendrick N.; Gallant, Jack L.; Yu, Bin. Encoding and decoding V1 fMRI responses to natural images with sparse nonparametric models. Ann. Appl. Stat. 5 (2011), no. 2B, 1159--1182. doi:10.1214/11-AOAS476.

Export citation


  • Adelson, E. H. and Bergen, J. R. (1985). Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Amer. A 2 284–299.
  • Albrecht, D. G. and Hamilton, D. B. (1982). Striate cortex of monkey and cat: Contrast response function. Journal of Neurophysiology 48 217–237.
  • Buxton, R. B., Wong, E. C. and Frank, L. R. (1998). Dynamics of blood flow and oxygenation changes during brain activation: The balloon model. Magnetic Resonance in Medicine 39 855–864.
  • Buxton, R. B., Uludag, K., Dubowitz, D. J. and Liu, T. T. (2004). Modeling the hemodynamic response to brain activation. NeuroImage 23 S220–S233.
  • Carandini, M., Heeger, D. J. and Movshon, J. A. (1997). Linearity and normalization in simple cells of the macaque primary visual cortex. Journal of Neuroscience 17 8621–8644.
  • Cleveland, W. S. and Devlin, S. J. (1988). Locally weighted regression: An approach to regression analysis by local fitting. J. Amer. Statist. Assoc. 83 596–610.
  • De Valois, R. L. and De Valois, K. K. (1990). Spatial Vision. Oxford Univ. Press, New York.
  • Frahm, H. D., Stephan, H. and Stephan, M. (1982). Comparison of brain structure volumes in Insectivora and Primates. I. Neocortex. Journal für Hirnforschung 23 375–389.
  • Friedman, J. H. and Popescu, B. E. (2004). Gradient directed regularization for linear regression and classification. Technical report, Dept. Statistics, Stanford Univ.
  • Friedman, J. H. and Stuetzle, W. (1981). Projection pursuit regression. J. Amer. Statist. Assoc. 76 817–823.
  • Friston, K. J., Jezzard, P. and Turner, R. (1994). Analysis of functional MRI time-series. Human Brain Mapping 1 153–171.
  • Hastie, T. and Tibshirani, R. (1990). Generalized Additive Models. Chapman & Hall, Boca Raton, FL.
  • Heeger, D. J. (1992). Normalization of cell responses in cat striate cortex. Visual Neuroscience 9 181–197.
  • Hofman, M. A. (1989). On the evolution and geometry of the brain in mammals. Progress in Neurobiology 32 137–158.
  • Jones, J. P. and Palmer, L. A. (1987). An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. Journal of Neurophysiology 58 1233–1258.
  • Kafadar, K. and Wegman, E. J. (2006). Visualizing “typical” and “exotic” internet traffic data. Comput. Statist. Data Anal. 50 3721–3743.
  • Kay, K. N., Naselaris, T., Prenger, R. J. and Gallant, J. L. (2008a). Identifying natural images from human brain activity. Nature 452 352–355.
  • Kay, K. N., David, S. V., Prenger, R. J., Hansen, K. A. and Gallant, J. L. (2008b). Modeling low-frequency fluctuation and hemodynamic response timecourse in event-related fMRI. Human Brain Mapping 29 142–156.
  • Lauritzen, M. (2005). Reading vascular changes in brain imaging: Is dendritic calcium the key? Nat. Rev. Neurosci. 6 77–85.
  • Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M. and Gallant, J. L. (2009). Bayesian reconstruction of natural images from human brain activity. Neuron 63 902–915.
  • Naselaris, T., Kay, K. N., Nishimoto, S. and Gallant, J. L. (2011). Encoding and decoding in fMRI. NeuroImage 56 400–410.
  • Olman, C. A., Ugurbil, K., Schrater, P. and Kersten, D. (2004). BOLD fMRI and psychophysical measurements of contrast response to broadband images. Vision Research 44 669–683.
  • Olshausen, B. A. and Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381 607–609.
  • Radic, P. (1995). A small step for the cell, a giant leap for mankind: A hypothesis of neocortical expansion during evolution. Trends in Neurosciences 18 383–388.
  • Raizada, R. D. S., Tsao, F.-M., Liu, H.-M. and Kuhl, P. K. (2010). Quantifying the adequacy of neural representations for a cross-language phonetic discrimination task: Prediction of individual differences. Cerebral Cortex 20 1–12.
  • Ravikumar, P., Vu, V. Q., Yu, B., Naselaris, T., Kay, K. and Gallant, J. (2009a). Nonparametric sparse hierarchical models describe V1 fMRI responses to natural images. In Advances in Neural Information Processing Systems ( D. Koller, D. Schuurmans, Y. Bengio and L. Bottou, eds.) 21 1337–1344. Curran Associates, Inc., Redhook, NY.
  • Ravikumar, P., Lafferty, J., Liu, H. and Wasserman, L. (2009b). Sparse additive models. J. Roy. Statist. Soc. Ser. B 71 1009–1030.
  • Sclar, G., Maunsell, J. H. R. and Lennie, P. (1990). Coding of image contrast in central visual pathways of the macaque monkey. Vision Research 30 1–10.
  • Sharpee, T. O., Miller, K. D. and Stryker, M. P. (2008). On the importance of static nonlinearity in estimating spatiotemporal neural filters with natural stimuli. Journal of Neurophysiology 99 2496–2509.
  • Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267–288.
  • Touryan, J., Lau, B. and Dan, Y. (2002). Isolation of relevant visual features from random stimuli for cortical complex cells. Journal of Neuroscience 22 10811–10818.
  • Van Essen, D. C. (1997). A tension-based theory of morphogenesis and compact wiring in the central nervous system. Nature 385 313–318.
  • Vinje, W. E. and Gallant, J. L. (2000). Sparse coding and decorrelation in primary visual cortex during natural vision. Science 287 1273–1276.
  • Walther, D. B., Caddigan, E., Fei-Fei, L. and Beck, D. M. (2009). Natural scene categories revealed in distributed patterns of activity in the human brain. Journal of Neuroscience 29 10573–10581.
  • Williams, M. A., Dang, S. and Kanwisher, N. G. (2007). Only some spatial patterns of fMRI response are read out in task performance. Nature Neuroscience 10 685–686.
  • Zhang, K. and Sejnowski, T. J. (1999). Neuronal tuning: To sharpen or broaden? Neural Comput. 11 75–84.