Open Access
December 2022 Informative Bayesian Neural Network Priors for Weak Signals
Tianyu Cui, Aki Havulinna, Pekka Marttinen, Samuel Kaski
Author Affiliations +
Bayesian Anal. 17(4): 1121-1151 (December 2022). DOI: 10.1214/21-BA1291


Encoding domain knowledge into the prior over the high-dimensional weight space of a neural network is challenging but essential in applications with limited data and weak signals. Two types of domain knowledge are commonly available in scientific applications: 1. feature sparsity (fraction of features deemed relevant); 2. signal-to-noise ratio, quantified, for instance, as the proportion of variance explained. We show how to encode both types of domain knowledge into the widely used Gaussian scale mixture priors with Automatic Relevance Determination. Specifically, we propose a new joint prior over the local (i.e., feature-specific) scale parameters that encodes knowledge about feature sparsity, and a Stein gradient optimization to tune the hyperparameters in such a way that the distribution induced on the model’s proportion of variance explained matches the prior distribution. We show empirically that the new prior improves prediction accuracy compared to existing neural network priors on publicly available datasets and in a genetics application where signals are weak and sparse, often outperforming even computationally intensive cross-validation for hyperparameter tuning.

Funding Statement

This work was supported by the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence, FCAI, grants 319264, 292334, 286607, 294015, 336033, 315896, 341763), and EU Horizon 2020 (INTERVENE, grant no. 101016775). We also acknowledge the computational resources provided by the Aalto Science-IT Project from Computer Science IT.


Download Citation

Tianyu Cui. Aki Havulinna. Pekka Marttinen. Samuel Kaski. "Informative Bayesian Neural Network Priors for Weak Signals." Bayesian Anal. 17 (4) 1121 - 1151, December 2022.


Published: December 2022
First available in Project Euclid: 27 September 2021

MathSciNet: MR4506024
Digital Object Identifier: 10.1214/21-BA1291

Keywords: informative prior , neural network , proportion of variance explained , Sparsity

Vol.17 • No. 4 • December 2022
Back to Top