Showing 1–1 of 1 results for author: Smolenski-Jensen, P
-
One Simple Trick to Fix Your Bayesian Neural Network
Authors:
Piotr Tempczyk,
Ksawery SmoczyĆski,
Philip Smolenski-Jensen,
Marek Cygan
Abstract:
One of the most popular estimation methods in Bayesian neural networks (BNN) is mean-field variational inference (MFVI). In this work, we show that neural networks with ReLU activation function induce posteriors, that are hard to fit with MFVI. We provide a theoretical justification for this phenomenon, study it empirically, and report the results of a series of experiments to investigate the effe…
▽ More
One of the most popular estimation methods in Bayesian neural networks (BNN) is mean-field variational inference (MFVI). In this work, we show that neural networks with ReLU activation function induce posteriors, that are hard to fit with MFVI. We provide a theoretical justification for this phenomenon, study it empirically, and report the results of a series of experiments to investigate the effect of activation function on the calibration of BNNs. We find that using Leaky ReLU activations leads to more Gaussian-like weight posteriors and achieves a lower expected calibration error (ECE) than its ReLU-based counterpart.
△ Less
Submitted 26 July, 2022;
originally announced July 2022.