Draft:Bayesian Neural Network

Draft article not currently submitted for review. This is a draft Articles for creation (AfC) submission. It is not currently pending review. While there are no deadlines, abandoned drafts may be deleted after six months. To edit or make changes to this draft, simply click on the "Edit" tab at the top of the window. To be accepted, a draft should: Show the subject qualifies for a Wikipedia article by using multiple sources that meet four criteria. The sources should be (1) reliable (2) secondary (3) independent of the subject (4) talk about the subject in some depth. For some topics, there are alternative criteria. Be written from a neutral point of view Respect copyright and do not plagiarize. Do not copy-paste. It is strongly discouraged to write about either yourself or your business or employer. If you do so, you must declare it. Where to get help If you need help editing or submitting your draft, please ask us a question at the AfC Help Desk or get live help from experienced editors. These venues are only for help with editing and the submission process, not to get reviews. If you need feedback on your draft, or if the review is taking a lot of time, you can try asking for help on the talk page of a relevant WikiProject. Some WikiProjects are more active than others so a speedy reply is not guaranteed. How to improve a draft Wikipedia:Contributing to Wikipedia – a basic overview on how to edit Wikipedia. Help:Wikitext – how to use the markup Help:Referencing for beginners – how to include references Wikipedia:Article development – how to develop your article Wikipedia:Writing better articles – how to improve your article Wikipedia:Verifiability – make sure your article includes reliable third-party sources You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Add tags to your draft Editor resources Easy tools: Citation bot (help) | Advanced: Fix bare URLs Last edited by Citation bot (talk | contribs) 36 days ago. (Update) Submit the draft for review!

A Bayesian Neural Network (BNN) is a neural network, that trains a distribution over its network parameters using Bayesian inference. Once trained it uses this distribution over parameters to predict a probability distribution in the output space for a single input. BNNs are used in applications which require the quantification of uncertainties or in which multimodal distributions are expected that point-estimate predictions cannot express.^[1] The design of a BNN entails the choice of a functional model ${\textstyle m_{\omega }}$ (the neural network, with model parameters ${\textstyle \omega }$), and a stochastic model which contains both priors ${\textstyle p(\omega )}$ and ${\textstyle p(y|x,\omega )}$.^[2]

Not to be confused with Bayesian Network.

The first use of BNNs was in 1993 by Hinton and Van Camp^[3].

From a Bayesian point of view, the model parameters ${\textstyle \omega }$ of a BNN are treated as latent random variables and the training process is their inference, conditional to the (observed) training data ${\textstyle D}$. The distribution of the completely trained model parameters is directly given by Bayes' theorem

${\displaystyle p(\omega |D)={\frac {p(\omega )\,p(D|\omega )}{p(D)}}}$

In practice, the marginal is often intractable, which makes it necessary to adapt strategies to approximate the true posterior. Once the parameter posterior is computed, other quantities of interest can be computed via marginalisation. For example, the predictive posterior is given by

${\displaystyle p(y|x,D)=\int p(y|x,\omega )\,p(\omega |D)d\omega }$

where ${\textstyle p(y|x,\omega )=m_{\omega }(x)}$is provided by the functional model.

The numerical difficulties that come with the computation of Bayes theorem have given rise to a family of BNN approaches, which can roughly be separated into parametric methods (Variational Inference (VI), Bayes by Backprop) and nonparametric methods (free-form VI, Monte Carlo Dropout (MCD), Direct sampling).

A simplification often performed is to introduce a surrogate posterior ${\textstyle q_{\theta }}$ with closed form and minimize its Kullback-Leibler (KL) divergence to the true parameter posterior with respect to the variational parameters ${\textstyle \theta }$. This is possible because the objective function can be rewritten such that the intractable log marginal likelihood ${\textstyle p(D)}$ is separated from the variational term:${\displaystyle {\begin{aligned}{\text{KL}}(q_{\theta }(\omega )\mid \mid p(\omega |D))&={\text{KL}}(q_{\theta }(\omega )\mid \mid p(\omega ))-\mathbb {E} _{q}[\log p(D|\omega )]+\log p(D)\\&=\underbrace {\mathbb {E} _{q}[\log p(D|\omega )-\log q_{\theta }(\omega )+\log p(\omega )]} _{-{\text{ELBO}}}+\log p(D)\end{aligned}}}$

The quantity to maximize therefore becomes the evidence lower bound (ELBO).

Using the reparameterization trick, ...

This allows the reformulation of each Bayesian update to an optimization problem, which can be solved using established gradient descent methods. This method is known as variational inference.

Common choices for the closed form include the mean-field approach, which assumes a complete factorisation of q, and the more general multivariate normal distribution with a low-rank covariance matrix. [Barber and Bishop (Ensemble Learning in Bayesian Neural Networks)] This choice is the equivalent of a loss function in point estimate ML.^[2]

In cases where the computation of the complete log-likelihood becomes infeasible due to the large volume of training data, VI also works with stochastic gradient descent (stochastic variational inference), where in each Bayesian update only a subset of data is used to approximate the likelihood term.

Another approximation technique is Monte Carlo Dropout (MCD)^[4] where conventional dropout layers are kept enabled during inference time. This allows the sampling of predictions.

In contrast to VI, samplers like MCMC converge to the true posterior without assumptions about its form.[Goan]

Often used Hamiltonian Monte Carlo (HMC) or Langevin Monte Carlo

d^[5]

Neural network Gaussian process

...

BNN are usually computation-heavy in training and inference, due to their need to generate many samples from the posterior distribution. For this reason, the VI method is often only suitable in its mean field form, which trades expressiveness for lower computational complexity.^[5] Further, it was found that VI approaches often underestimate the variance of their predictions. Direct sampling using Markov Chain Monte Carlo (MCMC) scales poorly with the amount of data as by default it requires the processing of the entire training dataset to perform an update. Stochastic MCMC methods, which consume only a subset of the data, have been shown to introduce a bias to the posterior.^[5]

VI: Placed assumptions and restrictions about the form may introduce a bias and induce inaccuracies in predictions.

Category:Bayesian statistics