Bayesian Machine Learning¶

Preliminaries¶

• Goals
  • Introduction to Bayesian (i.e., probabilistic) modeling
• Materials
  • Mandatory
    • These lecture notes
  • Optional
    • Bishop pp. 68-74 (on the coin toss example)
    • Ariel Caticha - 2012 - Entropic Inference and the Foundations of Physics, pp.35-44 (section 2.9, on deriving Bayes rule for updating probabilities)

Challenge: Predicting a Coin Toss¶

• Problem: We observe the following sequence of heads (outcome $=1$) and tails (outcome $=0$) when tossing the same coin repeatedly $$D=\{1011001\}\,.$$

• What is the probability that heads comes up next?

• Solution: later in this lecture.

The Bayesian Machine Learning Framework¶

• Suppose that your application is to predict a future observation $x$, based on $N$ past observations $D=\{x_1,\dotsc,x_N\}$.

• The Bayesian design approach to solving this task involves four stages:

• In principle, based on the model evaluation results, you may want to re-specify your model and repeat the design process (a few times), until model performance is acceptable.

• Next, we discuss these four stages in a bit more detail.

(1) Model specification¶

• Your first task is to propose a probabilistic model for generating the observations $x$.

• A probabilistic model $m$ consists of a joint distribution $p(x,\theta|m)$ that relates observations $x$ to model parameters $\theta$. Usually, the model is proposed in the form of a data generating distribution $p(x|\theta,m)$ and a prior $p(\theta|m)$.

• You are responsible to choose the data generating distribution $p(x|\theta)$ based on your physical understanding of the data generating process. (For brevity, if we are working on one given model $m$ with no alternative models, we usually drop the given dependency on $m$ from the notation).

• You must also choose the prior $p(\theta)$ to reflect what you know about the parameter values before you see the data $D$.

(2) Parameter estimation¶

• Note that, for a given data set $D=\{x_1,x_2,\dots,x_N\}$ with independent observations $x_n$, the likelihood factorizes as

$$ p(D|\theta) = \prod_{n=1}^N p(x_n|\theta)\,,$$

so usually you select a model for generating one observation $x_n$ and then use (in-)dependence assumptions to combine these models into a likelihood function for the model parameters.

• The likelihood and prior both contain information about the model parameters. Next, you use Bayes rule to fuse these two information sources into a posterior distribution for the parameters:

$$\begin{align*} \underbrace{p(\theta|D) }_{\text{posterior}} &= \frac{p(D|\theta) p(\theta)}{p(D)} \\ &= \frac{p(D|\theta) p(\theta)}{\int p(D|\theta) p(\theta) \mathrm{d}\theta} \end{align*}$$

• Note that there's no need for you to design some clever parameter estimation algorithm. Bayes rule is the parameter estimation algorithm, which can be entirely expressed in terms of the likelihood and prior. The only complexity lies in the computational issues!

• This parameter estimation "recipe" works if the right-hand side (RHS) factors can be evaluated; the computational details can be quite challenging and this is what machine learning is about.

• $\Rightarrow$ Machine learning is EASY, apart from computational details :)

(3) Model Evaluation¶

• In the framework above, parameter estimation was executed by "perfect" Bayesian reasoning. So is everything settled now?

• No, there appears to be one remaining problem: how good really were our model assumptions $p(x|\theta)$ and $p(\theta)$? We want to "score" the model performance.

• Note that this question is only interesting in practice if we have alternative models to choose from. After all, if you don't have an alternative model, any value for the model evidence would still not lead you to switch to another model.

• Let's assume that we have more candidate models, say $\mathcal{M} = \{m_1,\ldots,m_K\}$ where each model relates to specific prior $p(\theta|m_k)$ and likelihood $p(D|\theta,m_k)$? Can we evaluate the relative performance of a model against another model from the set?

• Start again with model specification. You must now specify a prior $p(m_k)$ (next to the likelihood $p(D|\theta,m_k)$ and prior $p(\theta|m_k)$) for each of the models and then solve the desired inference problem:

$$\begin{align*} \underbrace{p(m_k|D)}_{\substack{\text{model}\\\text{posterior}}} &= \frac{p(D|m_k) p(m_k)}{p(D)} \\ &\propto p(m_k) \cdot p(D|m_k) \\ &= p(m_k)\cdot \int_\theta p(D,\theta|m_k) \,\mathrm{d}\theta\\ &= \underbrace{p(m_k)}_{\substack{\text{model}\\\text{prior}}}\cdot \underbrace{\int_\theta \underbrace{p(D|\theta,m_k)}_{\text{likelihood}} \,\underbrace{p(\theta|m_k)}_{\text{prior}}\, \mathrm{d}\theta }_{\substack{\text{evidence }p(D|m_k)\\\text{= model likelihood}}}\\ \end{align*}$$

• You can evaluate the RHS of this equation since you selected the model priors $p(m_k)$, the parameter priors $p(\theta|m_k)$ and the likelihoods $p(D|\theta,m_k)$.

• You can now compare posterior distributions $p(m_k|D)$ for a set of models $\{m_k\}$ and decide on the merits of each model relative to alternative models. This procedure is called Bayesian model comparison.

• Note that, to evaluate the model posterior, you must calculate the "model evidence" $p(D|m_k)$, which can be interpreted as a likelihood function for model $m_k$.

• $\Rightarrow$ In a Bayesian framework, model estimation follows the same recipe as parameter estimation; it just works at one higher hierarchical level. Compare the required calulations:

$$\begin{align*} p(\theta|D) &\propto p(D|\theta) p(\theta) \; &&\text{(parameter estimation)} \\ p(m_k|D) &\propto p(D|m_k) p(m_k) \; &&\text{(model comparison)} \end{align*}$$

• Again, no need to invent a special algorithm for estimating the performance of your model. Straightforward application of probability theory takes care of all that.

• In principle, you could proceed with asking how good your choice for the candidate model set $\mathcal{M}$ was. You would have to provide a set of alternative model sets $\{\mathcal{M}_1,\mathcal{M}_2,\ldots,\mathcal{M}_M\}$ with priors $p(\mathcal{M}_m)$ for each set and compute posteriors $p(\mathcal{M}_m|D)$. And so forth ...

• With the (relative) performance evaluation scores of your model in hand, you could now re-specify your model (hopefully an improved model) and repeat the design process until the model performance score is acceptable.

Bayes Factors¶

• As an aside, in the (statistics and machine learning) literature, performance comparison between two models is often reported by the Bayes Factor, which is defined as the ratio of model evidences:

$$\begin{align*} \underbrace{\frac{p(D|m_1)}{p(D|m_2)}}_{\text{Bayes Factor}} &= \frac{\frac{p(D,m_1)}{p(m_1)}}{\frac{p(D,m_2)}{p(m_2)}} \\ &= \frac{p(D,m_1)}{p(m_1)} \cdot \frac{p(m_2)}{p(D,m_2)} \\ &= \frac{p(m_1|D) p(D)}{p(m_1)} \cdot \frac{p(m_2)}{p(m_2|D) p(D)} \\ &= \underbrace{\frac{p(m_1|D)}{p(m_2|D)}}_{\substack{\text{posterior} \\ \text{ratio}}} \cdot \underbrace{\frac{p(m_2)}{p(m_1)}}_{\substack{\text{prior} \\ \text{ratio}}} \end{align*}$$

• Hence, for equal model priors ($p(m_1)=p(m_2)=0.5$), the Bayes Factor reports the posterior probability ratio for the two models.

• In principle, any hard decision on which is the better model has to accept some ad hoc arguments, but Jeffreys (1961) advises the following interpretation of the log-Bayes factor

$$\log_{10} B_{12} =\log_{10}\frac{p(D|m_1)}{p(D|m_2)}$$

$\log_{10} B_{12}$	Evidence for $m_1$
0 to 0.5	not worth mentioning
0.5 to 1	substantial
1 to 2	strong
>2	decisive

(4) Prediction¶

• Once we are satisfied with the evidence for a (trained) model, we can apply the model to our prediction/classification/etc task.

• Given the data $D$, our knowledge about the yet unobserved datum $x$ is captured by (everything is conditioned on the selected model)

$$\begin{align*} p(x|D) &\stackrel{s}{=} \int p(x,\theta|D) \,\mathrm{d}\theta\\ &\stackrel{p}{=} \int p(x|\theta,D) p(\theta|D) \,\mathrm{d}\theta\\ &\stackrel{m}{=} \int \underbrace{p(x|\theta)}_{\text{data generation dist.}} \cdot \underbrace{p(\theta|D)}_{\text{posterior}} \,\mathrm{d}\theta\\ \end{align*}$$

• In the last equation, the simplification $p(x|\theta,D) = p(x|\theta)$ follows from our model specification. We assumed a parametric data generating distribution $p(x|\theta)$ with no explicit dependency on the data set $D$.

• Again, no need to invent a special prediction algorithm. Probability theory takes care of all that. The complexity of prediction is just computational: how to carry out the marginalization over $\theta$.

• Note that the application of the learned posterior $p(\theta|D)$ not necessarily has to be a prediction task. We use it here as an example, but other applications (e.g., classification, regression etc.) are of course also possible.

Prediction with multiple models¶

• When you have a posterior $p(m_k|D)$ for the models, you don't need to choose one model for the prediction task. You can do prediction by Bayesian model averaging, which combines the predictive power from all models:

$$\begin{align*} p(x|D) &= \sum_k \int p(x,\theta,m_k|D)\,\mathrm{d}\theta \\ &= \sum_k \int p(x|\theta,m_k) \,p(\theta|m_k,D)\, p(m_k|D) \,\mathrm{d}\theta \\ &= \sum_k \underbrace{p(m_k|D)}_{\substack{\text{model}\\\text{posterior}}} \cdot \int \underbrace{p(\theta|m_k,D)}_{\substack{\text{parameter}\\\text{posterior}}} \, \underbrace{p(x|\theta,m_k)}_{\substack{\text{data generating}\\\text{distribution}}} \,\mathrm{d}\theta \end{align*}$$

• Alternatively, if you do need to work with one model (e.g. due to computational resource constraints), you can for instance select the model with largest posterior $p(m_k|D)$ and use that model for prediction. This is called Bayesian model selection.

• Bayesian model averaging is the principal way to apply PT to machine learning. You don't throw away information by discarding lesser performant models, but rather use PT (marginalization of models) to compute $$p(\text{what-I-am-interested-in} \,|\, \text{all available information})$$ exactly.

We're Done!¶

• In principle, you now have the recipe in your hands now to solve all your prediction/classification/regression etc problems by the same method:

  1. specify a model
  2. train the model (by PT)
  3. evaluate the model (by PT); if not satisfied, goto 1
  4. apply the model (by PT)

• Crucially, there is no need to invent clever machine learning algorithms, and there is no need to invent a clever prediction algorithm nor a need to invent a model performance criterion. Instead, you propose a model and, from there on, you let PT reason about everything that you care about.

• Your problems are only of computational nature. Perhaps the integral to compute the evidence may not be analytically tractable, etc.

Bayesian Evidence as a Model Performance Criterion¶

• I'd like to convince you that Bayesian model evidence is an excellent criterion for assessing your model's performance. To do so, let us consider a decomposition that relates model evidence to other highly-valued criteria such as accuracy and model complexity.

• Consider a model $p(x,\theta|m)$ and a data set $D = \{x_1,x_2, \ldots,x_N\}$.

• Given the data set $D$, the log-evidence for model $m$ decomposes as follows (please check the derivation):

$$\begin{align*} \underbrace{\log p(D|m)}_{\text{log-evidence}} &= \log p(D|m) \cdot \underbrace{\int p(\theta|D,m)\mathrm{d}\theta}_{\text{evaluates to }1} \\ &= \int p(\theta|D,m) \log p(D|m) \mathrm{d}\theta \qquad \text{(move $\log p(D|m)$ into the integral)} \\ &= \int p(\theta|D,m) \log \underbrace{\frac{p(D|\theta,m) p(\theta|m)}{p(\theta|D,m)}}_{\text{by Bayes rule}} \mathrm{d}\theta \\ &= \underbrace{\int p(\theta|D,m) \log p(D|\theta,m) \mathrm{d}\theta}_{\text{accuracy (a.k.a. data fit)}} - \underbrace{\int p(\theta|D,m) \log \frac{p(\theta|D,m)}{p(\theta|m)} \mathrm{d}\theta}_{\text{complexity}} \end{align*}$$

• The "accuracy" term (also known as data fit) measures how well the model predicts the data set $D$. We want this term to be high because good models should predict the data $D$ well. Indeed, higher accuracy leads to higher model evidence. To achieve high accuracy, applying Bayes' rule will shift the posterior $p(\theta|D)$ away from the prior towards the likelihood function $p(D|\theta)$.

• The second term (complexity) is technically a Kullback-Leibler divergence (KLD) between the posterior and prior distributions, see OPTIONAL SLIDE. The KLD is an information-theoretic quantity that can be interpreted as a "distance" measure between two distributions. In other words, the complexity term measures how much the beliefs about $\theta$ changed, due to learning from the data $D$. Generally, we like the complexity term to be low, because moving away means forgetting previously acquired information represented by the prior. Indeed, lower complexity leads to higher model evidence.

• Models with high evidence $p(D|m)$ prefer both high accuracy and low complexity. Therefore, models with high evidence tend to predict the training data $D$ well (high accuracy), yet also try to preserve the information encoded by the prior (low complexity). These types of models are said to generalize well, since they can be applied to different data sets without specific adaptations for each data set.

• Focussing only on accuracy maximization could lead to overfitting. Focussing only on complexity minimization could lead to underfitting of the model. Bayesian ML attends to both terms and avoids both underfitting and overfitting.

• $\Rightarrow$ Bayesian learning automatically leads to models that generalize well. There is no need for early stopping or validation data sets. There is also no need for tuning parameters in the learning process. Just learn on the full data set and all behaves well.

• This latter point accentuates that the common practice in machine learning to divide a data set into a training, test and validation set is just an ad hoc mechanism that compensates for failing to frame the learning task as a Bayesian inference task.

Bayesian Machine Learning and the Scientific Method Revisited¶

• The Bayesian design process provides a unified framework for the Scientific Inquiry method. We can now add equations to the design loop. (Trial design to be discussed in Intelligent Agent lesson.)

Revisiting the Challenge: Predicting a Coin Toss¶

At the beginning of this lesson, we posed the following challenge:

• We observe a the following sequence of heads (outcome = $1$) and tails (outcome = $0$) when tossing the same coin repeatedly $$D=\{1011001\}\,.$$

• What is the probability that heads comes up next? We solve this in the next slides ...

Coin toss example (1): Model Specification¶

• We observe a sequence of $N$ coin tosses $D=\{x_1,\ldots,x_N\}$ with $n$ heads.

• Let us denote outcomes by

$$x_k = \begin{cases} 1 & \text{if heads comes up} \\ 0 & \text{otherwise (tails)} \end{cases} $$

Likelihood¶

• Assume a Bernoulli distributed variable $p(x_k=1|\mu)=\mu$ for a single coin toss, leading to

$$p(x_k|\mu)=\mu^{x_k} (1-\mu)^{1-x_k} \,.$$

• Assume $n$ times heads were thrown out of a total of $N$ throws. The likelihood function then follows a binomial distribution :

$$ p(D|\mu) = \prod_{k=1}^N p(x_k|\mu) = \mu^n (1-\mu)^{N-n} $$

Prior¶

• Assume the prior beliefs for $\mu$ are governed by a beta distribution

$$ p(\mu) = \mathrm{Beta}(\mu|\alpha,\beta) = \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} \mu^{\alpha-1}(1-\mu)^{\beta-1} $$

where the Gamma function is sort-of a generalized factorial function. In particular, if $\alpha,\beta$ are integers, then $$\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} = \frac{(\alpha+\beta-1)!}{(\alpha-1)!\,(\beta-1)!}$$

• A what distribution? Yes, the beta distribution is a conjugate prior for the binomial distribution, which means that

$$ \underbrace{\text{beta}}_{\text{posterior}} \propto \underbrace{\text{binomial}}_{\text{likelihood}} \times \underbrace{\text{beta}}_{\text{prior}} $$

so we get a closed-form posterior.

• $\alpha$ and $\beta$ are called hyperparameters, since they parameterize the distribution for another parameter ($\mu$). E.g., $\alpha=\beta=1$ leads to a uniform prior for $\mu$. Some other choices for $\alpha, \beta$ and the resulting prior for $\mu$ are shown in the following graph:

• (Bishop Fig.2.2). Plots of the beta distribution $\mathrm{Beta}(\mu|a, b)$ as a function of $\mu$ for various values of the hyperparameters $a$ and $b$.

• Before observing any data, you can express your state-of-knowledge about the coin by choosing values for $\alpha$ and $\beta$ that reflect your beliefs. Stronger yet, you must choose values for $\alpha$ and $\beta$, because the Bayesian framework does not allow you to walk away from your responsibility to explicitly state your beliefs before the experiment.

Coin toss example (2): Parameter estimation¶

• Infer posterior PDF over $\mu$ (and evidence) through Bayes rule

$$\begin{align*} p(\mu&|D) \cdot p(D) = p(D|\mu)\cdot p(\mu) \\ &= \underbrace{\biggl( \mu^n (1-\mu)^{N-n}\biggr)}_{\text{likelihood}} \cdot \underbrace{\biggl( \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} \mu^{\alpha-1}(1-\mu)^{\beta-1} \biggr)}_{\text{prior}} \\ &= \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} \mu^{n+\alpha-1} (1-\mu)^{N-n+\beta-1} \\ &= \underbrace{\biggl(\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} \frac{\Gamma(n+\alpha) \Gamma(N-n+\beta)}{\Gamma(N+\alpha+\beta)}\biggr)}_{\text{evidence }p(D)} \cdot \underbrace{\biggl( \frac{\Gamma(N+\alpha+\beta)}{\Gamma(n+\alpha)\Gamma(N-n+\beta)} \mu^{n+\alpha-1} (1-\mu)^{N-n+\beta-1}\biggr)}_{\text{posterior }p(\mu|D)=\mathrm{Beta}(\mu|n+\alpha, N-n+\beta)} \end{align*}$$

hence the posterior is also beta-distributed as

$$ p(\mu|D) = \mathrm{Beta}(\mu|\,n+\alpha, N-n+\beta) $$

Coin toss example (3): Model Evaluation¶

• It follow from the above calculation that the evidence for model $m$ can be analytically expressed as

$$ p(D|m) = \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} \frac{\Gamma(n+\alpha) \Gamma(N-n+\beta)}{\Gamma(N+\alpha+\beta)} $$

• The model evidence is a scalar. The absolute value is not important. However, you may want to compare the model evidence of this model to the evidence for another model on the same data set.

Coin Toss Example (4): Prediction¶

• Once we have accepted a model, let's apply it to the application, in this case, predicting future observations.

• Marginalize over the parameter posterior to get the predictive PDF for a new coin toss $x_\bullet$, given the data $D$,

$$\begin{align*} p(x_\bullet=1|D) &= \int_0^1 p(x_\bullet=1|\mu)\,p(\mu|D) \,\mathrm{d}\mu \\ &= \int_0^1 \mu \times \mathrm{Beta}(\mu|\,n+\alpha, N-n+\beta) \,\mathrm{d}\mu \\ &= \frac{n+\alpha}{N+\alpha+\beta} \end{align*}$$

• This result is known as Laplace's rule of succession.

• The above integral computes the mean of a beta distribution, which is given by $\mathbb{E}[x] = \frac{a}{a+b}$ for $x \sim \mathrm{Beta}(a,b)$, see wikipedia.

• Finally, we're ready to solve our challenge: for $D=\{1011001\}$ and uniform prior ($\alpha=\beta=1$), we get

$$ p(x_\bullet=1|D)=\frac{n+1}{N+2} = \frac{4+1}{7+2} = \frac{5}{9}$$

• In other words, given the model assumptions (the Bernoulli data-generating distribution and Beta prior as specified above), and the observations $D=\{1011001\}$, the probability for observing heads (outcome=$1$) on the next toss is $\frac{5}{9}$.

• Be aware that there is no such thing as an "objective" or "correct" prediction. Every prediction is conditional on the selected model and the used data set.

Coin Toss Example: What did we learn from the data?¶

• What did we learn from the data? Before seeing any data, we think that the probability for throwing heads is $$\left. p(x_\bullet=1|D) \right|_{n=N=0} = \left.\frac{n+\alpha}{N+\alpha+\beta}\right|_{n=N=0} = \frac{\alpha}{\alpha + \beta}\,.$$

• Hence, $\alpha$ and $\beta$ (to be precise: $\alpha-1$ and $\beta-1$) can be interpreted as prior pseudo-counts for heads and tails, respectively.

• If we were to assume zero pseudo-counts, i.e. $\alpha=\beta=1$, then our prediction for throwing heads after $N$ coin tosses is completely based on the data, given by

$$\left. p(x_\bullet=1|D) \right|_{\alpha=\beta=1} = \left.\frac{n+\alpha}{N+\alpha+\beta}\right|_{\alpha=\beta=1} = \frac{n}{N}\,.$$

• Note the following decomposition

$$\begin{align*} p(x_\bullet=1|\,D) &= \frac{n+\alpha}{N+\alpha+\beta} \\ &= \frac{\alpha}{N+\alpha+\beta} + \frac{n}{N+\alpha+\beta} \\ &= \frac{\alpha}{N+\alpha+\beta}\cdot \frac{\alpha+\beta}{\alpha+\beta} + \frac{n}{N+\alpha+\beta}\cdot \frac{N}{N} \\ &= \frac{\alpha}{\alpha+\beta}\cdot \frac{\alpha+\beta}{N+\alpha+\beta} + \frac{N}{N+\alpha+\beta}\cdot \frac{n}{N} \\ &= \frac{\alpha}{\alpha+\beta}\cdot \biggl(1-\frac{N}{N+\alpha+\beta} \biggr) + \frac{N}{N+\alpha+\beta}\cdot \frac{n}{N} \\ &= \underbrace{\frac{\alpha}{\alpha+\beta}}_{\substack{\text{prior}\\\text{prediction}}} + \underbrace{\underbrace{\frac{N}{N+\alpha+\beta}}_{\text{gain}}\cdot \underbrace{\biggl( \underbrace{\frac{n}{N}}_{\substack{\text{data-based}\\\text{prediction}}} - \underbrace{\frac{\alpha}{\alpha+\beta}}_{\substack{\text{prior}\\\text{prediction}}} \biggr)}_{\text{prediction error}}}_{\text{correction}} \end{align*}$$

• Let's interpret this decomposition of the posterior prediction. Before the data $D$ was observed, our model generated a prior prediction $ p(x_\bullet=1) = \frac{\alpha}{\alpha+\beta}$. Next, the degree to which the actually observed data matches this prediction is represented by the prediction error $ \frac{n}{N} - \frac{\alpha}{\alpha-\beta}$. The prior prediction is then updated to a posterior prediction $p(x_\bullet=1|D)$ by adding a fraction of the prediction error to the prior prediction. Hence, the data plays the role of "correcting" the prior prediction.

• Note that, since $0\leq \underbrace{\frac{N}{N+\alpha+\beta}}_{\text{gain}} \lt 1$, the Bayesian prediction lies between (fuses) the prior and data-based predictions.

• For large $N$, the gain goes to $1$ and $\left. p(x_\bullet=1|D)\right|_{N\rightarrow \infty} \rightarrow \frac{n}{N}$ goes to the data-based prediction (the observed relative frequency).

Code Example: Bayesian evolution for the coin toss¶

• Next, we code an example for a sequence of coin tosses, where we assume that the true coin generates data $x_n \in \{0,1\}$ by a Bernoulli distribution:

$$ p(x_n|\mu=0.4)=0.4^{x_n} \cdot 0.6^{1-x_n} $$

• So, this coin is biased!

• In order predict the outcomes of future coin tosses, we'll compare two models.

• All models have the same data generating distribution (also Bernoulli)

$$ p(x_n|\mu,m_k) = \mu^{x_n} (1-\mu)^{1-x_n} \quad \text{for }k=1,2 $$

but they have different priors: $$\begin{aligned} p(\mu|m_1) &= \mathrm{Beta}(\mu|\alpha=100,\beta=500) \\ p(\mu|m_2) &= \mathrm{Beta}(\mu|\alpha=8,\beta=13) \\ \end{aligned}$$

• You can verify that model $m_2$ has the best prior, since

$$\begin{align*} p(x_n=1|m_1) &= \left.\frac{\alpha}{\alpha+\beta}\right|_{m_1} = 100/600 \approx 0.17 \\ p(x_n=1|m_2) &= \left.\frac{\alpha}{\alpha+\beta}\right|_{m_2} = 8/21 \approx 0.38 \,, \end{align*}$$

( but you are not supposed to know that the real coin has probability for heads $p(x_n=1|\mu) = 0.4$ ).

• Let's run $500$ tosses:

• For each model, as a function of the number of coin tosses, we plot the evolution of the parameter posteriors

$$p(\mu|D_n,m_\bullet) $$

(This is an animation. If you are viewing these notes in PDF format you can see the animation at https://nbviewer.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Bayesian-Machine-Learning.ipynb .)

• Note that both posteriors move toward the "correct" value ($\mu=0.4$). However, the posterior for $m_1$ (blue) moves much slower because we assumed far more pseudo-observations for $m_1$ than for $m_2$.
• As we get more observations, the influence of the prior diminishes.

• We have an intuition that $m_2$ is superior over $m_1$. Let's check this by plotting over time the relative Bayesian evidences for each model:

$$\frac{p(D_n|m_i)}{\sum_{i=1}^2 p(D_n|m_i)}$$

• Over time, the relative evidence of model $m_1$ converges to 0. Can you explain this behavior?

From Posterior to Point-Estimate¶

• In the example above, Bayesian parameter estimation and prediction were tractable in closed-form. This is often not the case. We will need to approximate some of the computations.

• Recall Bayesian prediction

$$ p(x|D) = \int p(x|\theta)p(\theta|D)\,\mathrm{d}{\theta} $$

• If we approximate posterior $p(\theta|D)$ by a delta function for one 'best' value $\hat\theta$, then the predictive distribution collapses to

$$ p(x|D)= \int p(x|\theta)\,\delta(\theta-\hat\theta)\,\mathrm{d}{\theta} = p(x|\hat\theta) $$

• This is just the data generating distribution $p(x|\theta)$ evaluated at $\theta=\hat\theta$, which is easy to evaluate.

• The next question is how to get the parameter estimate $\hat{\theta}$? (See next slide).

Some Well-known Point-Estimates¶

• Bayes estimate (the mean of the posterior)

$$ \hat \theta_{bayes} = \int \theta \, p\left( \theta |D \right) \,\mathrm{d}{\theta} $$

• Maximum A Posteriori (MAP) estimate

$$ \hat \theta_{\text{map}}= \arg\max _{\theta} p\left( \theta |D \right) = \arg \max_{\theta} p\left(D |\theta \right) \, p\left(\theta \right) $$

• Maximum Likelihood (ML) estimate

$$ \hat \theta_{ml} = \arg \max_{\theta} p\left(D |\theta\right) $$

• Note that Maximum Likelihood is MAP with uniform prior
• ML is the most common approximation to the full Bayesian posterior.

Bayesian vs Maximum Likelihood Learning¶

Consider the task: predict a datum $x$ from an observed data set $D$.

	Bayesian	Maximum Likelihood
1. Model Specification	Choose a model $m$ with data generating distribution $p(x|\theta,m)$ and parameter prior $p(\theta|m)$	Choose a model $m$ with same data generating distribution $p(x|\theta,m)$. No need for priors.
2. Learning	use Bayes rule to find the parameter posterior, $$ p(\theta|D) \propto p(D|\theta) p(\theta) $$	By Maximum Likelihood (ML) optimization, $$ \hat \theta = \arg \max_{\theta} p(D |\theta) $$
3. Prediction	$$ p(x|D) = \int p(x|\theta) p(\theta|D) \,\mathrm{d}\theta $$	$$ p(x|D) = p(x|\hat\theta) $$

Report Card on Maximum Likelihood Estimation¶

• Maximum Likelihood (ML) is MAP with uniform prior. MAP is sometimes called a 'penalized' ML procedure:

$$ \hat \theta_{map} = \arg \max _\theta \{ \underbrace{\log p\left( D|\theta \right)}_{\text{log-likelihood}} + \underbrace{\log p\left( \theta \right)}_{\text{penalty}} \} $$

• (good!). ML works rather well if we have a lot of data because the influence of the prior diminishes with more data.

• (good!). Computationally often do-able. Useful fact that makes the optimization easier (since $\log$ is monotonously increasing):

$$\arg\max_\theta \log p(D|\theta) = \arg\max_\theta p(D|\theta)$$

• (bad). Cannot be used for model comparison! When doing ML estimation, the Bayesian model evidence always evaluates to zero because the prior probability mass under the likelihood function goes to zero. Therefore, when doing ML estimation, Bayesian model evidence cannot be used to evaluate model performance:

$$\begin{align*} \underbrace{p(D|m)}_{\substack{\text{Bayesian}\\ \text{evidence}}} &= \int p(D|\theta) \cdot p(\theta|m)\,\mathrm{d}\theta \\ &= \lim_{(b-a)\rightarrow \infty} \int p(D|\theta)\cdot \text{Uniform}(\theta|a,b)\,\mathrm{d}\theta \\ &= \lim_{(b-a)\rightarrow \infty} \frac{1}{b-a}\underbrace{\int_a^b p(D|\theta)\,\mathrm{d}\theta}_{<\infty} \\ &= 0 \end{align*}$$

• In fact, this is a serious problem because Bayesian evidence is fundamentally the correct performance assessment criterion that follows from straighforward PT. In practice, when estimating parameters by maximum likelihood, we often evaluate model performance by an ad hoc performance measure such as mean-squared-error on a testing data set.

$\Rightarrow$ Maximum Likelihood estimation is at best an approximation to Bayesian learning, but for good reason a very popular learning method when faced with lots of available data.

OPTIONAL SLIDES¶

The Kullback-Leibler Divergence¶

• The Kullback-Leibler Divergence (a.k.a. relative entropy) between two distributions $q$ and $p$ is defined as

$$ D_{\text{KL}}[q,p] \equiv \sum_z q(z) \log \frac{q(z)}{p(z)} $$

• The following Gibbs Inequality holds (see wikipedia for proof): $$D_{\text{KL}}[q,p] \geq 0 \quad \text{with equality only if } p=q $$

• The KL divergence can be interpreted as a distance between two probability distributions.

• As an aside, note that $D_{\text{KL}}[q,p] \neq D_{\text{KL}}[p,q]$. Both divergences are relevant.

• Here is an animation that shows the KL divergence between two Gaussian distributions:

• Note that this is an animation. If you are viewing these notes in PDF format you can see the animation at https://nbviewer.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Bayesian-Machine-Learning.ipynb