Continuous Data and the Gaussian Distribution

  • [1] (##) We are given an IID data set $D = \{x_1,x_2,\ldots,x_N\}$, where $x_n \in \mathbb{R}^M$. Let's assume that the data were drawn from a multivariate Gaussian (MVG), $$\begin{align*} p(x_n|\theta) = \mathcal{N}(x_n|\,\mu,\Sigma) = |2 \pi \Sigma|^{-\frac{1}{2}} \exp\left\{-\frac{1}{2}(x_n-\mu)^T \Sigma^{-1} (x_n-\mu) \right\} \end{align*}$$
    (a) Derive the log-likelihood of the parameters for these data.
    (b) Derive the maximum likelihood estimates for the mean $\mu$ and variance $\Sigma$ by setting the derivative of the log-likelihood to zero.

(a) Let $\theta ={\mu,\Sigma}$. Then the log-likelihood can be worked out as $$\begin{align*} \log p(D|\theta) &= \log \prod_n p(x_n|\theta) \\ &= \log \prod_n \mathcal{N}(x_n|\mu, \Sigma) \\ &= \log \prod_n (2\pi)^{-M/2} |\Sigma|^{-1/2} \exp\left( -\frac{1}{2}(x_n-\mu)^T \Sigma^{-1}(x_n-\mu)\right) \\ &= \sum_n \left( \log (2\pi)^{-M/2} + \log |\Sigma|^{-1/2} -\frac{1}{2}(x_n-\mu)^T \Sigma^{-1}(x_n-\mu)\right) \\ &\propto \frac{N}{2}\log |\Sigma|^{-1} - \frac{1}{2}\sum_n (x-\mu)^T \Sigma^{-1}(x-\mu) \end{align*}$$ (b) First we take the derivative with respect to the mean. $$\begin{align*} \nabla_{\mu} \log p(D|\theta) &\propto - \sum_n \nabla_{\mu} \left(x_n-\mu \right)^T\Sigma^{-1}\left(x_n-\mu \right) \\ &= - \sum_n \nabla_{\mu} \mathrm{Tr}\left[-2 \mu^T\Sigma^{-1}x_n + \mu^T \Sigma^{-1}\mu \right] \\ &= - \sum_n \left(-2 \Sigma^{-1}x_n + 2\Sigma^{-1}\mu \right) \\ &= \Sigma^{-1} \sum_n (x_n - \mu) \end{align*}$$ Setting the derivative to zeros leads to $\hat{\mu} = \frac{1}{N}\sum_n x_n$. The derivative with respect to covariance is a bit more involved. It's actually easier to compute this by taking the derivative to the precision: $$\begin{align*} \nabla_{\Sigma^{-1}} \log p(D|\theta) &= \nabla_{\Sigma^{-1}} \left( \frac{N}{2} \log |\Sigma| ^{-1} -\frac{1}{2}\sum_n (x_n-\mu)^T \Sigma^{-1} (x_n-\mu)\right) \\ &= \nabla_{\Sigma^{-1}} \left( \frac{N}{2} \log |\Sigma| ^{-1} - \frac{1}{2}\sum_n \mathrm{Tr}\left[(x_n-\mu) (x_n-\mu)^T \Sigma^{-1} \right]\right) \\ &=\frac{N}{2}\Sigma - \frac{1}{2}\sum_n (x_n-\mu) (x_n-\mu)^T \end{align*}$$ Setting the derivative to zero leads to $\hat{\Sigma} = \frac{1}{N}\sum_n (x_n-\hat{\mu}) (x_n-\hat{\mu})^T$.

  • [2] (#) Shortly explain why the Gaussian distribution is often preferred as a prior distribution over other distributions with the same support? You can get this answer straight from the lession notebook. Aside from the computational advantages (operations on distributions tends to make them more Gaussian, and Gaussians tends to remain Gaussians in computational manipulations), the Gaussian distribution is also the maximum-entropy distribution among distributions that are defined over real numbers. This means that there is no distribution with the same variance that assumes less information about its argument.
  • [3] (###) Proof that the Gaussian distribution is the maximum entropy distribution over the reals with specified mean and variance.

    This is a challenging question (e.g., too diffucult for a written exam:) which requires calculus of variations to solve rigorously. We will show how to maximize the entropy functional which is $-\int q(x) \log q(x) \mathrm{d}x$ with the specified constraints. We have three constraints: (1) we require $q(x)$ to be normalized, (2) $\mathbb{E}[x] = m$ and (3) $\mathbb{E}[x^2] = m^2+\sigma^2$, where $m \in \mathbb{R}$ and $\sigma^2 \in \mathbb{R}^{+}$ are arbitrary. Let us write entropy with the given constraints with undetermined multipliers as a Lagrangian $$\begin{align*} L[q] = -\int q(x)\log q(x)\mathrm{d}x + \lambda \left(\int xq(x)\mathrm{d}x - m\right) + \gamma \left(\int x^2q(x)\mathrm{d}x - (\sigma^2+m^2)\right) + \psi \left(\int q(x)\mathrm{d}x -1 \right). \end{align*}$$ We are searching for a distribution in a space of functions that minimizes the above Lagrangian. This is is a functional minimization problem that is defined over a function space as opposed to ordinary ($\mathbb{R}^N$). Even though the computational mechanics are somewhat different the idea is same with ordinary minimization problems. We look at the functional derivative that has a similar interpretation as a gradient(It can be thought of as the derivative of a functional with respect to a function). We want to solve $$\begin{align*} \frac{\delta L[q]}{\delta q} &= 0 \\ -\log q(x) + \psi + \lambda x + \gamma x^2 &= 0 \\ q(x) &= \exp(+\psi +\lambda x + \gamma x^2) \end{align*}$$ where $\frac{\delta L[q]}{\delta q} $ is the functional derivative. We can plug $q(x)$ back into the constraints and solve for the multipliers. Doing that we obtain $\lambda=\frac{m}{\sigma^2}$,$\gamma = -\frac{1}{2\sigma^2}$ and $\psi = -\frac{m^2}{2\sigma^2}-\log \sqrt{2\pi \sigma^2}$. This means the distribution that maximizes entropy, $q(x)$, is a Gaussian distribution.

  • [4] (##) Proof that a linear transformation $z=Ax+b$ of a Gaussian variable $\mathcal{N}(x|\mu,\Sigma)$ is Gaussian distributed as $$ p(z) = \mathcal{N} \left(z \,|\, A\mu+b, A\Sigma A^T \right) $$

First, we show that a linear transformation of a Gaussian is a Gaussian. In general, the transformed distribution of $z=g(x)$ is given by $$ p_Z(z) = \frac{p_X(g^{-1}(z))}{\mathrm{det}[g(z)]}\,.$$ Since the transformation is linear, $\mathrm{det}[g] = \mathrm{det}[A]$, which is independent of $z$, and consequently $p_Z(z)$ has the same functional form as $p_X(x)$, i.e. $p_Z(z)$ is a also Gaussian. The mean and variance can easily be determined by the calculation that we used in question 8 of the Probability Theory exercises. This results in $p(z) = \mathcal{N} \left(z \,|\, A\mu+b, A\Sigma A^T \right)$.

  • [5] (#) Given independent variables $x \sim \mathcal{N}(\mu_x,\sigma_x^2)$ and $y \sim \mathcal{N}(\mu_y,\sigma_y^2)$, what is the PDF for $z = A\cdot(x -y) + b$?

    $z$ is also Gaussian with $$ p_z(z) = \mathcal{N}(z \,|\, A(\mu_x-\mu_y)+b, \, A (\sigma_x^2 + \sigma_y^2) A^T) $$

  • [6] (###) Compute
\begin{equation*} \int_{-\infty}^{\infty} \exp(-x^2)\mathrm{d}x \,. \end{equation*}

For a Gaussian with zero mean and varance equal to $1$ we have $$ \int \frac{1}{\sqrt{2\pi}}\exp(-\frac{1}{2}x^2) = 1 $$ Substitution of $y = \sqrt{2}x$ with $\mathrm{d}x=\sqrt{2}\mathrm{d}y$ will simply lead you to $ \int_{-\infty}^{\infty} \exp(-x^2)\mathrm{d}x=\sqrt{\pi}$. If you don't want to use the result of the Gaussian integral, you can still do this integral, see youtube clip.

In [ ]: