Introduction to Random Variables and Statistical Distributions¶

Discrete random variables¶

A discrete random variable can take a finite number of possible values. A toss of a two-sided coin can be thought of as a random variable, or a roll of a 6 sided dice.

Probability distributions for discrete random variables¶

A random variable is generated from a probability distribution. There are different types of distributions defined for discrete random variables. These include:

• Bernoulli distribution
• Binomial distribution
• Multinoulli distribution
• Poisson distribution

Bernoulli distribution¶

Bernoulli distribution represents a binary random variable which takes value 1 with success probability $p$ and value 0 with failure probability $q=1-p$. A bernoulli distribution has only one parameter: $p$.

Binomial distribution¶

Another popular distribution for a discrete random variable is the binomial distribution. A binomial distribution has two parameters $n$ and $\theta$, where $0 \le \theta \le 1$. The sample generated by a binomial distribution denotes the number of successes observed in a sequence of $n$ binary trials (e.g., toss of a coin) when the probability of each success is $\theta$.

The samples that are drawn from a binomial distribution range between 0 and $n$.

The probability distribution is defined as: \begin{equation} p(k;n,\theta) = P(X = k) = \binom{n}{k}\theta^k (1 - \theta)^{n-k} \end{equation}

Multinoulli distribution¶

A multinoulli distribution is a generalization of Bernoulli distribution for trials which can take more than two possibilities ($k > 2$). The parameter for multinoulli distribution is a vector ${\bf \theta}$ which has $k$ entries. Each entry $\theta_i$ indicates the probability of observing the category $i$ in a single trial.

Multinomial distribution¶

A multinomial distribution is a generalization of Binomial distribution for trials which can take more than two possibilities. The parameters for the multinomial distribution is a vector ${\bf \theta}$ and $n$.

Poisson distribution¶

Typically used to model counts. Domain ($\chi$) is $0 ... \infty$. It has one parameter, $\lambda$.

The probability mass function is given as: $P(X = k) = \frac{\lambda^ke^{-\lambda}}{k!}$.

The expected value or mean of $X$ is $\lambda$.

Continuous random variables¶

A continuous random variable can take an infinite number of possible values. Several interesting distributions exist:

• alpha An alpha continuous random variable.
• beta A beta continuous random variable.
• gamma A gamma continuous random variable.
• expon An exponential continuous random variable.
• gauss Gaussian random variable

Gaussian distribution¶

One of the most popular distribution is the Gaussian distribution. This distribution is defined for any number of variables. For a single variable case, the distribution is defined using two parameters: $\mu$ and $\sigma$. $\mu$ or the mean can take any value and $\sigma$ or the standard deviation is $\ge 0$.

For a continuous distribution, you cannot compute the probability mass at any value of the random variable. Instead, you can compute the density using the probability density function: $$p(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp[-\frac{1}{2}(\frac{x - \mu}{\sigma})^2]$$ The random variable represented using a Gaussian distribution can take any value from $-\infty$ to $\infty$.

Real-world example¶

Consider heights and weights of a population sample

Multi-dimensional or multivariate Gaussian distribution¶

A distribution can be defined for multivariate random variables. One example is the multivariate Gaussian. In general, the random variable is a $D$ length vector ${\bf X}$. The two parameters of this distribution are a mean vector ${\bf \mu}$ and a covariance matrix $\Sigma$. The pdf at any value of ${\bf X}$ is given by: $$ \mathcal{N}({\bf X}| {\bf \mu,\Sigma}) \triangleq \frac{1}{(2\pi)^{D/2}|{\bf \Sigma}|^{D/2}}exp\left[-\frac{1}{2}{\bf (x-\mu)^\top\Sigma^{-1}(x-\mu)}\right] $$ Note that if $D = 1$, it reduces to a univariate Gaussian distribution.