ECE 046211- Technion - Deep Learning

---

Tal Daniel¶

Tutorial 02 - Single Neuron¶

---

Agenda

---

• Discriminative Models
• The Perceptron
• Logistic Regression
  • Logistic Regression with PyTorch
• Multi-Class (Multinomial) Logistic Regression - Softmax Regression
• Activation Functions
• Recommended Videos
• Credits

Additional Packages for Google Colab

---

If you are using Google Colab, you have to install additional packages. To do this, simply run the following cell.

Discriminative Models

---

• Discriminative models are a class of models used in statistical classification, especially in supervised machine learning. A discriminative classifier tries to build a model just by depending on the observed data while learning how to do the classification from the given statistics.
  • Compared to generative models, discriminative models make fewer assumptions on the distributions but depend heavily on the quality of the data.
  • For example, given a set of labeled pictures of dog and rabbit, discriminative models will be matching a new, unlabeled picture to a most similar labeled picture and then give out the label class, a dog or a rabbit.
• The typical discriminative learning approaches include Logistic Regression (LR), Support Vector Machine (SVM), conditional random fields (CRFs) (specified over an undirected graph), and others.

The Perceptron

---

• One of the first and simplest linear model.
• Based on a linear threshold unit (LTU): the input and output are numbers (not binary values), and each connection is associated with a weight.
• The LTU computes a weighted sum of its inputs: $z = w_1x_1 + w_2x_2 +....+w_nx_n = w^Tx$, and then it applies a step function to that sum and outputs the result: $$ h_w(x) = step(z) = step(w^Tx) $$

• Illustration:

* The most common step function used is the *Heaviside step function* but sometimes the *sign function* is used (as is the illustration).

• Perceptron Training draws inspiration from biological neurons: the connection weight between two neurons is increased whenever they have the same output. Perceptrons are trained by considering the error made.
  • At each iteration, the Perceptron is fed with one training instance and makes a prediction for it.
  • For every output that produced a wrong prediction, it reinforces the connection weights from the inputs that would have contributed to the correct prediction.
  • Criterion: $ E^{perc}(w) = - \sum_{i \in D_{miss}}w^T(x^iy^i) $
• Perceptron Learning Rule (weight update): $$ w_{t+1} = w_t +\eta(y_i -sign(w_t^Tx_i))x_i $$
  • $\eta$ is the learing rate (hyper-parameter).
• The decision boundary learned is linear, the Perceptron is incapable of learning complex patterns.

• Perceptron Convergence Theorem: If the training instances are linearly seperable, the algorithm would converge to a solution.
  • There can be multiple solutions (multiple hyperplanes).
• Perceptrons do not output a class probability, they just make predicitons based on a hard threshold.

• Pseudocode:
  • Require: Learning rate $\eta$
  • Require: Initial parameter $w$
  • While stopping criterion not met do
    • For $i=1,...,m$:
      • $ w_{t+1} \leftarrow w_t +\eta(y_i -sign(w_t^Tx_i))x_i $
    • $t \leftarrow t + 1$
  • end while

MLE with Bernoulli Assumption

---

• Recall that there is a connection between maximum likelihood estimation (MLE) and linear regression when we assume that the data can be created as follows: $y=\theta^Tx + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma^2) \to y \sim \mathcal{N}(\theta^Tx, \sigma^2)$.

  • In this case, minimizing the negative log-likelihood (NLL): $-\log P(y|x;\theta)$ results in the MSE error $(y-\theta^Tx)^2$, and minimizing the NLL is the same as maximizing $\log P(y|x;\theta)$, which is exactly the MLE!

• When we assume that the data is created in a different way, we get different loss functions, as we will now demonstrate. But the idea is the same --

  maximizing the log-likelihood (MLE) = minimizing the negative log-likelihood (NLL). $$ \log P(y|x;\theta) = \log \left[\frac{1}{\sqrt{2 \pi \sigma^2}} \exp{\left(-\frac{(y - \theta^Tx)^2}{2\sigma^2}\right)} \right] $$ $$ = -0.5\log(2\pi\sigma^2) -\frac{1}{2\sigma^2}(y -\theta^Tx)^2$$ $$ \to \max_{\theta}\log P(y|x;\theta) = \min_{\theta}-\log P(y|x;\theta) = \min_{\theta}\frac{1}{2}(y -\theta^Tx)^2 = \min_{\theta} MSE $$

• The Sigmoid function (also the Logistic Function): $$ \sigma(x) = \frac{1}{1+e^{-x}} = \frac{e^x}{1+e^x} $$

  • The output is in $[0,1]$, which is exactly what we need to model a probability distribution.

• We assume that: $$ P(y|x,\theta) = Bern(y|\sigma(\theta^Tx)) $$

  • Bernoulli Distribution (coin flip): $$ P(y) = p^y(1-p)^{1-y} $$
  • $p = \sigma(\theta^Tx) \in [0,1]$

• We will use the following notations:

$$P(y_i|x_i, w) = \begin{cases} \pi_{i1} = \sigma(w^Tx) = \frac{1}{1+e^{-x}} & \quad \text{if } y_i=1 \\ \pi_{i0} = 1 - \sigma(w^Tx) = 1 - \frac{1}{1+e^{-x}} & \quad \text{if } y_i = 0 \end{cases} $$

Logistic Regression

---

• Some regression algorithms can be used for classification as well.
• Logistic Regression is commonly used to estimate the probability that an instance belongs to a particular class.
  • Typically, if the estimated proabibility is greater than 50%, then the model predicts that the instance belongs to that class (called the positive class, labeled "1"), or else it predicts that it does not - a binary classifier.
• Estimating Probabilities - Similarly to Linear Regression, a Logistic Regression model computes a weighted sum of the input features (plus a bias term), but unlike Linear Regression, it outputs the logistic of the weighted sum - $\sigma(w^Tx)$, which is a number between 0 and 1.

Training and Cost Function¶

---

• The objective of training is to set the parameter vector $\theta$ (or $w$) so that the model estimates high probabilities for positive instances ($y=1$) and low probabilities for negative instances ($y=0$)
• Expanding the expression using the negative log-likelihood (NLL): $$ P(y|x,\theta) = Bern(y|\sigma(\theta^Tx)) \rightarrow NLL(\theta) = -\frac{1}{m}\sum_{i=1}^m \log \sigma(\theta^Tx_i)^{y_i}(1-\sigma(\theta^Tx_i))^{1-y_i} =- \frac{1}{m} \sum_{i=1}^m\log\pi_{i1}^{y_i}\pi_{i0}^{1-y_i} $$ $$ = -\frac{1}{m} \sum_{i=1}^m \left[y_i\log\pi_{i1} + (1-y_i)\log\pi_{i0} \right]$$

• This yields the Logistic Regression cost function (log loss): $$ J(\theta) = -\frac{1}{m} \sum_{i=1}^m \big[ y_i\log \pi_{i1} + (1-y_i)\log \pi_{i0} \big] = -\frac{1}{m} \sum_{i=1}^m \big[ y_i\log \pi_{i1} + (1-y_i)\log (1 - \pi_{i1}) \big] $$
  • Intuition: $-\log(t)$ grows very large when $t$ approaches 0, so the cost will be large if the model estimates a probability close to 0 for a positive instance, and it will also be very large if the estimated probability is close to 1 for a negative instance. On the other hand, $-log(t)$ is close to 0 when $t$ is close to 1, so the cost will be close to 0 if the estimated probability is close to 0 for a negative instance or close to 1 for a positive instance.
  • This expression is also called the binary cross-entropy (BCE) loss.
  • The cost function in the case of Logistic Regression is convex.

• Logistic cost function derivatives: $$ \frac{\partial}{\partial \theta_j}J(\theta) = \frac{1}{m}\sum_{i=1}^m \big( \sigma(\theta^Tx^{i}) - y_i \big) x_j^{i} $$
  • No closed-form solution.
  • Thanks to the convexity of the cost function (for the case of Logistic Regression), we can use Gradient Descent (or SGD, Mini-Batch GD).

Logistic Regression with PyTorch

---

• We will now get familiar with building neural networks with PyTorch.
• All neural network models inherit from a parent class nn.Module. The user must implement the __init__() and __forward()__ methods.
• In __init__() we initialize the parameters of the neural networks, e.g., number of parameters (such as number of hidden units/layers, type of layers and etc...)
• In __forward()__ we implement the forward pass of the network, i.e., what happens to the input.
  • For example, if in __init__() you defined a linear layer and a ReLU activation, then in __forward()__ you will define that the input goes first into the linear layer and then into the activation.

• Okay, so we have our network, now we need to train it.
• We need to define how to optimize the weights and other hyper-parameters, such as number of epochs.

Same as Scikit-Learn!

Multi-Class (Multinomial) Logistic Regression - Softmax Regression

---

• The Logistic Regression model can be generalized to support multiple classes.
• The idea: when given an instance $x$, the Softmax Regression model first computes a score $s_k(x)$ for each class $k$, then estimates a probability of each class by applying the softmax function (normalized exponential) to the scores.
• The Softmax score for class $k$: $$ s_k(x) = \big( \theta^{(k)} \big)^T \cdot x $$
  • Each class has its own dedicated parameter vector $\theta^{(k)}$, which is usually stored in a row of the parameter matrix $\Theta$.

• The Softmax Function: $$\hat{p}_k = p(y=k|x,\theta) = \sigma(s(x))_k = \frac{e^{s_k(x)}}{\sum_{j=1}^K e^{s_j(x)}} $$
  • $K$ is the number of classes.
  • $s(x)$ is a vector containing the scores of each class for the instance $x$.
  • $\sigma(s(x))_k$ is the estimated probability that the instance $x$ belongs to class $k$ given the scores of each class for that instance.
• The Softmax Regression classifier prediction: $$\hat{y} = \underset{k}{\mathrm{argmax}} \sigma(s(x))_k = \underset{k}{\mathrm{argmax}} s_k(x) = \underset{k}{\mathrm{argmax}} \big( (\theta^{(k)})^Tx \big) $$

• Cross-Entropy cost function: $$ J(\Theta) = -\frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K y_k^{(i)} \log(\hat{p}_k^{(i)}) $$
  • $y_k^{(i)}$ is equal to 1 if the target class for the $i^{th}$ instance is $k$, otherwise, it is 0.
  • When $K=2$ it is the BCE from the previous section.
• Cross-Entropy gradient vector for class $k$: $$ \nabla_{\theta^{(k)}}J(\Theta) = \frac{1}{m}\sum_{i=1}^m (\hat{p}_k^{(i)} - y_k^{(i)})x^{(i)} $$
  • Use Gradient Descent or its variants to solve
• In Scikit-Learn: LogisticRegression(multi_class="multinomial", solver="lbfgs", C=10)
  • $C$ is a regularization term: the inverse of regularization strength, smaller values specify stronger regularization.

Activation Functions

---

The key change made to the Perceptron that brought upon the era of deep learning is the addition of activation functions to the output of each neuron. These allow the learning of non-linear functions. We will use three popular activation functions:

1. Logistic function (sigmoid): $\sigma(z) = \frac{1}{1 + e^{-z}}$. The output is in $[0,1]$ which can be used for binary clssification or as a probability.
2. Hyperbolic tangent function: $tanh(z) = 2\sigma(2z) - 1$. The output is in $[-1,1]$ which tends to make each layer's output more or less normalized at the beginning of the training (which may speed up convergence).
3. ReLU (Rectified Linear Unit) function: $ReLU(z) = max(0,z)$. Continuous but not differentiable at $z=0$. However, it is the most common activation function as it is fast to compute and does not bound the output (which helps with some issues during Gradient Descent).

The activation functions derivatives (for the backpropagation):

1. $\frac{d\sigma(z)}{dz} = \sigma(z)(1-\sigma(z))$
2. $\frac{d tanh(z)}{dz} = 1 - tanh^2(z)$
3. We define the derivative at 0 to be zero: $\frac{d ReLU(z)}{dz} = 1$ if $x>0$ else $0$

Recommended Videos

---

Warning!

• These videos do not replace the lectures and tutorials.
• Please use these to get a better understanding of the material, and not as an alternative to the written material.

Video By Subject¶

• Pereceptron - Pereceptron
  • Perceptron Training
• Logistic Regression - Lecture 3 | Machine Learning (Stanford)
  • StatQuest: Logistic Regression
• Softmax Regression - Softmax Regression (C2W3L08)
• Activation Functions - Activation Functions (C1W3L06)
  • Why Non-linear Activation Functions (C1W3L07)

Credits

---

• Icons made by Becris from www.flaticon.com
• Icons from Icons8.com - https://icons8.com
• Datasets from Kaggle - https://www.kaggle.com/
• Examples and code snippets were taken from "Hands-On Machine Learning with Scikit-Learn and TensorFlow"