Deep learning¶

$$ \newcommand{\eg}{{\it e.g.}} \newcommand{\ie}{{\it i.e.}} \newcommand{\argmin}{\operatornamewithlimits{argmin}} \newcommand{\mc}{\mathcal} \newcommand{\mb}{\mathbb} \newcommand{\mf}{\mathbf} \newcommand{\minimize}{{\text{minimize}}} \newcommand{\diag}{{\text{diag}}} \newcommand{\cond}{{\text{cond}}} \newcommand{\rank}{{\text{rank }}} \newcommand{\range}{{\mathcal{R}}} \newcommand{\null}{{\mathcal{N}}} \newcommand{\tr}{{\text{trace}}} \newcommand{\dom}{{\text{dom}}} \newcommand{\dist}{{\text{dist}}} \newcommand{\R}{\mathbf{R}} \newcommand{\SM}{\mathbf{S}} \newcommand{\ball}{\mathcal{B}} \newcommand{\bmat}[1]{\begin{bmatrix}#1\end{bmatrix}} \newcommand{\loss}{\ell} \newcommand{\eloss}{\mathbf{L}} \newcommand{\abs}[1]{| #1 |} \newcommand{\norm}[1]{\| #1 \|} \newcommand{\tp}{T} $$

Jong-Han Kim (jonghank@inha.ac.kr)

Jiwoo Choi (jiwoochoi@inha.edu)

---

Artificial Intelligence (AI)¶

• Computer systems designed to mimic human intelligence.
• Includes traditional methods like rule-based systems and classical computer vision techniques.
  • Examples: Edge detection, Camera Calibration
• Encompasses subsets such as machine learning and reinforcement learning.

Machine Learning¶

• A subset of AI that enables computers to perform specific tasks by learning from data.
• Involves extracting features and using them to carry out tasks:
  • The process of feature extraction is often performed manually by humans.
  • Examples include decision trees, support vector machines, and basic clustering techniques.

In the previous class, we discussed machine learning. Today, we will focus on deep learning.

Deep Learning¶

• A specialized branch of machine learning.

• Relies on deep neural networks to learn and perform tasks directly from raw data.
• The ultimate goal of deep learning is to find the optimal parameters $\theta^*$ that best approximate the mapping from input $x$ to output $y$:

$$\theta^* = \underset{\theta}{\text{argmin}} \ \mathbf{L}(y, f_\text{NN}(x; \theta))$$

• We can find $\theta^*$ through a process called learning.
• Learning? This involves optimizing the model parameters $\theta$ by minimizing the loss function $\mathbf{L}$, which measures the difference between the predicted output and the true output.
• Usually gradient decent method is applied to find $\theta^*$. The pseudo code of the gradient descent algorithm for finding a (locally) optimal solution minimizing $\mathbf{L}$ is as follows,

$$ \begin{array}{l} \textbf{initialize } x, \textbf{and set } \text{lr}>0\\ \textbf{for } i \leftarrow 1 \textbf{ to } \text{maxiter} \textbf{ do} \\ \quad \theta \leftarrow \theta - \text{lr} \nabla_\theta \mathbf{L}(x; \theta); \\ \textbf{end} \end{array} $$

• Automates the process of feature extraction by using layered architectures.
  • Multi-Layer Perceptron (MLP), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs) etc.
  • Today, we will use the MLP architecture to design our model

Multi-Layer Perceptron (MLP)¶

• One of the most well-known deep learning architectures.
• Composed of a combination of linear transformations and activation functions to capture non-linear relationships in the data:
  • Activation functions introduce non-linearity, enabling the MLP to learn more complex patterns in the data.
  • Compared to linear models like Support Vector Machines (SVMs) used in traditional machine learning, MLPs can represent more complex decision boundaries and adapt to a wider variety of tasks.
  • Common activation functions include:
    • ReLU: Rectified Linear Unit, widely used for its simplicity and effectiveness.
    • GELU: Gaussian Error Linear Unit, popular in transformer architectures.
    • Sigmoid: Used for mapping outputs to probabilities in certain contexts.
    • Tanh: Similar to sigmoid but outputs values between -1 and 1.

---

Supervised Learning¶

Learning is largely diveded into two parts: Supervised Learning and Unspervised Learning. We will focus on unsupervised learning in the assignment.

Supervised Learning is a methodology where a model is trained using input data $x$ and the corresponding labels $y$ to predict the correct output for a given input. In supervised learning, the dataset includes labeled data, and the model learns the relationship between the inputs and outputs based on these labels.

When paired data $(x, y)$(input and corresponding output) is given, the objective is to find the optimal parameters $\theta$ of a model $f_\text{NN}$ that best approximates the mapping from $x$ to $y$. Depending on the nature of the output $y$ supervised learning can be broadly categorized into two types:

---

1. Classification¶

• Objective: Predict the class to which the data belongs.

• Output: Binary classification or multi-class classification.

• Examples:

  • Determining whether an email is spam or not.
  • Classifying images into categories such as cats, dogs, and rabbits.

• Loss Function: Commonly, Cross-Entropy Loss is used, defined as:

  
  $$\mathbf{L}(Y, P ) = -\frac{1}{N} \sum_{i=1}^N\sum_{j=1}^{N_c} y_{ij} \log p_{ij}$$
  

  where,

  • $N$: Number of data samples.

  • $N_c$: Number of classes

  • $Y = \bmat{y_1 & y_2& \cdots& y_N}^T$ where $y_i \in \mathbb{R}^{N_c}$ is the one-hot encoded true label vector for the $i$-th data sample.

  • $P = \bmat{p_1 & p_2& \cdots& p_N}^T$ where $p_i \in \mathbb{R}^{N_c}$ is the predicted probability vector for each class for $i$-th data sample.

Here is the example of Classification.

The Stellar Classification Dataset (SDSS17) contains spectroscopic data of stars, galaxies, and quasars collected from the Sloan Digital Sky Survey (SDSS). It is commonly used for classifying astronomical objects. Below are the details for processing the dataset and building the training dataset.

Feature¶

1. obj_ID: Unique ID of the object.
2. alpha: Right Ascension (RA) of the celestial sphere.
3. delta: Declination (Dec) of the celestial sphere.
4. u, g, r, i, z: Magnitudes measured through five filters.
5. class: Classification of the object (STAR, GALAXY, QSO).
6. Miscellaneous (plate, MJD, fiberID): Reference data for SDSS.

1) Data Preprocessing¶

1. Load the archive.zip File

   • Extract and load the required data for processing.

2. Split data into train data and test data.

3. Normalization

   • Perform normalization on the features: 'u', 'g', 'r', 'i', 'z', 'alpha', 'delta', 'redshift'.
   • The normalization process scales the data values to the range of 0 to 1 using the formula:

$$\hat{x} = \frac{x - x_\text{min}}{x_\text{max} - x_\text{min}}$$

- This ensures that no particular feature dominates the learning process, improving model stability and preventing overfitting.
- Use the minimum and maximum values from the training set to normalize both the training and testing sets. This ensures consistency and prevents information leakage from the test set into the training process.

4. Convert numpy to torch
   • Convert the normalized features into torch.FloatTensor for compatibility with PyTorch.
   • Convert the class column into torch.LongTensor as it represents categorical data for classification.

2) Define the model, loss function and optimizer.¶

• Model

  • We use a Multi-Layer Perceptron (MLP) architecture for the model.
  • Input: the model takes the normalized features of 'u', 'g', 'r', 'i', 'z', 'alpha', 'delta', 'redshift'
  • output: the model outputs logits for 3 classes (STAR, GALAXY, QSO).

• Loss function

  • We use Cross-Entropy Loss torch.nn.CrossEntropyLoss to train the model.
  • As mentioned before, this loss function is suitable for multi-class classification tasks as it computes the difference between predicted probabilities and true labels.

• Optimizer

  • We use the Adam optimizer torch.optim.Adam to optimize the model parameters. Adam is widely used for its adaptability and efficiency in deep learning tasks.

3) Training process repeats the following.¶

1. Gradient Initialization: Use optimizer.zero_grad() to reset the gradients calculated in the previous step.
2. Forward: pass the input data through the model to compute the prediction
3. Loss calculation: Calculate the loss by comparing the model's predictions with the actual labels.
4. Backward: Call loss.backward() to compute the gradients of the model parameters with respect to the loss.
5. Parameter Update: Use optimizer.step() to update the model parameters based on the computed gradients.

4) Predict the values with trained model¶

---

2. Regression¶

• Objective: Predict continuous values.

• Output: Real-valued numbers.

• Examples:

  • Predicting weather conditions in a specific region.
  • Forecasting stock price movements.
  • Approximating a continuous function.

• Loss function:

  • Commonly used loss functions for regression tasks include:

    Mean Squared Error(MSE)

    $$\mathbf{L}$(y, \hat{y}) = \frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2$$

    Mean Absolute Error (MAE):

  $$\mathbf{L}(y, \hat{y}) = \frac{1}{N} \sum_{i=1}^N |y_i - \hat{y}_i|$$

<ㅠㄱ>

Let's train a deep model to approximate the given spiral defined by the parametric equations:

$$ \begin{aligned} x &= \frac{1}{12} \theta \cos{\theta}\\ y &= \frac{1}{12} \theta \sin{\theta} \end{aligned} $$
Our objective is to design a model that takes $\theta$ as input and predicts the corresponding $(x, y)$ coordinates of the spiral.

1) Generate a training data¶

To start, we generate the spiral dataset using the parametric equations provided above. This dataset will serve as the ground truth for training the model. $\theta$ will be sampled over a specified range, and the corresponding $(x, y)$ coordinates will be computed.

2) Define the model, loss function and optimizer.¶

We define a neural network model to map $\theta$ to $(x, y)$. we use nn.MSELoss(), which minimizes the mean squared error between the predicted and actual coordinates. The optimizer is identical to the one used previously.

3) Train model.¶

The training process follows a structure similar to the above classification task.

4) Predict the values with trained model¶

Using the trained model, we predict the values of $(x, y)$ for both the trained points and a new set of test points.