The perceptron is a simple supervised machine learning algorithm and one of the earliest neural network architectures. It was introduced by Rosenblatt in the late 1950s. A perceptron represents a binary linear classifier that maps a set of training examples (of $d$ dimensional input vectors) onto binary output values using a $d-1$ dimensional hyperplane.
The perceptron as follows.
Given:
The perceptron is a very simple neural network:
A perceptron is trained using gradient descent. The training algorithm has different steps. In the beginning (step 0) the model parameters are initialized. The other steps (see below) are repeated for a specified number of training iterations or until the parameters have converged.
Step 0: Initialize the weight vector and bias with zeros (or small random values).
Step 1: Compute a linear combination of the input features and weights. This can be done in one step for all training examples, using vectorization and broadcasting: $\boldsymbol{a} = \boldsymbol{X} \cdot \boldsymbol{w} + b$
where $\boldsymbol{X}$ is a matrix of shape $(n_{samples}, n_{features})$ that holds all training examples, and $\cdot$ denotes the dot product.
Step 2: Apply the Heaviside function, which returns binary values:
$\hat{y}^{(i)} = 1 \, if \, a^{(i)} \geq 0, \, else \, 0$
** Step 3: ** Compute the weight updates using the perceptron learning rule
\begin{equation} \Delta \boldsymbol{w} = \eta \, \boldsymbol{X}^T \cdot \big(\boldsymbol{\hat{y}} - \boldsymbol{y} \big) \end{equation}$$ \Delta b = \eta \, \big(\boldsymbol{\hat{y}} - \boldsymbol{y} \big) $$where $\eta$ is the learning rate.
** Step 4: ** Update the weights and bias
\begin{equation} \boldsymbol{w} = \boldsymbol{w} + \Delta \boldsymbol{w} \end{equation}$$ b = b + \Delta b $$import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
np.random.seed(123)
%matplotlib inline
X, y = make_blobs(n_samples=1000, centers=2)
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.title("Dataset")
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.show()
y_true = y[:, np.newaxis]
X_train, X_test, y_train, y_test = train_test_split(X, y_true)
print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape})')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
Shape X_train: (750, 2) Shape y_train: (750, 1)) Shape X_test: (250, 2) Shape y_test: (250, 1)
class Perceptron():
def __init__(self):
pass
def train(self, X, y, learning_rate=0.05, n_iters=100):
n_samples, n_features = X.shape
# Step 0: Initialize the parameters
self.weights = np.zeros((n_features,1))
self.bias = 0
for i in range(n_iters):
# Step 1: Compute the activation
a = np.dot(X, self.weights) + self.bias
# Step 2: Compute the output
y_predict = self.step_function(a)
# Step 3: Compute weight updates
delta_w = learning_rate * np.dot(X.T, (y - y_predict))
delta_b = learning_rate * np.sum(y - y_predict)
# Step 4: Update the parameters
self.weights += delta_w
self.bias += delta_b
return self.weights, self.bias
def step_function(self, x):
return np.array([1 if elem >= 0 else 0 for elem in x])[:, np.newaxis]
def predict(self, X):
a = np.dot(X, self.weights) + self.bias
return self.step_function(a)
p = Perceptron()
w_trained, b_trained = p.train(X_train, y_train,learning_rate=0.05, n_iters=500)
y_p_train = p.predict(X_train)
y_p_test = p.predict(X_test)
print(f"training accuracy: {100 - np.mean(np.abs(y_p_train - y_train)) * 100}%")
print(f"test accuracy: {100 - np.mean(np.abs(y_p_test - y_test)) * 100}%")
training accuracy: 100.0% test accuracy: 100.0%
def plot_hyperplane(X, y, weights, bias):
"""
Plots the dataset and the estimated decision hyperplane
"""
slope = - weights[0]/weights[1]
intercept = - bias/weights[1]
x_hyperplane = np.linspace(-10,10,10)
y_hyperplane = slope * x_hyperplane + intercept
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.plot(x_hyperplane, y_hyperplane, '-')
plt.title("Dataset and fitted decision hyperplane")
plt.xlabel("First feature")
plt.ylabel("Second feature")
plt.show()
plot_hyperplane(X, y, w_trained, b_trained)