In this notebook, we present how to build predictive models on tabular datasets, with only numerical features.
In particular we highlight:
.fit(X, y)
/.predict(X)
/.score(X, y)
;Here API stands for "Application Programming Interface" and refers to a set of conventions to build self-consistent software. Notice that you can visit the Glossary for more info on technical jargon.
We use the "adult_census" dataset described in the previous notebook. For more details about the dataset see http://www.openml.org/d/1590.
Numerical data is the most natural type of data used in machine learning and can (almost) directly be fed into predictive models. Here we load a subset of the original data with only the numerical columns.
import pandas as pd
adult_census = pd.read_csv("../datasets/adult-census-numeric.csv")
Let's have a look at the first records of this dataframe:
adult_census.head()
We see that this CSV file contains all information: the target that we would
like to predict (i.e. "class"
) and the data that we want to use to train our
predictive model (i.e. the remaining columns). The first step is to separate
columns to get on one side the target and on the other side the data.
target_name = "class"
target = adult_census[target_name]
target
data = adult_census.drop(columns=[target_name])
data.head()
We can now linger on the variables, also denominated features, that we later use to build our predictive model. In addition, we can also check how many samples are available in our dataset.
data.columns
print(
f"The dataset contains {data.shape[0]} samples and "
f"{data.shape[1]} features"
)
We now build a classification model using the "K-nearest neighbors" strategy.
To predict the target of a new sample, a k-nearest neighbors takes into
account its k
closest samples in the training set and predicts the majority
target of these samples.
Caution!
We use a K-nearest neighbors here. However, be aware that it is seldom useful in practice. We use it because it is an intuitive algorithm. In the next notebook, we will introduce better models.
The fit
method is called to train the model from the input (features) and
target data.
from sklearn.neighbors import KNeighborsClassifier
model = KNeighborsClassifier()
_ = model.fit(data, target)
Learning can be represented as follows:
In scikit-learn an object that has a fit
method is called an estimator.
The method fit
is composed of two elements: (i) a learning algorithm and
(ii) some model states. The learning algorithm takes the training data and
training target as input and sets the model states. These model states are
later used to either predict (for classifiers and regressors) or transform
data (for transformers).
Both the learning algorithm and the type of model states are specific to each type of model.
Note
Here and later, we use the name data and target to be explicit. In scikit-learn documentation, data is commonly named X and target is commonly called y.
Let's use our model to make some predictions using the same dataset.
target_predicted = model.predict(data)
An estimator (an object with a fit
method) with a predict
method is called
a predictor. We can illustrate the prediction mechanism as follows:
To predict, a model uses a prediction function that uses the input data together with the model states. As for the learning algorithm and the model states, the prediction function is specific for each type of model.
Let's now have a look at the computed predictions. For the sake of simplicity, we look at the five first predicted targets.
target_predicted[:5]
Indeed, we can compare these predictions to the actual data...
target[:5]
...and we could even check if the predictions agree with the real targets:
target[:5] == target_predicted[:5]
print(
"Number of correct prediction: "
f"{(target[:5] == target_predicted[:5]).sum()} / 5"
)
Here, we see that our model makes a mistake when predicting for the first sample.
To get a better assessment, we can compute the average success rate.
(target == target_predicted).mean()
This result means that the model makes a correct prediction for approximately 82 samples out of 100. Note that we used the same data to train and evaluate our model. Can this evaluation be trusted or is it too good to be true?
When building a machine learning model, it is important to evaluate the trained model on data that was not used to fit it, as generalization is more than memorization (meaning we want a rule that generalizes to new data, without comparing to data we memorized). It is harder to conclude on never-seen instances than on already seen ones.
Correct evaluation is easily done by leaving out a subset of the data when training the model and using it afterwards for model evaluation. The data used to fit a model is called training data while the data used to assess a model is called testing data.
We can load more data, which was actually left-out from the original data set.
adult_census_test = pd.read_csv("../datasets/adult-census-numeric-test.csv")
From this new data, we separate our input features and the target to predict, as in the beginning of this notebook.
target_test = adult_census_test[target_name]
data_test = adult_census_test.drop(columns=[target_name])
We can check the number of features and samples available in this new set.
print(
f"The testing dataset contains {data_test.shape[0]} samples and "
f"{data_test.shape[1]} features"
)
Instead of computing the prediction and manually computing the average success
rate, we can use the method score
. When dealing with classifiers this method
returns their performance metric.
accuracy = model.score(data_test, target_test)
model_name = model.__class__.__name__
print(f"The test accuracy using a {model_name} is {accuracy:.3f}")
We use the generic term model for objects whose goodness of fit can be
measured using the score
method. Let's check the underlying mechanism when
calling score
:
To compute the score, the predictor first computes the predictions (using the
predict
method) and then uses a scoring function to compare the true target
y
and the predictions. Finally, the score is returned.
If we compare with the accuracy obtained by wrongly evaluating the model on the training set, we find that this evaluation was indeed optimistic compared to the score obtained on a held-out test set.
It shows the importance to always testing the generalization performance of predictive models on a different set than the one used to train these models. We will discuss later in more detail how predictive models should be evaluated.
Note
In this MOOC, we refer to generalization performance of a model when referring to the test score or test error obtained by comparing the prediction of a model and the true targets. Equivalent terms for generalization performance are predictive performance and statistical performance. We refer to computational performance of a predictive model when assessing the computational costs of training a predictive model or using it to make predictions.
In this notebook we:
.fit(X, y)
(to train a model),
.predict(X)
(to make predictions) and .score(X, y)
(to evaluate a
model);