MatchVariables() ensures that the columns in the test set are identical to those in the train set.
If the test set contains additional columns, they are dropped. Alternatively, if the test set lacks columns that were present in the train set, they will be added with a value determined by the user, for example np.nan.
import numpy as np
import pandas as pd
from feature_engine.preprocessing import MatchVariables
# Load titanic dataset from OpenML
def load_titanic():
data = pd.read_csv('https://www.openml.org/data/get_csv/16826755/phpMYEkMl')
data = data.replace('?', np.nan)
data['cabin'] = data['cabin'].astype(str).str[0]
data['pclass'] = data['pclass'].astype('O')
data['age'] = data['age'].astype('float')
data['fare'] = data['fare'].astype('float')
data['embarked'].fillna('C', inplace=True)
data.drop(
labels=['name', 'ticket', 'boat', 'body', 'home.dest'],
axis=1, inplace=True,
)
return data
data = load_titanic()
data.head()
pclass | survived | sex | age | sibsp | parch | fare | cabin | embarked | |
---|---|---|---|---|---|---|---|---|---|
0 | 1 | 1 | female | 29.0000 | 0 | 0 | 211.3375 | B | S |
1 | 1 | 1 | male | 0.9167 | 1 | 2 | 151.5500 | C | S |
2 | 1 | 0 | female | 2.0000 | 1 | 2 | 151.5500 | C | S |
3 | 1 | 0 | male | 30.0000 | 1 | 2 | 151.5500 | C | S |
4 | 1 | 0 | female | 25.0000 | 1 | 2 | 151.5500 | C | S |
# separate the dataset into train and test
train = data.iloc[0:1000, :]
test = data.iloc[1000:, :]
train.shape, test.shape
((1000, 9), (309, 9))
# set up the transformer
match_cols = MatchVariables(missing_values="ignore")
# learn the variables in the train set
match_cols.fit(train)
MatchVariables(missing_values='ignore')
# the transformer stores the input variables
match_cols.input_features_
['pclass', 'survived', 'sex', 'age', 'sibsp', 'parch', 'fare', 'cabin', 'embarked']
# Let's drop some columns in the test set for the demo
test_t = test.drop(["sex", "age"], axis=1)
test_t.head()
pclass | survived | sibsp | parch | fare | cabin | embarked | |
---|---|---|---|---|---|---|---|
1000 | 3 | 1 | 0 | 0 | 7.7500 | n | Q |
1001 | 3 | 1 | 2 | 0 | 23.2500 | n | Q |
1002 | 3 | 1 | 2 | 0 | 23.2500 | n | Q |
1003 | 3 | 1 | 2 | 0 | 23.2500 | n | Q |
1004 | 3 | 1 | 0 | 0 | 7.7875 | n | Q |
# Let's drop some columns in the test set for the demo
test_t = test.drop(["sex", "age"], axis=1)
test_t.head()
pclass | survived | sibsp | parch | fare | cabin | embarked | |
---|---|---|---|---|---|---|---|
1000 | 3 | 1 | 0 | 0 | 7.7500 | n | Q |
1001 | 3 | 1 | 2 | 0 | 23.2500 | n | Q |
1002 | 3 | 1 | 2 | 0 | 23.2500 | n | Q |
1003 | 3 | 1 | 2 | 0 | 23.2500 | n | Q |
1004 | 3 | 1 | 0 | 0 | 7.7875 | n | Q |
# the transformer adds the columns back
test_tt = match_cols.transform(test_t)
print()
test_tt.head()
The following variables are added to the DataFrame: ['sex', 'age']
pclass | survived | sex | age | sibsp | parch | fare | cabin | embarked | |
---|---|---|---|---|---|---|---|---|---|
1000 | 3 | 1 | NaN | NaN | 0 | 0 | 7.7500 | n | Q |
1001 | 3 | 1 | NaN | NaN | 2 | 0 | 23.2500 | n | Q |
1002 | 3 | 1 | NaN | NaN | 2 | 0 | 23.2500 | n | Q |
1003 | 3 | 1 | NaN | NaN | 2 | 0 | 23.2500 | n | Q |
1004 | 3 | 1 | NaN | NaN | 0 | 0 | 7.7875 | n | Q |
Note how the missing columns were added back to the transformed test set, with missing values, in the position (i.e., order) in which they were in the train set.
Similarly, if the test set contained additional columns, those would be removed:
test_t.loc[:, "new_col1"] = 5
test_t.loc[:, "new_col2"] = "test"
test_t.head()
pclass | survived | sibsp | parch | fare | cabin | embarked | new_col1 | new_col2 | |
---|---|---|---|---|---|---|---|---|---|
1000 | 3 | 1 | 0 | 0 | 7.7500 | n | Q | 5 | test |
1001 | 3 | 1 | 2 | 0 | 23.2500 | n | Q | 5 | test |
1002 | 3 | 1 | 2 | 0 | 23.2500 | n | Q | 5 | test |
1003 | 3 | 1 | 2 | 0 | 23.2500 | n | Q | 5 | test |
1004 | 3 | 1 | 0 | 0 | 7.7875 | n | Q | 5 | test |
# set up the transformer with different
# fill value
match_cols = MatchVariables(
fill_value=0, missing_values="ignore",
)
# learn the variables in the train set
match_cols.fit(train)
MatchVariables(fill_value=0, missing_values='ignore')
test_tt = match_cols.transform(test_t)
print()
test_tt.head()
The following variables are added to the DataFrame: ['sex', 'age'] The following variables are dropped from the DataFrame: ['new_col2', 'new_col1']
pclass | survived | sex | age | sibsp | parch | fare | cabin | embarked | |
---|---|---|---|---|---|---|---|---|---|
1000 | 3 | 1 | 0 | 0 | 0 | 0 | 7.7500 | n | Q |
1001 | 3 | 1 | 0 | 0 | 2 | 0 | 23.2500 | n | Q |
1002 | 3 | 1 | 0 | 0 | 2 | 0 | 23.2500 | n | Q |
1003 | 3 | 1 | 0 | 0 | 2 | 0 | 23.2500 | n | Q |
1004 | 3 | 1 | 0 | 0 | 0 | 0 | 7.7875 | n | Q |
Note how the columns that were present in the test set but not in train set were dropped. And now, the missing variables were added back into the dataset with the value 0.