DecisionTreeEncoder¶

The DecisionTreeEncoder() encodes categorical variables with predictions of a decision tree model.

In [1]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from feature_engine.encoding import DecisionTreeEncoder

In [2]:

# Load titanic dataset from OpenML

def load_titanic():
    data = pd.read_csv('https://www.openml.org/data/get_csv/16826755/phpMYEkMl')
    data = data.replace('?', np.nan)
    data['cabin'] = data['cabin'].astype(str).str[0]
    data['pclass'] = data['pclass'].astype('O')
    data['age'] = data['age'].astype('float')
    data['fare'] = data['fare'].astype('float')
    data['embarked'].fillna('C', inplace=True)
    data.drop(labels=['boat', 'body', 'home.dest'], axis=1, inplace=True)
    return data

In [3]:

data = load_titanic()
data.head()

Out[3]:

	pclass	survived	name	sex	age	sibsp	parch	ticket	fare	cabin	embarked
0	1	1	Allen, Miss. Elisabeth Walton	female	29.0000	0	0	24160	211.3375	B	S
1	1	1	Allison, Master. Hudson Trevor	male	0.9167	1	2	113781	151.5500	C	S
2	1	0	Allison, Miss. Helen Loraine	female	2.0000	1	2	113781	151.5500	C	S
3	1	0	Allison, Mr. Hudson Joshua Creighton	male	30.0000	1	2	113781	151.5500	C	S
4	1	0	Allison, Mrs. Hudson J C (Bessie Waldo Daniels)	female	25.0000	1	2	113781	151.5500	C	S

In [4]:

X = data.drop(['survived', 'name', 'ticket'], axis=1)
y = data.survived

In [5]:

# we will encode the below variables, they have no missing values
X[['cabin', 'pclass', 'embarked']].isnull().sum()

Out[5]:

cabin       0
pclass      0
embarked    0
dtype: int64

In [6]:

''' Make sure that the variables are type (object).
if not, cast it as object , otherwise the transformer will either send an error (if we pass it as argument) 
or not pick it up (if we leave variables=None). '''

X[['cabin', 'pclass', 'embarked']].dtypes

Out[6]:

cabin       object
pclass      object
embarked    object
dtype: object

In [7]:

# let's separate into training and testing set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

X_train.shape, X_test.shape

Out[7]:

((916, 8), (393, 8))

The categorical variable will be first encoded into integers with the OrdinalEncoder(). The integers can be assigned arbitrarily to the categories or following the mean value of the target in each category.

Then a decision tree will be fit using the resulting numerical variable to predict the target variable. Finally, the original categorical variable values will be replaced by the predictions of the decision tree.

In [8]:

'''
Parameters
    ----------

    encoding_method: str, default='arbitrary'
        The categorical encoding method that will be used to encode the original
        categories to numerical values.

        'ordered': the categories are numbered in ascending order according to
        the target mean value per category.

        'arbitrary' : categories are numbered arbitrarily.

    cv : int, default=3
        Desired number of cross-validation fold to be used to fit the decision
        tree.

    scoring: str, default='neg_mean_squared_error'
        Desired metric to optimise the performance for the tree. Comes from
        sklearn metrics. See the DecisionTreeRegressor or DecisionTreeClassifier
        model evaluation documentation for more options:
        https://scikit-learn.org/stable/modules/model_evaluation.html

    regression : boolean, default=True
        Indicates whether the encoder should train a regression or a classification
        decision tree.

    param_grid : dictionary, default=None
        The list of parameters over which the decision tree should be optimised
        during the grid search. The param_grid can contain any of the permitted
        parameters for Scikit-learn's DecisionTreeRegressor() or
        DecisionTreeClassifier().

        If None, then param_grid = {'max_depth': [1, 2, 3, 4]}.

    random_state : int, default=None
        The random_state to initialise the training of the decision tree. It is one
        of the parameters of the Scikit-learn's DecisionTreeRegressor() or
        DecisionTreeClassifier(). For reproducibility it is recommended to set
        the random_state to an integer.

    variables : list, default=None
        The list of categorical variables that will be encoded. If None, the
        encoder will find and select all object type variables.
'''

Out[8]:

"\nParameters\n    ----------\n\n    encoding_method: str, default='arbitrary'\n        The categorical encoding method that will be used to encode the original\n        categories to numerical values.\n\n        'ordered': the categories are numbered in ascending order according to\n        the target mean value per category.\n\n        'arbitrary' : categories are numbered arbitrarily.\n\n    cv : int, default=3\n        Desired number of cross-validation fold to be used to fit the decision\n        tree.\n\n    scoring: str, default='neg_mean_squared_error'\n        Desired metric to optimise the performance for the tree. Comes from\n        sklearn metrics. See the DecisionTreeRegressor or DecisionTreeClassifier\n        model evaluation documentation for more options:\n        https://scikit-learn.org/stable/modules/model_evaluation.html\n\n    regression : boolean, default=True\n        Indicates whether the encoder should train a regression or a classification\n        decision tree.\n\n    param_grid : dictionary, default=None\n        The list of parameters over which the decision tree should be optimised\n        during the grid search. The param_grid can contain any of the permitted\n        parameters for Scikit-learn's DecisionTreeRegressor() or\n        DecisionTreeClassifier().\n\n        If None, then param_grid = {'max_depth': [1, 2, 3, 4]}.\n\n    random_state : int, default=None\n        The random_state to initialise the training of the decision tree. It is one\n        of the parameters of the Scikit-learn's DecisionTreeRegressor() or\n        DecisionTreeClassifier(). For reproducibility it is recommended to set\n        the random_state to an integer.\n\n    variables : list, default=None\n        The list of categorical variables that will be encoded. If None, the\n        encoder will find and select all object type variables.\n"

In [9]:

tree_enc = DecisionTreeEncoder(encoding_method='arbitrary',
                               cv=3,
                               scoring = 'roc_auc',
                               param_grid = {'max_depth': [1, 2, 3, 4]},
                               regression = False,
                               variables=['cabin', 'pclass', 'embarked']
                              )

tree_enc.fit(X_train,y_train) # to fit you need to pass the target y

Out[9]:

DecisionTreeEncoder(param_grid={'max_depth': [1, 2, 3, 4]}, regression=False,
                    scoring='roc_auc',
                    variables=['cabin', 'pclass', 'embarked'])

In [10]:

tree_enc.encoder_

Out[10]:

Pipeline(steps=[('categorical_encoder',
                 OrdinalEncoder(encoding_method='arbitrary',
                                variables=['cabin', 'pclass', 'embarked'])),
                ('tree_discretiser',
                 DecisionTreeDiscretiser(param_grid={'max_depth': [1, 2, 3, 4]},
                                         regression=False, scoring='roc_auc',
                                         variables=['cabin', 'pclass',
                                                    'embarked']))])

In [11]:

# transform and visualise the data

train_t = tree_enc.transform(X_train)
test_t = tree_enc.transform(X_test)

test_t.sample(5)

Out[11]:

	pclass	sex	age	sibsp	parch	fare	cabin	embarked
233	0.617391	female	56.0	0	1	83.1583	0.611650	0.558011
725	0.259036	female	22.0	0	0	7.7500	0.304843	0.373494
934	0.259036	female	4.0	0	2	22.0250	0.304843	0.338957
241	0.617391	male	NaN	0	0	50.0000	0.698113	0.338957
466	0.436170	male	34.0	1	0	26.0000	0.304843	0.338957

Automatically select the variables¶

This encoder will select all categorical variables to encode, when no variables are specified when calling the encoder.

In [12]:

tree_enc = DecisionTreeEncoder(encoding_method='arbitrary',
                               cv=3,
                               scoring = 'roc_auc',
                               param_grid = {'max_depth': [1, 2, 3, 4]},
                               regression = False,
                              )

tree_enc.fit(X_train,y_train) # to fit you need to pass the target y

Out[12]:

DecisionTreeEncoder(param_grid={'max_depth': [1, 2, 3, 4]}, regression=False,
                    scoring='roc_auc')

In [13]:

tree_enc.encoder_

Out[13]:

Pipeline(steps=[('categorical_encoder',
                 OrdinalEncoder(encoding_method='arbitrary',
                                variables=['pclass', 'sex', 'cabin',
                                           'embarked'])),
                ('tree_discretiser',
                 DecisionTreeDiscretiser(param_grid={'max_depth': [1, 2, 3, 4]},
                                         regression=False, scoring='roc_auc',
                                         variables=['pclass', 'sex', 'cabin',
                                                    'embarked']))])

In [14]:

# transform and visualise the data

train_t = tree_enc.transform(X_train)
test_t = tree_enc.transform(X_test)

test_t.sample(5)

Out[14]:

	pclass	sex	age	sibsp	parch	fare	cabin	embarked
753	0.259036	0.187608	22.0	0	0	8.0500	0.304843	0.338957
39	0.617391	0.187608	48.0	0	0	50.4958	0.698113	0.558011
900	0.259036	0.187608	NaN	1	2	23.4500	0.304843	0.338957
1148	0.259036	0.187608	35.0	0	0	7.1250	0.304843	0.338957
187	0.617391	0.728358	16.0	0	1	39.4000	0.698113	0.338957

In [ ]: