(Click the icon below to open this notebook in Colab)

Overview¶

In our last class, we explored the Titanic dataset, examined it from multiple perspectives, and applied various feature engineering techniques to enhance its explanatory variables. Today, we will continue working with the Titanic dataset, focusing on model training and evaluation techniques to gain deeper insights into predictive modeling.

Load the dataset¶

https://www.kaggle.com/competitions/titanic/data

The Titanic https://en.wikipedia.org/wiki/Titanic

Variable	Definition	Key
survival	Survival	0 = No, 1 = Yes
pclass	Ticket class	1 = 1st, 2 = 2nd, 3 = 3rd
sex	Sex
Age	Age in years
sibsp	# of siblings / spouses aboard the Titanic
parch	# of parents / children aboard the Titanic
ticket	Ticket number
fare	Passenger fare
cabin	Cabin number
embarked	Port of Embarkation	C = Cherbourg, Q = Queenstown, S = Southampton

In [1]:

import pandas as pd

pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)

In [2]:

train = pd.read_csv('../data/titanic/train.csv')
test = pd.read_csv('../data/titanic/test.csv')

# convert all column names to lower cases
train.columns = train.columns.str.lower()
test.columns = test.columns.str.lower()

In [3]:

train.head(3)

Out[3]:

	passengerid	survived	pclass	name	sex	age	sibsp	ticket	fare	cabin	embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	STON/O2. 3101282	7.9250	NaN	S

In [4]:

test.head(3)

Out[4]:

	passengerid	pclass	name	sex	age	sibsp	ticket	fare	cabin	embarked
0	892	3	Kelly, Mr. James	male	34.5	0	330911	7.8292	NaN	Q
1	893	3	Wilkes, Mrs. James (Ellen Needs)	female	47.0	1	363272	7.0000	NaN	S
2	894	2	Myles, Mr. Thomas Francis	male	62.0	0	240276	9.6875	NaN	Q

In [ ]:

Streamline the data transformations¶

Here are the data exploration and transformation strategies we used so far:

Quick survey across key variables
Detect and address data anomalies
- Missing values
- Outliers
Feature engineering
- Encode categorical variables
- Normalize numerical variables
- Create new features with stronger predictive power

Data exploration process is typically iterative and complex. Once we have a good understanding of the data and some potential strategies to apply in the feature engineering process, we need to make sure these transformation strategies can be easily and consistently applied to new datasets, such as the test set and new batches of data for model retraining. This requires a systematic approach to streamline the data transformations so that we don't need to start from scratch and repeat the same steps for each new dataset. This is especially important in the real-world scenario where we want to productionalize and automate the data transformation process.

In [5]:

train.isnull().sum()

Out[5]:

passengerid      0
survived         0
pclass           0
name             0
sex              0
age            177
sibsp            0
parch            0
ticket           0
fare             0
cabin          687
embarked         2
dtype: int64

Missing value imputation¶

In [6]:

def impute_missing_num_values(df):
    """
    Impute missing values in numerical columns of a DataFrame using the median of each column.

    Args:
    df (pandas.DataFrame): The DataFrame to impute missing values in.

    Returns:
    pandas.DataFrame: The DataFrame with missing values imputed.
    """
    # Select only the numerical columns
    num_cols = df.select_dtypes(include=['float64', 'int64']).columns
    # Impute missing values with the median of each column
    for col in num_cols:
        df[col] = df[col].fillna(df[col].median())
    return df

In [7]:

train = impute_missing_num_values(train)
train.isnull().sum()

Out[7]:

passengerid      0
survived         0
pclass           0
name             0
sex              0
age              0
sibsp            0
parch            0
ticket           0
fare             0
cabin          687
embarked         2
dtype: int64

In [8]:

# categorical variables could have missing values too
# https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.mode.html

def impute_missing_cat_values(df, ignore_list):
    """
    Impute missing categorical values with the most frequent value.

    Args:
        df (pd.DataFrame): DataFrame containing the data.
        ignore_list (list): List of column names to ignore.    
    Returns:
        pd.DataFrame: DataFrame with imputed missing categorical values.
    """
    for col in df.columns:
        if col not in ignore_list:
            if df[col].dtype == 'object':
                df[col] = df[col].fillna(df[col].mode()[0])
    return df

In [9]:

train = impute_missing_cat_values(train, ignore_list=['cabin'])
train.isnull().sum()

Out[9]:

passengerid      0
survived         0
pclass           0
name             0
sex              0
age              0
sibsp            0
parch            0
ticket           0
fare             0
cabin          687
embarked         0
dtype: int64

⚠️ Attention: We will treat the missing values in cabin in the feature engineering step!!

Feature Engineering¶

Encode categorical features
Normalize numerical features
Create new features

In [10]:

# The categorical variables in the datasets

cat_cols = [
    col
    for col in train.columns if train[col].dtype == "object"
] 
print(cat_cols)

['name', 'sex', 'ticket', 'cabin', 'embarked']

In [11]:

# apply onehot encoding to the categorical columns
# use the sklearn library

from sklearn.preprocessing import OneHotEncoder

def onehot_encode(df, ignore_list):
    cat_cols = [
        col for col in df.columns if col not in ignore_list and df[col].dtype == 'object'
    ]
    encoder = OneHotEncoder()
    encoded = encoder.fit_transform(df[cat_cols])
    encoded_df = pd.DataFrame(encoded.toarray(), columns=encoder.get_feature_names_out(cat_cols))
    df = pd.concat([df, encoded_df], axis=1)
    df = df.drop(cat_cols, axis=1)
    return df

In [12]:

train = onehot_encode(train, ignore_list=['cabin', 'name', 'ticket'])

In [13]:

train.head(3)

Out[13]:

	passengerid	survived	pclass	name	age	sibsp	ticket	fare	cabin	sex_female	sex_male	embarked_C	embarked_S
0	1	0	3	Braund, Mr. Owen Harris	22.0	1	A/5 21171	7.2500	NaN	0.0	1.0	0.0	1.0
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	38.0	1	PC 17599	71.2833	C85	1.0	0.0	1.0	0.0
2	3	1	3	Heikkinen, Miss. Laina	26.0	0	STON/O2. 3101282	7.9250	NaN	1.0	0.0	0.0	1.0

In [ ]:

In [14]:

# transform numeric features, log transform `fare`
import numpy as np

def log_transform(df, features, drop=False):
    for feature in features:
        df[feature+'_log'] = np.log1p(df[feature]) 
    if drop:
        df = df.drop(features, axis=1)
    return df

In [15]:

train = log_transform(train, features=['fare'])
train.columns

Out[15]:

Index(['passengerid', 'survived', 'pclass', 'name', 'age', 'sibsp', 'parch',
       'ticket', 'fare', 'cabin', 'sex_female', 'sex_male', 'embarked_C',
       'embarked_Q', 'embarked_S', 'fare_log'],
      dtype='object')

In [16]:

# Create new features
import re

def create_features(df):
    df['has_cabin'] = df['cabin'].apply(lambda x: 0 if type(x) == float else 1)
    df['family_size'] = df['sibsp'] + df['parch'] + 1
    df['is_alone'] = df['family_size'].apply(lambda x: 1 if x == 1 else 0)
    df['title'] = df['name'].apply(lambda x: re.search('([A-Z][a-z]+)\\.', x).group(1))
    df['cabin'] = df['cabin'].fillna('U0')
    df['deck'] = df['cabin'].apply(lambda x: re.search('([A-Z]+)', x).group(1))
    df['name_len_cat'] = df['name'].apply(lambda x: 0 if len(x) <= 23 else 1 if len(x) <= 28 else 2 if len(x) <= 40 else 3)
    df['age_cat'] = df['age'].apply(lambda x: 0 if x <= 14 else 1 if x <= 30 else 2 if x <= 40 else 3 if x <= 50 else 4 if x <= 60 else 5)
    df['fare_log_cat'] = df['fare_log'].apply(lambda x: 0 if x <= 2.7 else 1 if x <= 3.2 else 2 if x <= 3.6 else 3)
    return df

In [17]:

train = create_features(train)
train.columns

Out[17]:

Index(['passengerid', 'survived', 'pclass', 'name', 'age', 'sibsp', 'parch',
       'ticket', 'fare', 'cabin', 'sex_female', 'sex_male', 'embarked_C',
       'embarked_Q', 'embarked_S', 'fare_log', 'has_cabin', 'family_size',
       'is_alone', 'title', 'deck', 'name_len_cat', 'age_cat', 'fare_log_cat'],
      dtype='object')

Put all together¶

In [18]:

import re
import pandas as pd
import numpy as np
from sklearn.preprocessing import OneHotEncoder

def impute_missing_num_values(df):
    """
    Impute missing values in numerical columns of a DataFrame using the median of each column.

    Args:
    df (pandas.DataFrame): The DataFrame to impute missing values in.

    Returns:
    pandas.DataFrame: The DataFrame with missing values imputed.
    """
    # Select only the numerical columns
    num_cols = df.select_dtypes(include=['float64', 'int64']).columns
    # Impute missing values with the median of each column
    for col in num_cols:
        df[col] = df[col].fillna(df[col].median())
    return df

def impute_missing_cat_values(df, ignore_list):
    """
    Impute missing categorical values with the most frequent value.

    Args:
        df (pd.DataFrame): DataFrame containing the data.
        ignore_list (list): List of column names to ignore.    
    Returns:
        pd.DataFrame: DataFrame with imputed missing categorical values.
    """
    for col in df.columns:
        if col not in ignore_list:
            if df[col].dtype == 'object':
                df[col] = df[col].fillna(df[col].mode()[0])
    return df

def log_transform(df, features, drop=False):
    for feature in features:
        df[feature+'_log'] = np.log1p(df[feature]) 
    if drop:
        df = df.drop(features, axis=1)
    return df

def create_features(df):
    df['has_cabin'] = df['cabin'].apply(lambda x: 0 if type(x) == float else 1)
    df['family_size'] = df['sibsp'] + df['parch'] + 1
    df['is_alone'] = df['family_size'].apply(lambda x: 1 if x == 1 else 0)
    df['title'] = df['name'].apply(lambda x: re.search('([A-Z][a-z]+)\\.', x).group(1))
    df['cabin'] = df['cabin'].fillna('U0')
    df['deck'] = df['cabin'].apply(lambda x: re.search('([A-Z]+)', x).group(1))
    df['name_len_cat'] = df['name'].apply(lambda x: 0 if len(x) <= 23 else 1 if len(x) <= 28 else 2 if len(x) <= 40 else 3)
    df['age_cat'] = df['age'].apply(lambda x: 0 if x <= 14 else 1 if x <= 30 else 2 if x <= 40 else 3 if x <= 50 else 4 if x <= 60 else 5)
    df['fare_log_cat'] = df['fare_log'].apply(lambda x: 0 if x <= 2.7 else 1 if x <= 3.2 else 2 if x <= 3.6 else 3)
    return df

def load_data():
    train = pd.read_csv('../data/titanic/train.csv')
    test = pd.read_csv('../data/titanic/test.csv')
    # convert all column names to lower cases
    train.columns = train.columns.str.lower()
    test.columns = test.columns.str.lower()    
    return train, test

def transform_data(df, encoder=None):
    df = impute_missing_num_values(df)
    df = impute_missing_cat_values(df, ['cabin', 'embarked'])
    df = log_transform(df, ['fare'])
    df = create_features(df)
    
    cat_attributes = ['sex', 'embarked', 'title', 'deck']
    if not encoder:
        encoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
        encoder.fit(df[cat_attributes])
    encoded = encoder.transform(df[cat_attributes])
    df = pd.concat([df, pd.DataFrame(encoded, columns=encoder.get_feature_names_out(cat_attributes))], axis=1)
    
    # drop columns that are not needed
    df = df.drop([
        'name', 'ticket', 'cabin', 'fare'
        # , 'age', 'fare', 'sibsp', 'parch'
    ] + cat_attributes, axis=1)
    return df, encoder

train, test = load_data()
train, encoder = transform_data(train)
test, _ = transform_data(test, encoder)

In [19]:

train.head(2)

Out[19]:

	passengerid	survived	pclass	age	sibsp	parch	fare_log	has_cabin	family_size	is_alone	name_len_cat	age_cat	fare_log_cat	sex_female	sex_male	embarked_C	embarked_Q	embarked_S	embarked_nan	title_Capt	title_Col	title_Countess	title_Don	title_Dr	title_Jonkheer	title_Lady	title_Major	title_Master	title_Miss	title_Mlle	title_Mme	title_Mr	title_Mrs	title_Ms	title_Rev	title_Sir	deck_A	deck_B	deck_C	deck_D	deck_E	deck_F	deck_G	deck_T	deck_U
0	1	0	3	22.0	1	0	2.110213	0	2	0	0	1	0	0.0	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0
1	2	1	1	38.0	1	0	4.280593	1	2	0	3	2	3	1.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0

In [20]:

test.shape

Out[20]:

(418, 44)

In [21]:

test.head(3)

Out[21]:

	passengerid	pclass	age	sibsp	fare_log	family_size	is_alone	name_len_cat	age_cat	sex_female	sex_male	embarked_Q	embarked_S	title_Mr	title_Mrs	deck_U
0	892	3	34.5	0	2.178064	1	1	0	2	0.0	1.0	1.0	0.0	1.0	0.0	1.0
1	893	3	47.0	1	2.079442	2	0	2	3	1.0	0.0	0.0	1.0	0.0	1.0	1.0
2	894	2	62.0	0	2.369075	1	1	1	5	0.0	1.0	1.0	0.0	1.0	0.0	1.0

In [ ]:

For standard transfromations, you could also use the pipeline modules from sklearn

https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

In [ ]:

Train a simple model¶

In [22]:

# train a simple logistic regression model to predict the survival label
from sklearn.linear_model import LogisticRegression

lr = LogisticRegression()
lr.fit(
    train.drop(columns=['survived', 'passengerid']), # everything except the survival label
    train['survived'] # the survival label
)

pred = lr.predict(test.drop(columns=['passengerid']))

/Users/xiangshiyin/Documents/Teaching/machine-learning-for-actuarial-science/.venv/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:465: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(

In [23]:

type(pred)

Out[23]:

numpy.ndarray

In [24]:

pred.shape

Out[24]:

(418,)

In [25]:

pred[:10]

Out[25]:

array([0, 1, 0, 0, 1, 0, 1, 0, 1, 0])

In [ ]:

In [26]:

df_submission = pd.concat([test['passengerid'], pd.DataFrame(pred, columns=['survived'])], axis=1)
df_submission.head(3)

Out[26]:

	passengerid	survived
0	892	0
1	893	1
2	894	0

In [27]:

df_submission.to_csv('../data/titanic/submission.csv', index=False)

In [ ]:

Evaluate the prediction results¶

https://www.kaggle.com/competitions/titanic/overview/evaluation

AutoML exploration¶

In [28]:

X_train = train.drop(columns=['survived', 'passengerid'])
y_train = train['survived']

In [29]:

X_train.head(2)

Out[29]:

	pclass	age	sibsp	parch	fare_log	has_cabin	family_size	is_alone	name_len_cat	age_cat	fare_log_cat	sex_female	sex_male	embarked_C	embarked_Q	embarked_S	embarked_nan	title_Capt	title_Col	title_Countess	title_Don	title_Dr	title_Jonkheer	title_Lady	title_Major	title_Master	title_Miss	title_Mlle	title_Mme	title_Mr	title_Mrs	title_Ms	title_Rev	title_Sir	deck_A	deck_B	deck_C	deck_D	deck_E	deck_F	deck_G	deck_T	deck_U
0	3	22.0	1	0	2.110213	0	2	0	0	1	0	0.0	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0
1	1	38.0	1	0	4.280593	1	2	0	3	2	3	1.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0

FLAML - https://github.com/microsoft/FLAML/tree/main

pip install flaml[automl]
Documentation
Best practices [link]

In [ ]:

In [30]:

from flaml import AutoML

automl = AutoML()
settings = {
    "time_budget": 60,  # total running time in seconds
    "metric": 'accuracy', 
                        # check the documentation for options of metrics (https://microsoft.github.io/FLAML/docs/Use-Cases/Task-Oriented-AutoML#optimization-metric)
    "task": 'classification',  # task type
    "log_file_name": 'automl_experiment.log',  # flaml log file
    "seed": 7654321,    # random seed
    "ensemble": False,
}
automl.fit(X_train, y_train, **settings)

[flaml.automl.logger: 02-17 20:34:43] {1728} INFO - task = classification
[flaml.automl.logger: 02-17 20:34:43] {1739} INFO - Evaluation method: cv
[flaml.automl.logger: 02-17 20:34:43] {1838} INFO - Minimizing error metric: 1-accuracy
[flaml.automl.logger: 02-17 20:34:43] {1955} INFO - List of ML learners in AutoML Run: ['lgbm', 'rf', 'xgboost', 'extra_tree', 'xgb_limitdepth', 'sgd', 'lrl1']
[flaml.automl.logger: 02-17 20:34:43] {2258} INFO - iteration 0, current learner lgbm
[flaml.automl.logger: 02-17 20:34:43] {2393} INFO - Estimated sufficient time budget=1181s. Estimated necessary time budget=27s.
[flaml.automl.logger: 02-17 20:34:43] {2442} INFO -  at 0.2s,	estimator lgbm's best error=0.2189,	best estimator lgbm's best error=0.2189
[flaml.automl.logger: 02-17 20:34:43] {2258} INFO - iteration 1, current learner lgbm
[flaml.automl.logger: 02-17 20:34:43] {2442} INFO -  at 0.3s,	estimator lgbm's best error=0.2189,	best estimator lgbm's best error=0.2189
[flaml.automl.logger: 02-17 20:34:43] {2258} INFO - iteration 2, current learner lgbm
[flaml.automl.logger: 02-17 20:34:43] {2442} INFO -  at 0.3s,	estimator lgbm's best error=0.1728,	best estimator lgbm's best error=0.1728
[flaml.automl.logger: 02-17 20:34:43] {2258} INFO - iteration 3, current learner sgd
[flaml.automl.logger: 02-17 20:34:43] {2442} INFO -  at 0.5s,	estimator sgd's best error=0.3704,	best estimator lgbm's best error=0.1728
[flaml.automl.logger: 02-17 20:34:43] {2258} INFO - iteration 4, current learner lgbm
[flaml.automl.logger: 02-17 20:34:44] {2442} INFO -  at 0.6s,	estimator lgbm's best error=0.1683,	best estimator lgbm's best error=0.1683
[flaml.automl.logger: 02-17 20:34:44] {2258} INFO - iteration 5, current learner lgbm
[flaml.automl.logger: 02-17 20:34:44] {2442} INFO -  at 0.9s,	estimator lgbm's best error=0.1683,	best estimator lgbm's best error=0.1683
[flaml.automl.logger: 02-17 20:34:44] {2258} INFO - iteration 6, current learner lgbm
[flaml.automl.logger: 02-17 20:34:44] {2442} INFO -  at 1.0s,	estimator lgbm's best error=0.1683,	best estimator lgbm's best error=0.1683
[flaml.automl.logger: 02-17 20:34:44] {2258} INFO - iteration 7, current learner lgbm
[flaml.automl.logger: 02-17 20:34:44] {2442} INFO -  at 1.1s,	estimator lgbm's best error=0.1650,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:44] {2258} INFO - iteration 8, current learner lgbm
[flaml.automl.logger: 02-17 20:34:44] {2442} INFO -  at 1.1s,	estimator lgbm's best error=0.1650,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:44] {2258} INFO - iteration 9, current learner sgd
[flaml.automl.logger: 02-17 20:34:44] {2442} INFO -  at 1.3s,	estimator sgd's best error=0.3704,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:44] {2258} INFO - iteration 10, current learner xgboost
[flaml.automl.logger: 02-17 20:34:44] {2442} INFO -  at 1.5s,	estimator xgboost's best error=0.2189,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:44] {2258} INFO - iteration 11, current learner xgboost
[flaml.automl.logger: 02-17 20:34:45] {2442} INFO -  at 1.7s,	estimator xgboost's best error=0.2189,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:45] {2258} INFO - iteration 12, current learner xgboost
[flaml.automl.logger: 02-17 20:34:45] {2442} INFO -  at 1.9s,	estimator xgboost's best error=0.1762,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:45] {2258} INFO - iteration 13, current learner extra_tree
[flaml.automl.logger: 02-17 20:34:45] {2442} INFO -  at 2.2s,	estimator extra_tree's best error=0.2043,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:45] {2258} INFO - iteration 14, current learner rf
[flaml.automl.logger: 02-17 20:34:45] {2442} INFO -  at 2.4s,	estimator rf's best error=0.2177,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:45] {2258} INFO - iteration 15, current learner rf
[flaml.automl.logger: 02-17 20:34:46] {2442} INFO -  at 2.8s,	estimator rf's best error=0.2054,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:46] {2258} INFO - iteration 16, current learner extra_tree
[flaml.automl.logger: 02-17 20:34:46] {2442} INFO -  at 3.0s,	estimator extra_tree's best error=0.2043,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:46] {2258} INFO - iteration 17, current learner rf
[flaml.automl.logger: 02-17 20:34:46] {2442} INFO -  at 3.3s,	estimator rf's best error=0.2054,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:46] {2258} INFO - iteration 18, current learner sgd
[flaml.automl.logger: 02-17 20:34:47] {2442} INFO -  at 3.9s,	estimator sgd's best error=0.3199,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:47] {2258} INFO - iteration 19, current learner lgbm
[flaml.automl.logger: 02-17 20:34:47] {2442} INFO -  at 4.0s,	estimator lgbm's best error=0.1650,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:47] {2258} INFO - iteration 20, current learner extra_tree
[flaml.automl.logger: 02-17 20:34:47] {2442} INFO -  at 4.3s,	estimator extra_tree's best error=0.2020,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:47] {2258} INFO - iteration 21, current learner lgbm
[flaml.automl.logger: 02-17 20:34:47] {2442} INFO -  at 4.4s,	estimator lgbm's best error=0.1650,	best estimator lgbm's best error=0.1650
[flaml.automl.logger: 02-17 20:34:47] {2258} INFO - iteration 22, current learner xgboost
[flaml.automl.logger: 02-17 20:34:48] {2442} INFO -  at 4.8s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:48] {2258} INFO - iteration 23, current learner xgboost
[flaml.automl.logger: 02-17 20:34:48] {2442} INFO -  at 5.0s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:48] {2258} INFO - iteration 24, current learner xgboost
[flaml.automl.logger: 02-17 20:34:48] {2442} INFO -  at 5.3s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:48] {2258} INFO - iteration 25, current learner xgboost
[flaml.automl.logger: 02-17 20:34:49] {2442} INFO -  at 5.6s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:49] {2258} INFO - iteration 26, current learner sgd
[flaml.automl.logger: 02-17 20:34:49] {2442} INFO -  at 6.0s,	estimator sgd's best error=0.3199,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:49] {2258} INFO - iteration 27, current learner xgboost
[flaml.automl.logger: 02-17 20:34:49] {2442} INFO -  at 6.2s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:49] {2258} INFO - iteration 28, current learner xgboost
[flaml.automl.logger: 02-17 20:34:50] {2442} INFO -  at 6.6s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:50] {2258} INFO - iteration 29, current learner extra_tree
[flaml.automl.logger: 02-17 20:34:50] {2442} INFO -  at 6.9s,	estimator extra_tree's best error=0.2020,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:50] {2258} INFO - iteration 30, current learner sgd
[flaml.automl.logger: 02-17 20:34:50] {2442} INFO -  at 7.0s,	estimator sgd's best error=0.3199,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:50] {2258} INFO - iteration 31, current learner lgbm
[flaml.automl.logger: 02-17 20:34:50] {2442} INFO -  at 7.1s,	estimator lgbm's best error=0.1650,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:50] {2258} INFO - iteration 32, current learner lgbm
[flaml.automl.logger: 02-17 20:34:50] {2442} INFO -  at 7.2s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:50] {2258} INFO - iteration 33, current learner lgbm
[flaml.automl.logger: 02-17 20:34:50] {2442} INFO -  at 7.3s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:50] {2258} INFO - iteration 34, current learner xgboost
[flaml.automl.logger: 02-17 20:34:50] {2442} INFO -  at 7.6s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:50] {2258} INFO - iteration 35, current learner lgbm
[flaml.automl.logger: 02-17 20:34:51] {2442} INFO -  at 7.7s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:51] {2258} INFO - iteration 36, current learner lgbm
[flaml.automl.logger: 02-17 20:34:51] {2442} INFO -  at 7.9s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:51] {2258} INFO - iteration 37, current learner sgd
[flaml.automl.logger: 02-17 20:34:51] {2442} INFO -  at 8.0s,	estimator sgd's best error=0.3199,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:51] {2258} INFO - iteration 38, current learner lgbm
[flaml.automl.logger: 02-17 20:34:51] {2442} INFO -  at 8.1s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:51] {2258} INFO - iteration 39, current learner rf
[flaml.automl.logger: 02-17 20:34:51] {2442} INFO -  at 8.4s,	estimator rf's best error=0.1975,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:51] {2258} INFO - iteration 40, current learner lgbm
[flaml.automl.logger: 02-17 20:34:52] {2442} INFO -  at 8.6s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:52] {2258} INFO - iteration 41, current learner xgboost
[flaml.automl.logger: 02-17 20:34:52] {2442} INFO -  at 9.0s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:52] {2258} INFO - iteration 42, current learner xgboost
[flaml.automl.logger: 02-17 20:34:52] {2442} INFO -  at 9.4s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:52] {2258} INFO - iteration 43, current learner lgbm
[flaml.automl.logger: 02-17 20:34:52] {2442} INFO -  at 9.5s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:52] {2258} INFO - iteration 44, current learner xgboost
[flaml.automl.logger: 02-17 20:34:53] {2442} INFO -  at 9.9s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:53] {2258} INFO - iteration 45, current learner sgd
[flaml.automl.logger: 02-17 20:34:53] {2442} INFO -  at 10.1s,	estimator sgd's best error=0.2222,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:53] {2258} INFO - iteration 46, current learner sgd
[flaml.automl.logger: 02-17 20:34:53] {2442} INFO -  at 10.2s,	estimator sgd's best error=0.2222,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:53] {2258} INFO - iteration 47, current learner lgbm
[flaml.automl.logger: 02-17 20:34:53] {2442} INFO -  at 10.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:53] {2258} INFO - iteration 48, current learner sgd
[flaml.automl.logger: 02-17 20:34:53] {2442} INFO -  at 10.5s,	estimator sgd's best error=0.2222,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:53] {2258} INFO - iteration 49, current learner lgbm
[flaml.automl.logger: 02-17 20:34:54] {2442} INFO -  at 10.7s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:54] {2258} INFO - iteration 50, current learner rf
[flaml.automl.logger: 02-17 20:34:54] {2442} INFO -  at 11.0s,	estimator rf's best error=0.1852,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:54] {2258} INFO - iteration 51, current learner sgd
[flaml.automl.logger: 02-17 20:34:54] {2442} INFO -  at 11.2s,	estimator sgd's best error=0.2222,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:54] {2258} INFO - iteration 52, current learner lgbm
[flaml.automl.logger: 02-17 20:34:54] {2442} INFO -  at 11.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:54] {2258} INFO - iteration 53, current learner rf
[flaml.automl.logger: 02-17 20:34:55] {2442} INFO -  at 11.7s,	estimator rf's best error=0.1762,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:55] {2258} INFO - iteration 54, current learner lgbm
[flaml.automl.logger: 02-17 20:34:55] {2442} INFO -  at 11.8s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:55] {2258} INFO - iteration 55, current learner lgbm
[flaml.automl.logger: 02-17 20:34:55] {2442} INFO -  at 11.9s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:55] {2258} INFO - iteration 56, current learner xgboost
[flaml.automl.logger: 02-17 20:34:55] {2442} INFO -  at 12.2s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:55] {2258} INFO - iteration 57, current learner lgbm
[flaml.automl.logger: 02-17 20:34:55] {2442} INFO -  at 12.5s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:55] {2258} INFO - iteration 58, current learner rf
[flaml.automl.logger: 02-17 20:34:56] {2442} INFO -  at 12.7s,	estimator rf's best error=0.1762,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:56] {2258} INFO - iteration 59, current learner rf
[flaml.automl.logger: 02-17 20:34:56] {2442} INFO -  at 13.1s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:56] {2258} INFO - iteration 60, current learner lgbm
[flaml.automl.logger: 02-17 20:34:56] {2442} INFO -  at 13.2s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:56] {2258} INFO - iteration 61, current learner xgboost
[flaml.automl.logger: 02-17 20:34:56] {2442} INFO -  at 13.5s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:56] {2258} INFO - iteration 62, current learner lgbm
[flaml.automl.logger: 02-17 20:34:57] {2442} INFO -  at 13.6s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:57] {2258} INFO - iteration 63, current learner xgboost
[flaml.automl.logger: 02-17 20:34:57] {2442} INFO -  at 13.9s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:57] {2258} INFO - iteration 64, current learner sgd
[flaml.automl.logger: 02-17 20:34:57] {2442} INFO -  at 14.1s,	estimator sgd's best error=0.2222,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:57] {2258} INFO - iteration 65, current learner lgbm
[flaml.automl.logger: 02-17 20:34:57] {2442} INFO -  at 14.3s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:57] {2258} INFO - iteration 66, current learner rf
[flaml.automl.logger: 02-17 20:34:57] {2442} INFO -  at 14.6s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:57] {2258} INFO - iteration 67, current learner sgd
[flaml.automl.logger: 02-17 20:34:58] {2442} INFO -  at 14.9s,	estimator sgd's best error=0.2222,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:58] {2258} INFO - iteration 68, current learner rf
[flaml.automl.logger: 02-17 20:34:58] {2442} INFO -  at 15.3s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:58] {2258} INFO - iteration 69, current learner lgbm
[flaml.automl.logger: 02-17 20:34:58] {2442} INFO -  at 15.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:58] {2258} INFO - iteration 70, current learner xgboost
[flaml.automl.logger: 02-17 20:34:59] {2442} INFO -  at 15.6s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:59] {2258} INFO - iteration 71, current learner xgboost
[flaml.automl.logger: 02-17 20:34:59] {2442} INFO -  at 15.9s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:59] {2258} INFO - iteration 72, current learner xgboost
[flaml.automl.logger: 02-17 20:34:59] {2442} INFO -  at 16.2s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:34:59] {2258} INFO - iteration 73, current learner sgd
[flaml.automl.logger: 02-17 20:35:00] {2442} INFO -  at 16.7s,	estimator sgd's best error=0.2222,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:00] {2258} INFO - iteration 74, current learner xgboost
[flaml.automl.logger: 02-17 20:35:00] {2442} INFO -  at 17.1s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:00] {2258} INFO - iteration 75, current learner sgd
[flaml.automl.logger: 02-17 20:35:00] {2442} INFO -  at 17.2s,	estimator sgd's best error=0.2144,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:00] {2258} INFO - iteration 76, current learner lgbm
[flaml.automl.logger: 02-17 20:35:00] {2442} INFO -  at 17.3s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:00] {2258} INFO - iteration 77, current learner lgbm
[flaml.automl.logger: 02-17 20:35:00] {2442} INFO -  at 17.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:00] {2258} INFO - iteration 78, current learner xgboost
[flaml.automl.logger: 02-17 20:35:01] {2442} INFO -  at 17.7s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:01] {2258} INFO - iteration 79, current learner lgbm
[flaml.automl.logger: 02-17 20:35:01] {2442} INFO -  at 17.9s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:01] {2258} INFO - iteration 80, current learner lgbm
[flaml.automl.logger: 02-17 20:35:01] {2442} INFO -  at 18.1s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:01] {2258} INFO - iteration 81, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:01] {2442} INFO -  at 18.5s,	estimator extra_tree's best error=0.2020,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:01] {2258} INFO - iteration 82, current learner lgbm
[flaml.automl.logger: 02-17 20:35:02] {2442} INFO -  at 18.7s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:02] {2258} INFO - iteration 83, current learner xgboost
[flaml.automl.logger: 02-17 20:35:02] {2442} INFO -  at 19.1s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:02] {2258} INFO - iteration 84, current learner xgboost
[flaml.automl.logger: 02-17 20:35:02] {2442} INFO -  at 19.4s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:02] {2258} INFO - iteration 85, current learner xgboost
[flaml.automl.logger: 02-17 20:35:03] {2442} INFO -  at 19.7s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:03] {2258} INFO - iteration 86, current learner rf
[flaml.automl.logger: 02-17 20:35:03] {2442} INFO -  at 20.1s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:03] {2258} INFO - iteration 87, current learner xgboost
[flaml.automl.logger: 02-17 20:35:03] {2442} INFO -  at 20.4s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:03] {2258} INFO - iteration 88, current learner lgbm
[flaml.automl.logger: 02-17 20:35:03] {2442} INFO -  at 20.5s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:03] {2258} INFO - iteration 89, current learner lgbm
[flaml.automl.logger: 02-17 20:35:03] {2442} INFO -  at 20.5s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:03] {2258} INFO - iteration 90, current learner xgboost
[flaml.automl.logger: 02-17 20:35:04] {2442} INFO -  at 20.8s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:04] {2258} INFO - iteration 91, current learner rf
[flaml.automl.logger: 02-17 20:35:04] {2442} INFO -  at 21.2s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:04] {2258} INFO - iteration 92, current learner sgd
[flaml.automl.logger: 02-17 20:35:04] {2442} INFO -  at 21.5s,	estimator sgd's best error=0.2144,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:04] {2258} INFO - iteration 93, current learner rf
[flaml.automl.logger: 02-17 20:35:05] {2442} INFO -  at 21.9s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:05] {2258} INFO - iteration 94, current learner rf
[flaml.automl.logger: 02-17 20:35:05] {2442} INFO -  at 22.2s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:05] {2258} INFO - iteration 95, current learner lgbm
[flaml.automl.logger: 02-17 20:35:05] {2442} INFO -  at 22.3s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:05] {2258} INFO - iteration 96, current learner sgd
[flaml.automl.logger: 02-17 20:35:05] {2442} INFO -  at 22.4s,	estimator sgd's best error=0.2144,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:05] {2258} INFO - iteration 97, current learner rf
[flaml.automl.logger: 02-17 20:35:06] {2442} INFO -  at 22.8s,	estimator rf's best error=0.1684,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:06] {2258} INFO - iteration 98, current learner rf
[flaml.automl.logger: 02-17 20:35:06] {2442} INFO -  at 23.2s,	estimator rf's best error=0.1661,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:06] {2258} INFO - iteration 99, current learner lgbm
[flaml.automl.logger: 02-17 20:35:06] {2442} INFO -  at 23.3s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:06] {2258} INFO - iteration 100, current learner lgbm
[flaml.automl.logger: 02-17 20:35:06] {2442} INFO -  at 23.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:06] {2258} INFO - iteration 101, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:07] {2442} INFO -  at 23.7s,	estimator extra_tree's best error=0.2020,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:07] {2258} INFO - iteration 102, current learner rf
[flaml.automl.logger: 02-17 20:35:07] {2442} INFO -  at 24.0s,	estimator rf's best error=0.1661,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:07] {2258} INFO - iteration 103, current learner lgbm
[flaml.automl.logger: 02-17 20:35:07] {2442} INFO -  at 24.2s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:07] {2258} INFO - iteration 104, current learner xgboost
[flaml.automl.logger: 02-17 20:35:07] {2442} INFO -  at 24.5s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:07] {2258} INFO - iteration 105, current learner lgbm
[flaml.automl.logger: 02-17 20:35:08] {2442} INFO -  at 24.7s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:08] {2258} INFO - iteration 106, current learner rf
[flaml.automl.logger: 02-17 20:35:08] {2442} INFO -  at 24.9s,	estimator rf's best error=0.1661,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:08] {2258} INFO - iteration 107, current learner xgboost
[flaml.automl.logger: 02-17 20:35:08] {2442} INFO -  at 25.3s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:08] {2258} INFO - iteration 108, current learner lgbm
[flaml.automl.logger: 02-17 20:35:08] {2442} INFO -  at 25.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:08] {2258} INFO - iteration 109, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:09] {2442} INFO -  at 25.7s,	estimator extra_tree's best error=0.2020,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:09] {2258} INFO - iteration 110, current learner xgboost
[flaml.automl.logger: 02-17 20:35:09] {2442} INFO -  at 26.0s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:09] {2258} INFO - iteration 111, current learner lgbm
[flaml.automl.logger: 02-17 20:35:09] {2442} INFO -  at 26.1s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:09] {2258} INFO - iteration 112, current learner sgd
[flaml.automl.logger: 02-17 20:35:09] {2442} INFO -  at 26.3s,	estimator sgd's best error=0.2144,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:09] {2258} INFO - iteration 113, current learner xgboost
[flaml.automl.logger: 02-17 20:35:10] {2442} INFO -  at 26.7s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:10] {2258} INFO - iteration 114, current learner sgd
[flaml.automl.logger: 02-17 20:35:10] {2442} INFO -  at 27.0s,	estimator sgd's best error=0.2144,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:10] {2258} INFO - iteration 115, current learner lgbm
[flaml.automl.logger: 02-17 20:35:10] {2442} INFO -  at 27.2s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:10] {2258} INFO - iteration 116, current learner lgbm
[flaml.automl.logger: 02-17 20:35:10] {2442} INFO -  at 27.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:10] {2258} INFO - iteration 117, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:11] {2442} INFO -  at 27.7s,	estimator extra_tree's best error=0.1964,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:11] {2258} INFO - iteration 118, current learner xgboost
[flaml.automl.logger: 02-17 20:35:11] {2442} INFO -  at 28.0s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:11] {2258} INFO - iteration 119, current learner lgbm
[flaml.automl.logger: 02-17 20:35:11] {2442} INFO -  at 28.1s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:11] {2258} INFO - iteration 120, current learner xgboost
[flaml.automl.logger: 02-17 20:35:11] {2442} INFO -  at 28.4s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:11] {2258} INFO - iteration 121, current learner xgboost
[flaml.automl.logger: 02-17 20:35:12] {2442} INFO -  at 28.8s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:12] {2258} INFO - iteration 122, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:12] {2442} INFO -  at 29.1s,	estimator extra_tree's best error=0.1964,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:12] {2258} INFO - iteration 123, current learner xgboost
[flaml.automl.logger: 02-17 20:35:12] {2442} INFO -  at 29.5s,	estimator xgboost's best error=0.1582,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:12] {2258} INFO - iteration 124, current learner lgbm
[flaml.automl.logger: 02-17 20:35:13] {2442} INFO -  at 29.6s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:13] {2258} INFO - iteration 125, current learner lgbm
[flaml.automl.logger: 02-17 20:35:13] {2442} INFO -  at 29.8s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:13] {2258} INFO - iteration 126, current learner lgbm
[flaml.automl.logger: 02-17 20:35:13] {2442} INFO -  at 29.9s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:13] {2258} INFO - iteration 127, current learner lgbm
[flaml.automl.logger: 02-17 20:35:13] {2442} INFO -  at 30.0s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:13] {2258} INFO - iteration 128, current learner lgbm
[flaml.automl.logger: 02-17 20:35:13] {2442} INFO -  at 30.4s,	estimator lgbm's best error=0.1583,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:13] {2258} INFO - iteration 129, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:14] {2442} INFO -  at 30.7s,	estimator extra_tree's best error=0.1964,	best estimator xgboost's best error=0.1582
[flaml.automl.logger: 02-17 20:35:14] {2258} INFO - iteration 130, current learner rf
[flaml.automl.logger: 02-17 20:35:14] {2442} INFO -  at 31.0s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:14] {2258} INFO - iteration 131, current learner rf
[flaml.automl.logger: 02-17 20:35:14] {2442} INFO -  at 31.3s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:14] {2258} INFO - iteration 132, current learner rf
[flaml.automl.logger: 02-17 20:35:15] {2442} INFO -  at 31.7s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:15] {2258} INFO - iteration 133, current learner lgbm
[flaml.automl.logger: 02-17 20:35:15] {2442} INFO -  at 32.2s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:15] {2258} INFO - iteration 134, current learner rf
[flaml.automl.logger: 02-17 20:35:15] {2442} INFO -  at 32.5s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:15] {2258} INFO - iteration 135, current learner rf
[flaml.automl.logger: 02-17 20:35:16] {2442} INFO -  at 32.9s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:16] {2258} INFO - iteration 136, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:16] {2442} INFO -  at 33.1s,	estimator extra_tree's best error=0.1841,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:16] {2258} INFO - iteration 137, current learner rf
[flaml.automl.logger: 02-17 20:35:16] {2442} INFO -  at 33.4s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:16] {2258} INFO - iteration 138, current learner lgbm
[flaml.automl.logger: 02-17 20:35:16] {2442} INFO -  at 33.6s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:16] {2258} INFO - iteration 139, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:17] {2442} INFO -  at 33.9s,	estimator extra_tree's best error=0.1841,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:17] {2258} INFO - iteration 140, current learner rf
[flaml.automl.logger: 02-17 20:35:17] {2442} INFO -  at 34.3s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:17] {2258} INFO - iteration 141, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:18] {2442} INFO -  at 34.6s,	estimator extra_tree's best error=0.1807,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:18] {2258} INFO - iteration 142, current learner rf
[flaml.automl.logger: 02-17 20:35:18] {2442} INFO -  at 35.0s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:18] {2258} INFO - iteration 143, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:18] {2442} INFO -  at 35.2s,	estimator extra_tree's best error=0.1807,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:18] {2258} INFO - iteration 144, current learner xgboost
[flaml.automl.logger: 02-17 20:35:18] {2442} INFO -  at 35.5s,	estimator xgboost's best error=0.1582,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:18] {2258} INFO - iteration 145, current learner rf
[flaml.automl.logger: 02-17 20:35:19] {2442} INFO -  at 35.9s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:19] {2258} INFO - iteration 146, current learner lgbm
[flaml.automl.logger: 02-17 20:35:19] {2442} INFO -  at 36.0s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:19] {2258} INFO - iteration 147, current learner lgbm
[flaml.automl.logger: 02-17 20:35:19] {2442} INFO -  at 36.2s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:19] {2258} INFO - iteration 148, current learner lgbm
[flaml.automl.logger: 02-17 20:35:19] {2442} INFO -  at 36.4s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:19] {2258} INFO - iteration 149, current learner lgbm
[flaml.automl.logger: 02-17 20:35:19] {2442} INFO -  at 36.5s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:19] {2258} INFO - iteration 150, current learner lgbm
[flaml.automl.logger: 02-17 20:35:20] {2442} INFO -  at 36.6s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:20] {2258} INFO - iteration 151, current learner rf
[flaml.automl.logger: 02-17 20:35:20] {2442} INFO -  at 37.1s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:20] {2258} INFO - iteration 152, current learner xgboost
[flaml.automl.logger: 02-17 20:35:20] {2442} INFO -  at 37.4s,	estimator xgboost's best error=0.1582,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:20] {2258} INFO - iteration 153, current learner rf
[flaml.automl.logger: 02-17 20:35:21] {2442} INFO -  at 37.7s,	estimator rf's best error=0.1560,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:21] {2258} INFO - iteration 154, current learner lgbm
[flaml.automl.logger: 02-17 20:35:21] {2442} INFO -  at 37.9s,	estimator lgbm's best error=0.1583,	best estimator rf's best error=0.1560
[flaml.automl.logger: 02-17 20:35:21] {2258} INFO - iteration 155, current learner lgbm
[flaml.automl.logger: 02-17 20:35:21] {2442} INFO -  at 38.1s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:21] {2258} INFO - iteration 156, current learner sgd
[flaml.automl.logger: 02-17 20:35:21] {2442} INFO -  at 38.2s,	estimator sgd's best error=0.2144,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:21] {2258} INFO - iteration 157, current learner rf
[flaml.automl.logger: 02-17 20:35:22] {2442} INFO -  at 38.6s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:22] {2258} INFO - iteration 158, current learner lgbm
[flaml.automl.logger: 02-17 20:35:22] {2442} INFO -  at 38.8s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:22] {2258} INFO - iteration 159, current learner rf
[flaml.automl.logger: 02-17 20:35:22] {2442} INFO -  at 39.1s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:22] {2258} INFO - iteration 160, current learner xgboost
[flaml.automl.logger: 02-17 20:35:23] {2442} INFO -  at 39.6s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:23] {2258} INFO - iteration 161, current learner lgbm
[flaml.automl.logger: 02-17 20:35:23] {2442} INFO -  at 39.7s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:23] {2258} INFO - iteration 162, current learner rf
[flaml.automl.logger: 02-17 20:35:23] {2442} INFO -  at 40.1s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:23] {2258} INFO - iteration 163, current learner sgd
[flaml.automl.logger: 02-17 20:35:23] {2442} INFO -  at 40.2s,	estimator sgd's best error=0.2144,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:23] {2258} INFO - iteration 164, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:23] {2442} INFO -  at 40.6s,	estimator extra_tree's best error=0.1762,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:23] {2258} INFO - iteration 165, current learner rf
[flaml.automl.logger: 02-17 20:35:24] {2442} INFO -  at 40.9s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:24] {2258} INFO - iteration 166, current learner rf
[flaml.automl.logger: 02-17 20:35:24] {2442} INFO -  at 41.4s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:24] {2258} INFO - iteration 167, current learner xgboost
[flaml.automl.logger: 02-17 20:35:25] {2442} INFO -  at 41.6s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:25] {2258} INFO - iteration 168, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:25] {2442} INFO -  at 42.0s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:25] {2258} INFO - iteration 169, current learner rf
[flaml.automl.logger: 02-17 20:35:25] {2442} INFO -  at 42.3s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:25] {2258} INFO - iteration 170, current learner lgbm
[flaml.automl.logger: 02-17 20:35:25] {2442} INFO -  at 42.5s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:25] {2258} INFO - iteration 171, current learner lgbm
[flaml.automl.logger: 02-17 20:35:26] {2442} INFO -  at 42.6s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:26] {2258} INFO - iteration 172, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:26] {2442} INFO -  at 43.0s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:26] {2258} INFO - iteration 173, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:26] {2442} INFO -  at 43.2s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:26] {2258} INFO - iteration 174, current learner xgboost
[flaml.automl.logger: 02-17 20:35:27] {2442} INFO -  at 43.7s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:27] {2258} INFO - iteration 175, current learner rf
[flaml.automl.logger: 02-17 20:35:27] {2442} INFO -  at 44.1s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:27] {2258} INFO - iteration 176, current learner sgd
[flaml.automl.logger: 02-17 20:35:27] {2442} INFO -  at 44.2s,	estimator sgd's best error=0.2144,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:27] {2258} INFO - iteration 177, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:27] {2442} INFO -  at 44.6s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:27] {2258} INFO - iteration 178, current learner rf
[flaml.automl.logger: 02-17 20:35:28] {2442} INFO -  at 44.8s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:28] {2258} INFO - iteration 179, current learner xgboost
[flaml.automl.logger: 02-17 20:35:28] {2442} INFO -  at 45.2s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:28] {2258} INFO - iteration 180, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:28] {2442} INFO -  at 45.5s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:28] {2258} INFO - iteration 181, current learner xgboost
[flaml.automl.logger: 02-17 20:35:29] {2442} INFO -  at 45.8s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:29] {2258} INFO - iteration 182, current learner rf
[flaml.automl.logger: 02-17 20:35:29] {2442} INFO -  at 46.1s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:29] {2258} INFO - iteration 183, current learner lgbm
[flaml.automl.logger: 02-17 20:35:29] {2442} INFO -  at 46.3s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:29] {2258} INFO - iteration 184, current learner rf
[flaml.automl.logger: 02-17 20:35:30] {2442} INFO -  at 46.6s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:30] {2258} INFO - iteration 185, current learner xgboost
[flaml.automl.logger: 02-17 20:35:30] {2442} INFO -  at 47.0s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:30] {2258} INFO - iteration 186, current learner lgbm
[flaml.automl.logger: 02-17 20:35:30] {2442} INFO -  at 47.1s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:30] {2258} INFO - iteration 187, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:30] {2442} INFO -  at 47.5s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:30] {2258} INFO - iteration 188, current learner lgbm
[flaml.automl.logger: 02-17 20:35:31] {2442} INFO -  at 47.6s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:31] {2258} INFO - iteration 189, current learner lgbm
[flaml.automl.logger: 02-17 20:35:31] {2442} INFO -  at 47.7s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:31] {2258} INFO - iteration 190, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:31] {2442} INFO -  at 48.0s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:31] {2258} INFO - iteration 191, current learner lgbm
[flaml.automl.logger: 02-17 20:35:31] {2442} INFO -  at 48.2s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:31] {2258} INFO - iteration 192, current learner xgboost
[flaml.automl.logger: 02-17 20:35:32] {2442} INFO -  at 48.7s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:32] {2258} INFO - iteration 193, current learner rf
[flaml.automl.logger: 02-17 20:35:32] {2442} INFO -  at 49.0s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:32] {2258} INFO - iteration 194, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:32] {2442} INFO -  at 49.3s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:32] {2258} INFO - iteration 195, current learner lgbm
[flaml.automl.logger: 02-17 20:35:32] {2442} INFO -  at 49.6s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:32] {2258} INFO - iteration 196, current learner lgbm
[flaml.automl.logger: 02-17 20:35:33] {2442} INFO -  at 49.7s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:33] {2258} INFO - iteration 197, current learner xgboost
[flaml.automl.logger: 02-17 20:35:33] {2442} INFO -  at 50.1s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:33] {2258} INFO - iteration 198, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:33] {2442} INFO -  at 50.3s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:33] {2258} INFO - iteration 199, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:34] {2442} INFO -  at 50.7s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:34] {2258} INFO - iteration 200, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:34] {2442} INFO -  at 51.1s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:34] {2258} INFO - iteration 201, current learner lgbm
[flaml.automl.logger: 02-17 20:35:34] {2442} INFO -  at 51.2s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:34] {2258} INFO - iteration 202, current learner lgbm
[flaml.automl.logger: 02-17 20:35:34] {2442} INFO -  at 51.4s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:34] {2258} INFO - iteration 203, current learner rf
[flaml.automl.logger: 02-17 20:35:35] {2442} INFO -  at 51.8s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:35] {2258} INFO - iteration 204, current learner sgd
[flaml.automl.logger: 02-17 20:35:35] {2442} INFO -  at 51.9s,	estimator sgd's best error=0.2144,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:35] {2258} INFO - iteration 205, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:35] {2442} INFO -  at 52.2s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:35] {2258} INFO - iteration 206, current learner rf
[flaml.automl.logger: 02-17 20:35:35] {2442} INFO -  at 52.6s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:35] {2258} INFO - iteration 207, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:36] {2442} INFO -  at 52.9s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:36] {2258} INFO - iteration 208, current learner xgboost
[flaml.automl.logger: 02-17 20:35:36] {2442} INFO -  at 53.2s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:36] {2258} INFO - iteration 209, current learner rf
[flaml.automl.logger: 02-17 20:35:36] {2442} INFO -  at 53.6s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:36] {2258} INFO - iteration 210, current learner rf
[flaml.automl.logger: 02-17 20:35:37] {2442} INFO -  at 54.0s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:37] {2258} INFO - iteration 211, current learner rf
[flaml.automl.logger: 02-17 20:35:37] {2442} INFO -  at 54.3s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:37] {2258} INFO - iteration 212, current learner rf
[flaml.automl.logger: 02-17 20:35:38] {2442} INFO -  at 54.6s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:38] {2258} INFO - iteration 213, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:38] {2442} INFO -  at 54.9s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:38] {2258} INFO - iteration 214, current learner extra_tree
[flaml.automl.logger: 02-17 20:35:38] {2442} INFO -  at 55.3s,	estimator extra_tree's best error=0.1684,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:38] {2258} INFO - iteration 215, current learner lgbm
[flaml.automl.logger: 02-17 20:35:38] {2442} INFO -  at 55.4s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:38] {2258} INFO - iteration 216, current learner sgd
[flaml.automl.logger: 02-17 20:35:38] {2442} INFO -  at 55.5s,	estimator sgd's best error=0.2144,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:38] {2258} INFO - iteration 217, current learner lgbm
[flaml.automl.logger: 02-17 20:35:39] {2442} INFO -  at 55.7s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:39] {2258} INFO - iteration 218, current learner lgbm
[flaml.automl.logger: 02-17 20:35:39] {2442} INFO -  at 56.0s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:39] {2258} INFO - iteration 219, current learner lgbm
[flaml.automl.logger: 02-17 20:35:39] {2442} INFO -  at 56.1s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:39] {2258} INFO - iteration 220, current learner rf
[flaml.automl.logger: 02-17 20:35:39] {2442} INFO -  at 56.5s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:39] {2258} INFO - iteration 221, current learner lgbm
[flaml.automl.logger: 02-17 20:35:40] {2442} INFO -  at 56.7s,	estimator lgbm's best error=0.1526,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:40] {2258} INFO - iteration 222, current learner sgd
[flaml.automl.logger: 02-17 20:35:40] {2442} INFO -  at 56.8s,	estimator sgd's best error=0.2144,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:40] {2258} INFO - iteration 223, current learner xgboost
[flaml.automl.logger: 02-17 20:35:40] {2442} INFO -  at 57.1s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1526
[flaml.automl.logger: 02-17 20:35:40] {2258} INFO - iteration 224, current learner lgbm
[flaml.automl.logger: 02-17 20:35:40] {2442} INFO -  at 57.5s,	estimator lgbm's best error=0.1515,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:40] {2258} INFO - iteration 225, current learner lgbm
[flaml.automl.logger: 02-17 20:35:41] {2442} INFO -  at 57.6s,	estimator lgbm's best error=0.1515,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:41] {2258} INFO - iteration 226, current learner lgbm
[flaml.automl.logger: 02-17 20:35:41] {2442} INFO -  at 57.8s,	estimator lgbm's best error=0.1515,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:41] {2258} INFO - iteration 227, current learner lgbm
[flaml.automl.logger: 02-17 20:35:41] {2442} INFO -  at 57.9s,	estimator lgbm's best error=0.1515,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:41] {2258} INFO - iteration 228, current learner rf
[flaml.automl.logger: 02-17 20:35:41] {2442} INFO -  at 58.2s,	estimator rf's best error=0.1560,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:41] {2258} INFO - iteration 229, current learner lgbm
[flaml.automl.logger: 02-17 20:35:41] {2442} INFO -  at 58.5s,	estimator lgbm's best error=0.1515,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:41] {2258} INFO - iteration 230, current learner xgboost
[flaml.automl.logger: 02-17 20:35:42] {2442} INFO -  at 58.9s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:42] {2258} INFO - iteration 231, current learner lgbm
[flaml.automl.logger: 02-17 20:35:42] {2442} INFO -  at 59.1s,	estimator lgbm's best error=0.1515,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:42] {2258} INFO - iteration 232, current learner xgboost
[flaml.automl.logger: 02-17 20:35:42] {2442} INFO -  at 59.5s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:42] {2258} INFO - iteration 233, current learner xgboost
[flaml.automl.logger: 02-17 20:35:43] {2442} INFO -  at 59.9s,	estimator xgboost's best error=0.1582,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:43] {2258} INFO - iteration 234, current learner xgb_limitdepth
[flaml.automl.logger: 02-17 20:35:43] {2442} INFO -  at 60.2s,	estimator xgb_limitdepth's best error=0.2099,	best estimator lgbm's best error=0.1515
[flaml.automl.logger: 02-17 20:35:43] {2685} INFO - retrain lgbm for 0.1s
[flaml.automl.logger: 02-17 20:35:43] {2688} INFO - retrained model: LGBMClassifier(colsample_bytree=np.float64(0.9263510142224147),
               learning_rate=np.float64(0.11073419548910175), max_bin=1023,
               min_child_samples=7, n_estimators=26, n_jobs=-1, num_leaves=35,
               reg_alpha=np.float64(0.8491669558916534),
               reg_lambda=np.float64(0.08792333670079354), verbose=-1)
[flaml.automl.logger: 02-17 20:35:43] {1985} INFO - fit succeeded
[flaml.automl.logger: 02-17 20:35:43] {1986} INFO - Time taken to find the best model: 57.45935106277466

In [31]:

X_test = test.drop(columns=['passengerid'])
y_test_pred = automl.predict(X_test)
df_submission = pd.concat([test['passengerid'], pd.DataFrame(y_test_pred, columns=['survived'])], axis=1)
df_submission.head(3)

Out[31]:

	passengerid	survived
0	892	0
1	893	0
2	894	0

In [32]:

df_submission.to_csv('../data/titanic/submission.csv', index=False)

In [33]:

automl.best_estimator

Out[33]:

'lgbm'

In [34]:

automl.best_config

Out[34]:

{'n_estimators': 26,
 'num_leaves': 35,
 'min_child_samples': 7,
 'learning_rate': np.float64(0.11073419548910175),
 'log_max_bin': 10,
 'colsample_bytree': np.float64(0.9263510142224147),
 'reg_alpha': np.float64(0.8491669558916534),
 'reg_lambda': np.float64(0.08792333670079354)}

In [35]:

automl.model.estimator.feature_importances_

Out[35]:

array([ 32, 262,   1,   4, 292,  16,  28,   0,  29,   1,   2,  17,   3,
        14,   0,  24,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         1,   0,   0,  26,   1,   0,   0,   0,   2,   0,   8,   3,   8,
         0,   0,   0,   5], dtype=int32)

In [36]:

# Get feature importance values
importances = automl.model.estimator.feature_importances_
# Get feature names from the dataset
feature_names = X_train.columns
# Create a DataFrame for better visualization
feature_importance_df = pd.DataFrame({'Feature': feature_names, 'Importance': importances})
# Sort by importance (highest first)
feature_importance_df = feature_importance_df.sort_values(by='Importance', ascending=False)

feature_importance_df

Out[36]:

	Feature	Importance
4	fare_log	292
1	age	262
0	pclass	32
8	name_len_cat	29
6	family_size	28
29	title_Mr	26
15	embarked_S	24
11	sex_female	17
5	has_cabin	16
13	embarked_C	14
36	deck_C	8
38	deck_E	8
42	deck_U	5
3	parch	4
12	sex_male	3
37	deck_D	3
34	deck_A	2
10	fare_log_cat	2
2	sibsp	1
26	title_Miss	1
9	age_cat	1
30	title_Mrs	1
7	is_alone	0
18	title_Col	0
14	embarked_Q	0
17	title_Capt	0
16	embarked_nan	0
25	title_Master	0
24	title_Major	0
23	title_Lady	0
22	title_Jonkheer	0
20	title_Don	0
21	title_Dr	0
19	title_Countess	0
28	title_Mme	0
33	title_Sir	0
32	title_Rev	0
31	title_Ms	0
27	title_Mlle	0
35	deck_B	0
39	deck_F	0
40	deck_G	0
41	deck_T	0

In [37]:

import matplotlib.pyplot as plt

# Plot feature importance
plt.figure(figsize=(10, 6))  # Adjust figure size for better readability
plt.bar(feature_importance_df['Feature'], feature_importance_df['Importance'], color='skyblue')

# Rotate feature names for readability
plt.xticks(rotation=45, ha='right')  # Tilt labels 45 degrees and align to the right

# Labels and title
plt.xlabel('Feature Name')
plt.ylabel('Importance Score')
plt.title('Feature Importance')

plt.show()

Model Evaluations¶

Model evaluations can be approached from multiple perspectives:

From the Perspective of Evaluation Metrics:
- Prediction Quality: How well does the model predict or classify new data?
- Interpretability: How easily can the model’s predictions be understood and explained?
From the Perspective of the ML Workflow:
- Offline Evaluations: Assessment of model performance on the training and test datasets.
- Online Evaluations: Evaluation of model performance using live, production data.

Prediction Quality¶

Classification Problems¶

Accuracy¶

$$ \mathrm{Accuracy} = \frac{\mathrm{TP} + \mathrm{TN}}{\mathrm{TP} + \mathrm{FP} + \mathrm{FN} + \mathrm{TN}} $$

In [38]:

# Cross Validation Classification Accuracy
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = pd.read_csv(url, names=names)
array = dataframe.values
X = array[:,0:8]
Y = array[:,8]

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=7)
lr = LogisticRegression()
lr.fit(X_train, Y_train)

Y_test_pred = lr.predict(X_test)
accuracy = accuracy_score(Y_test, Y_test_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))

Accuracy: 79.22%

In [40]:

np.sum(Y_test == Y_test_pred) / len(Y_test)

Out[40]:

np.float64(0.7922077922077922)

In [ ]:

In [41]:

from sklearn.metrics import precision_score, recall_score

precision = precision_score(Y_test, Y_test_pred)
recall = recall_score(Y_test, Y_test_pred)

print(f"precision: {precision}")
print(f"recall: {recall}")

precision: 0.7906976744186046
recall: 0.5964912280701754

In [42]:

from sklearn.metrics import f1_score
f1_score = f1_score(Y_test, Y_test_pred)
print(f"f1 score: {f1_score}")

f1 score: 0.68

In [43]:

import matplotlib.pyplot as plt
import numpy as np

precision = np.linspace(0, 1, 1000)
recalls = np.linspace(0, 1, 10)
f1s = [
    2 * precision * recall / (precision + recall)
    for recall in recalls
]
for f1, recall in zip(f1s, recalls):
    plt.plot(precision, f1, label=f"recall={recall:.1f}%")

plt.xlabel("Precision")
plt.ylabel("F1 Score")
plt.legend()
plt.show()

/var/folders/78/njcscll93_s6cc27zw_h0pmr0000gn/T/ipykernel_50750/2920054557.py:7: RuntimeWarning: invalid value encountered in divide
  2 * precision * recall / (precision + recall)

In [ ]:

Logistic Loss (Log Loss)¶

For binary classification problem, the log loss is defined as: $$ L(y, p) = - \frac{1}{N} \sum_{i=1}^N \left[ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right] $$

In [47]:

# Calculate the score from scratch
import numpy as np
Y_test_pred_prob = lr.predict_proba(X_test)[:, 1]
log_loss_score = np.sum(-np.log(np.where(Y_test == 1, Y_test_pred_prob, 1 - Y_test_pred_prob))) / len(Y_test)
print(f"Calculated Log Loss Score: {log_loss_score}")

Calculated Log Loss Score: 0.495027137138449

In [46]:

from sklearn.metrics import log_loss

log_loss_score2 = log_loss(Y_test, Y_test_pred_prob)
print(f"Log Loss Score: {log_loss_score2:.4f}")

Log Loss Score: 0.4950

Area under ROC curve¶

In [48]:

from sklearn.metrics import roc_curve, auc

fpr, tpr, _ = roc_curve(Y_test, Y_test_pred_prob)
roc_auc = auc(fpr, tpr)

# plot the ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='blue', lw=2, label=f'ROC Curve (AUC = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='gray', linestyle='--')  # Diagonal line for random guessing
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic (ROC) Curve')
plt.legend(loc='lower right')
plt.grid()
plt.show()

Confusion Matrix¶

In [49]:

from sklearn.metrics import confusion_matrix

matrix = confusion_matrix(Y_test, Y_test_pred)
print(matrix)

[[88  9]
 [23 34]]

Classification Report¶

https://scikit-learn.org/stable/modules/model_evaluation.html#classification-report

In [50]:

from sklearn.metrics import classification_report

report = classification_report(Y_test, Y_test_pred)
print(report)

              precision    recall  f1-score   support

         0.0       0.79      0.91      0.85        97
         1.0       0.79      0.60      0.68        57

    accuracy                           0.79       154
   macro avg       0.79      0.75      0.76       154
weighted avg       0.79      0.79      0.78       154

Regression Problems¶

In [51]:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Generate a synthetic regression dataset
X, y = make_regression(n_samples=100, n_features=1, noise=10, random_state=42)

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a linear regression model
model = LinearRegression()
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Calculate Mean Squared Error
mse = mean_squared_error(y_test, y_pred)

# Print the result
print(f"Mean Squared Error: {mse:.2f}")

# Plot the regression line
plt.scatter(X_test, y_test, color='blue', label="Actual values")
plt.plot(X_test, y_pred, color='red', linewidth=2, label="Predicted values")
plt.xlabel("Feature")
plt.ylabel("Target")
plt.title("Linear Regression with MSE Calculation")
plt.legend()
plt.show()

Mean Squared Error: 104.20

In [54]:

print(f"MSE: {np.sqrt(mse)}")

MSE: 10.207949183448665

In [ ]:

In [52]:

from sklearn.metrics import mean_absolute_error

mae = mean_absolute_error(y_test, y_pred)
print(f"MAE: {mae}")

MAE: 8.416659922209051

In [55]:

from sklearn.metrics import r2_score

r2 = r2_score(y_test, y_pred)
print(f"R2 score: {r2}")

R2 score: 0.9374151607623286

In [ ]:

`k-fold` cross-validation¶

In [56]:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold

# Generate a synthetic regression dataset
X, y = make_regression(n_samples=100, n_features=1, noise=10, random_state=42)

# Define the number of folds
kf = KFold(n_splits=5, shuffle=True, random_state=42)

# Store MSE for each fold
mse_values = []

# Perform K-Fold Cross-Validation
for fold, (train_index, test_index) in enumerate(kf.split(X), 1):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    # Train a linear regression model
    model = LinearRegression()
    model.fit(X_train, y_train)

    # Make predictions
    y_pred = model.predict(X_test)

    # Calculate Mean Squared Error
    mse = mean_squared_error(y_test, y_pred)
    mse_values.append(mse)

    print(f"Fold {fold}: MSE = {mse:.2f}")

# Calculate overall mean MSE
overall_mse = np.mean(mse_values)
print(f"\nOverall Mean Squared Error across all folds: {overall_mse:.2f}")

# Plot MSE values for each fold
plt.figure(figsize=(8, 5))
plt.plot(range(1, len(mse_values) + 1), mse_values, marker='o', linestyle='-', color='b', label="MSE per fold")
plt.axhline(y=overall_mse, color='r', linestyle='--', label=f"Overall Mean MSE = {overall_mse:.2f}")
plt.xlabel("Fold Number")
plt.ylabel("Mean Squared Error")
plt.title("K-Fold Cross-Validation MSE")
plt.legend()
plt.grid()
plt.show()

Fold 1: MSE = 104.20
Fold 2: MSE = 66.52
Fold 3: MSE = 63.17
Fold 4: MSE = 69.47
Fold 5: MSE = 105.17

Overall Mean Squared Error across all folds: 81.71

In [58]:

from sklearn import model_selection

# Define the number of folds
kf = KFold(n_splits=5, shuffle=True, random_state=42)

LR = LinearRegression()
k_fold_mse = model_selection.cross_val_score(
    LR, X, y, cv=kf, scoring='neg_mean_squared_error')
print("MSE: %.3f (%.3f)" % (-1 * k_fold_mse.mean(), k_fold_mse.std()))

MSE: 81.709 (18.871)

Model interpretation¶

Random Forest model¶

In [59]:

from sklearn.model_selection import train_test_split

X = train.drop(columns=['survived', 'passengerid'])
y = train['survived']

In [60]:

X.shape

Out[60]:

(891, 43)

In [61]:

from sklearn.ensemble import RandomForestClassifier

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=123)
rf = RandomForestClassifier(n_estimators=5, max_depth=3, random_state=234)
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

accuracy = accuracy_score(y_test, y_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))

Accuracy: 80.00%

In [62]:

importances = rf.feature_importances_
df_importances = pd.DataFrame({
    'Feature': X.columns,
    'Importance': importances
})
df_importances = df_importances.sort_values(by='Importance', ascending=False)
df_importances

Out[62]:

	Feature	Importance
12	sex_male	0.463926
29	title_Mr	0.136208
30	title_Mrs	0.133414
5	has_cabin	0.078299
10	fare_log_cat	0.057256
1	age	0.022328
42	deck_U	0.021360
37	deck_D	0.019512
0	pclass	0.015956
6	family_size	0.013974
13	embarked_C	0.013360
8	name_len_cat	0.008225
39	deck_F	0.004893
9	age_cat	0.002641
7	is_alone	0.002355
24	title_Major	0.001872
14	embarked_Q	0.001702
3	parch	0.001431
40	deck_G	0.001100
38	deck_E	0.000188
2	sibsp	0.000000
4	fare_log	0.000000
18	title_Col	0.000000
15	embarked_S	0.000000
16	embarked_nan	0.000000
17	title_Capt	0.000000
11	sex_female	0.000000
26	title_Miss	0.000000
25	title_Master	0.000000
23	title_Lady	0.000000
22	title_Jonkheer	0.000000
20	title_Don	0.000000
21	title_Dr	0.000000
19	title_Countess	0.000000
28	title_Mme	0.000000
34	deck_A	0.000000
33	title_Sir	0.000000
32	title_Rev	0.000000
31	title_Ms	0.000000
27	title_Mlle	0.000000
36	deck_C	0.000000
35	deck_B	0.000000
41	deck_T	0.000000

In [51]:

X_test2 = test.drop(columns=['passengerid'])
y_pred2 = automl.predict(X_test2)
df_submission = pd.concat([test['passengerid'], pd.DataFrame(y_pred2, columns=['survived'])], axis=1)
df_submission.to_csv('../data/titanic/submission.csv', index=False)

SHAP¶

In [63]:

import xgboost
import shap

# train an XGBoost model
X, y = shap.datasets.california()
model = xgboost.XGBRegressor().fit(X, y)

# explain the model's predictions using SHAP
# (same syntax works for LightGBM, CatBoost, scikit-learn, transformers, Spark, etc.)
explainer = shap.Explainer(model)
shap_values = explainer(X)

# visualize the first prediction's explanation
shap.plots.waterfall(shap_values[0])

/Users/xiangshiyin/Documents/Teaching/machine-learning-for-actuarial-science/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

In [65]:

y[:5]

Out[65]:

array([4.526, 3.585, 3.521, 3.413, 3.422])

In [ ]:

In [66]:

import shap

explainer = shap.Explainer(rf)
shap_values = explainer.shap_values(X_test)

# Extract SHAP values for the positive class (class 1)
shap_values_class1 = shap_values[:,:,1]

# Visualize global feature importance for class 1
shap.summary_plot(shap_values_class1, X_test, feature_names=feature_names, plot_type="bar")

In [67]:

explainer = shap.Explainer(rf)
shap_values = explainer.shap_values(X_test)

shap_values_class1 = shap_values[:,:,1]
explanation = shap.Explanation(
    values=shap_values_class1,
    base_values=explainer.expected_value[1],  # Base value for class 1
    data=X_test.values,  # Input data (as a NumPy array)
    feature_names=X_test.columns.tolist()  # Feature names
)
shap.plots.waterfall(explanation[1])

In [68]:

y_pred_prob = rf.predict_proba(X_test)
y_pred_prob[1,1]

Out[68]:

np.float64(0.16546512151307932)

In [ ]:

Overview¶

Load the dataset¶

Streamline the data transformations¶

Missing value imputation¶

Feature Engineering¶

Put all together¶

Train a simple model¶

Evaluate the prediction results¶

AutoML exploration¶

Model Evaluations¶

Prediction Quality¶

Classification Problems¶

Accuracy¶

Logistic Loss (Log Loss)¶

Area under ROC curve¶

Confusion Matrix¶

Classification Report¶

Regression Problems¶

k-fold cross-validation¶

Model interpretation¶

Random Forest model¶

SHAP¶

`k-fold` cross-validation¶