Last updated: 16 Feb 2023

👋 PyCaret Multiclass Classification Tutorial¶

PyCaret is an open-source, low-code machine learning library in Python that automates machine learning workflows. It is an end-to-end machine learning and model management tool that exponentially speeds up the experiment cycle and makes you more productive.

Compared with the other open-source machine learning libraries, PyCaret is an alternate low-code library that can be used to replace hundreds of lines of code with a few lines only. This makes experiments exponentially fast and efficient. PyCaret is essentially a Python wrapper around several machine learning libraries and frameworks, such as scikit-learn, XGBoost, LightGBM, CatBoost, spaCy, Optuna, Hyperopt, Ray, and a few more.

The design and simplicity of PyCaret are inspired by the emerging role of citizen data scientists, a term first used by Gartner. Citizen Data Scientists are power users who can perform both simple and moderately sophisticated analytical tasks that would previously have required more technical expertise.

💻 Installation¶

PyCaret is tested and supported on the following 64-bit systems:

Python 3.7 – 3.10
Python 3.9 for Ubuntu only
Ubuntu 16.04 or later
Windows 7 or later

You can install PyCaret with Python's pip package manager:

pip install pycaret

PyCaret's default installation will not install all the extra dependencies automatically. For that you will have to install the full version:

pip install pycaret[full]

or depending on your use-case you may install one of the following variant:

pip install pycaret[analysis]
pip install pycaret[models]
pip install pycaret[tuner]
pip install pycaret[mlops]
pip install pycaret[parallel]
pip install pycaret[test]

In [1]:

# check installed version
import pycaret
pycaret.__version__

Out[1]:

'3.0.0'

🚀 Quick start¶

PyCaret’s Classification Module is a supervised machine learning module that is used for classifying elements into groups. The goal is to predict the categorical class labels which are discrete and unordered.

Some common use cases include predicting customer default (Yes or No), predicting customer churn (customer will leave or stay), the disease found (positive or negative).

This module can be used for binary or multiclass problems. It provides several pre-processing features that prepare the data for modeling through the setup function. It has over 18 ready-to-use algorithms and several plots to analyze the performance of trained models.

A typical workflow in PyCaret consist of following 5 steps in this order:

Setup ➡️ Compare Models ➡️ Analyze Model ➡️ Prediction ➡️ Save Model¶

In [2]:

# loading sample dataset from pycaret dataset module
from pycaret.datasets import get_data
data = get_data('iris')

	sepal_length	sepal_width	petal_length	petal_width	species
0	5.1	3.5	1.4	0.2	Iris-setosa
1	4.9	3.0	1.4	0.2	Iris-setosa
2	4.7	3.2	1.3	0.2	Iris-setosa
3	4.6	3.1	1.5	0.2	Iris-setosa
4	5.0	3.6	1.4	0.2	Iris-setosa

Setup¶

This function initializes the training environment and creates the transformation pipeline. Setup function must be called before executing any other function in PyCaret. It only has two required parameters i.e. data and target. All the other parameters are optional.

In [3]:

# import pycaret classification and init setup
from pycaret.classification import *
s = setup(data, target = 'species', session_id = 123)

	Description	Value
0	Session id	123
1	Target	species
2	Target type	Multiclass
3	Target mapping	Iris-setosa: 0, Iris-versicolor: 1, Iris-virginica: 2
4	Original data shape	(150, 5)
5	Transformed data shape	(150, 5)
6	Transformed train set shape	(105, 5)
7	Transformed test set shape	(45, 5)
8	Numeric features	4
9	Preprocess	True
10	Imputation type	simple
11	Numeric imputation	mean
12	Categorical imputation	mode
13	Fold Generator	StratifiedKFold
14	Fold Number	10
15	CPU Jobs	-1
16	Use GPU	False
17	Log Experiment	False
18	Experiment Name	clf-default-name
19	USI	8d38

Once the setup has been successfully executed it shows the information grid containing experiment level information.

Session id: A pseudo-random number distributed as a seed in all functions for later reproducibility. If no session_id is passed, a random number is automatically generated that is distributed to all functions.

- **Target type:** Binary, Multiclass, or Regression. The Target type is automatically detected.

- **Label Encoding:** When the Target variable is of type string (i.e. 'Yes' or 'No') instead of 1 or 0, it automatically encodes the label into 1 and 0 and displays the mapping (0 : No, 1 : Yes) for reference. In this tutorial, no label encoding is required since the target variable is of numeric type.

- **Original data shape:** Shape of the original data prior to any transformations.

- **Transformed train set shape :** Shape of transformed train set

- **Transformed test set shape :** Shape of transformed test set

- **Numeric features :** The number of features considered as numerical.

- **Categorical features :** The number of features considered as categorical.

PyCaret has two set of API's that you can work with. (1) Functional (as seen above) and (2) Object Oriented API.

With Object Oriented API instead of executing functions directly you will import a class and execute methods of class.

In [4]:

# import ClassificationExperiment and init the class
from pycaret.classification import ClassificationExperiment
exp = ClassificationExperiment()

In [5]:

# check the type of exp
type(exp)

Out[5]:

pycaret.classification.oop.ClassificationExperiment

In [6]:

# init setup on exp
exp.setup(data, target = 'species', session_id = 123)

	Description	Value
0	Session id	123
1	Target	species
2	Target type	Multiclass
3	Target mapping	Iris-setosa: 0, Iris-versicolor: 1, Iris-virginica: 2
4	Original data shape	(150, 5)
5	Transformed data shape	(150, 5)
6	Transformed train set shape	(105, 5)
7	Transformed test set shape	(45, 5)
8	Numeric features	4
9	Preprocess	True
10	Imputation type	simple
11	Numeric imputation	mean
12	Categorical imputation	mode
13	Fold Generator	StratifiedKFold
14	Fold Number	10
15	CPU Jobs	-1
16	Use GPU	False
17	Log Experiment	False
18	Experiment Name	clf-default-name
19	USI	42d4

Out[6]:

<pycaret.classification.oop.ClassificationExperiment at 0x22362e4bd00>

You can use any of the two method i.e. Functional or OOP and even switch back and forth between two set of API's. The choice of method will not impact the results and has been tested for consistency.

Compare Models¶

This function trains and evaluates the performance of all the estimators available in the model library using cross-validation. The output of this function is a scoring grid with average cross-validated scores. Metrics evaluated during CV can be accessed using the get_metrics function. Custom metrics can be added or removed using add_metric and remove_metric function.

In [7]:

# compare baseline models
best = compare_models()

	Model	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC	TT (Sec)
lr	Logistic Regression	0.9718	0.9971	0.9718	0.9780	0.9712	0.9573	0.9609	0.9190
knn	K Neighbors Classifier	0.9718	0.9830	0.9718	0.9780	0.9712	0.9573	0.9609	0.0370
qda	Quadratic Discriminant Analysis	0.9718	0.9974	0.9718	0.9780	0.9712	0.9573	0.9609	0.0300
lda	Linear Discriminant Analysis	0.9718	1.0000	0.9718	0.9780	0.9712	0.9573	0.9609	0.0330
lightgbm	Light Gradient Boosting Machine	0.9536	0.9935	0.9536	0.9634	0.9528	0.9298	0.9356	0.3150
nb	Naive Bayes	0.9445	0.9868	0.9445	0.9525	0.9438	0.9161	0.9207	0.0300
et	Extra Trees Classifier	0.9445	0.9935	0.9445	0.9586	0.9426	0.9161	0.9246	0.0880
catboost	CatBoost Classifier	0.9445	0.9922	0.9445	0.9586	0.9426	0.9161	0.9246	0.1220
gbc	Gradient Boosting Classifier	0.9355	0.9792	0.9355	0.9416	0.9325	0.9023	0.9083	0.1360
xgboost	Extreme Gradient Boosting	0.9355	0.9868	0.9355	0.9440	0.9343	0.9023	0.9077	0.0710
dt	Decision Tree Classifier	0.9264	0.9429	0.9264	0.9502	0.9201	0.8886	0.9040	0.0270
rf	Random Forest Classifier	0.9264	0.9909	0.9264	0.9343	0.9232	0.8886	0.8956	0.0900
ada	Ada Boost Classifier	0.9155	0.9947	0.9155	0.9401	0.9097	0.8720	0.8873	0.0580
ridge	Ridge Classifier	0.8227	0.0000	0.8227	0.8437	0.8186	0.7320	0.7454	0.0220
svm	SVM - Linear Kernel	0.7618	0.0000	0.7618	0.6655	0.6888	0.6333	0.7048	0.0300
dummy	Dummy Classifier	0.2864	0.5000	0.2864	0.0822	0.1277	0.0000	0.0000	0.0490

Processing:   0%|          | 0/69 [00:00<?, ?it/s]

In [8]:

# compare models using OOP
exp.compare_models()

	Model	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC	TT (Sec)
lr	Logistic Regression	0.9718	0.9971	0.9718	0.9780	0.9712	0.9573	0.9609	0.0430
knn	K Neighbors Classifier	0.9718	0.9830	0.9718	0.9780	0.9712	0.9573	0.9609	0.0520
qda	Quadratic Discriminant Analysis	0.9718	0.9974	0.9718	0.9780	0.9712	0.9573	0.9609	0.0420
lda	Linear Discriminant Analysis	0.9718	1.0000	0.9718	0.9780	0.9712	0.9573	0.9609	0.0550
lightgbm	Light Gradient Boosting Machine	0.9536	0.9935	0.9536	0.9634	0.9528	0.9298	0.9356	0.0550
nb	Naive Bayes	0.9445	0.9868	0.9445	0.9525	0.9438	0.9161	0.9207	0.0380
et	Extra Trees Classifier	0.9445	0.9935	0.9445	0.9586	0.9426	0.9161	0.9246	0.1430
catboost	CatBoost Classifier	0.9445	0.9922	0.9445	0.9586	0.9426	0.9161	0.9246	0.0480
gbc	Gradient Boosting Classifier	0.9355	0.9792	0.9355	0.9416	0.9325	0.9023	0.9083	0.1850
xgboost	Extreme Gradient Boosting	0.9355	0.9868	0.9355	0.9440	0.9343	0.9023	0.9077	0.0600
dt	Decision Tree Classifier	0.9264	0.9429	0.9264	0.9502	0.9201	0.8886	0.9040	0.0370
rf	Random Forest Classifier	0.9264	0.9909	0.9264	0.9343	0.9232	0.8886	0.8956	0.1440
ada	Ada Boost Classifier	0.9155	0.9947	0.9155	0.9401	0.9097	0.8720	0.8873	0.0850
ridge	Ridge Classifier	0.8227	0.0000	0.8227	0.8437	0.8186	0.7320	0.7454	0.0330
svm	SVM - Linear Kernel	0.7618	0.0000	0.7618	0.6655	0.6888	0.6333	0.7048	0.0320
dummy	Dummy Classifier	0.2864	0.5000	0.2864	0.0822	0.1277	0.0000	0.0000	0.0430

Processing:   0%|          | 0/69 [00:00<?, ?it/s]

Out[8]:

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=1000,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=123, solver='lbfgs', tol=0.0001, verbose=0,
                   warm_start=False)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Notice that the output between functional and OOP API is consistent. Rest of the functions in this notebook will only be shown using functional API only.

Analyze Model¶

You can use the plot_model function to analyzes the performance of a trained model on the test set. It may require re-training the model in certain cases.

In [9]:

# plot confusion matrix
plot_model(best, plot = 'confusion_matrix')

In [10]:

# plot AUC
plot_model(best, plot = 'auc')

In [11]:

# plot feature importance
plot_model(best, plot = 'feature')

In [12]:

# check docstring to see available plots 
help(plot_model)

Help on function plot_model in module pycaret.classification.functional:

plot_model(estimator, plot: str = 'auc', scale: float = 1, save: bool = False, fold: Union[int, Any, NoneType] = None, fit_kwargs: Union[dict, NoneType] = None, plot_kwargs: Union[dict, NoneType] = None, groups: Union[str, Any, NoneType] = None, use_train_data: bool = False, verbose: bool = True, display_format: Union[str, NoneType] = None) -> Union[str, NoneType]
    This function analyzes the performance of a trained model on holdout set.
    It may require re-training the model in certain cases.
    
    Example
    -------
    >>> from pycaret.datasets import get_data
    >>> juice = get_data('juice')
    >>> from pycaret.classification import *
    >>> exp_name = setup(data = juice,  target = 'Purchase')
    >>> lr = create_model('lr')
    >>> plot_model(lr, plot = 'auc')
    
    
    estimator: scikit-learn compatible object
        Trained model object
    
    
    plot: str, default = 'auc'
        List of available plots (ID - Name):
    
        * 'pipeline' - Schematic drawing of the preprocessing pipeline
        * 'auc' - Area Under the Curve
        * 'threshold' - Discrimination Threshold
        * 'pr' - Precision Recall Curve
        * 'confusion_matrix' - Confusion Matrix
        * 'error' - Class Prediction Error
        * 'class_report' - Classification Report
        * 'boundary' - Decision Boundary
        * 'rfe' - Recursive Feature Selection
        * 'learning' - Learning Curve
        * 'manifold' - Manifold Learning
        * 'calibration' - Calibration Curve
        * 'vc' - Validation Curve
        * 'dimension' - Dimension Learning
        * 'feature' - Feature Importance
        * 'feature_all' - Feature Importance (All)
        * 'parameter' - Model Hyperparameter
        * 'lift' - Lift Curve
        * 'gain' - Gain Chart
        * 'tree' - Decision Tree
        * 'ks' - KS Statistic Plot
    
    
    scale: float, default = 1
        The resolution scale of the figure.
    
    
    save: bool, default = False
        When set to True, plot is saved in the current working directory.
    
    
    fold: int or scikit-learn compatible CV generator, default = None
        Controls cross-validation. If None, the CV generator in the ``fold_strategy``
        parameter of the ``setup`` function is used. When an integer is passed,
        it is interpreted as the 'n_splits' parameter of the CV generator in the
        ``setup`` function.
    
    
    fit_kwargs: dict, default = {} (empty dict)
        Dictionary of arguments passed to the fit method of the model.
    
    
    plot_kwargs: dict, default = {} (empty dict)
        Dictionary of arguments passed to the visualizer class.
            - pipeline: fontsize -> int
    
    
    groups: str or array-like, with shape (n_samples,), default = None
        Optional group labels when GroupKFold is used for the cross validation.
        It takes an array with shape (n_samples, ) where n_samples is the number
        of rows in training dataset. When string is passed, it is interpreted as
        the column name in the dataset containing group labels.
    
    
    use_train_data: bool, default = False
        When set to true, train data will be used for plots, instead
        of test data.
    
    
    verbose: bool, default = True
        When set to False, progress bar is not displayed.
    
    
    display_format: str, default = None
        To display plots in Streamlit (https://www.streamlit.io/), set this to 'streamlit'.
        Currently, not all plots are supported.
    
    
    Returns:
        Path to saved file, if any.
    
    
    Warnings
    --------
    -   Estimators that does not support 'predict_proba' attribute cannot be used for
        'AUC' and 'calibration' plots.
    
    -   When the target is multiclass, 'calibration', 'threshold', 'manifold' and 'rfe'
        plots are not available.
    
    -   When the 'max_features' parameter of a trained model object is not equal to
        the number of samples in training set, the 'rfe' plot is not available.

An alternate to plot_model function is evaluate_model. It can only be used in Notebook since it uses ipywidget.

In [13]:

evaluate_model(best)

interactive(children=(ToggleButtons(description='Plot Type:', icons=('',), options=(('Pipeline Plot', 'pipelin…

Prediction¶

The predict_model function returns prediction_label and prediction_score (probability of the predicted class) as new columns in dataframe. When data is None (default), it uses the test set (created during the setup function) for scoring.

In [14]:

# predict on test set
holdout_pred = predict_model(best)

	Model	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
0	Logistic Regression	0.9778	0.9985	0	0	0	0.9667	0.9674

In [15]:

# show predictions df
holdout_pred.head()

Out[15]:

	sepal_length	sepal_width	petal_length	petal_width	species	prediction_label	prediction_score
105	6.3	2.5	4.9	1.5	Iris-versicolor	Iris-versicolor	0.5204
106	7.2	3.2	6.0	1.8	Iris-virginica	Iris-virginica	0.9503
107	5.5	2.4	3.8	1.1	Iris-versicolor	Iris-versicolor	0.9334
108	6.7	3.1	4.7	1.5	Iris-versicolor	Iris-versicolor	0.7321
109	7.7	3.8	6.7	2.2	Iris-virginica	Iris-virginica	0.9952

The same function works for predicting the labels on unseen dataset. Let's create a copy of original data and drop the Class variable. We can then use the new data frame without labels for scoring.

In [16]:

# copy data and drop Class variable

new_data = data.copy()
new_data.drop('species', axis=1, inplace=True)
new_data.head()

Out[16]:

	sepal_length	sepal_width	petal_length	petal_width
0	5.1	3.5	1.4	0.2
1	4.9	3.0	1.4	0.2
2	4.7	3.2	1.3	0.2
3	4.6	3.1	1.5	0.2
4	5.0	3.6	1.4	0.2

In [17]:

# predict model on new_data
predictions = predict_model(best, data = new_data)
predictions.head()

Out[17]:

	sepal_length	sepal_width	petal_length	petal_width	prediction_label	prediction_score
0	5.1	3.5	1.4	0.2	Iris-setosa	0.9775
1	4.9	3.0	1.4	0.2	Iris-setosa	0.9678
2	4.7	3.2	1.3	0.2	Iris-setosa	0.9820
3	4.6	3.1	1.5	0.2	Iris-setosa	0.9719
4	5.0	3.6	1.4	0.2	Iris-setosa	0.9813

Save Model¶

Finally, you can save the entire pipeline on disk for later use, using pycaret's save_model function.

In [18]:

# save pipeline
save_model(best, 'my_first_pipeline')

Transformation Pipeline and Model Successfully Saved

Out[18]:

(Pipeline(memory=FastMemory(location=C:\Users\owner\AppData\Local\Temp\joblib),
          steps=[('label_encoding',
                  TransformerWrapperWithInverse(exclude=None, include=None,
                                                transformer=LabelEncoder())),
                 ('numerical_imputer',
                  TransformerWrapper(exclude=None,
                                     include=['sepal_length', 'sepal_width',
                                              'petal_length', 'petal_width'],
                                     transformer=SimpleImputer(add_indicator=F...
                                                               fill_value=None,
                                                               missing_values=nan,
                                                               strategy='most_frequent',
                                                               verbose='deprecated'))),
                 ('trained_model',
                  LogisticRegression(C=1.0, class_weight=None, dual=False,
                                     fit_intercept=True, intercept_scaling=1,
                                     l1_ratio=None, max_iter=1000,
                                     multi_class='auto', n_jobs=None,
                                     penalty='l2', random_state=123,
                                     solver='lbfgs', tol=0.0001, verbose=0,
                                     warm_start=False))],
          verbose=False),
 'my_first_pipeline.pkl')

In [19]:

# load pipeline
loaded_best_pipeline = load_model('my_first_pipeline')
loaded_best_pipeline

👇 Detailed function-by-function overview¶

✅ Setup¶

This function initializes the experiment in PyCaret and creates the transformation pipeline based on all the parameters passed in the function. Setup function must be called before executing any other function. It takes two required parameters: data and target. All the other parameters are optional and are used for configuring data preprocessing pipeline.

In [20]:

s = setup(data, target = 'species', session_id = 123)

	Description	Value
0	Session id	123
1	Target	species
2	Target type	Multiclass
3	Target mapping	Iris-setosa: 0, Iris-versicolor: 1, Iris-virginica: 2
4	Original data shape	(150, 5)
5	Transformed data shape	(150, 5)
6	Transformed train set shape	(105, 5)
7	Transformed test set shape	(45, 5)
8	Numeric features	4
9	Preprocess	True
10	Imputation type	simple
11	Numeric imputation	mean
12	Categorical imputation	mode
13	Fold Generator	StratifiedKFold
14	Fold Number	10
15	CPU Jobs	-1
16	Use GPU	False
17	Log Experiment	False
18	Experiment Name	clf-default-name
19	USI	35bc

To access all the variables created by the setup function such as transformed dataset, random_state, etc. you can use get_config method.

In [21]:

# check all available config
get_config()

Out[21]:

{'USI',
 'X',
 'X_test',
 'X_test_transformed',
 'X_train',
 'X_train_transformed',
 'X_transformed',
 '_available_plots',
 '_ml_usecase',
 'data',
 'dataset',
 'dataset_transformed',
 'exp_id',
 'exp_name_log',
 'fix_imbalance',
 'fold_generator',
 'fold_groups_param',
 'fold_shuffle_param',
 'gpu_n_jobs_param',
 'gpu_param',
 'html_param',
 'idx',
 'is_multiclass',
 'log_plots_param',
 'logging_param',
 'memory',
 'n_jobs_param',
 'pipeline',
 'seed',
 'target_param',
 'test',
 'test_transformed',
 'train',
 'train_transformed',
 'variable_and_property_keys',
 'variables',
 'y',
 'y_test',
 'y_test_transformed',
 'y_train',
 'y_train_transformed',
 'y_transformed'}

In [22]:

# lets access X_train_transformed
get_config('X_train_transformed')

Out[22]:

	sepal_length	sepal_width	petal_length	petal_width
0	5.0	2.0	3.5	1.0
1	5.4	3.9	1.3	0.4
2	5.6	3.0	4.1	1.3
3	7.4	2.8	6.1	1.9
4	4.6	3.4	1.4	0.3
...	...	...	...	...
100	6.6	2.9	4.6	1.3
101	4.5	2.3	1.3	0.3
102	4.8	3.0	1.4	0.1
103	5.4	3.4	1.7	0.2
104	6.2	3.4	5.4	2.3

105 rows × 4 columns

In [23]:

# another example: let's access seed
print("The current seed is: {}".format(get_config('seed')))

# now lets change it using set_config
set_config('seed', 786)
print("The new seed is: {}".format(get_config('seed')))

The current seed is: 123
The new seed is: 786

All the preprocessing configurations and experiment settings/parameters are passed into the setup function. To see all available parameters, check the docstring:

In [24]:

# help(setup)

In [25]:

# init setup with normalize = True

s = setup(data, target = 'species', session_id = 123,
          normalize = True, normalize_method = 'minmax')

	Description	Value
0	Session id	123
1	Target	species
2	Target type	Multiclass
3	Target mapping	Iris-setosa: 0, Iris-versicolor: 1, Iris-virginica: 2
4	Original data shape	(150, 5)
5	Transformed data shape	(150, 5)
6	Transformed train set shape	(105, 5)
7	Transformed test set shape	(45, 5)
8	Numeric features	4
9	Preprocess	True
10	Imputation type	simple
11	Numeric imputation	mean
12	Categorical imputation	mode
13	Normalize	True
14	Normalize method	minmax
15	Fold Generator	StratifiedKFold
16	Fold Number	10
17	CPU Jobs	-1
18	Use GPU	False
19	Log Experiment	False
20	Experiment Name	clf-default-name
21	USI	3b39

In [26]:

# lets check the X_train_transformed to see effect of params passed
get_config('X_train_transformed')['sepal_length'].hist()

Out[26]:

<AxesSubplot:>

Notice that all the values are between 0 and 1 - that is because we passed normalize=True in the setup function. If you don't remember how it compares to actual data, no problem - we can also access non-transformed values using get_config and then compare. See below and notice the range of values on x-axis and compare it with histogram above.

In [27]:

get_config('X_train')['sepal_length'].hist()

Out[27]:

<AxesSubplot:>

✅ Compare Models¶

This function trains and evaluates the performance of all estimators available in the model library using cross-validation. The output of this function is a scoring grid with average cross-validated scores. Metrics evaluated during CV can be accessed using the get_metrics function. Custom metrics can be added or removed using add_metric and remove_metric function.

In [28]:

best = compare_models()

	Model	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC	TT (Sec)
qda	Quadratic Discriminant Analysis	0.9718	0.9974	0.9718	0.9780	0.9712	0.9573	0.9609	0.0380
lda	Linear Discriminant Analysis	0.9718	1.0000	0.9718	0.9780	0.9712	0.9573	0.9609	0.0440
knn	K Neighbors Classifier	0.9636	0.9844	0.9636	0.9709	0.9631	0.9450	0.9494	0.0510
lightgbm	Light Gradient Boosting Machine	0.9536	0.9857	0.9536	0.9634	0.9528	0.9298	0.9356	0.0500
nb	Naive Bayes	0.9445	0.9868	0.9445	0.9525	0.9438	0.9161	0.9207	0.0380
et	Extra Trees Classifier	0.9445	0.9935	0.9445	0.9586	0.9426	0.9161	0.9246	0.1240
catboost	CatBoost Classifier	0.9445	0.9922	0.9445	0.9586	0.9426	0.9161	0.9246	0.0390
xgboost	Extreme Gradient Boosting	0.9355	0.9868	0.9355	0.9440	0.9343	0.9023	0.9077	0.0480
dt	Decision Tree Classifier	0.9264	0.9429	0.9264	0.9502	0.9201	0.8886	0.9040	0.0340
rf	Random Forest Classifier	0.9264	0.9903	0.9264	0.9343	0.9232	0.8886	0.8956	0.1210
gbc	Gradient Boosting Classifier	0.9264	0.9688	0.9264	0.9343	0.9232	0.8886	0.8956	0.1500
ada	Ada Boost Classifier	0.9155	0.9843	0.9155	0.9401	0.9097	0.8720	0.8873	0.0690
lr	Logistic Regression	0.9073	0.9751	0.9073	0.9159	0.9064	0.8597	0.8645	0.0400
ridge	Ridge Classifier	0.8318	0.0000	0.8318	0.8545	0.8281	0.7459	0.7595	0.0370
svm	SVM - Linear Kernel	0.8100	0.0000	0.8100	0.7831	0.7702	0.7125	0.7527	0.0350
dummy	Dummy Classifier	0.2864	0.5000	0.2864	0.0822	0.1277	0.0000	0.0000	0.0380

Processing:   0%|          | 0/69 [00:00<?, ?it/s]

compare_models by default uses all the estimators in model library (all except models with Turbo=False) . To see all available models you can use the function models()

In [29]:

# check available models
models()

Out[29]:

	Name	Reference	Turbo
ID
lr	Logistic Regression	sklearn.linear_model._logistic.LogisticRegression	True
knn	K Neighbors Classifier	sklearn.neighbors._classification.KNeighborsCl...	True
nb	Naive Bayes	sklearn.naive_bayes.GaussianNB	True
dt	Decision Tree Classifier	sklearn.tree._classes.DecisionTreeClassifier	True
svm	SVM - Linear Kernel	sklearn.linear_model._stochastic_gradient.SGDC...	True
rbfsvm	SVM - Radial Kernel	sklearn.svm._classes.SVC	False
gpc	Gaussian Process Classifier	sklearn.gaussian_process._gpc.GaussianProcessC...	False
mlp	MLP Classifier	sklearn.neural_network._multilayer_perceptron....	False
ridge	Ridge Classifier	sklearn.linear_model._ridge.RidgeClassifier	True
rf	Random Forest Classifier	sklearn.ensemble._forest.RandomForestClassifier	True
qda	Quadratic Discriminant Analysis	sklearn.discriminant_analysis.QuadraticDiscrim...	True
ada	Ada Boost Classifier	sklearn.ensemble._weight_boosting.AdaBoostClas...	True
gbc	Gradient Boosting Classifier	sklearn.ensemble._gb.GradientBoostingClassifier	True
lda	Linear Discriminant Analysis	sklearn.discriminant_analysis.LinearDiscrimina...	True
et	Extra Trees Classifier	sklearn.ensemble._forest.ExtraTreesClassifier	True
xgboost	Extreme Gradient Boosting	xgboost.sklearn.XGBClassifier	True
lightgbm	Light Gradient Boosting Machine	lightgbm.sklearn.LGBMClassifier	True
catboost	CatBoost Classifier	catboost.core.CatBoostClassifier	True
dummy	Dummy Classifier	sklearn.dummy.DummyClassifier	True

You can use the include and exclude parameter in the compare_models to train only select model or exclude specific models from training by passing the model id's in exclude parameter.

In [30]:

compare_tree_models = compare_models(include = ['dt', 'rf', 'et', 'gbc', 'xgboost', 'lightgbm', 'catboost'])

	Model	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC	TT (Sec)
lightgbm	Light Gradient Boosting Machine	0.9536	0.9857	0.9536	0.9634	0.9528	0.9298	0.9356	0.0520
et	Extra Trees Classifier	0.9445	0.9935	0.9445	0.9586	0.9426	0.9161	0.9246	0.1190
catboost	CatBoost Classifier	0.9445	0.9922	0.9445	0.9586	0.9426	0.9161	0.9246	0.0390
xgboost	Extreme Gradient Boosting	0.9355	0.9868	0.9355	0.9440	0.9343	0.9023	0.9077	0.0580
dt	Decision Tree Classifier	0.9264	0.9429	0.9264	0.9502	0.9201	0.8886	0.9040	0.0370
rf	Random Forest Classifier	0.9264	0.9903	0.9264	0.9343	0.9232	0.8886	0.8956	0.1200
gbc	Gradient Boosting Classifier	0.9264	0.9688	0.9264	0.9343	0.9232	0.8886	0.8956	0.1510

Processing:   0%|          | 0/33 [00:00<?, ?it/s]

In [31]:

compare_tree_models

Out[31]:

LGBMClassifier(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,
               importance_type='split', learning_rate=0.1, max_depth=-1,
               min_child_samples=20, min_child_weight=0.001, min_split_gain=0.0,
               n_estimators=100, n_jobs=-1, num_leaves=31, objective=None,
               random_state=123, reg_alpha=0.0, reg_lambda=0.0, silent='warn',
               subsample=1.0, subsample_for_bin=200000, subsample_freq=0)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

The function above has return trained model object as an output. The scoring grid is only displayed and not returned. If you need access to the scoring grid you can use pull function to access the dataframe.

In [32]:

compare_tree_models_results = pull()
compare_tree_models_results

Out[32]:

	Model	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC	TT (Sec)
lightgbm	Light Gradient Boosting Machine	0.9536	0.9857	0.9536	0.9634	0.9528	0.9298	0.9356	0.052
et	Extra Trees Classifier	0.9445	0.9935	0.9445	0.9586	0.9426	0.9161	0.9246	0.119
catboost	CatBoost Classifier	0.9445	0.9922	0.9445	0.9586	0.9426	0.9161	0.9246	0.039
xgboost	Extreme Gradient Boosting	0.9355	0.9868	0.9355	0.9440	0.9343	0.9023	0.9077	0.058
dt	Decision Tree Classifier	0.9264	0.9429	0.9264	0.9502	0.9201	0.8886	0.9040	0.037
rf	Random Forest Classifier	0.9264	0.9903	0.9264	0.9343	0.9232	0.8886	0.8956	0.120
gbc	Gradient Boosting Classifier	0.9264	0.9688	0.9264	0.9343	0.9232	0.8886	0.8956	0.151

By default compare_models return the single best performing model based on the metric defined in the sort parameter. Let's change our code to return 3 top models based on Recall.

In [33]:

best_recall_models_top3 = compare_models(sort = 'Recall', n_select = 3)

	Model	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC	TT (Sec)
qda	Quadratic Discriminant Analysis	0.9718	0.9974	0.9718	0.9780	0.9712	0.9573	0.9609	0.0400
lda	Linear Discriminant Analysis	0.9718	1.0000	0.9718	0.9780	0.9712	0.9573	0.9609	0.0380
knn	K Neighbors Classifier	0.9636	0.9844	0.9636	0.9709	0.9631	0.9450	0.9494	0.0490
lightgbm	Light Gradient Boosting Machine	0.9536	0.9857	0.9536	0.9634	0.9528	0.9298	0.9356	0.0490
nb	Naive Bayes	0.9445	0.9868	0.9445	0.9525	0.9438	0.9161	0.9207	0.0380
et	Extra Trees Classifier	0.9445	0.9935	0.9445	0.9586	0.9426	0.9161	0.9246	0.1180
catboost	CatBoost Classifier	0.9445	0.9922	0.9445	0.9586	0.9426	0.9161	0.9246	0.0390
xgboost	Extreme Gradient Boosting	0.9355	0.9868	0.9355	0.9440	0.9343	0.9023	0.9077	0.0460
dt	Decision Tree Classifier	0.9264	0.9429	0.9264	0.9502	0.9201	0.8886	0.9040	0.0330
rf	Random Forest Classifier	0.9264	0.9903	0.9264	0.9343	0.9232	0.8886	0.8956	0.1170
gbc	Gradient Boosting Classifier	0.9264	0.9688	0.9264	0.9343	0.9232	0.8886	0.8956	0.1450
ada	Ada Boost Classifier	0.9155	0.9843	0.9155	0.9401	0.9097	0.8720	0.8873	0.0680
lr	Logistic Regression	0.9073	0.9751	0.9073	0.9159	0.9064	0.8597	0.8645	0.0420
ridge	Ridge Classifier	0.8318	0.0000	0.8318	0.8545	0.8281	0.7459	0.7595	0.0310
svm	SVM - Linear Kernel	0.8100	0.0000	0.8100	0.7831	0.7702	0.7125	0.7527	0.0350
dummy	Dummy Classifier	0.2864	0.5000	0.2864	0.0822	0.1277	0.0000	0.0000	0.0360

Processing:   0%|          | 0/71 [00:00<?, ?it/s]

In [34]:

# list of top 3 models by Recall
best_recall_models_top3

Out[34]:

[QuadraticDiscriminantAnalysis(priors=None, reg_param=0.0,
                               store_covariance=False, tol=0.0001),
 LinearDiscriminantAnalysis(covariance_estimator=None, n_components=None,
                            priors=None, shrinkage=None, solver='svd',
                            store_covariance=False, tol=0.0001),
 KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                      metric_params=None, n_jobs=-1, n_neighbors=5, p=2,
                      weights='uniform')]

Some other parameters that you might find very useful in compare_models are:

fold
cross_validation
budget_time
errors
probability_threshold
parallel

You can check the docstring of the function for more info.

In [35]:

# help(compare_models)

✅ Experiment Logging¶

PyCaret integrates with many different type of experiment loggers (default = 'mlflow'). To turn on experiment tracking in PyCaret you can set log_experiment and experiment_name parameter. It will automatically track all the metrics, hyperparameters, and artifacts based on the defined logger.

In [36]:

# from pycaret.classification import *
# s = setup(data, target = 'Class variable', log_experiment='mlflow', experiment_name='iris_experiment')

In [37]:

# compare models
# best = compare_models()

In [38]:

# start mlflow server on localhost:5000
# !mlflow ui

By default PyCaret uses MLFlow logger that can be changed using log_experiment parameter. Following loggers are available:

- mlflow
- wandb
- comet_ml
- dagshub

Other logging related parameters that you may find useful are:

experiment_custom_tags
log_plots
log_data
log_profile

For more information check out the docstring of the setup function.

In [39]:

# help(setup)

✅ Create Model¶

This function trains and evaluates the performance of a given estimator using cross-validation. The output of this function is a scoring grid with CV scores by fold. Metrics evaluated during CV can be accessed using the get_metrics function. Custom metrics can be added or removed using add_metric and remove_metric function. All the available models can be accessed using the models function.

In [40]:

# check all the available models
models()

Out[40]:

	Name	Reference	Turbo
ID
lr	Logistic Regression	sklearn.linear_model._logistic.LogisticRegression	True
knn	K Neighbors Classifier	sklearn.neighbors._classification.KNeighborsCl...	True
nb	Naive Bayes	sklearn.naive_bayes.GaussianNB	True
dt	Decision Tree Classifier	sklearn.tree._classes.DecisionTreeClassifier	True
svm	SVM - Linear Kernel	sklearn.linear_model._stochastic_gradient.SGDC...	True
rbfsvm	SVM - Radial Kernel	sklearn.svm._classes.SVC	False
gpc	Gaussian Process Classifier	sklearn.gaussian_process._gpc.GaussianProcessC...	False
mlp	MLP Classifier	sklearn.neural_network._multilayer_perceptron....	False
ridge	Ridge Classifier	sklearn.linear_model._ridge.RidgeClassifier	True
rf	Random Forest Classifier	sklearn.ensemble._forest.RandomForestClassifier	True
qda	Quadratic Discriminant Analysis	sklearn.discriminant_analysis.QuadraticDiscrim...	True
ada	Ada Boost Classifier	sklearn.ensemble._weight_boosting.AdaBoostClas...	True
gbc	Gradient Boosting Classifier	sklearn.ensemble._gb.GradientBoostingClassifier	True
lda	Linear Discriminant Analysis	sklearn.discriminant_analysis.LinearDiscrimina...	True
et	Extra Trees Classifier	sklearn.ensemble._forest.ExtraTreesClassifier	True
xgboost	Extreme Gradient Boosting	xgboost.sklearn.XGBClassifier	True
lightgbm	Light Gradient Boosting Machine	lightgbm.sklearn.LGBMClassifier	True
catboost	CatBoost Classifier	catboost.core.CatBoostClassifier	True
dummy	Dummy Classifier	sklearn.dummy.DummyClassifier	True

In [41]:

# train logistic regression with default fold=10
lr = create_model('lr')

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	0.9091	1.0000	0.9091	0.9273	0.9076	0.8625	0.8735
1	0.8182	0.9221	0.8182	0.8182	0.8182	0.7250	0.7250
2	0.9091	0.9610	0.9091	0.9273	0.9076	0.8625	0.8735
3	0.6364	0.8961	0.6364	0.6364	0.6364	0.4500	0.4500
4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
5	0.9000	0.9714	0.9000	0.9250	0.8971	0.8485	0.8616
6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	0.9000	1.0000	0.9000	0.9250	0.8971	0.8485	0.8616
9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Mean	0.9073	0.9751	0.9073	0.9159	0.9064	0.8597	0.8645
Std	0.1076	0.0360	0.1076	0.1079	0.1077	0.1628	0.1628

Processing:   0%|          | 0/4 [00:00<?, ?it/s]

The function above has return trained model object as an output. The scoring grid is only displayed and not returned. If you need access to the scoring grid you can use pull function to access the dataframe.

In [42]:

lr_results = pull()
print(type(lr_results))
lr_results

<class 'pandas.core.frame.DataFrame'>

Out[42]:

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	0.9091	1.0000	0.9091	0.9273	0.9076	0.8625	0.8735
1	0.8182	0.9221	0.8182	0.8182	0.8182	0.7250	0.7250
2	0.9091	0.9610	0.9091	0.9273	0.9076	0.8625	0.8735
3	0.6364	0.8961	0.6364	0.6364	0.6364	0.4500	0.4500
4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
5	0.9000	0.9714	0.9000	0.9250	0.8971	0.8485	0.8616
6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	0.9000	1.0000	0.9000	0.9250	0.8971	0.8485	0.8616
9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Mean	0.9073	0.9751	0.9073	0.9159	0.9064	0.8597	0.8645
Std	0.1076	0.0360	0.1076	0.1079	0.1077	0.1628	0.1628

In [43]:

# train logistic regression with fold=3
lr = create_model('lr', fold=3)

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	0.9143	0.9730	0.9143	0.9158	0.9140	0.8712	0.8722
1	0.8857	0.9764	0.8857	0.8922	0.8849	0.8284	0.8325
2	0.9714	0.9988	0.9714	0.9736	0.9713	0.9571	0.9582
Mean	0.9238	0.9827	0.9238	0.9272	0.9234	0.8856	0.8877
Std	0.0356	0.0115	0.0356	0.0342	0.0359	0.0535	0.0525

Processing:   0%|          | 0/4 [00:00<?, ?it/s]

In [44]:

# train logistic regression with specific model parameters
create_model('lr', C = 0.5, l1_ratio = 0.15)

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	0.9091	1.0000	0.9091	0.9273	0.9076	0.8625	0.8735
1	0.8182	0.9091	0.8182	0.8182	0.8182	0.7250	0.7250
2	0.9091	0.9351	0.9091	0.9273	0.9076	0.8625	0.8735
3	0.7273	0.8831	0.7273	0.7333	0.7229	0.5875	0.5950
4	0.9091	1.0000	0.9091	0.9273	0.9076	0.8625	0.8735
5	0.9000	0.9714	0.9000	0.9250	0.8971	0.8485	0.8616
6	0.9000	1.0000	0.9000	0.9250	0.8971	0.8485	0.8616
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	0.9000	0.9857	0.9000	0.9250	0.8971	0.8485	0.8616
9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Mean	0.8973	0.9684	0.8973	0.9108	0.8955	0.8445	0.8525
Std	0.0753	0.0415	0.0753	0.0758	0.0762	0.1139	0.1130

Processing:   0%|          | 0/4 [00:00<?, ?it/s]

Out[44]:

LogisticRegression(C=0.5, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=0.15, max_iter=1000,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=123, solver='lbfgs', tol=0.0001, verbose=0,
                   warm_start=False)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

In [45]:

# train lr and return train score as well alongwith CV
create_model('lr', return_train_score=True)

		Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Split	Fold
CV-Train	0	0.9149	0.9882	0.9149	0.9159	0.9148	0.8723	0.8729
	1	0.9255	0.9892	0.9255	0.9258	0.9255	0.8883	0.8884
	2	0.9255	0.9887	0.9255	0.9279	0.9254	0.8883	0.8896
	3	0.9468	0.9883	0.9468	0.9471	0.9468	0.9202	0.9204
	4	0.9043	0.9855	0.9043	0.9065	0.9040	0.8564	0.8577
	5	0.9158	0.9878	0.9158	0.9168	0.9157	0.8737	0.8743
	6	0.9053	0.9843	0.9053	0.9074	0.9051	0.8579	0.8592
	7	0.9158	0.9863	0.9158	0.9158	0.9158	0.8737	0.8737
	8	0.9158	0.9848	0.9158	0.9168	0.9157	0.8737	0.8743
	9	0.9053	0.9858	0.9053	0.9074	0.9051	0.8579	0.8592
CV-Val	0	0.9091	1.0000	0.9091	0.9273	0.9076	0.8625	0.8735
	1	0.8182	0.9221	0.8182	0.8182	0.8182	0.7250	0.7250
	2	0.9091	0.9610	0.9091	0.9273	0.9076	0.8625	0.8735
	3	0.6364	0.8961	0.6364	0.6364	0.6364	0.4500	0.4500
	4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
	5	0.9000	0.9714	0.9000	0.9250	0.8971	0.8485	0.8616
	6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
	7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
	8	0.9000	1.0000	0.9000	0.9250	0.8971	0.8485	0.8616
	9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
CV-Train	Mean	0.9175	0.9869	0.9175	0.9187	0.9174	0.8762	0.8770
CV-Train	Std	0.0122	0.0017	0.0122	0.0117	0.0122	0.0182	0.0180
CV-Val	Mean	0.9073	0.9751	0.9073	0.9159	0.9064	0.8597	0.8645
CV-Val	Std	0.1076	0.0360	0.1076	0.1079	0.1077	0.1628	0.1628
Train	nan	0.9143	0.9873	0.0000	0.0000	0.0000	0.8714	0.8715

Processing:   0%|          | 0/4 [00:00<?, ?it/s]

Out[45]:

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                   intercept_scaling=1, l1_ratio=None, max_iter=1000,
                   multi_class='auto', n_jobs=None, penalty='l2',
                   random_state=123, solver='lbfgs', tol=0.0001, verbose=0,
                   warm_start=False)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Some other parameters that you might find very useful in create_model are:

cross_validation
engine
fit_kwargs
groups

You can check the docstring of the function for more info.

In [46]:

# help(create_model)

✅ Tune Model¶

This function tunes the hyperparameters of the model. The output of this function is a scoring grid with cross-validated scores by fold. The best model is selected based on the metric defined in optimize parameter. Metrics evaluated during cross-validation can be accessed using the get_metrics function. Custom metrics can be added or removed using add_metric and remove_metric function.

In [47]:

# train a dt model with default params
dt = create_model('dt')

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	0.8182	0.8571	0.8182	0.8788	0.8061	0.7250	0.7642
1	0.9091	0.9286	0.9091	0.9273	0.9076	0.8625	0.8735
2	0.9091	0.9286	0.9091	0.9273	0.9076	0.8625	0.8735
3	0.7273	0.7857	0.7273	0.8442	0.6826	0.5875	0.6674
4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
5	0.9000	0.9286	0.9000	0.9250	0.8971	0.8485	0.8616
6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Mean	0.9264	0.9429	0.9264	0.9502	0.9201	0.8886	0.9040
Std	0.0893	0.0700	0.0893	0.0552	0.1011	0.1351	0.1119

Processing:   0%|          | 0/4 [00:00<?, ?it/s]

In [48]:

# tune hyperparameters of dt
tuned_dt = tune_model(dt)

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
1	0.9091	0.9481	0.9091	0.9273	0.9076	0.8625	0.8735
2	0.9091	0.9481	0.9091	0.9273	0.9076	0.8625	0.8735
3	0.7273	0.8442	0.7273	0.8442	0.6826	0.5875	0.6674
4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
5	0.9000	0.9286	0.9000	0.9250	0.8971	0.8485	0.8616
6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Mean	0.9445	0.9669	0.9445	0.9624	0.9395	0.9161	0.9276
Std	0.0838	0.0488	0.0838	0.0513	0.0958	0.1267	0.1046

Processing:   0%|          | 0/7 [00:00<?, ?it/s]

Fitting 10 folds for each of 10 candidates, totalling 100 fits

Metric to optimize can be defined in optimize parameter (default = 'Accuracy'). Also, a custom tuned grid can be passed with custom_grid parameter.

In [49]:

dt

Out[49]:

DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                       max_depth=None, max_features=None, max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_samples_leaf=1,
                       min_samples_split=2, min_weight_fraction_leaf=0.0,
                       random_state=123, splitter='best')

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

In [50]:

# define tuning grid
dt_grid = {'max_depth' : [None, 2, 4, 6, 8, 10, 12]}

# tune model with custom grid and metric = F1
tuned_dt = tune_model(dt, custom_grid = dt_grid, optimize = 'F1')

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	0.9091	0.9481	0.9091	0.9273	0.9076	0.8625	0.8735
1	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
2	0.9091	0.9286	0.9091	0.9273	0.9076	0.8625	0.8735
3	0.7273	0.8442	0.7273	0.8442	0.6826	0.5875	0.6674
4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
5	0.9000	0.9571	0.9000	0.9250	0.8971	0.8485	0.8616
6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Mean	0.9445	0.9678	0.9445	0.9624	0.9395	0.9161	0.9276
Std	0.0838	0.0485	0.0838	0.0513	0.0958	0.1267	0.1046

Processing:   0%|          | 0/7 [00:00<?, ?it/s]

Fitting 10 folds for each of 7 candidates, totalling 70 fits

In [51]:

# to access the tuner object you can set return_tuner = True
tuned_dt, tuner = tune_model(dt, return_tuner=True)

	Accuracy	AUC	Recall	Prec.	F1	Kappa	MCC
Fold
0	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
1	0.9091	0.9481	0.9091	0.9273	0.9076	0.8625	0.8735
2	0.9091	0.9481	0.9091	0.9273	0.9076	0.8625	0.8735
3	0.7273	0.8442	0.7273	0.8442	0.6826	0.5875	0.6674
4	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
5	0.9000	0.9286	0.9000	0.9250	0.8971	0.8485	0.8616
6	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
7	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
8	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
9	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
Mean	0.9445	0.9669	0.9445	0.9624	0.9395	0.9161	0.9276
Std	0.0838	0.0488	0.0838	0.0513	0.0958	0.1267	0.1046

Processing:   0%|          | 0/7 [00:00<?, ?it/s]

Fitting 10 folds for each of 10 candidates, totalling 100 fits

In [52]:

	sepal_length	sepal_width	petal_length	petal_width
0	5.1	3.5	1.4	0.2
1	4.9	3.0	1.4	0.2
2	4.7	3.2	1.3	0.2
3	4.6	3.1	1.5	0.2
4	5.0	3.6	1.4	0.2

	sepal_length	sepal_width	petal_length	petal_width
0	5.0	2.0	3.5	1.0
1	5.4	3.9	1.3	0.4
2	5.6	3.0	4.1	1.3
3	7.4	2.8	6.1	1.9
4	4.6	3.4	1.4	0.3
...	...	...	...	...
100	6.6	2.9	4.6	1.3
101	4.5	2.3	1.3	0.3
102	4.8	3.0	1.4	0.1
103	5.4	3.4	1.7	0.2
104	6.2	3.4	5.4	2.3

	sepal_length	sepal_width	petal_length	petal_width
0	5.1	3.5	1.4	0.2
1	4.9	3.0	1.4	0.2
2	4.7	3.2	1.3	0.2
3	4.6	3.1	1.5	0.2
4	5.0	3.6	1.4	0.2

	sepal_length	sepal_width	petal_length	petal_width
0	5.0	2.0	3.5	1.0
1	5.4	3.9	1.3	0.4
2	5.6	3.0	4.1	1.3
3	7.4	2.8	6.1	1.9
4	4.6	3.4	1.4	0.3
...	...	...	...	...
100	6.6	2.9	4.6	1.3
101	4.5	2.3	1.3	0.3
102	4.8	3.0	1.4	0.1
103	5.4	3.4	1.7	0.2
104	6.2	3.4	5.4	2.3

	sepal_length	sepal_width	petal_length	petal_width
0	5.1	3.5	1.4	0.2
1	4.9	3.0	1.4	0.2
2	4.7	3.2	1.3	0.2
3	4.6	3.1	1.5	0.2
4	5.0	3.6	1.4	0.2

	sepal_length	sepal_width	petal_length	petal_width
0	5.0	2.0	3.5	1.0
1	5.4	3.9	1.3	0.4
2	5.6	3.0	4.1	1.3
3	7.4	2.8	6.1	1.9
4	4.6	3.4	1.4	0.3
...	...	...	...	...
100	6.6	2.9	4.6	1.3
101	4.5	2.3	1.3	0.3
102	4.8	3.0	1.4	0.1
103	5.4	3.4	1.7	0.2
104	6.2	3.4	5.4	2.3