#!/usr/bin/env python # coding: utf-8 # Last updated: 16 Feb 2023 # # # πŸ‘‹ PyCaret Multiclass Classification Tutorial # # PyCaret is an open-source, low-code machine learning library in Python that automates machine learning workflows. It is an end-to-end machine learning and model management tool that exponentially speeds up the experiment cycle and makes you more productive. # # Compared with the other open-source machine learning libraries, PyCaret is an alternate low-code library that can be used to replace hundreds of lines of code with a few lines only. This makes experiments exponentially fast and efficient. PyCaret is essentially a Python wrapper around several machine learning libraries and frameworks, such as scikit-learn, XGBoost, LightGBM, CatBoost, spaCy, Optuna, Hyperopt, Ray, and a few more. # # The design and simplicity of PyCaret are inspired by the emerging role of citizen data scientists, a term first used by Gartner. Citizen Data Scientists are power users who can perform both simple and moderately sophisticated analytical tasks that would previously have required more technical expertise. # # # πŸ’» Installation # # PyCaret is tested and supported on the following 64-bit systems: # - Python 3.7 – 3.10 # - Python 3.9 for Ubuntu only # - Ubuntu 16.04 or later # - Windows 7 or later # # You can install PyCaret with Python's pip package manager: # # `pip install pycaret` # # PyCaret's default installation will not install all the extra dependencies automatically. For that you will have to install the full version: # # `pip install pycaret[full]` # # or depending on your use-case you may install one of the following variant: # # - `pip install pycaret[analysis]` # - `pip install pycaret[models]` # - `pip install pycaret[tuner]` # - `pip install pycaret[mlops]` # - `pip install pycaret[parallel]` # - `pip install pycaret[test]` # In[1]: # check installed version import pycaret pycaret.__version__ # # πŸš€ Quick start # PyCaret’s Classification Module is a supervised machine learning module that is used for classifying elements into groups. The goal is to predict the categorical class labels which are discrete and unordered. # # Some common use cases include predicting customer default (Yes or No), predicting customer churn (customer will leave or stay), the disease found (positive or negative). # # This module can be used for binary or multiclass problems. It provides several pre-processing features that prepare the data for modeling through the setup function. It has over 18 ready-to-use algorithms and several plots to analyze the performance of trained models. # # A typical workflow in PyCaret consist of following 5 steps in this order: # # ## **Setup** ➑️ **Compare Models** ➑️ **Analyze Model** ➑️ **Prediction** ➑️ **Save Model** # In[2]: # loading sample dataset from pycaret dataset module from pycaret.datasets import get_data data = get_data('iris') # ## Setup # This function initializes the training environment and creates the transformation pipeline. Setup function must be called before executing any other function in PyCaret. It only has two required parameters i.e. `data` and `target`. All the other parameters are optional. # In[3]: # import pycaret classification and init setup from pycaret.classification import * s = setup(data, target = 'species', session_id = 123) # Once the setup has been successfully executed it shows the information grid containing experiment level information. # # - **Session id:** A pseudo-random number distributed as a seed in all functions for later reproducibility. If no `session_id` is passed, a random number is automatically generated that is distributed to all functions.
#
# - **Target type:** Binary, Multiclass, or Regression. The Target type is automatically detected.
#
# - **Label Encoding:** When the Target variable is of type string (i.e. 'Yes' or 'No') instead of 1 or 0, it automatically encodes the label into 1 and 0 and displays the mapping (0 : No, 1 : Yes) for reference. In this tutorial, no label encoding is required since the target variable is of numeric type.
#
# - **Original data shape:** Shape of the original data prior to any transformations.
#
# - **Transformed train set shape :** Shape of transformed train set
#
# - **Transformed test set shape :** Shape of transformed test set
#
# - **Numeric features :** The number of features considered as numerical.
#
# - **Categorical features :** The number of features considered as categorical.
# PyCaret has two set of API's that you can work with. (1) Functional (as seen above) and (2) Object Oriented API. # # With Object Oriented API instead of executing functions directly you will import a class and execute methods of class. # In[4]: # import ClassificationExperiment and init the class from pycaret.classification import ClassificationExperiment exp = ClassificationExperiment() # In[5]: # check the type of exp type(exp) # In[6]: # init setup on exp exp.setup(data, target = 'species', session_id = 123) # You can use any of the two method i.e. Functional or OOP and even switch back and forth between two set of API's. The choice of method will not impact the results and has been tested for consistency. # ## Compare Models # # This function trains and evaluates the performance of all the estimators available in the model library using cross-validation. The output of this function is a scoring grid with average cross-validated scores. Metrics evaluated during CV can be accessed using the `get_metrics` function. Custom metrics can be added or removed using `add_metric` and `remove_metric` function. # In[7]: # compare baseline models best = compare_models() # In[8]: # compare models using OOP exp.compare_models() # Notice that the output between functional and OOP API is consistent. Rest of the functions in this notebook will only be shown using functional API only. # ## Analyze Model # You can use the `plot_model` function to analyzes the performance of a trained model on the test set. It may require re-training the model in certain cases. # In[9]: # plot confusion matrix plot_model(best, plot = 'confusion_matrix') # In[10]: # plot AUC plot_model(best, plot = 'auc') # In[11]: # plot feature importance plot_model(best, plot = 'feature') # In[12]: # check docstring to see available plots help(plot_model) # An alternate to `plot_model` function is `evaluate_model`. It can only be used in Notebook since it uses ipywidget. # In[13]: evaluate_model(best) # ## Prediction # The `predict_model` function returns `prediction_label` and `prediction_score` (probability of the predicted class) as new columns in dataframe. When data is `None` (default), it uses the test set (created during the setup function) for scoring. # In[14]: # predict on test set holdout_pred = predict_model(best) # In[15]: # show predictions df holdout_pred.head() # The same function works for predicting the labels on unseen dataset. Let's create a copy of original data and drop the `Class variable`. We can then use the new data frame without labels for scoring. # In[16]: # copy data and drop Class variable new_data = data.copy() new_data.drop('species', axis=1, inplace=True) new_data.head() # In[17]: # predict model on new_data predictions = predict_model(best, data = new_data) predictions.head() # ## Save Model # Finally, you can save the entire pipeline on disk for later use, using pycaret's `save_model` function. # In[18]: # save pipeline save_model(best, 'my_first_pipeline') # In[19]: # load pipeline loaded_best_pipeline = load_model('my_first_pipeline') loaded_best_pipeline # # πŸ‘‡ Detailed function-by-function overview # ## βœ… Setup # This function initializes the experiment in PyCaret and creates the transformation pipeline based on all the parameters passed in the function. Setup function must be called before executing any other function. It takes two required parameters: `data` and `target`. All the other parameters are optional and are used for configuring data preprocessing pipeline. # In[20]: s = setup(data, target = 'species', session_id = 123) # To access all the variables created by the setup function such as transformed dataset, random_state, etc. you can use `get_config` method. # In[21]: # check all available config get_config() # In[22]: # lets access X_train_transformed get_config('X_train_transformed') # In[23]: # another example: let's access seed print("The current seed is: {}".format(get_config('seed'))) # now lets change it using set_config set_config('seed', 786) print("The new seed is: {}".format(get_config('seed'))) # All the preprocessing configurations and experiment settings/parameters are passed into the `setup` function. To see all available parameters, check the docstring: # In[24]: # help(setup) # In[25]: # init setup with normalize = True s = setup(data, target = 'species', session_id = 123, normalize = True, normalize_method = 'minmax') # In[26]: # lets check the X_train_transformed to see effect of params passed get_config('X_train_transformed')['sepal_length'].hist() # Notice that all the values are between 0 and 1 - that is because we passed `normalize=True` in the `setup` function. If you don't remember how it compares to actual data, no problem - we can also access non-transformed values using `get_config` and then compare. See below and notice the range of values on x-axis and compare it with histogram above. # In[27]: get_config('X_train')['sepal_length'].hist() # ## βœ… Compare Models # This function trains and evaluates the performance of all estimators available in the model library using cross-validation. The output of this function is a scoring grid with average cross-validated scores. Metrics evaluated during CV can be accessed using the `get_metrics` function. Custom metrics can be added or removed using `add_metric` and `remove_metric` function. # In[28]: best = compare_models() # `compare_models` by default uses all the estimators in model library (all except models with `Turbo=False`) . To see all available models you can use the function `models()` # In[29]: # check available models models() # You can use the `include` and `exclude` parameter in the `compare_models` to train only select model or exclude specific models from training by passing the model id's in `exclude` parameter. # In[30]: compare_tree_models = compare_models(include = ['dt', 'rf', 'et', 'gbc', 'xgboost', 'lightgbm', 'catboost']) # In[31]: compare_tree_models # The function above has return trained model object as an output. The scoring grid is only displayed and not returned. If you need access to the scoring grid you can use `pull` function to access the dataframe. # In[32]: compare_tree_models_results = pull() compare_tree_models_results # By default `compare_models` return the single best performing model based on the metric defined in the `sort` parameter. Let's change our code to return 3 top models based on `Recall`. # In[33]: best_recall_models_top3 = compare_models(sort = 'Recall', n_select = 3) # In[34]: # list of top 3 models by Recall best_recall_models_top3 # Some other parameters that you might find very useful in `compare_models` are: # # - fold # - cross_validation # - budget_time # - errors # - probability_threshold # - parallel # # You can check the docstring of the function for more info. # In[35]: # help(compare_models) # ## βœ… Experiment Logging # PyCaret integrates with many different type of experiment loggers (default = 'mlflow'). To turn on experiment tracking in PyCaret you can set `log_experiment` and `experiment_name` parameter. It will automatically track all the metrics, hyperparameters, and artifacts based on the defined logger. # In[36]: # from pycaret.classification import * # s = setup(data, target = 'Class variable', log_experiment='mlflow', experiment_name='iris_experiment') # In[37]: # compare models # best = compare_models() # In[38]: # start mlflow server on localhost:5000 # !mlflow ui # By default PyCaret uses `MLFlow` logger that can be changed using `log_experiment` parameter. Following loggers are available: # # - mlflow # - wandb # - comet_ml # - dagshub # # Other logging related parameters that you may find useful are: # # - experiment_custom_tags # - log_plots # - log_data # - log_profile # # For more information check out the docstring of the `setup` function. # In[39]: # help(setup) # ## βœ… Create Model # This function trains and evaluates the performance of a given estimator using cross-validation. The output of this function is a scoring grid with CV scores by fold. Metrics evaluated during CV can be accessed using the `get_metrics` function. Custom metrics can be added or removed using `add_metric` and `remove_metric` function. All the available models can be accessed using the models function. # In[40]: # check all the available models models() # In[41]: # train logistic regression with default fold=10 lr = create_model('lr') # The function above has return trained model object as an output. The scoring grid is only displayed and not returned. If you need access to the scoring grid you can use `pull` function to access the dataframe. # In[42]: lr_results = pull() print(type(lr_results)) lr_results # In[43]: # train logistic regression with fold=3 lr = create_model('lr', fold=3) # In[44]: # train logistic regression with specific model parameters create_model('lr', C = 0.5, l1_ratio = 0.15) # In[45]: # train lr and return train score as well alongwith CV create_model('lr', return_train_score=True) # Some other parameters that you might find very useful in `create_model` are: # # - cross_validation # - engine # - fit_kwargs # - groups # # You can check the docstring of the function for more info. # In[46]: # help(create_model) # ## βœ… Tune Model # # This function tunes the hyperparameters of the model. The output of this function is a scoring grid with cross-validated scores by fold. The best model is selected based on the metric defined in optimize parameter. Metrics evaluated during cross-validation can be accessed using the `get_metrics` function. Custom metrics can be added or removed using `add_metric` and `remove_metric` function. # In[47]: # train a dt model with default params dt = create_model('dt') # In[48]: # tune hyperparameters of dt tuned_dt = tune_model(dt) # Metric to optimize can be defined in `optimize` parameter (default = 'Accuracy'). Also, a custom tuned grid can be passed with `custom_grid` parameter. # In[49]: dt # In[50]: # define tuning grid dt_grid = {'max_depth' : [None, 2, 4, 6, 8, 10, 12]} # tune model with custom grid and metric = F1 tuned_dt = tune_model(dt, custom_grid = dt_grid, optimize = 'F1') # In[51]: # to access the tuner object you can set return_tuner = True tuned_dt, tuner = tune_model(dt, return_tuner=True) # In[52]: # model object tuned_dt # In[53]: # tuner object tuner # The default search algorithm is `RandomizedSearchCV` from `sklearn`. This can be changed by using `search_library` and `search_algorithm` parameter. # In[54]: # tune dt using optuna tuned_dt = tune_model(dt, search_library = 'optuna') # For more details on all available `search_library` and `search_algorithm` please check the docstring. Some other parameters that you might find very useful in `tune_model` are: # # - choose_better # - n_iter # - early_stopping # - groups # # You can check the docstring of the function for more info. # In[55]: # help(tune_model) # ## βœ… Ensemble Model # This function ensembles a given estimator. The output of this function is a scoring grid with CV scores by fold. Metrics evaluated during CV can be accessed using the `get_metrics` function. Custom metrics can be added or removed using `add_metric` and `remove_metric` function. # In[56]: # ensemble with bagging ensemble_model(dt, method = 'Bagging') # In[57]: # ensemble with boosting ensemble_model(dt, method = 'Boosting') # Some other parameters that you might find very useful in `ensemble_model` are: # # - choose_better # - n_estimators # - groups # - fit_kwargs # - probability_threshold # - return_train_score # # You can check the docstring of the function for more info. # In[58]: # help(ensemble_model) # ## βœ… Blend Models # This function trains a Soft Voting / Majority Rule classifier for select models passed in the estimator_list parameter. The output of this function is a scoring grid with CV scores by fold. Metrics evaluated during CV can be accessed using the `get_metrics` function. Custom metrics can be added or removed using `add_metric` and `remove_metric` function. # In[59]: # top 3 models based on recall best_recall_models_top3 # In[60]: # blend top 3 models blend_models(best_recall_models_top3) # Some other parameters that you might find very useful in `blend_models` are: # # - choose_better # - method # - weights # - fit_kwargs # - probability_threshold # - return_train_score # # You can check the docstring of the function for more info. # In[61]: # help(blend_models) # ## βœ… Stack Models # This function trains a meta-model over select estimators passed in the estimator_list parameter. The output of this function is a scoring grid with CV scores by fold. Metrics evaluated during CV can be accessed using the `get_metrics` function. Custom metrics can be added or removed using `add_metric` and `remove_metric` function. # In[62]: # stack models stack_models(best_recall_models_top3) # Some other parameters that you might find very useful in `stack_models` are: # # - choose_better # - meta_model # - method # - restack # - probability_threshold # - return_train_score # # You can check the docstring of the function for more info. # In[63]: # help(stack_models) # ## βœ… Plot Model # This function analyzes the performance of a trained model on the hold-out set. It may require re-training the model in certain cases. # In[64]: # plot class report plot_model(best, plot = 'class_report') # In[65]: # to control the scale of plot plot_model(best, plot = 'class_report', scale = 2) # In[66]: # to save the plot plot_model(best, plot = 'class_report', save=True) # Some other parameters that you might find very useful in `plot_model` are: # # - fit_kwargs # - plot_kwargs # - groups # - display_format # # You can check the docstring of the function for more info. # In[67]: # help(plot_model) # ## βœ… Interpret Model # This function analyzes the predictions generated from a trained model. Most plots in this function are implemented based on the SHAP (Shapley Additive exPlanations). For more info on this, please see https://shap.readthedocs.io/en/latest/ # In[68]: # train lightgbm model lightgbm = create_model('lightgbm') # In[69]: # interpret summary model interpret_model(lightgbm, plot = 'summary') # In[70]: # reason plot for test set observation 1 interpret_model(lightgbm, plot = 'reason', observation = 1) # Some other parameters that you might find very useful in `interpret_model` are: # # - plot # - feature # - use_train_data # - X_new_sample # - y_new_sample # - save # # You can check the docstring of the function for more info. # In[71]: # help(interpret_model) # ## βœ… Get Leaderboard # This function returns the leaderboard of all models trained in the current setup. # In[72]: # get leaderboard lb = get_leaderboard() lb # In[73]: # select the best model based on F1 lb.sort_values(by='F1', ascending=False)['Model'].iloc[0] # Some other parameters that you might find very useful in `get_leaderboard` are: # # - finalize_models # - fit_kwargs # - model_only # - groups # # You can check the docstring of the function for more info. # In[74]: # help(get_leaderboard) # ## βœ… AutoML # This function returns the best model out of all trained models in the current setup based on the optimize parameter. Metrics evaluated can be accessed using the `get_metrics` function. # In[75]: automl() # ## βœ… Dashboard # The dashboard function generates the interactive dashboard for a trained model. The dashboard is implemented using `ExplainerDashboard`. For more information check out [Explainer Dashboard.](explainerdashboard.readthedocs.io) # In[76]: # dashboard function dashboard(dt, display_format ='inline') # ## βœ…Create App # This function creates a basic gradio app for inference. # In[79]: # create gradio app create_app(best) # ## βœ… Create API # This function takes an input model and creates a POST API for inference. # In[80]: # create api create_api(best, api_name = 'my_first_api') # In[81]: # !python my_first_api.py # In[82]: # check out the .py file created with this magic command # %load my_first_api.py # ## βœ… Create Docker # This function creates a `Dockerfile` and `requirements.txt` for productionalizing API end-point. # In[83]: create_docker('my_first_api') # In[84]: # check out the DockerFile file created with this magic command # %load DockerFile # In[85]: # check out the requirements file created with this magic command # %load requirements.txt # ## βœ… Finalize Model # This function trains a given model on the entire dataset including the hold-out set. # In[86]: final_best = finalize_model(best) # In[87]: final_best # ## βœ… Convert Model # This function transpiles the trained machine learning model's decision function in different programming languages such as Python, C, Java, Go, C#, etc. It is very useful if you want to deploy models into environments where you can't install your normal Python stack to support model inference. # In[88]: # transpiles learned function to java print(convert_model(dt, language = 'java')) # ## βœ… Deploy Model # This function deploys the entire ML pipeline on the cloud. # # **AWS:** When deploying model on AWS S3, environment variables must be configured using the command-line interface. To configure AWS environment variables, type `aws configure` in terminal. The following information is required which can be generated using the Identity and Access Management (IAM) portal of your amazon console account: # # - AWS Access Key ID # - AWS Secret Key Access # - Default Region Name (can be seen under Global settings on your AWS console) # - Default output format (must be left blank) # # **GCP:** To deploy a model on Google Cloud Platform ('gcp'), the project must be created using the command-line or GCP console. Once the project is created, you must create a service account and download the service account key as a JSON file to set environment variables in your local environment. Learn more about it: https://cloud.google.com/docs/authentication/production # # **Azure:** To deploy a model on Microsoft Azure ('azure'), environment variables for the connection string must be set in your local environment. Go to settings of storage account on Azure portal to access the connection string required. # AZURE_STORAGE_CONNECTION_STRING (required as environment variable) # Learn more about it: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?toc=%2Fpython%2Fazure%2FTOC.json # In[89]: # deploy model on aws s3 # deploy_model(best, model_name = 'my_first_platform_on_aws', # platform = 'aws', authentication = {'bucket' : 'pycaret-test'}) # In[90]: # load model from aws s3 # loaded_from_aws = load_model(model_name = 'my_first_platform_on_aws', platform = 'aws', # authentication = {'bucket' : 'pycaret-test'}) # loaded_from_aws # ## βœ… Save / Load Model # This function saves the transformation pipeline and a trained model object into the current working directory as a pickle file for later use. # In[91]: # save model save_model(best, 'my_first_model') # In[92]: # load model loaded_from_disk = load_model('my_first_model') loaded_from_disk # ## βœ… Save / Load Experiment # This function saves all the experiment variables on disk, allowing to later resume without rerunning the setup function. # In[93]: # save experiment save_experiment('my_experiment') # In[94]: # load experiment from disk exp_from_disk = load_experiment('my_experiment', data=data)