Analyze Errors and Explore Interpretability of Models¶

This notebook demonstrates how to use the Responsible AI Widget's Error Analysis dashboard to understand a model trained on the regression california housing dataset. The goal of this sample notebook is to predict housing prices with scikit-learn and explore model errors and explanations:

Train an SVM classification model using Scikit-learn
Run Interpret-Community's 'explain_model' globally and locally to generate model explanations.
Visualize model errors and global and local explanations with the Error Analysis visualization dashboard.

Install Required Packages¶

In [ ]:

# %pip install --upgrade interpret-community
# %pip install --upgrade raiwidgets

Import required packages¶

In [ ]:

from sklearn.datasets import fetch_california_housing
from sklearn import svm

# Imports for SHAP MimicExplainer with LightGBM surrogate model
from interpret.ext.blackbox import MimicExplainer
from interpret.ext.glassbox import LGBMExplainableModel

Load the wine data¶

In [ ]:

housing = fetch_california_housing()
X = housing['data']
y = housing['target']
feature_names = housing['feature_names']

In [ ]:

# Split data into train and test
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)

Train a SVM classification model¶

In [ ]:

clf = svm.SVR(gamma=0.001, C=100.)
model = clf.fit(X_train, y_train)

In [ ]:

# Notice the model makes a fair amount of error
print("average abs error on test dataset: " + str(sum(abs(model.predict(X_test) - y_test))/y_test.shape[0]))

Load simple ErrorAnalysis view without explanations¶

In [ ]:

from raiwidgets import ErrorAnalysisDashboard
predictions = model.predict(X_test)
ErrorAnalysisDashboard(dataset=X_test, true_y=y_test, features=feature_names, pred_y=predictions, model_task='regression')

Train a surrogate model to explain the original blackbox model¶

In [ ]:

from interpret_community.common.constants import ModelTask
# Train the LightGBM surrogate model using MimicExplaner
model_task = ModelTask.Regression
explainer = MimicExplainer(model, X_train, LGBMExplainableModel,
                           augment_data=True, max_num_of_augmentations=10,
                           features=feature_names, model_task=model_task)

Generate global explanations¶

Explain overall model predictions (global explanation)

In [ ]:

# Passing in test dataset for evaluation examples - note it must be a representative sample of the original data
# X_train can be passed as well, but with more examples explanations will take longer although they may be more accurate
global_explanation = explainer.explain_global(X_test)

In [ ]:

# Print out a dictionary that holds the sorted feature importance names and values
print('global importance rank: {}'.format(global_explanation.get_feature_importance_dict()))

Visualize¶

Analyze model errors and explanations using Error Analysis dashboard¶

In [ ]:

from raiwidgets import ErrorAnalysisDashboard

In [ ]:

ErrorAnalysisDashboard(global_explanation, model, dataset=X_test, true_y=y_test, model_task='regression')