In [1]:
%reload_ext autoreload
%matplotlib inline
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID";
os.environ["CUDA_VISIBLE_DEVICES"]="0"


# Explainable AI in ktrain¶

Deep neural networks are sometimes called "black boxes", as it is not always clear how such models are using data to make a decision or prediction. Explainable AI (XAI) involves methods and techniques to help understand how an AI model reach particular conclusions. Although XAI is an open and actively researched problem, there are some existing methods that can be practically applied now.

In previous tutorials, we saw that, for both image classification and text classification, we can invoke the ktrain.get_predictor function and obtain a Predictor object to make predictions on new raw data. For instance, with text data, one can easily make predictions from the raw and unprocessed text of a document as follows:

predictor = ktrain.get_predictor(learner.model, preproc=preproc)
predictor.predict(document_text)

In this notebook, we show how one can invoke the explain method of Predictor objects to help understand how those predictions were made. This is particularly useful in understanding misclassifications. We start with image classification.

## Explaining Image Classification¶

Let's begin by using ktrain to train an image classifier for a single epoch on the publicly available Kaggle Dogs vs. Cats dataset, as we did in the tutorial 3 notebook.

In [2]:
# imports
import ktrain
from ktrain import vision as vis

(train_data, val_data, preproc) = vis.images_from_folder(
data_aug = vis.get_data_aug(horizontal_flip=True),
train_test_names=['train', 'valid'],
target_size=(224,224), color_mode='rgb')

# STEP 2: define model
model = vis.image_classifier('pretrained_resnet50', train_data, val_data, freeze_layers=15)
learner = ktrain.get_learner(model=model, train_data=train_data, val_data=val_data,
workers=8, use_multiprocessing=False, batch_size=64)

# STEP 3:  train
learner.fit_onecycle(1e-4, 1)

Using TensorFlow backend.

Found 23000 images belonging to 2 classes.
Found 23000 images belonging to 2 classes.
Found 2000 images belonging to 2 classes.
The normalization scheme has been changed for use with a pretrained_resnet50 model. If you decide to use a different model, please reload your dataset with a ktrain.vision.data.images_from* function.

Is Multi-Label? False
pretrained_resnet50 model created.

begin training using onecycle policy with max lr of 0.0001...
Epoch 1/1
359/359 [==============================] - 127s 353ms/step - loss: 0.2548 - acc: 0.9270 - val_loss: 0.0486 - val_acc: 0.9874

Out[2]:
<keras.callbacks.History at 0x7f69710eefd0>

The view_top_losses method in ktrain identifies those examples in the validation set that were misclassified and sorts them by loss such that examples at the top of the list are the most severely misclassified. Let's invoke view_top_losses to see the top 3 most misclassified images by our model.

In [5]:
learner.view_top_losses(n=1, preproc=preproc)


The top most misclassified image depicts both a dog and a cat but is labeled as belonging to only the cats category. Our classifier's prediction for this image is dogs probably because it is focusing mostly on the dog in this image. This can be verified by invoking the explain method.

In [6]:
predictor = ktrain.get_predictor(learner.model, preproc)
predictor.explain('data/dogscats/valid/cats/cat.5583.jpg')

Out[6]:

The explain method displays the image and highlights the area on which the classifier focuses to make the prediction. As expected, our model is focusing on the dog in this picture when predicting the label dogs. In this case, Explainable AI was not really necessary to understand why our classifier generated its prediction. However, such techniques can be helpful in other cases where the decision-making process is not as evident.

Consider this image in the validation set, for example:

In [7]:
vis.show_image('data/dogscats/valid/cats/cat.92.jpg')

Out[7]:
<matplotlib.image.AxesImage at 0x7f697209d400>

The image (i.e., cat.92.jpg) is interesting in that there are no real dogs or cats in the picture. Nevertheless, the image has a ground truth label of cats. Perhaps this is because there is a stuffed cat in the photo. There is, however, also a cartoon dog in the background.

Our classifier is understandably confused and placed the picture in the dogs category.

In [8]:
predictor.predict_filename('data/dogscats/valid/cats/cat.92.jpg')

Out[8]:
['dogs']

But, why has our classifier predicted dogs for this photo? Is it mistaking the stuffed cat for a dog or is it focusing on the cartoon dog in the background? We can find the answer by creating a Predictor object and invoking the explain method.

In [9]:
predictor.explain('data/dogscats/valid/cats/cat.92.jpg')

Out[9]:

It is clear from the visualization that our classifier is focusing on the stuffed cat and mistaking it for a dog. Moreover, it appears to be particuarly to focused on the ears of the stuffed animal, which do sort of look like dog ears.

These visualizations are based on the Grad-CAM technique and is supported in ktrain via the eli5 library.

## Explaining Text Classification¶

We can also visualize which words text classifiers tend to focus on when making predictions. As before, let's use ktrain to quickly build a text classifier to classify IMDb movie reivews as positive or negative.

In [10]:
# imports
import ktrain
from ktrain import text

# STEP 1: load and preprocess text data
(x_train, y_train), (x_test, y_test), preproc = text.texts_from_folder('data/aclImdb',
max_features=20000, maxlen=400,
ngram_range=1,
train_test_names=['train', 'test'],
classes=['pos', 'neg'])

# STEP 2: define a Keras text classification model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding, GlobalAveragePooling1D
model = Sequential()
learner = ktrain.get_learner(model, train_data=(x_train, y_train), val_data=(x_test, y_test))

# STEP 3: train
learner.autofit(0.005, 2)

Word Counts: 88582
Nrows: 25000
25000 train sequences
Average train sequence length: 231
x_train shape: (25000,400)
y_train shape: (25000,2)
25000 test sequences
Average test sequence length: 224
x_test shape: (25000,400)
y_test shape: (25000,2)

begin training using triangular learning rate policy with max lr of 0.005...
Train on 25000 samples, validate on 25000 samples
Epoch 1/2
25000/25000 [==============================] - 9s 357us/step - loss: 0.4473 - acc: 0.7881 - val_loss: 0.3067 - val_acc: 0.8809
Epoch 2/2
25000/25000 [==============================] - 8s 317us/step - loss: 0.2328 - acc: 0.9122 - val_loss: 0.2794 - val_acc: 0.8890

Out[10]:
<keras.callbacks.History at 0x7f693d5d2dd8>

As we did above, we invoke view_top_losses to view the most misclassified review in the validation set.

In [11]:
learner.view_top_losses(n=1, preproc=preproc)

----------
id:8244 | loss:11.57 | true:neg | pred:pos)

is not like mickey rourke ever really disappeared he has had a steady string of appearances before he burst back on the scene he was memorable in domino sin city man on fire once upon a time in mexico and get carter but in his powerful dramatic performance in the wrestler 2008 we see a full blown presentation of the character only hinted at in get carter whenever we get to know him rourke remains a cool but sleazy muscle bound slim ball br br this is an leonard story and production leonard wrote such notable movies as western thriller 3 10 to yuma be cool jackie brown get 52 pick up and joe this means that we get tough guys some good some not so good br br it also means we get tight realistic plots with characters doing what is best for them in each situation weaving complications into violent conclusions is no different tough slim ball killer rourke stalks unhappily married witness lane think history of violence meets no country for old men it is not as intense bloody or gory as those two but it is almost as good if you like those two including david equally wonderful eastern promises you will like also br br director john has not done a lot of movies his last few were enjoyable if not successful proof captain and shakespeare in love br br diana lane hasn't had a powerful movie role since she and richard gere gave incredible performances in unfaithful lately she is charming and appealing in romantic stories such as nights in must love dogs and under the sun here she is right on mark balancing her sexy appeal with reserved tension br br this is a small part for rosario dawson yet dawson does a good job with it you see a lot more of lane including an underwear scene to rival sigourney weaver in aliens and nicole kidman in eyes wide shut br br while you are in the crime drama section also pick up kiss kiss bang bang and gone baby gone and before the devil knows your dead the last has wonderful performances by phillip seymour hoffman ethan hawke marisa tomei and albert finney br br flopped at the box office more is our luck it is certainly worth a 3 4 dollar rental if you like this genre 6 20 2009


The review is evidently about a movie starring Mickey Rourke, but it is difficult to follow. This is partly because the text displayed is post-processed in that punctuation and rare words are omitted (including the actual title of the movie!). Let's use the ID of this document (i.e., id=8244 as shown above) to retrieve the original text and supply that as input to predictor.explain.

In [10]:
from sklearn.datasets import load_files
test_b = load_files(os.path.join('data/aclImdb',  'test'), shuffle=False, categories=['neg', 'pos'])
doc = test_b.data[8244].decode('utf-8')

In [11]:
predictor = ktrain.get_predictor(learner.model, preproc)


NOTE: Output of explain below does not render correctly through GitHub's notebook viewer (colors are . View directly through Jupyter notebook for proper display.

In [12]:
predictor.explain(doc)

Out[12]:

y=pos (probability 1.000, score 12.473) top features

Contribution? Feature
+12.740 Highlighted in text (sum)
-0.267 <BIAS>

mickey rourke hunts diane lane in elmore leonard's killshot it is not like mickey rourke ever really disappeared. he has had a steady string of appearances before he burst back on the scene. he was memorable in: domino, sin city, man on fire, once upon a time in mexico, and get carter. but in his powerful dramatic performance in the wrestler (2008), we see a full blown presentation of the character only hinted at in get carter. whenever we get to know him, rourke remains a cool, but sleazy, muscle bound slim ball.<br /><br />this is an elmore leonard story, and production. leonard wrote such notable movies as taunt western thriller 3:10 to yuma, be cool, jackie brown, get shorty, 52 pick-up, and joe kidd. this means that we get tough guys, some good, some not so good.<br /><br />it also means we get tight, realistic plots with characters doing what is best for them in each situation, weaving complications into violent conclusions. killshot is no different. tough, slim ball killer rourke stalks unhappily married witness lane. think history of violence meets no country for old men. it is not as intense, bloody or gory as those two, but it is almost as good. if you like those two, including david croneberg's equally wonderful eastern promises, you will like killshot also.<br /><br />director john madden has not done a lot of movies. his last few were enjoyable, if not successful: proof, captain corelli's mandolin and shakespeare in love.<br /><br />diana lane hasn't had a powerful movie role since she and richard gere gave incredible performances in unfaithful. lately she is charming and appealing in romantic stories such as nights in rodanthe, must love dogs, and under the tuscan sun. here she is right on mark, balancing her sexy appeal with reserved tension.<br /><br />this is a small part for rosario dawson. yet dawson does a good job with it. you see a lot more of lane, including an underwear scene to rival sigourney weaver in aliens and nicole kidman in eyes wide shut.<br /><br />while you are in the crime drama section, also pick up kiss, kiss, bang, bang, and gone baby gone, and before the devil knows your dead. the last has wonderful performances by phillip seymour hoffman, ethan hawke, marisa tomei and albert finney.<br /><br />killshot flopped at the box office. more is our luck. it is certainly worth a 3-4 dollar rental, if you like this genre. 6/20/2009

The visualization is generated using a technique called LIME. The input is randomly perturbed to examine how the prediction changes. This is used to infer the relative importance of different words to the final prediction using a linear interpretable model. The GREEN words contribute to the incorrect classification. The RED (and PINK) words detract from our final prediction. (Shade of color denotes the strength or size of the coefficients in the inferred linear model.) By examining the GREEN words, we can see that the review is overall positive despite being given a ground truth label of negative. In fact, the last line recommends this movie as a rental. The review is also odd in that it spends a lot of time praising past performances of the actors in previous movies. The PINK words show us that there is not very much there in terms of negative words that would lead us to conclude that this is a negative review. For these reasons, we can probably forgive our simple model for misclassifying this particular review as positive, as many humans would probably classify this as positive, as well.

## Explaining Tabular Models¶

Let's train a model to predict Survival using Kaggle's Titatnic dataset.

After training the model, we will explain the model's prediction for a specific example.

In [2]:
import pandas as pd
import numpy as np
train_df = train_df.drop('Name', 1)
train_df = train_df.drop('Ticket', 1)
train_df = train_df.drop('Cabin', 1)

np.random.seed(42)
p = 0.1 # 10% for test set
prop = 1-p
df = train_df.copy()
msk = np.random.rand(len(df)) < prop
train_df = df[msk]
test_df = df[~msk]

import ktrain
from ktrain import tabular
trn, val, preproc = tabular.tabular_from_df(train_df, label_columns=['Survived'], random_state=42)
model = tabular.tabular_classifier('mlp', trn)
learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=32)
learner.fit_onecycle(1e-3, 25)

processing train: 717 rows x 8 columns

The following integer column(s) are being treated as categorical variables:
['Pclass', 'SibSp', 'Parch']
To treat any of these column(s) as numerical, cast the column to float in DataFrame or CSV
and re-run tabular_from* function.

processing test: 82 rows x 8 columns
Is Multi-Label? False
done.

begin training using onecycle policy with max lr of 0.001...
Train for 23 steps, validate for 3 steps
Epoch 1/25
23/23 [==============================] - 2s 81ms/step - loss: 0.9956 - accuracy: 0.6444 - val_loss: 0.6542 - val_accuracy: 0.5976
Epoch 2/25
23/23 [==============================] - 1s 26ms/step - loss: 0.8901 - accuracy: 0.6876 - val_loss: 0.6740 - val_accuracy: 0.5976
Epoch 3/25
23/23 [==============================] - 1s 27ms/step - loss: 0.8071 - accuracy: 0.7113 - val_loss: 0.7263 - val_accuracy: 0.5976
Epoch 4/25
23/23 [==============================] - 1s 26ms/step - loss: 0.7878 - accuracy: 0.7252 - val_loss: 0.7907 - val_accuracy: 0.5976
Epoch 5/25
23/23 [==============================] - 1s 26ms/step - loss: 0.6747 - accuracy: 0.7531 - val_loss: 0.8780 - val_accuracy: 0.5976
Epoch 6/25
23/23 [==============================] - 1s 27ms/step - loss: 0.7568 - accuracy: 0.7322 - val_loss: 0.9060 - val_accuracy: 0.5976
Epoch 7/25
23/23 [==============================] - 1s 25ms/step - loss: 0.7112 - accuracy: 0.7490 - val_loss: 0.9270 - val_accuracy: 0.5976
Epoch 8/25
23/23 [==============================] - 1s 26ms/step - loss: 0.7759 - accuracy: 0.7211 - val_loss: 0.9231 - val_accuracy: 0.5976
Epoch 9/25
23/23 [==============================] - 1s 25ms/step - loss: 0.6682 - accuracy: 0.7322 - val_loss: 0.8639 - val_accuracy: 0.5976
Epoch 10/25
23/23 [==============================] - 1s 26ms/step - loss: 0.6515 - accuracy: 0.7531 - val_loss: 0.8748 - val_accuracy: 0.5976
Epoch 11/25
23/23 [==============================] - 1s 27ms/step - loss: 0.6163 - accuracy: 0.7545 - val_loss: 0.8865 - val_accuracy: 0.5976
Epoch 12/25
23/23 [==============================] - 1s 26ms/step - loss: 0.6151 - accuracy: 0.7545 - val_loss: 0.7569 - val_accuracy: 0.5976
Epoch 13/25
23/23 [==============================] - 1s 25ms/step - loss: 0.5756 - accuracy: 0.7699 - val_loss: 0.7045 - val_accuracy: 0.5976
Epoch 14/25
23/23 [==============================] - 1s 26ms/step - loss: 0.5726 - accuracy: 0.7601 - val_loss: 0.6486 - val_accuracy: 0.5976
Epoch 15/25
23/23 [==============================] - 1s 28ms/step - loss: 0.5173 - accuracy: 0.7880 - val_loss: 0.6483 - val_accuracy: 0.5976
Epoch 16/25
23/23 [==============================] - 1s 25ms/step - loss: 0.5761 - accuracy: 0.7768 - val_loss: 0.6257 - val_accuracy: 0.5976
Epoch 17/25
23/23 [==============================] - 1s 27ms/step - loss: 0.4843 - accuracy: 0.7992 - val_loss: 0.5562 - val_accuracy: 0.6341
Epoch 18/25
23/23 [==============================] - 1s 27ms/step - loss: 0.4905 - accuracy: 0.7894 - val_loss: 0.5312 - val_accuracy: 0.6829
Epoch 19/25
23/23 [==============================] - 1s 27ms/step - loss: 0.4682 - accuracy: 0.8103 - val_loss: 0.5137 - val_accuracy: 0.6951
Epoch 20/25
23/23 [==============================] - 1s 27ms/step - loss: 0.5161 - accuracy: 0.7713 - val_loss: 0.4818 - val_accuracy: 0.7439
Epoch 21/25
23/23 [==============================] - 1s 26ms/step - loss: 0.5027 - accuracy: 0.7810 - val_loss: 0.4391 - val_accuracy: 0.8049
Epoch 22/25
23/23 [==============================] - 1s 26ms/step - loss: 0.5012 - accuracy: 0.7810 - val_loss: 0.4115 - val_accuracy: 0.7927
Epoch 23/25
23/23 [==============================] - 1s 26ms/step - loss: 0.4683 - accuracy: 0.8006 - val_loss: 0.3888 - val_accuracy: 0.8049
Epoch 24/25
23/23 [==============================] - 1s 27ms/step - loss: 0.4762 - accuracy: 0.7852 - val_loss: 0.3725 - val_accuracy: 0.8659
Epoch 25/25
23/23 [==============================] - 1s 26ms/step - loss: 0.4649 - accuracy: 0.8075 - val_loss: 0.3573 - val_accuracy: 0.8659

Out[2]:
<tensorflow.python.keras.callbacks.History at 0x7f66d826d6d8>
In [3]:
predictor = ktrain.get_predictor(learner.model, preproc)
preds = predictor.predict(test_df, return_proba=True)
df = test_df.copy()[[c for c in test_df.columns.values if c != 'Survived']]
df['Survived'] = test_df['Survived']
df['predicted_Survived'] = np.argmax(preds, axis=1)

Out[3]:
Pclass Sex Age SibSp Parch Fare Embarked Survived predicted_Survived
PassengerId
2 1 female 38.0 1 0 71.2833 C 1 1
12 1 female 58.0 0 0 26.5500 S 1 1
34 2 male 66.0 0 0 10.5000 S 0 0
35 1 male 28.0 1 0 82.1708 C 0 0
44 2 female 3.0 1 2 41.5792 C 1 1

Let's explain the male passenger with PassengerID=5 using the shape library.

In [4]:
predictor.explain(test_df, row_index=35, class_id=1)

Explanation for class = Survived (PassengerId=35):


From the visualization above, we can see that his First class status (Pclass=1) and his higher-than-average Fare price (suggesting that he i is wealthy) are pushing the model higher towards predicting Survived. At the same time, the fact that he is a Male pushes the model to lower its prediction towards NOT Survived. For these reasons, this is a border-line and uncertain prediction.