Notebook

ImageNet Example All Metrics¶

This notebook shows the functionality of the various metrics included in the library.

For this purpose, we use a pre-trained PyTorch ResNet-18 model and ImageNet dataset.

Make sure to have GPUs enabled to speed up computation.
Alternatively, on OSX you could you MPS acceleration, more about MPS here.

If you have access to another hardware acceleration supported by PyTorch, add it to choose_device function. List of all supported backends is available here.

In [1]:

from IPython.display import clear_output
!pip install torch torchvision captum xmltodict quantus
clear_output()

In [1]:

import random
import numpy as np
import matplotlib.pyplot as plt
import torch
import torchvision
import quantus
from typing import Union

2022-12-14 14:23:42.073314: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

In [2]:

# Choose hardware acceleration if available
def choose_device() -> str:
    if torch.cuda.is_available():
        return "cuda:0"
    if hasattr(torch.backends, "mps"):
        if torch.backends.mps.is_available():
            return "mps"
    return "cpu"


device = torch.device(choose_device())
device

Out[2]:

device(type='mps')

1) Preliminaries¶

1.1 Load datasets¶

We have prepared a small subset of ImageNet images which can be downloaded at: https://github.com/understandable-machine-intelligence-lab/Quantus/tree/main/tutorials/assets/imagenet_samples/. Please make sure to download the contents of the folder, that are, the inputs x_batch.pt, outputs y_batch.pt and segmentation masks s_batch.pt. (A description for how to download the full dataset can be found here: https://image-net.org/download.php.)

If you use Google Colab, you might want to add the following:

# Mount Google Drive.
from google.colab import drive
drive.mount('/content/drive', force_remount=True)

In [3]:

# Adjust this path.
path_to_files = "./assets/imagenet_samples"

# Load test data and make loaders.
x_batch = torch.load(f'{path_to_files}/x_batch.pt')
y_batch = torch.load(f'{path_to_files}/y_batch.pt')
s_batch = torch.load(f'{path_to_files}/s_batch.pt')
x_batch, s_batch, y_batch = x_batch.to(device), s_batch.to(device), y_batch.to(device)
print(f"{len(x_batch)} matches found.")

17 matches found.

In [4]:

def normalize_image(arr: Union[np.ndarray, torch.Tensor]) -> np.ndarray:
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    if isinstance(arr, torch.Tensor):
        arr_copy = arr.clone().cpu().numpy()
    else:
        arr_copy = arr.copy()

    arr_copy = quantus.normalise_func.denormalise(arr_copy, mean=mean, std=std)
    arr_copy  = np.moveaxis(arr_copy, 0, -1)
    arr_copy = (arr_copy * 255.).astype(np.uint8)
    return arr_copy

# Plot some inputs!
nr_images = 5
fig, axes = plt.subplots(nrows=1, ncols=nr_images, figsize=(nr_images*3, int(nr_images*2/3)))
for i in range(nr_images):
    axes[i].imshow(normalize_image(x_batch[i]), vmin=0.0, vmax=1.0, cmap="gray")
    axes[i].title.set_text(f"ImageNet class - {y_batch[i].item()}")
    axes[i].axis("off")
plt.show()

1.2 Load a pre-trained model¶

In this example we load a pre-trained ResNet-18 model but it goes without saying that any model works.

In [5]:

# Load pre-trained ResNet18 model.
model = torchvision.models.resnet18(pretrained=True)
model = model.to(device)

/Users/artemsereda/anaconda3/envs/quantus/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/Users/artemsereda/anaconda3/envs/quantus/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)

1.3 Generate attribution-based explanations¶

There exist multiple ways to generate explanations for neural network models e.g., using captum or innvestigate libraries. In this example, we rely on the quantus.explain functionality (a simple wrapper around captum) however use whatever approach or library you'd like to create your explanations.

In [6]:

# Generate Integrated Gradients attributions of the first batch of the test set.
a_batch = quantus.explain(model, x_batch, y_batch, method="IntegratedGradients")

# Save x_batch and y_batch as numpy arrays that will be used to call metric instances.
x_batch, s_batch, y_batch = x_batch.cpu().numpy(), s_batch.cpu().numpy(), y_batch.cpu().numpy()

# Make sure s_batch is of correct shape.
s_batch = s_batch.reshape(len(x_batch), 1, 224, 224)

# Quick assert.
assert [isinstance(obj, np.ndarray) for obj in [x_batch, y_batch, s_batch, a_batch]]

Visulise attributions given model and pairs of input-output.

In [7]:

index = random.randint(0, len(x_batch)-1)

# Plot examplary explanations!
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(8, 5))
axes[0].imshow(normalize_image(x_batch[index]), vmin=0.0, vmax=1.0)
axes[0].title.set_text(f"ImageNet class {y_batch[index].item()}")
exp = axes[1].imshow(a_batch[index].reshape(224, 224), cmap="seismic")
fig.colorbar(exp, fraction=0.03, pad=0.05); 
axes[0].axis("off"); axes[1].axis("off"); plt.show();

2) Evaluation using `Quantus`¶

As a preliminary step to evaluating the explanations using all the available metrics in the library, we will first view what categories and metrics we have available.

In [8]:

# Let's view what metrics categories are available.
quantus.helpers.constants.available_categories()

Out[8]:

['Faithfulness',
 'Robustness',
 'Localisation',
 'Complexity',
 'Randomisation',
 'Axiomatic']

In [9]:

# In each category, let's view the available metrics.
quantus.helpers.constants.available_metrics()

Out[9]:

{'Faithfulness': ['Faithfulness Correlation',
  'Faithfulness Estimate',
  'Pixel-Flipping',
  'Region Segmentation',
  'Monotonicity-Arya',
  'Monotonicity-Nguyen',
  'Selectivity',
  'SensitivityN',
  'IROF',
  'ROAD',
  'Infidelity',
  'Sufficiency'],
 'Robustness': ['Continuity Test',
  'Local Lipschitz Estimate',
  'Max-Sensitivity',
  'Avg-Sensitivity',
  'Consistency',
  'Relative Input Stability',
  'Relative Output Stability',
  'Relative Representation Stability'],
 'Localisation': ['Pointing Game',
  'Top-K Intersection',
  'Relevance Mass Accuracy',
  'Relevance Rank Accuracy',
  'Attribution Localisation ',
  'AUC',
  'Focus'],
 'Complexity': ['Sparseness', 'Complexity', 'Effective Complexity'],
 'Randomisation': ['Model Parameter Randomisation', 'Random Logit'],
 'Axiomatic': ['Completeness', 'NonSensitivity', 'InputInvariance']}

2.1 Faithfulness metrics¶

Score explanations using the following faithfulness metrics:

Faithfulness Correlation (Bhatt et al., 2020)
Faithfulness Estimate (Alvarez-Melis et al., 2018, 2019)
Pixel-Flipping (Bach et al., 2015)
Region Segmentation (Samek et al., 2015)
Monotonicity-Arya (Arya et al., 2019)
Monotonicity-Nguyen (Nguyen et al., 2019)
Selectivity (Montavan et al., 2018)
SensitivityN (Ancona et al., 2018)
IROF (Rieger et al., 2020)
Infidelity (Yeh et al., 2019)
ROAD (Rong, Leehmann et al., 2022)
Sufficiency (Dasgupta et al., 2022)

In [ ]:

# Return faithfulness correlation scores in an one-liner - by calling the metric instance.
quantus.FaithfulnessCorrelation(
    nr_runs=100,  
    subset_size=224,  
    perturb_baseline="black",
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
    similarity_func=quantus.similarity_func.correlation_pearson,  
    abs=False,  
    return_aggregate=False,
)(model=model, 
   x_batch=x_batch, 
   y_batch=y_batch,
   a_batch=a_batch,
   device=device)

Warnings and information:
 (1) The Faithfulness Correlation metric is likely to be sensitive to the choice of baseline value 'perturb_baseline', size of subset |S| 'subset_size' and the number of runs (for each input and explanation pair) 'nr_runs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Bhatt, Umang, Adrian Weller, and José MF Moura. 'Evaluating and aggregating feature-based model explanations.' arXiv preprint arXiv:2005.00631 (2020).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[-0.03852115130120209,
 0.05681358501733275,
 -0.023240354085029294,
 -0.007335645392576588,
 0.07114167908973591,
 0.10796982893359308,
 -0.13866391825714072,
 -0.023036546248715033,
 0.01251575207587471,
 0.05595064896365906,
 0.00884666287591645,
 0.009519672411087364,
 0.22290830979836326,
 -0.1186819643123575,
 -0.09198627498464948,
 -0.025075883597024583,
 -0.04287550840012317,
 -0.2242524663613466,
 -0.20020833827481757,
 -0.006196208583163412]

In [ ]:

# Return faithfulness estimate scores in an one-liner - by calling the metric instance.
quantus.FaithfulnessEstimate(
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
    similarity_func=quantus.similarity_func.correlation_pearson,
    features_in_step=224,  
    perturb_baseline="black",
)(model=model, 
   x_batch=x_batch, 
   y_batch=y_batch,
   a_batch=a_batch,
   device=device,)

Warnings and information:
 (1) The Faithfulness Estimate metric is likely to be sensitive to the choice of baseline value 'perturb_baseline' and similarity function 'similarity_func'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Alvarez-Melis, David, and Tommi S. Jaakkola. 'Towards robust interpretability with self-explaining neural networks.' arXiv preprint arXiv:1806.07538 (2018).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[-0.15454811443663005,
 -0.30933664258003946,
 -0.15927610181143093,
 -0.6879649617539156,
 0.08609686619002017,
 -0.20238983957329543,
 -0.3439131733828681,
 -0.3528094861960702,
 -0.07530687223381272,
 -0.3005613698366527,
 -0.5438176985228981,
 -0.09018626782454381,
 -0.0684177220189427,
 -0.03195967660623574,
 -0.3405809252387466,
 -0.42783366828762304,
 -0.36181560432334536,
 -0.5092673365063154,
 -0.023251215571104628,
 -0.0019366275390666939]

In [ ]:

# Return monotonicity scores in an one-liner - by calling the metric instance.
quantus.Monotonicity(
    features_in_step=224,
    perturb_baseline="black",
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
)(model=model, 
   x_batch=x_batch, 
   y_batch=y_batch,
   a_batch=a_batch,
   device=device,)

Warnings and information:
 (1) The Monotonicity metric is likely to be sensitive to the choice of baseline value 'perturb_baseline', also, the monotonicity constraint between your given model and explanation method should be assessed. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Arya, Vijay, et al. 'One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques.' arXiv preprint arXiv:1909.03012 (2019).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True]

In [ ]:

# Return monotonicity scores in an one-liner - by calling the metric instance.
quantus.MonotonicityCorrelation(
    nr_samples=10,
    features_in_step=3136,
    perturb_baseline="uniform",
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
    similarity_func=quantus.similarity_func.correlation_spearman,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=a_batch,
   device=device,)

Warnings and information:
 (1) The Monotonicity Correlation metric is likely to be sensitive to the choice of baseline value 'perturb_baseline', threshold value 'eps' and number of samples to iterate over 'nr_samples'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Nguyen, An-phi, and María Rodríguez Martínez. 'On quantitative aspects of model interpretability.' arXiv preprint arXiv:2007.07584 (2020).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.5411764705882353,
 0.85,
 -0.01764705882352941,
 -0.4264705882352941,
 -0.10588235294117648,
 0.1411764705882353,
 0.7764705882352941,
 0.8558823529411764,
 0.42352941176470593,
 0.9205882352941176,
 -0.1264705882352941,
 0.03529411764705882,
 -0.4764705882352941,
 0.4911764705882353,
 0.3941176470588235,
 0.23823529411764705,
 0.6235294117647059,
 -0.27058823529411763,
 0.3852941176470588,
 -0.6588235294117648]

In [ ]:

# Create the pixel-flipping experiment.
pixel_flipping = quantus.PixelFlipping(
    features_in_step=224,
    perturb_baseline="black",
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
)
    
# Call the metric instance to produce scores.
scores = pixel_flipping(model=model,
                        x_batch=x_batch,
                        y_batch=y_batch,
                        a_batch=a_batch,
                        device=device,)

# Plot example!
pixel_flipping.plot(y_batch=y_batch, scores=scores)

Warnings and information:
 (1) The Pixel Flipping metric is likely to be sensitive to the choice of baseline value 'perturb_baseline'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Bach, Sebastian, et al. 'On pixel-wise explanations for non-linear classifier decisions by layer - wise relevance propagation.' PloS one 10.7 (2015) e0130140.
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

In [ ]:

# Create the region perturbation experiment.
region_perturb = quantus.RegionPerturbation(
    patch_size=14,
    regions_evaluation=10,
    perturb_baseline="uniform",  
    normalise=True,
)
    
# Call the metric instance to produce scores.
results = {method: region_perturb(model=model, 
                                  x_batch=x_batch,
                                  y_batch=y_batch,
                                  a_batch=None,
                                  device=device,
                                  explain_func=quantus.explain, 
                                  explain_func_kwargs={"method": method}) for method in ["Saliency", "IntegratedGradients"]}

# Plot example!
#region_perturb.plot(results=results) 
results

Out[ ]:

{'Saliency': [[-1.2341888622202646e-10,
   -3.0112020099615222e-09,
   -8.062674583531049e-09,
   -1.0355286989005563e-07,
   -2.477114465215813e-07,
   -5.810901116398703e-07,
   -1.4528842116612317e-06,
   -2.4284706428567837e-06,
   -1.587000233060909e-06,
   -1.4986879564582445e-06],
  [-1.4817697291391596e-08,
   1.1729239801638869e-08,
   1.6422561088802468e-09,
   1.0249014970042936e-08,
   -3.736788656283352e-09,
   -8.82089778997397e-09,
   -2.9589486416625732e-08,
   -4.8234912952693776e-08,
   -9.290443969689477e-08,
   -2.1076973055755843e-07],
  [-3.345280128996819e-05,
   -8.964914741227403e-06,
   3.043276592507027e-06,
   -8.38581399875693e-07,
   6.1557475419249386e-06,
   1.379104196530534e-05,
   1.1652271496132016e-05,
   9.352199413115159e-06,
   1.2568179045047145e-05,
   1.386103485856438e-05],
  [1.623800471861614e-07,
   2.393142040091334e-07,
   1.9211461221857462e-07,
   8.462849905299663e-07,
   7.022431987024902e-07,
   4.0584370708529605e-07,
   3.5472737636155216e-07,
   7.008790703366685e-07,
   6.432670716094435e-07,
   2.621168277983088e-07],
  [1.5202567205108153e-08,
   2.189906389560292e-08,
   2.5420725968672286e-08,
   2.7108789213414752e-08,
   2.7694198490024746e-08,
   2.7769138544186944e-08,
   2.8130892837374688e-08,
   2.8017513975520103e-08,
   2.1215900147808497e-08,
   2.2417790734863274e-08],
  [1.1709211378274631e-08,
   4.6065999015354464e-08,
   5.391957103029199e-08,
   2.185197445214726e-08,
   -4.413472254327644e-09,
   -9.255960975451671e-08,
   -9.811567025508339e-08,
   -1.999570287125607e-08,
   -1.0759206503507812e-07,
   -2.0749887852389293e-07],
  [-1.243091729812712e-13,
   -1.421220725741218e-13,
   -2.0847164055801115e-13,
   -2.7807525162723556e-13,
   -3.6514310997569355e-12,
   -5.520396585603077e-12,
   -6.222214095959883e-12,
   -7.58535768376721e-11,
   -9.31548062429663e-12,
   -1.2205517792209658e-11],
  [4.0009581425692886e-07,
   9.61412297328934e-07,
   7.239392289193347e-06,
   1.5583264030283317e-05,
   1.6597834473941475e-05,
   1.609719492989825e-05,
   1.6530524590052664e-05,
   1.528307984699495e-05,
   1.0935371392406523e-05,
   1.2461968253774103e-05],
  [-6.028410126646122e-07,
   -1.382592490983825e-05,
   -1.3519571623987758e-05,
   -4.775741402340827e-06,
   -4.868818180625567e-06,
   -1.202579660741776e-05,
   -2.541545400447376e-05,
   -7.054733687539283e-05,
   -6.768445128102485e-05,
   -7.504683271974955e-05],
  [-5.939584934999687e-10,
   -1.4647498214337418e-09,
   -5.043670980864334e-09,
   -9.727234739909107e-09,
   -7.988737393826995e-09,
   -7.094577092470189e-09,
   -1.5135222575235474e-08,
   -1.3647283914064445e-08,
   -8.943127632887737e-08,
   -4.427718786059742e-08],
  [2.718303804272182e-09,
   1.4850209950623139e-09,
   1.1007781353100654e-09,
   -3.0948692497645425e-09,
   -9.47898648462342e-10,
   -1.5025288568892847e-08,
   -2.0928482946658278e-08,
   -2.5486486698866884e-08,
   -3.650177493597084e-08,
   -6.417557418814113e-08],
  [9.424490064768344e-15,
   1.2723032873020353e-14,
   1.3978843073586631e-14,
   6.2267882229528546e-15,
   1.0659025338798436e-14,
   1.4087411521600951e-15,
   -1.10884913718471e-14,
   7.698821370997685e-15,
   6.496930746754774e-15,
   8.666358307526066e-15],
  [-3.838991460725083e-06,
   -1.335703450422443e-05,
   -4.512268674261577e-05,
   -5.5609247965549e-05,
   -5.1427143944238196e-05,
   -0.00010299511154698848,
   -0.00010680632362891629,
   -0.00013460912555274263,
   -8.9061638163912e-05,
   -9.725510130920156e-05],
  [-1.3976881457766765e-10,
   -1.787608047698619e-10,
   -2.6347790546554783e-09,
   -2.6123556295248873e-07,
   -3.8642941290606414e-07,
   -3.2601901266832156e-07,
   -4.419218058090979e-07,
   -5.636210076362391e-07,
   -5.878115771837389e-07,
   -3.4622852140772653e-06],
  [-3.0018847650692404e-15,
   -7.737889918656843e-15,
   -7.462965046834726e-15,
   -9.32995610565708e-15,
   -1.2282416498254033e-14,
   -1.4479587776179692e-14,
   -1.3541912139173815e-14,
   1.2403337439496399e-14,
   1.2737503583891245e-14,
   -4.720804300316177e-15],
  [2.0579193460434908e-08,
   3.0055938182727004e-08,
   3.5706518630718165e-08,
   3.15482306945114e-08,
   3.794387382072273e-08,
   3.383000191092833e-08,
   1.2413796213195383e-08,
   2.7932962609611423e-09,
   -9.718164051264466e-10,
   -1.2977793062418641e-08],
  [-0.00034493525163270533,
   -0.01968873749137856,
   -0.04062196324230172,
   -0.2144915171957109,
   -0.037035283545264974,
   -0.03965739990235306,
   -0.023978960613021627,
   -0.010467168787727132,
   -0.00594050288782455,
   -0.0049536632432136685],
  [-6.779620059793601e-10,
   -7.817257813291789e-10,
   -3.0185691585193908e-09,
   -9.728943414777369e-09,
   -1.157863359357858e-08,
   -1.1360865567744405e-08,
   -3.700488936109192e-08,
   -4.152401932022709e-08,
   -1.600530151468771e-08,
   -2.738714453831559e-08],
  [-1.6145799974776054e-10,
   -2.1613839884526698e-10,
   -6.397778140598831e-11,
   -1.2698661566723501e-11,
   -1.5375024864372477e-10,
   -3.661818487321611e-10,
   -4.4925700659614876e-10,
   -7.564249088209962e-10,
   -9.856012533893121e-10,
   -1.1262705310044474e-08],
  [-1.8063087736663874e-06,
   -5.625906851491891e-09,
   -1.2983050510229077e-06,
   -1.879511728475336e-06,
   -6.1119699239498004e-06,
   -8.924896974349394e-06,
   -3.509420366754057e-06,
   -3.3141031963168643e-06,
   -5.398571829573484e-06,
   -4.376894594315672e-06],
  [-1.7719810330163455e-07,
   -9.792000810193713e-08,
   -3.156678758386988e-07,
   -1.2271993341528287e-06,
   -1.5684362892898207e-06,
   -1.7103031382248446e-06,
   -3.6236492064745107e-06,
   -4.04168980594477e-06,
   -4.66166915202848e-06,
   -6.13632067825165e-06]],
 'IntegratedGradients': [[-9.317657201357576e-10,
   -4.5785525371044855e-10,
   -2.436711554754112e-09,
   -5.935369140619429e-09,
   -3.951483196207306e-09,
   -2.1848172715444036e-09,
   -4.923298269066123e-09,
   -8.281486219008372e-09,
   -6.144357522774868e-09,
   -1.1035125041525262e-07],
  [-7.098130083704746e-09,
   -2.0285810364839563e-08,
   -1.2258657378083626e-07,
   -1.2958006223584562e-07,
   -1.4018913319091553e-07,
   -1.0405260653101323e-07,
   -5.50161267653948e-08,
   -7.340426222413043e-08,
   -9.652650945213281e-08,
   -2.0523312471709687e-07],
  [8.485973012284376e-06,
   1.1541943422344048e-05,
   1.540021457913099e-05,
   1.3649886568600778e-05,
   1.3381097232922912e-05,
   1.2886566764791496e-05,
   1.7224785551661626e-05,
   2.0284539459680673e-05,
   1.671507015998941e-05,
   1.3416673027677462e-05],
  [3.0281682938948506e-07,
   5.207595563661016e-07,
   5.545939529838506e-08,
   7.740482033113949e-09,
   8.37979882817308e-07,
   6.386369477695553e-07,
   6.938935825928638e-07,
   9.075841091998882e-07,
   9.522375705728336e-07,
   9.850918445408752e-07],
  [-5.643061840032715e-08,
   2.0490699803588086e-08,
   2.3386843572836824e-08,
   -5.0694243469706635e-08,
   -1.1352463857861039e-07,
   -2.4100841145013874e-07,
   -1.2722241038432003e-07,
   -1.0526942517685711e-07,
   -7.889131481420009e-08,
   -1.1698390167680373e-07],
  [-4.8467093449744425e-08,
   -1.3418223687722275e-07,
   -9.318113569634079e-08,
   -5.544591630268769e-08,
   -3.4094327361344767e-09,
   -1.0804833294741911e-07,
   -1.0113297577163394e-07,
   -1.5859730950751327e-07,
   -1.5390960328431902e-07,
   -2.2219611395257743e-07],
  [-2.0177601451655536e-13,
   -2.673776456317556e-13,
   -1.5753801800404144e-13,
   -2.468544032380171e-14,
   -1.4544734774218915e-14,
   -8.244841983619244e-13,
   -2.735645206457943e-12,
   -2.54390240968759e-11,
   -4.3829089608950905e-11,
   -5.858005394784524e-11],
  [7.164589987951331e-06,
   1.3736479559156578e-05,
   1.7516687876195647e-05,
   2.114385824825149e-05,
   2.1129860215296503e-05,
   2.163399949495215e-05,
   2.0163162844255567e-05,
   1.8934125364467036e-05,
   1.683904474703013e-05,
   1.9100825738860294e-05],
  [-3.203030729537204e-07,
   -2.2249302858767805e-05,
   -3.2843539159443935e-05,
   -9.109165803522501e-05,
   -0.00024050159954924766,
   -0.0002235285221132699,
   -0.00025548454250667874,
   -0.00020092401355498168,
   -0.00029831577667494,
   -0.00022660360373549793],
  [-3.8357289566803843e-10,
   -2.5044779028426944e-09,
   -8.297479758834214e-09,
   -5.638247813077868e-09,
   -3.929685188364118e-09,
   -3.229600309406777e-09,
   -8.377189331110202e-09,
   -1.008164013382995e-08,
   -1.4032909667616167e-08,
   -3.606077009221664e-08],
  [2.9117774857922996e-10,
   5.779936529393126e-10,
   -1.2612685118540412e-08,
   -1.6884860620791642e-08,
   -4.1096247294092336e-08,
   -8.947612517573589e-08,
   -1.0418897122832504e-07,
   -1.5611024029027476e-07,
   -2.4922038566188576e-07,
   -1.263786311156423e-07],
  [2.852219125487089e-15,
   3.603894797605395e-15,
   5.0259513076963275e-15,
   1.5011721825732103e-14,
   1.997694341220914e-14,
   2.1144239093303816e-14,
   2.236077788182007e-14,
   2.4792057843180267e-14,
   2.5470752415409313e-14,
   2.4732557590283915e-14],
  [5.224922006163979e-08,
   -3.9934320739121176e-07,
   -4.1124246763502015e-07,
   -4.4969431201025145e-07,
   -5.784211225545732e-07,
   -2.736655346780026e-07,
   2.284889433212811e-08,
   -1.0907717751251766e-07,
   -3.4400034110149136e-07,
   -3.2011456596592325e-07],
  [1.4833594422225538e-12,
   -4.12141223926632e-11,
   -1.0278497497573369e-10,
   -5.937263944377769e-10,
   -8.209416410553949e-10,
   -1.8118180220016988e-09,
   -5.43915445322396e-09,
   -4.855321915353539e-09,
   -6.19696737674813e-09,
   -1.1202586407133008e-08],
  [1.1098419868538342e-14,
   1.080158860711786e-14,
   8.559729564960959e-15,
   -3.698100209577077e-13,
   -2.4118892172635305e-13,
   -5.885901473514938e-13,
   -2.369937569351695e-12,
   -3.5756202584555103e-12,
   -4.219420375839141e-12,
   -6.470238831068594e-12],
  [-8.25772232815325e-08,
   -3.080766930452228e-08,
   2.7930049384394806e-09,
   -2.405335663979713e-08,
   -1.169028962522134e-08,
   -1.1776023711718153e-08,
   6.884697256737127e-10,
   -2.021366896087784e-08,
   -3.793299896415192e-08,
   -5.6607362353133794e-08],
  [-0.0034659217635635287,
   -0.034284655848750845,
   -0.0463096582388971,
   -0.05455023134709336,
   -0.02867096188128926,
   -0.024853511626133695,
   -0.025764224032172933,
   -0.021107097069034353,
   -0.01372556752176024,
   -0.010947444388875738],
  [-5.023896576528131e-11,
   -1.617583983426485e-10,
   -4.2019281110139417e-10,
   -3.4374157686922757e-10,
   -1.9362493691721383e-09,
   -1.8923600325626566e-09,
   -3.2821576428920807e-09,
   -5.75099433353099e-09,
   -6.2608034173683436e-09,
   -6.859259812919127e-09],
  [1.8041662087908694e-10,
   1.7969435314735271e-10,
   1.772613496958675e-10,
   1.8602176916904245e-10,
   1.5477756576398605e-10,
   -4.5158714268023203e-10,
   2.9106855814475807e-12,
   -2.182606734235648e-10,
   -1.0290226309939143e-09,
   -7.745927094182647e-10],
  [-1.8836753952200525e-06,
   -4.184284534858307e-06,
   -5.256827989796875e-06,
   -9.738034350448288e-06,
   -8.635539415990934e-06,
   -4.890474428975722e-06,
   -2.501486051187385e-06,
   -4.947638444718905e-06,
   -5.797092853754293e-06,
   -2.5188125619024504e-05],
  [-5.528420388145605e-08,
   -3.281120370957069e-07,
   -4.4292119127931073e-07,
   -4.0619761421112344e-07,
   -2.671005177035113e-07,
   -1.1301915492367698e-07,
   -8.331818435181049e-08,
   -2.190424766013166e-07,
   -4.534010145107459e-07,
   -2.040754168319836e-06]]}

In [ ]:

# Create the selectivity experiment.
selectivity = quantus.Selectivity(
    patch_size=56,
    perturb_baseline="black",  
)
    
# Call the metric instance to produce scores.
results = {method: selectivity(model=model, 
                               x_batch=x_batch,
                               y_batch=y_batch,
                               a_batch=None,
                               device=device,
                               explain_func=quantus.explain, 
                               explain_func_kwargs={"method": method}) for method in ["Saliency", "IntegratedGradients"]}

# Plot example!
selectivity.plot(results=results)
results

In [ ]:

# Create the sensitivity-n experiment.
sensitivity_n = quantus.SensitivityN(
    features_in_step=224,
    n_max_percentage=0.8,
    similarity_func=quantus.similarity_func.correlation_pearson,
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
    perturb_baseline="uniform",  
    return_aggregate=False,
)
# Call the metric instance to produce scores.
results = sensitivity_n(model=model, 
                        x_batch=x_batch,
                        y_batch=y_batch,
                        a_batch=None,
                        device=device,
                        explain_func=quantus.explain, 
                        explain_func_kwargs={"method": "IntegratedGradients"})
                        

# Plot example!
sensitivity_n.plot(results=results)

WARNING:matplotlib.legend:No handles with labels found to put in legend.

In [ ]:

# Return IFOF scores in an one-liner - by calling the metric instance.
scores = {method: quantus.IROF(
    segmentation_method="slic",
    perturb_baseline="mean",
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
    return_aggregate=False,
)(model=model,
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   device=device,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": method}) for method in ["Saliency", "IntegratedGradients"]}

scores # felzenswalb

Warnings and information:
 (1) The IROF metric is likely to be sensitive to the choice of baseline value 'perturb_baseline' and the method to segment the image 'segmentation_method' (including all its associated hyperparameters), also, IROF only works with image data. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Rieger, Laura, and Lars Kai Hansen. 'Irof: a low resource evaluation metric for explanation methods.' arXiv preprint arXiv:2003.08747 (2020).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/usr/local/lib/python3.7/dist-packages/captum/_utils/gradient.py:59: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  "required_grads has been set automatically." % index

Warnings and information:
 (1) The IROF metric is likely to be sensitive to the choice of baseline value 'perturb_baseline' and the method to segment the image 'segmentation_method' (including all its associated hyperparameters), also, IROF only works with image data. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Rieger, Laura, and Lars Kai Hansen. 'Irof: a low resource evaluation metric for explanation methods.' arXiv preprint arXiv:2003.08747 (2020).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

{'Saliency': [-2450.2173034978723,
  -16.374110163577612,
  -28.863229005189176,
  3.4739964811507207,
  -11461.351480565618,
  -4.350771566117068,
  -129.88071852734805,
  -2097.1381447828126,
  -35.36820374450471,
  -156.78475600201338,
  -454.90247938336296,
  -399.9201570419748,
  -92.66307640776316,
  -4.130042044001463,
  -102.80285010028393,
  -825076.2370666246,
  -3.325261169068547,
  -100.30074095804162],
 'IntegratedGradients': [-2974.3905808129384,
  -14.957913740375133,
  -31.597699001362372,
  3.399904352201858,
  -11461.002755258021,
  -4.255324475896472,
  -134.6151674311129,
  -2035.673341709858,
  -27.785719333962703,
  -170.03504515459696,
  -455.42920531676276,
  -401.69614527525783,
  -93.4874045457903,
  -3.977154934199362,
  -102.60959033992808,
  -412540.1193502592,
  -2.043219271533964,
  -101.26936100151497]}

In [ ]:

# Return Infidelity scores in an one-liner - by calling the metric instance.
quantus.Infidelity(
    perturb_baseline="uniform",
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
    n_perturb_samples=5,
    perturb_patch_sizes=[56], 
    display_progressbar=True,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   device=device,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "IntegratedGradients"})

Warnings and information:
 (1) The Infidelity metric is likely to be sensitive to the choice of baseline value 'perturb_baseline', perturbation function 'perturb_func',number of perturbed samples 'n_perturb_samples', the loss function 'loss_func' aggregation boolean 'aggregate'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Chih-Kuan, Yeh, et al. 'On the (In)fidelity and Sensitivity of Explanations'arXiv:1901.09392 (2019).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Evaluating Infidelity:   0%|          | 0/19 [00:00<?, ?it/s]

Out[ ]:

[1796483.4443198796,
 12416903.33174478,
 14158807.670184037,
 363631.479612375,
 159359.25294499582,
 1113157.2950333632,
 9323674.929579282,
 1388681.5510537962,
 8303985.413801474,
 1070049.3484908913,
 907060.4465064537,
 8370813.388146776,
 3530213.983484649,
 4468296.148278534,
 16211488.48945864,
 30363664.373251207,
 3374637.8717216603,
 58902321.66314578,
 626065.2206585895]

In [ ]:

# Return ROAD scores in an one-liner - by calling the metric instance.
quantus.ROAD(
    noise=0.01,
    perturb_func=quantus.perturb_func.noisy_linear_imputation,
    percentages=list(range(1, 50, 2)),
    display_progressbar=False,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=a_batch,
   device=device,)

Warnings and information:
 (1) The ROAD metric is likely to be sensitive to the choice of baseline value 'perturb_baseline', perturbation function 'perturb_func',percentage of pixels k removed per iteration 'percentage_in_step'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Rong, Leemann, et al. 'Evaluating Feature Attribution: An Information-Theoretic Perspective.' arXiv:2202.00449 (2022).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

{1: 0.0,
 3: 0.0,
 5: 0.0,
 7: 0.0,
 9: 0.0,
 11: 0.0,
 13: 0.0,
 15: 0.0,
 17: 0.0,
 19: 0.0,
 21: 0.0,
 23: 0.0,
 25: 0.0,
 27: 0.0,
 29: 0.0,
 31: 0.0,
 33: 0.0,
 35: 0.0,
 37: 0.0,
 39: 0.0,
 41: 0.0,
 43: 0.0,
 45: 0.0,
 47: 0.0,
 49: 0.0}

In [ ]:

# Return Sufficiency scores in an one-liner - by calling the metric instance.
quantus.Sufficiency(
    threshold=0.6,
    return_aggregate=False,
)(model=model,
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=a_batch,
   device=device,)

Warnings and information:
 (1) The Sufficiency metric is likely to be sensitive to the choice of distance threshold that determines if images share an attribute 'threshold', distance function 'distance_func'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. 'Framework for Evaluating Faithfulness of Explanations.' arXiv preprint arXiv:2202.00734 (2022).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.0,
 0.0,
 0.0,
 0.0625,
 0,
 0.0,
 0.0,
 0.058823529411764705,
 0.0,
 0,
 0.1111111111111111,
 0.06666666666666667,
 0.0,
 0.09090909090909091,
 0.0,
 0.0,
 0.058823529411764705,
 0.058823529411764705,
 0,
 0.0,
 0.058823529411764705]

2.2 Robustness metrics¶

Score explanations using the following robustness metrics:

Local Lipschitz Estimate (Alvarez-Melis et al., 2018; 2019)
Max-Sensitivity (Yeh at el., 2019)
Avg-Sensitivity (Yeh at el., 2019)
Continuity (Montavan et al., 2017)
Input Independence Rate (Yang et al., 2019)
Consistency (Dasgupta et al., 2022)
Relative Input Stability (Chirag Agarwal, et. al.)
Relative Output Stability (Chirag Agarwal, et. al.)
Relative Representation Stability (Chirag Agarwal, et. al.)

In [ ]:

# Return local lipschitzEstimate estimate scores in an one-liner - by calling the metric instance.
quantus.LocalLipschitzEstimate(
    nr_samples=10,
    perturb_std=0.2,
    perturb_mean=0.0,
    norm_numerator=quantus.similarity_func.distance_euclidean,
    norm_denominator=quantus.similarity_func.distance_euclidean,    
    perturb_func=quantus.perturb_func.gaussian_noise,
    similarity_func=quantus.similarity_func.lipschitz_constant,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,    
   device=device,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"})

Warnings and information:
 (1) The Local Lipschitz Estimate metric is likely to be sensitive to the choice of amount of noise added 'perturb_std', the number of samples iterated over 'nr_samples', the function to perturb the input 'perturb_func', the similarity metric 'similarity_func' as well as norm calculations on the numerator and denominator of the lipschitz equation i.e., 'norm_numerator' and 'norm_denominator'.  
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Alvarez-Melis, David, and Tommi S. Jaakkola. 'On the robustness of interpretability methods.' arXiv preprint arXiv:1806.08049 (2018). and Alvarez-Melis, David, and Tommi S. Jaakkola. 'Towards robust interpretability with self-explaining neural networks.' arXiv preprint arXiv:1806.07538 (2018).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[57.52864385277926,
 38.43395822304614,
 48.92632463386785,
 52.40489011736009,
 58.048339504677806,
 55.3765551198521,
 68.49756565151691,
 47.63754124498776,
 52.923116734879784,
 40.284309498958436,
 68.23067270175258,
 79.5426162423288,
 49.8548827037425,
 46.968453472503185,
 50.83593110524175,
 58.43350321279524,
 32.325763130076226,
 52.06859743343769,
 42.800074602238816,
 34.26269973908186,
 70.26931166247635]

In [ ]:

# Return max sensitivity scores in an one-liner - by calling the metric instance.
quantus.MaxSensitivity(
    nr_samples=10,
    lower_bound=0.2,
    norm_numerator=quantus.norm_func.fro_norm,
    norm_denominator=quantus.norm_func.fro_norm,
    perturb_func=quantus.perturb_func.uniform_noise,
    similarity_func=quantus.similarity_func.difference,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   device=device,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Gradient"})

Warnings and information:
 (1) The Max Sensitivity metric is likely to be sensitive to the choice of amount of noise added 'lower_bound' and 'upper_bound', the number of samples iterated over 'nr_samples', the function to perturb the input 'perturb_func', the similarity metric 'similarity_func' as well as norm calculations on the numerator and denominator of the sensitivity equation i.e., 'norm_numerator' and 'norm_denominator'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Yeh, Chih-Kuan, et al. 'On the (in) fidelity and sensitivity for explanations.' arXiv preprint arXiv:1901.09392 (2019).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.014146048,
 0.011706328,
 0.016862085,
 0.016317796,
 0.018953297,
 0.015230152,
 0.013311529,
 0.012152915,
 0.045168135,
 0.010675438,
 0.011723739,
 0.009206712,
 0.011673499,
 0.026382558,
 0.01727627,
 0.020674637,
 0.0151237175,
 0.032591835,
 0.008827085]

In [ ]:

# Return avg sensitivity scores in an one-liner - by calling the metric instance.
quantus.AvgSensitivity(
    nr_samples=10,
    lower_bound=0.2,
    norm_numerator=quantus.norm_func.fro_norm,
    norm_denominator=quantus.norm_func.fro_norm,
    perturb_func=quantus.perturb_func.uniform_noise,
    similarity_func=quantus.similarity_func.difference,
)(model=model, 
  x_batch=x_batch,
  y_batch=y_batch,
  a_batch=None,
  explain_func=quantus.explain, 
  explain_func_kwargs={"method": "Saliency"}, 
  device=device)

Warnings and information:
 (1) The Avg Sensitivity metric is likely to be sensitive to the choice of amount of noise added 'lower_bound' and 'upper_bound', the number of samples iterated over 'nr_samples', the function to perturb the input 'perturb_func', the similarity metric 'similarity_func' as well as norm calculations on the numerator and denominator of the sensitivity equation i.e., 'norm_numerator' and 'norm_denominator'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Yeh, Chih-Kuan, et al. 'On the (in) fidelity and sensitivity for explanations.' arXiv preprint arXiv:1901.09392 (2019).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.009212605655193329,
 0.007392846047878265,
 0.010924143716692924,
 0.010722446255385876,
 0.01237289048731327,
 0.009490462020039558,
 0.008615540340542793,
 0.009117192588746548,
 0.028994327411055565,
 0.007189278490841389,
 0.007913483306765556,
 0.006138150580227375,
 0.007428924553096294,
 0.017067646607756615,
 0.0115343127399683,
 0.01353415660560131,
 0.00984276831150055,
 0.019707508385181427,
 0.006467340048402548]

In [ ]:

# Return continuity scores in an one-liner - by calling the metric instance.
continuity_test = quantus.Continuity(
    patch_size=56,
    nr_steps=10,
    perturb_baseline="uniform",
    similarity_func=quantus.similarity_func.correlation_spearman,
)
scores = continuity_test(model=model, 
                         x_batch=x_batch,
                         y_batch=y_batch,
                         a_batch=None,
                         explain_func=quantus.explain, 
                         explain_func_kwargs={"method": "Saliency"},
                         device=device)
scores

Warnings and information:
 (1) The Continuity metric is likely to be sensitive to the choice of how many patches to split the input image to 'nr_patches', the number of steps to iterate over 'nr_steps', the value to replace the masking with 'perturb_baseline' and in what direction to translate the image 'perturb_func'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Montavon, Grégoire, Wojciech Samek, and Klaus-Robert Müller. 'Methods for interpreting and understanding deep neural networks.' Digital Signal Processing 73, 1-15 (2018.
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[{0: [273.5874387155054,
   455.2056053960696,
   326.9445907329209,
   342.9486792665557,
   426.5160544118844,
   263.75143982539885,
   383.92685332219116,
   290.0035306008067,
   363.05295708123595,
   415.1640391598921],
  1: [302.90579543611966,
   295.62067048763856,
   397.3570333789103,
   559.9538742862642,
   505.4255059901625,
   276.57958492072066,
   518.9767691520974,
   304.63772350968793,
   485.83839229214936,
   468.4251905635465],
  2: [406.21301000914536,
   430.5600434700027,
   355.8602512963116,
   358.2736078592716,
   520.7333955168724,
   283.554831141606,
   565.1769892955199,
   305.1036934246076,
   458.1406351821497,
   491.00463343109004],
  3: [329.5788051141426,
   291.3178534603212,
   342.0037309064064,
   297.40532647888176,
   398.6346804823261,
   229.73999873991124,
   404.53727704961784,
   323.40004774415866,
   369.76184817170724,
   575.8938621589914],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-1.2097055912017822,
   -0.9220386743545532,
   -0.8316829204559326,
   -1.0296118259429932,
   -0.6991629600524902,
   -1.1555286645889282,
   -0.6219760179519653,
   -0.7323520183563232,
   -1.043391227722168,
   -0.4717656970024109]},
 {0: [414.6850174397696,
   458.8128752212506,
   380.0978690444026,
   330.65754413884133,
   417.75901556410827,
   424.96391449146904,
   382.0020867921412,
   332.48235988244414,
   342.50689973589033,
   280.5749183408916],
  1: [477.6477234396152,
   538.7512440639548,
   416.9236962797586,
   434.10041219671257,
   413.4850391461514,
   395.98363698343746,
   430.94773954781704,
   415.88946350989863,
   407.41644880367676,
   291.3468050372321],
  2: [486.4654781925492,
   482.36623754445463,
   359.9380922042765,
   453.3989008574281,
   352.87961548171006,
   399.3077908733394,
   365.1301050661132,
   391.02318022958934,
   345.7633836849127,
   265.0337930644164],
  3: [366.28675630898215,
   440.917700964259,
   480.33613350475207,
   310.4148136731237,
   396.9428047067486,
   383.19229011260904,
   271.11338608979713,
   358.916311933659,
   517.0985291688703,
   218.25206147902645],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.003216157201677561,
   -0.28884240984916687,
   -0.5477739572525024,
   -0.4929467439651489,
   -0.5524523854255676,
   -0.03071552701294422,
   0.0713300034403801,
   -0.36552178859710693,
   -0.5519583821296692,
   -0.11873912066221237]},
 {0: [565.7101983150933,
   470.8451354377903,
   498.28246775991283,
   293.79199856496416,
   438.9810785672162,
   319.6328416315373,
   379.5415778965689,
   406.99174244492315,
   291.54834521678276,
   524.0550692388788],
  1: [631.6157954456285,
   450.12878728006035,
   482.6174080474302,
   609.3157598236576,
   490.2323216776131,
   478.76489397580735,
   409.6791979686823,
   431.38810750469565,
   383.9553777943365,
   517.9085912476294],
  2: [594.5562296425924,
   414.72869075328344,
   454.73242306918837,
   557.5289401537739,
   479.72480283491313,
   585.7746501229703,
   335.45958741684444,
   590.3902723807842,
   362.902007073164,
   626.2963709142059],
  3: [456.23499445803463,
   330.2850660751574,
   412.1244127252139,
   420.85491246334277,
   315.03226582868956,
   587.6426017885096,
   420.5646642050706,
   645.743737062905,
   311.53851483622566,
   594.6157937310636],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [2.517960786819458,
   2.185148239135742,
   2.5571043491363525,
   2.5357282161712646,
   2.5464820861816406,
   2.419518232345581,
   2.2540714740753174,
   2.2392330169677734,
   1.9839845895767212,
   2.3962392807006836]},
 {0: [428.95255806774367,
   464.29943378805183,
   461.9032464369666,
   230.67815538158175,
   426.6792754633352,
   505.4294127402827,
   488.9049205300398,
   541.469139682129,
   303.5245394918602,
   528.3223115953151],
  1: [372.30878607183695,
   593.0422458648682,
   450.91299134260044,
   635.9020072952844,
   472.83370002917945,
   705.1481659207493,
   428.342535420903,
   439.65832418296486,
   408.26872562337667,
   552.0903167945798],
  2: [360.1304812853923,
   544.0356028163806,
   447.48031982500106,
   450.1458997707814,
   455.9176049709786,
   501.51767874811776,
   396.651933406014,
   424.1321711852215,
   489.4324380690232,
   506.53020617645234],
  3: [415.74799215013627,
   470.34871097654104,
   431.37682150746696,
   487.3095533438027,
   541.6356074516661,
   578.7666758252308,
   487.32854068093,
   496.3270350215025,
   490.96816085185856,
   485.1545635026414],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [2.4482178688049316,
   2.507836103439331,
   2.5995895862579346,
   2.710893154144287,
   2.307528018951416,
   2.13710618019104,
   2.603586435317993,
   2.5479984283447266,
   2.4903924465179443,
   2.4746298789978027]},
 {0: [402.50214359513484,
   431.07796085625887,
   420.7369622567203,
   432.2202229225077,
   472.8755989279598,
   377.6782356845215,
   445.77030216995627,
   420.4356659450568,
   495.52728563011624,
   380.1900938539766],
  1: [464.79698015283793,
   538.7726141612511,
   382.6340006168466,
   467.7587452218868,
   441.69278127280995,
   416.7301000193693,
   393.5422766129486,
   387.14875293755904,
   542.2820619395934,
   467.18813609238714],
  2: [434.23459493997507,
   490.83225347381085,
   331.49777292762883,
   576.6374650560319,
   469.1254790809471,
   393.4829561859369,
   333.42976340441965,
   310.37435553595424,
   489.4191075505223,
   534.6090030670166],
  3: [432.5019363538595,
   348.08501855959184,
   328.36190614895895,
   479.4015111280605,
   440.61134039843455,
   329.4287241626298,
   295.6243396870559,
   366.4505392268766,
   426.77262411033735,
   376.01026777597144],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.27492308616638184,
   -0.06998560577630997,
   -0.37051522731781006,
   -0.16171549260616302,
   -0.18212352693080902,
   -0.48728513717651367,
   -0.37097790837287903,
   -0.11774283647537231,
   -0.5449292063713074,
   -0.15978802740573883]},
 {0: [380.1979755247012,
   328.78074523154646,
   284.6356462116819,
   411.61696681845933,
   495.1461123812478,
   495.6294262185693,
   369.11902499827556,
   349.8274118367117,
   428.3278079109732,
   326.6529924290953],
  1: [401.20981367724016,
   395.61591734783724,
   468.17263001808897,
   366.03752605372574,
   608.9849770937581,
   507.26853853976354,
   282.996162942145,
   372.4218942571897,
   493.22190614417195,
   382.51494143833406],
  2: [371.969972909661,
   377.41508609149605,
   444.8272484487388,
   368.3109368022997,
   611.6452283095568,
   371.2080269004218,
   283.1125865478534,
   422.97709854878485,
   442.5650317450054,
   405.67563557613175],
  3: [381.9793953304179,
   343.7312836162746,
   533.6126385023817,
   403.34393188357353,
   316.3900161727215,
   311.6675717611797,
   313.87714593391865,
   400.2248813393526,
   363.1654870172497,
   461.108315164689],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-1.9877647161483765,
   -1.7036365270614624,
   -0.7665746212005615,
   -1.9130903482437134,
   -0.7835344076156616,
   -1.1507247686386108,
   -1.2199584245681763,
   -1.8488670587539673,
   -1.5594919919967651,
   -1.7522382736206055]},
 {0: [503.9435661570169,
   543.7304392731749,
   571.8189722336829,
   403.3958586747758,
   482.07759636756964,
   485.52477137790993,
   425.02120294841006,
   538.3753820927814,
   472.4715402191505,
   517.8415425936691],
  1: [623.9131138036028,
   506.1689339405857,
   743.1645285971463,
   487.38260487793013,
   469.4964758376591,
   547.2887933976017,
   574.4632256305777,
   607.7919731670991,
   522.8662115233019,
   430.83910508593544],
  2: [561.594575649593,
   462.0180809814483,
   750.5908211134374,
   404.68761745328084,
   493.3660018662922,
   501.1025924102869,
   587.7116265008226,
   483.84871718334034,
   567.4707936285995,
   355.5678667781176],
  3: [460.9229008718394,
   424.4135105526075,
   529.8679581703618,
   371.8243444238324,
   482.0696777580306,
   406.3704572930001,
   400.3933055056259,
   619.2394692050293,
   551.1750586051494,
   361.18030748330057],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.4076906442642212,
   -0.1947755515575409,
   -0.7539382576942444,
   -0.053013257682323456,
   -0.13897870481014252,
   -0.14921793341636658,
   -0.763436496257782,
   0.02823125757277012,
   -0.009320003911852837,
   -1.1806789636611938]},
 {0: [322.9521375214681,
   421.688683077693,
   425.3828803594224,
   482.3189717163332,
   424.6959170522168,
   455.9308214908233,
   410.63486607465893,
   488.43289825413376,
   439.5011379318312,
   548.2519948389381],
  1: [403.4961816268042,
   513.3285972615704,
   441.7457961961627,
   483.13053310336545,
   451.3834473718889,
   379.7332265423611,
   476.05435832729563,
   422.17212201002985,
   447.3045447602635,
   496.38941959291697],
  2: [392.60290370415896,
   523.268286596518,
   427.6777235744521,
   724.4794667316601,
   437.85089823941234,
   432.1534223440103,
   487.76852273079567,
   405.0452447066782,
   409.97444545151666,
   464.0481817740947],
  3: [325.82507949182764,
   468.831631551031,
   277.65863275318407,
   420.5150351654738,
   309.1344026206061,
   360.0619739703834,
   305.9223566915025,
   331.03839997877367,
   419.815604488831,
   512.7856196796056],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [2.4398744106292725,
   2.483368158340454,
   2.4505093097686768,
   2.5996129512786865,
   2.4807162284851074,
   2.584489345550537,
   2.182790994644165,
   2.0949196815490723,
   2.683905601501465,
   2.4815163612365723]},
 {0: [303.68235117639415,
   442.8567960144719,
   460.1531808376312,
   360.44391758553684,
   523.2503823689185,
   322.724678898463,
   346.31122907379176,
   527.9883633167483,
   496.8869036908727,
   457.8979062670842],
  1: [505.85732359811664,
   546.2954699303955,
   457.48179890657775,
   363.11944650975056,
   464.45964494900545,
   521.6114194344264,
   535.0293998839334,
   488.41924657905474,
   390.85160991433077,
   511.8401939673349],
  2: [442.2337143495679,
   516.1478663175367,
   478.109352737898,
   404.3967137141153,
   519.8255872419104,
   398.5881111335475,
   529.0044819777831,
   520.1784952920862,
   393.7034790259786,
   474.8515732518863],
  3: [362.1922239002306,
   413.25454691192135,
   372.1723985793069,
   314.45256841229275,
   437.1968528642319,
   321.7055111101363,
   510.8281853389926,
   423.50214957632124,
   412.5289841252379,
   365.403165658121],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.6538019180297852,
   -1.3673391342163086,
   -1.5838444232940674,
   -0.9775153398513794,
   -1.0199484825134277,
   -1.4586502313613892,
   -0.973930299282074,
   -0.9731709957122803,
   -1.2626399993896484,
   -1.7593485116958618]},
 {0: [568.3219100912102,
   477.28882299689576,
   482.35083340574056,
   401.42555933841504,
   428.5984730911441,
   463.0420811320655,
   469.69435246894136,
   506.67249140981585,
   595.1867219181731,
   368.0471724404488],
  1: [552.7808985104784,
   572.2836862420663,
   403.25628246879205,
   533.6173454574309,
   470.37949487403966,
   468.7218929112423,
   501.74377552838996,
   449.7247120072134,
   734.4406247558072,
   296.8627220108174],
  2: [522.7255448151845,
   485.52783423662186,
   526.456920097582,
   444.4061092354823,
   463.7722817303147,
   397.8048643779475,
   466.5553269293159,
   457.01110143307596,
   578.0636748909019,
   432.4561472637579],
  3: [395.87650249525905,
   332.4427158138715,
   514.960705047939,
   427.9393169400282,
   410.8455168006476,
   376.8038808021229,
   393.29551778174937,
   369.48662914917804,
   437.3210901156999,
   485.2176551744342],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.1903405487537384,
   0.11408751457929611,
   -0.6834728121757507,
   0.19051167368888855,
   -0.13057509064674377,
   -0.15139594674110413,
   0.4020000994205475,
   -0.2776026129722595,
   -0.5950244665145874,
   -0.5899246335029602]},
 {0: [383.7573481044965,
   306.1904980733525,
   324.67135315039195,
   330.20873834274244,
   283.3217910786043,
   397.38555049477145,
   395.98631172347814,
   259.74459882464726,
   389.5471702229697,
   383.4664949590806],
  1: [418.3013053636532,
   352.1965216845274,
   374.1949993360322,
   260.29519239335787,
   307.4029586951947,
   414.7710104584694,
   313.60767900093924,
   378.82216519350186,
   371.8054678863846,
   433.78998522041366],
  2: [427.2308762217872,
   324.2863796306774,
   424.6905319709331,
   250.2643345191609,
   287.1427992917015,
   346.4887033144478,
   484.0244100727141,
   352.9164009273518,
   365.4854488505516,
   424.5603284606477],
  3: [419.5107405362651,
   344.3938517838251,
   369.55868591787294,
   296.55200898018666,
   221.9560769230593,
   326.17692678631283,
   399.56410689558834,
   290.21211478684563,
   366.57913979759905,
   382.8266536538722],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.1441238522529602,
   -0.14028526842594147,
   -0.5342198610305786,
   -0.5005261301994324,
   -0.03247251734137535,
   0.06898530572652817,
   -0.455594539642334,
   0.11785054951906204,
   0.052794210612773895,
   -0.5195986032485962]},
 {0: [364.27690763166174,
   358.83583613997325,
   419.0221569838468,
   354.67437215917744,
   375.39285984728485,
   353.03799631725997,
   364.5510707949288,
   360.0513431751169,
   414.19722187600564,
   370.1722652919125],
  1: [321.09388499730267,
   377.3868599885609,
   514.200938639231,
   489.9471650933847,
   438.64409037842415,
   315.6091182976961,
   417.55727105308324,
   391.1879266980104,
   451.6637839451432,
   372.65098591987044],
  2: [300.96362484339625,
   375.3622933793813,
   539.7535748889204,
   470.43780345260166,
   363.00838872184977,
   340.4049666062929,
   401.4882551319897,
   432.1443500400055,
   425.5000650244765,
   431.97911131521687],
  3: [280.9617768251337,
   365.8545709088212,
   419.40500331146177,
   465.61686445912346,
   315.7593191913329,
   484.3106740619987,
   416.8306971839629,
   354.58739702473395,
   373.85859620780684,
   423.01751460065134],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.13092194497585297,
   -0.20033742487430573,
   -0.26423463225364685,
   -0.33478403091430664,
   -0.3414269983768463,
   -0.12700104713439941,
   -0.19338484108448029,
   -0.5133644938468933,
   0.1123470589518547,
   0.03640918806195259]},
 {0: [501.45079025113955,
   432.397135039093,
   557.9966710391454,
   464.72887251200154,
   384.54544717725366,
   500.8113085948862,
   572.3721282877959,
   447.2326789412182,
   486.2649252838455,
   512.0237528942525],
  1: [544.4174383841455,
   418.9443032592535,
   610.8169319578446,
   491.85326432064176,
   337.46217255387455,
   456.05758880637586,
   535.355451031588,
   489.9727056522388,
   533.2394016282633,
   578.5922062546015],
  2: [566.3501955289394,
   386.8848873535171,
   605.8637598315254,
   474.8542554678861,
   386.2323728064075,
   458.2207454442978,
   529.6602374878712,
   629.2084020678885,
   466.21635622298345,
   562.6341092502698],
  3: [402.9818065101281,
   466.31324889464304,
   396.2857061444083,
   335.43663534196094,
   470.428270236589,
   443.58490891056135,
   521.9916061877739,
   459.81650839652866,
   357.62730928859673,
   453.9594157175161],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [0.04363824427127838,
   -0.8483442664146423,
   -0.5926631689071655,
   -0.7101678252220154,
   -0.36224165558815,
   -1.00519859790802,
   0.010489566251635551,
   -0.931331217288971,
   -0.5399861335754395,
   -0.18164914846420288]},
 {0: [508.58916191663593,
   405.5007285864558,
   268.1370437843725,
   342.36892596865073,
   408.61128046386875,
   427.240528896451,
   414.8014075239189,
   493.991234194953,
   420.8982580269221,
   381.35163996426854],
  1: [421.1228253929876,
   493.03477809159085,
   362.11643215455115,
   383.6841262837406,
   369.6780721389223,
   503.52802923787385,
   556.2810994572937,
   495.90239276504144,
   326.70009781001136,
   464.3772868877277],
  2: [484.6606914966833,
   468.4978000699193,
   373.1866461785976,
   396.7378115473548,
   398.22256045416,
   474.6131032318808,
   532.7334362263791,
   677.3547234237194,
   423.510198074393,
   391.4674941422418],
  3: [325.19740689499304,
   348.6468405164778,
   357.09211390139535,
   240.82482293224894,
   287.14063311822247,
   398.2198575399816,
   393.68694214720745,
   412.65826538950205,
   251.4637292684056,
   408.9807439316064],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.8363890051841736,
   -0.9749091267585754,
   -0.616896390914917,
   -1.0745772123336792,
   -0.901199996471405,
   -0.9157782196998596,
   -0.9734123349189758,
   -0.7798977494239807,
   -0.884078860282898,
   -0.84311443567276]},
 {0: [432.4213484614156,
   515.1246303971857,
   563.9064058298245,
   495.30348029499874,
   473.42951928824186,
   453.60416858736426,
   507.3631537363399,
   551.4293531945441,
   532.6694440131541,
   533.7641508644447],
  1: [510.47954288474284,
   379.00458430079743,
   600.2505338024348,
   366.5718903283123,
   447.6951020853594,
   437.2458373811096,
   564.3722758581862,
   557.0789682380855,
   552.1502592470497,
   611.8250758950599],
  2: [508.353473785799,
   451.44699674425647,
   501.1098209163174,
   494.01932788826525,
   610.6707612306345,
   673.7488496596925,
   572.3511118274182,
   486.3050982498098,
   526.7992570591159,
   590.0423562419601],
  3: [415.31512577435933,
   595.3807220626622,
   549.9320014365949,
   476.26161610893905,
   404.9244924252853,
   483.70733347022906,
   522.0359012917615,
   361.9198465263471,
   416.55582806281745,
   575.0596274209674],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-0.5654199719429016,
   0.23509562015533447,
   0.06871291249990463,
   -0.14182178676128387,
   -0.7349326014518738,
   -0.6478922963142395,
   -0.812299907207489,
   0.48680579662323,
   -0.127858504652977,
   0.4161769151687622]},
 {0: [348.99233735119924,
   384.7924002283253,
   498.49565756181255,
   436.0462263398804,
   315.43092467729,
   358.697794866981,
   482.48830978618935,
   501.64638875529636,
   343.2568253977224,
   494.9207600974478],
  1: [405.87551436177455,
   492.81940323719755,
   381.30224399035797,
   403.2597479384858,
   499.124034602195,
   425.21777032106183,
   623.0862922789529,
   579.4061340414919,
   529.0136368516833,
   467.90277519868687],
  2: [403.1441297181882,
   396.6425908753881,
   434.4822793812491,
   426.12149431952275,
   600.7985621774569,
   464.28653013752773,
   520.8670782302506,
   521.8772648097947,
   531.3966020960361,
   362.78802875289693],
  3: [477.2413943610154,
   397.90159215184394,
   467.4993715300225,
   393.91219798265956,
   310.23580511508044,
   390.0671752842609,
   480.95958228595555,
   415.03984623448923,
   282.55496057588607,
   394.9300959170796],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-1.7257133722305298,
   -1.488253116607666,
   -1.2623231410980225,
   -1.2186577320098877,
   -0.9724174737930298,
   -0.7489832043647766,
   0.051319558173418045,
   -1.1228480339050293,
   -0.958098292350769,
   -0.036782119423151016]},
 {0: [291.5313561654184,
   303.4461125249509,
   341.7096032919362,
   309.0536185912788,
   393.66254468262196,
   468.1183115709573,
   488.39705473324284,
   456.05024351598695,
   440.2263098361436,
   439.5955519278068],
  1: [328.01866457238793,
   438.74844936304726,
   423.28193178446963,
   540.8296158560552,
   348.1551082765218,
   371.61455711792223,
   415.9188045905903,
   627.1664481358603,
   502.34468611550983,
   469.1831498022657],
  2: [410.3739943159744,
   314.8839095670264,
   415.4544989177957,
   371.0958461505361,
   451.42304462834727,
   389.23353134072386,
   424.56165799312294,
   464.58257098821923,
   460.3782592632342,
   489.3082982725464],
  3: [383.175292607164,
   377.8496737899259,
   442.4778149861377,
   349.4820033283904,
   373.8290848269826,
   427.1658533369191,
   336.23269980575424,
   382.52386926161125,
   354.3869279967621,
   383.18092749407515],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [0.4330769181251526,
   -0.5180585980415344,
   -0.6052035689353943,
   -0.33445674180984497,
   0.39380037784576416,
   -0.5349408388137817,
   0.37383824586868286,
   -0.10338664799928665,
   0.3928500711917877,
   -0.5178137421607971]},
 {0: [517.6454304759391,
   309.63826344604604,
   362.47068294265773,
   301.8684379165061,
   335.16998589690775,
   337.60434063873254,
   358.04129488556646,
   510.89775867713615,
   327.57471051765606,
   445.3224158519879],
  1: [383.22386716818437,
   464.1298499801196,
   632.979270123411,
   412.20092769758776,
   334.7783290888183,
   352.1428926852532,
   502.10397105431184,
   515.3268239749596,
   469.00200428487733,
   437.0318791260943],
  2: [357.8192686899565,
   412.2926475960994,
   552.4466906893067,
   364.9551156742964,
   339.71478369575925,
   328.62035946850665,
   330.16144884657115,
   469.84092010115273,
   476.08398100640625,
   456.23600250971504],
  3: [355.1342309236061,
   405.4363941887859,
   398.3735931487754,
   337.64816606452223,
   244.84058470185846,
   227.56098410557024,
   328.44800372840837,
   374.0653287091991,
   476.0921701043844,
   445.24360720207915],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [-1.6493664979934692,
   -1.5087156295776367,
   -1.931075096130371,
   -1.9934145212173462,
   -1.9502397775650024,
   -1.8897072076797485,
   -2.0675535202026367,
   -2.1240806579589844,
   -1.4070273637771606,
   -1.8302712440490723]},
 {0: [515.9972862848081,
   478.2855983125046,
   502.5610137847252,
   543.7503321678378,
   423.4067869549617,
   321.9647163783666,
   537.6839312864468,
   353.2152141057886,
   480.0061705787666,
   505.10933636897244],
  1: [541.480411733035,
   470.2641958859749,
   626.1035392526537,
   492.984906448517,
   517.2953160591424,
   349.6209752308205,
   448.67860909877345,
   487.1234259228222,
   442.4175401311368,
   715.9793232213706],
  2: [492.2086018100381,
   408.2415728571359,
   565.1203726185486,
   477.36874933331273,
   474.08236511796713,
   349.9713000964839,
   397.2983726975508,
   469.0810108995065,
   411.91924182418734,
   561.0066943103448],
  3: [355.2501658212859,
   329.5472632027231,
   415.0623317747377,
   584.2577058244497,
   361.2453079430852,
   374.5974660227075,
   373.9493454284966,
   382.7712612967007,
   573.9828780372627,
   397.4498406960629],
  4: [],
  5: [],
  6: [],
  7: [],
  8: [],
  9: [],
  10: [],
  11: [],
  12: [],
  13: [],
  14: [],
  15: [],
  16: [2.059391736984253,
   2.523526430130005,
   2.803926706314087,
   2.420689105987549,
   2.006521463394165,
   1.9579671621322632,
   2.3867132663726807,
   2.654944658279419,
   2.330371379852295,
   2.4710769653320312]}]

In [13]:

# Return Consistency scores in an one-liner - by calling the metric instance.
quantus.Consistency(
    discretise_func=quantus.discretise_func.top_n_sign,
    return_aggregate=False,
)(model=model,
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Consistency metric is likely to be sensitive to the choice of Function for discretisation of the explanation space 'discretise_func' (return hash value ofan np.array used for comparison)..  
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. 'Framework for Evaluating Faithfulness of Explanations.' arXiv preprint arXiv:2202.00734 (2022).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/Users/artemsereda/anaconda3/envs/quantus/lib/python3.9/site-packages/captum/_utils/gradient.py:56: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  warnings.warn(

Out[13]:

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

In [11]:

# Return Relative Input Stability scores in an one-liner - by calling the metric instance.
quantus.RelativeInputStability(nr_samples=5)(
    model=model,
    x_batch=x_batch,
    y_batch=y_batch,
    explain_func=quantus.explain,
    explain_func_kwargs={"method": "Saliency"},
    device=device
)

Warnings and information:
 (1) The Relative Input Stability metric is likely to be sensitive to the choice of function used to generate perturbations 'perturb_func' and parameters passed to it 'perturb_func_kwargs'number of times perturbations are sampled 'nr_samples'.  
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Chirag Agarwal, et. al., 2022. "Rethinking stability for attribution based explanations." https://arxiv.org/pdf/2203.06877.pdf.
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/Users/artemsereda/anaconda3/envs/quantus/lib/python3.9/site-packages/captum/_utils/gradient.py:56: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  warnings.warn(

Out[11]:

[0.9370962955664959,
 1.400342565815413,
 0.8362847066070609,
 0.9300563738327268,
 0.34368543547441677,
 1.80912062730827,
 0.6125925506956547,
 0.6466403537291399,
 0.6713025894279046,
 0.5676414947631669,
 0.49859090807245027,
 0.6546771069761413,
 0.4099039934179301,
 0.6446867059212764,
 0.5638572657567565,
 0.8077426449301647,
 0.6668300363658602]

In [12]:

# Return Relative Output Stability scores in an one-liner - by calling the metric instance.
quantus.RelativeOutputStability(nr_samples=5)(
    model=model,
    x_batch=x_batch,
    y_batch=y_batch,
    explain_func=quantus.explain,
    explain_func_kwargs={"method": "Saliency"},
    device=device
)

Warnings and information:
 (1) The Relative Output Stability metric is likely to be sensitive to the choice of function used to generate perturbations 'perturb_func' and parameters passed to it 'perturb_func_kwargs'number of times perturbations are sampled 'nr_samples'.  
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Chirag Agarwal, et. al., 2022. "Rethinking stability for attribution based explanations." https://arxiv.org/pdf/2203.06877.pdf.
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/Users/artemsereda/anaconda3/envs/quantus/lib/python3.9/site-packages/captum/_utils/gradient.py:56: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  warnings.warn(

Out[12]:

[49.901998793079876,
 53.24123943072003,
 39.92974167614703,
 42.65079933016114,
 38.65713318786283,
 46.6123852928791,
 59.79080469324747,
 46.94594190010342,
 51.627784929029886,
 39.626800133853514,
 56.78942750839559,
 58.43638515076417,
 39.36197897635456,
 41.64708400693029,
 36.29111286187993,
 50.444240504745466,
 45.176516758881334]

In [13]:

# Return Relative Representation Stability scores in an one-liner - by calling the metric instance.
quantus.RelativeRepresentationStability(nr_samples=5, layer_names=["layer4.1.conv2"])(
    model=model,
    x_batch=x_batch,
    y_batch=y_batch,
    explain_func=quantus.explain,
    explain_func_kwargs={"method": "Saliency"},
    device=device,
)

Warnings and information:
 (1) The Relative Representation Stability metric is likely to be sensitive to the choice of function used to generate perturbations 'perturb_func' and parameters passed to it 'perturb_func_kwargs'number of times perturbations are sampled 'nr_samples'choice which internal representations to use 'layer_names', 'layer_indices'.  
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Chirag Agarwal, et. al., 2022. "Rethinking stability for attribution based explanations." https://arxiv.org/pdf/2203.06877.pdf.
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/Users/artemsereda/anaconda3/envs/quantus/lib/python3.9/site-packages/captum/_utils/gradient.py:56: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  warnings.warn(

Out[13]:

[0.2919503263131499,
 0.032576507420171985,
 0.20695623932817356,
 0.18923498415818468,
 0.1035082179315342,
 0.11028891911386982,
 0.14181839842277552,
 0.11389669029822373,
 0.11882913572655851,
 0.2256330430473133,
 0.04239658104868257,
 0.22176035349575418,
 0.13256847929894863,
 0.18340840013237372,
 0.0030804017747872135,
 0.07369939671423234,
 0.0716779019528837]

2.3 Localisation metrics¶

Pointing Game (Zhang et al., 2018)
Attribution Localisation (Kohlbrenner et al., 2020)
TKI (Theiner et al., 2021)
Relevance Rank Accuracy (Arras et al., 2021)
Relevance Mass Accuracy (Arras et al., 2021)
AUC (Arras et al., 2021)

In [ ]:

# Return effective complexity scores in an one-liner - by calling the metric instance.
quantus.PointingGame(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   s_batch=s_batch,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Pointing Game metric is likely to be sensitive to the choice of ground truth mask i.e., the 's_batch' input as well as if the attributions are normalised 'normalise' (and 'normalise_func') and/ or taking absolute values of such 'abs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Zhang, Jianming, Baral, Sarah Adel, Lin, Zhe, Brandt, Jonathan, Shen, Xiaohui, and Sclaroff, Stan. 'Top-Down Neural Attention by Excitation Backprop.' International Journal of Computer Vision, 126:1084-1102 (2018).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[False,
 True,
 True,
 False,
 True,
 True,
 False,
 True,
 False,
 True,
 True,
 False,
 False,
 True,
 True,
 True,
 False,
 False,
 True,
 True]

In [ ]:

# Return attribution localisation scores in an one-liner - by calling the metric instance.
quantus.AttributionLocalisation(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   s_batch=s_batch,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"},
   device=device)

Warnings and information:
 (1) The Attribution Localisation metric is likely to be sensitive to the choice of ground truth mask i.e., the 's_batch', if size of the ground truth mask is taking into account 'weighted' as well as if attributions are normalised 'normalise' (and 'normalise_func') and/ or taking the absolute values of such 'abs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Kohlbrenner M., Bauer A., Nakajima S., Binder A., Wojciech S., Lapuschkin S. 'Towards Best Practice in Explaining Neural Network Decisions with LRP.arXiv preprint arXiv:1910.09840v2 (2020)..
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/usr/local/lib/python3.7/dist-packages/captum/_utils/gradient.py:59: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  "required_grads has been set automatically." % index

Out[ ]:

[0.5148162841796875,
 0.8962053060531616,
 0.5655674934387207,
 0.09387124329805374,
 0.4196097254753113,
 0.9943642616271973,
 0.5516382455825806,
 0.38236069679260254,
 0.26735198497772217,
 0.7278314232826233,
 0.6226062774658203,
 0.09952004253864288,
 0.001369201228953898,
 0.6229471564292908,
 0.9224637150764465,
 0.5589882135391235,
 0.29845502972602844,
 0.23174387216567993,
 0.7260722517967224,
 0.11046291142702103]

In [ ]:

# Return tki scores in an one-liner - by calling the metric instance.
quantus.TopKIntersection(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   s_batch=s_batch,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Top K Intersection metric is likely to be sensitive to the choice of ground truth mask i.e., the 's_batch', the number of features to consider 'k', if size of the ground truth mask is taking into account 'concept_influence' as well as if attributions are normalised 'normalise' (and 'normalise_func') and/ or taking absolute values of such 'abs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Theiner, Jonas, Müller-Budack Eric, and Ewerth, Ralph. 'Interpretable Semantic Photo Geolocalization.' arXiv preprint arXiv:2104.14995 (2021).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.386,
 0.996,
 0.706,
 0.055,
 0.547,
 0.997,
 0.5760000000000001,
 0.725,
 0.276,
 0.96,
 0.802,
 0.107,
 0.001,
 0.904,
 0.882,
 0.605,
 0.554,
 0.28400000000000003,
 0.913,
 0.343]

In [ ]:

# Return relevane rank accuracy scores in an one-liner - by calling the metric instance.
quantus.RelevanceRankAccuracy(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   s_batch=s_batch,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Relevance Rank Accuracy metric is likely to be sensitive to the choice of ground truth mask i.e., the 's_batch' as well as if the attributions are normalised 'normalise' (and 'normalise_func') and/ or taking absolute values of such 'abs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Arras, Leila, Osman, Ahmed, and Samek, Wojciech. 'Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI.' arXiv preprint, arXiv:2003.07258v2 (2021)..
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05,
 3.985969387755102e-05]

In [ ]:

# Return relevane mass accuracy scores in an one-liner - by calling the metric instance.
quantus.RelevanceMassAccuracy(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   s_batch=s_batch,
   explain_func=quantus.explain,
   explain_func_kwargs={"method": "Saliency"},
   device=device)

Warnings and information:
 (1) The Relevance Mass Accuracy metric is likely to be sensitive to the choice of ground truth mask i.e., the 's_batch' as well as if the attributions are normalised 'normalise' (and 'normalise_func') and/ or taking absolute values of such 'abs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Arras, Leila, Osman, Ahmed, and Samek, Wojciech. 'Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI.' arXiv preprint, arXiv:2003.07258v2 (2021)..
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.5148163,
 0.8962053,
 0.5655675,
 0.09387124,
 0.41960973,
 0.99436426,
 0.55163825,
 0.3823607,
 0.26735198,
 0.7278314,
 0.6226063,
 0.09952004,
 0.0013692012,
 0.62294716,
 0.9224637,
 0.5589882,
 0.29845503,
 0.23174387,
 0.72607225,
 0.11046291]

In [ ]:

# Return relevane mass accuracy scores in an one-liner - by calling the metric instance.
quantus.AUC(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   s_batch=s_batch,
   explain_func=quantus.explain, 
  explain_func_kwargs={"method": "Saliency"},
  device=device)

Warnings and information:
 (1) The AUC metric is likely to be sensitive to the choice of ground truth mask i.e., the 's_batch' input as well as if absolute values 'abs' are taken of the attributions . 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Fawcett, Tom. 'An introduction to ROC analysis' Pattern Recognition Letters Vol 27, Issue 8, (2006).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.5496032156149311,
 0.8378034883871343,
 0.5978047258304584,
 0.4810583667775295,
 0.6088005205623628,
 0.6562129605999191,
 0.5851947499344192,
 0.6883728302277395,
 0.6215463549723745,
 0.7019519834709333,
 0.7085925234251864,
 0.559065186133047,
 0.4976049816381926,
 0.610841403913073,
 0.48205099899315945,
 0.47184680236897103,
 0.6842717134624884,
 0.629719089623429,
 0.7285726396094813,
 0.8325527955872081]

2.4 Randomisation metrics¶

In [ ]:

# Return model parameter randomization scores in an one-liner - by calling the metric instance.
results = {method: quantus.ModelParameterRandomisation(
    layer_order="bottom_up",
    similarity_func=quantus.similarity_func.correlation_spearman,
    normalise=True,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": method}, 
   device=device) for method in ["Saliency", "Gradient"]}

quantus.plot_model_parameter_randomisation_experiment(results=results, methods=["Saliency", "Gradient"], similarity_metric=quantus.similarity_func.correlation_spearman.__name__.replace("_", "").capitalize())

Warnings and information:
 (1) The Model Parameter Randomisation metric is likely to be sensitive to the choice of similarity metric 'similarity_func' and the order of the layer randomisation 'layer_order'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. 'Sanity Checks for Saliency Maps.' arXiv preprint, arXiv:1810.073292v3 (2018).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/usr/local/lib/python3.7/dist-packages/captum/_utils/gradient.py:59: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  "required_grads has been set automatically." % index

Warnings and information:
 (1) The Model Parameter Randomisation metric is likely to be sensitive to the choice of similarity metric 'similarity_func' and the order of the layer randomisation 'layer_order'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. 'Sanity Checks for Saliency Maps.' arXiv preprint, arXiv:1810.073292v3 (2018).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

In [ ]:

# Return random logit scores in an one-liner - by calling the metric instance.
quantus.RandomLogit(
    num_classes=1000,
    similarity_func=quantus.similarity_func.ssim,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Random Logit metric is likely to be sensitive to the choice of similarity metric 'similarity_func'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Sixt, Leon, Granz, Maximilian, and Landgraf, Tim. 'When Explanations Lie: Why Many Modified BP Attributions Fail.' arXiv preprint, arXiv:1912.09818v6 (2020).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

[0.2785277241429084,
 0.7636847907139436,
 0.49716974138222436,
 0.4627671439594901,
 0.5571587781387842,
 0.6519391260950801,
 0.7168172416351766,
 0.8194065053937403,
 0.5382503839623709,
 0.4246806424579798,
 0.3732174574133432,
 0.5226632786691975,
 0.4919609282309529,
 0.49361720569078354,
 0.4881337121259406,
 0.645903774913976,
 0.3145225318147408,
 0.40125093337452256,
 0.4119223151408124,
 0.455222863450516]

2.5 Complexity metrics¶

Score explanations using the following complexity metrics:

Sparseness (Alvarez-Melis et al., 2018; 2019)
Complexity (Bhatt et al., 2020)
Effective complexity (Nguyen at el., 2020.)

In [ ]:

# Return sparseness scores in an one-liner - by calling the metric instance.
quantus.Sparseness(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Sparseness metric is likely to be sensitive to the choice of normalising 'normalise' (and 'normalise_func') and if taking absolute values of attributions 'abs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Chalasani, Prasad, et al. Concise explanations of neural networks using adversarial training.' International Conference on Machine Learning. PMLR, (2020).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/usr/local/lib/python3.7/dist-packages/captum/_utils/gradient.py:59: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  "required_grads has been set automatically." % index

Out[ ]:

[0.4038726505741572,
 0.4417775710056374,
 0.43847453487263954,
 0.46608260004703106,
 0.4141086790933253,
 0.45284584706748965,
 0.420271762129851,
 0.43028661260632267,
 0.41026565943758697,
 0.39818799234766555,
 0.4366796556762597,
 0.4486218971247016,
 0.43187905108771174,
 0.49551300177253116,
 0.4530533022476297,
 0.4529068350009297,
 0.39059322088333015,
 0.41575436635169527]

In [ ]:

# Return complexity scores in an one-liner - by calling the metric instance.
quantus.Complexity(
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"},
   device=device)

Warnings and information:
 (1) The Complexity metric is likely to be sensitive to the choice of normalising 'normalise' (and 'normalise_func') and if taking absolute values of attributions 'abs'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Bhatt, Umang, Adrian Weller, and José MF Moura. 'Evaluating and aggregating feature-based model explanations.' arXiv preprint arXiv:2005.00631 (2020).
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/usr/local/lib/python3.7/dist-packages/captum/_utils/gradient.py:59: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  "required_grads has been set automatically." % index

Out[ ]:

[10.44070243488689,
 10.420596971333223,
 10.484279878887667,
 10.56721230819776,
 10.540531584268715,
 10.536727456028371,
 10.500922736444046,
 10.51492837656417,
 10.512601491344476,
 10.557478675883386,
 10.40113705597035,
 10.486384748693812,
 10.551167703984634,
 10.520842420783698,
 10.547990322449959,
 10.467381022906697,
 10.339475828236742]

In [ ]:

# Return effective complexity scores in an one-liner - by calling the metric instance.
quantus.EffectiveComplexity(
    eps=1e-5,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Effective Complexity metric is likely to be sensitive to the choice of normalising 'normalise' (and 'normalise_func') and if taking absolute values of attributions 'abs' and the choice of threshold 'eps'. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Nguyen, An-phi, and María Rodríguez Martínez. 'On quantitative aspects of model interpretability.' arXiv preprint arXiv:2007.07584 (2020)..
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

Out[ ]:

2.6 Axiomatic metrics¶

Score explanations using the following axiomatic metrics:

Completeness (Montavon et al., 2018)
Non-sensitivity (Nguyen at el., 2020)
Input Invariance (Kindermans et al., 2017)

In [ ]:

# Return completeness scores in an one-liner - by calling the metric instance.
quantus.Completeness(
    abs=False,
    disable_warnings=True,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=a_batch,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Out[ ]:

[False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False,
 False]

In [ ]:

# Return non-sensitivity scores in an one-liner - by calling the metric instance.
quantus.NonSensitivity(
    abs=True,
    eps=1e-5,
    n_samples=10, 
    perturb_baseline="black",
    perturb_func=quantus.perturb_func.baseline_replacement_by_indices,
    features_in_step=6272,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Warnings and information:
 (1) The Non Sensitivity metric is likely to be sensitive to the choice of baseline value 'perturb_baseline', the number of samples to iterate over 'n_samples' and the threshold value function for the feature to be considered having an insignificant contribution to the model. 
 (2) If attributions are normalised or their absolute values are taken it may destroy or skew information in the explanation and as a result, affect the overall evaluation outcome.
 (3) Make sure to validate the choices for hyperparameters of the metric (by calling .get_params of the metric instance).
 (4) For further information, see original publication: Nguyen, An-phi, and María Rodríguez Martínez. 'On quantitative aspects of model interpretability.' arXiv preprint arXiv:2007.07584 (2020)..
 (5) To disable these warnings set 'disable_warnings' = True when initialising the metric.

/usr/local/lib/python3.7/dist-packages/captum/_utils/gradient.py:59: UserWarning: Input Tensor 0 did not already require gradients, required_grads has been set automatically.
  "required_grads has been set automatically." % index

Out[ ]:

[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]

In [ ]:

# Return completeness scores in an one-liner - by calling the metric instance.
quantus.InputInvariance(
    abs=False,
    disable_warnings=True,
)(model=model, 
   x_batch=x_batch,
   y_batch=y_batch,
   a_batch=None,
   explain_func=quantus.explain, 
   explain_func_kwargs={"method": "Saliency"}, 
   device=device)

Out[ ]:

[True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 True,
 False,
 True,
 True,
 True,
 True,
 True]

In [ ]:

ImageNet Example All Metrics¶

1) Preliminaries¶

1.1 Load datasets¶

1.2 Load a pre-trained model¶

1.3 Generate attribution-based explanations¶

2) Evaluation using Quantus¶

2.1 Faithfulness metrics¶

2.2 Robustness metrics¶

2.3 Localisation metrics¶

2.4 Randomisation metrics¶

2.5 Complexity metrics¶

2.6 Axiomatic metrics¶

2) Evaluation using `Quantus`¶