LIME explanations¶

In [1]:

import trustyai

trustyai.init()

Simple example¶

We start by defining our black-box model, typically represented by

$$ f(\mathbf{x}) = \mathbf{y} $$

Where $\mathbf{x}=\{x_1, x_2, \dots,x_m\}$ and $\mathbf{y}=\{y_1, y_2, \dots,y_n\}$.

Our example toy model, in this case, takes an all-numerical input $\mathbf{x}$ and return a $\mathbf{y}$ of either true or false if the sum of the $\mathbf{x}$ components is within a threshold $\epsilon$ of a point $\mathbf{C}$, that is:

$$ f(\mathbf{x}, \epsilon, \mathbf{C})=\begin{cases} \text{true},\qquad \text{if}\ \mathbf{C}-\epsilon<\sum_{i=1}^m x_i <\mathbf{C}+\epsilon \\ \text{false},\qquad \text{otherwise} \end{cases} $$

This model is provided in the TestUtils module. We instantiate with a $\mathbf{C}=500$ and $\epsilon=1.0$.

In [2]:

from trustyai.utils import TestUtils

center = 10.0
epsilon = 2.0

model = TestUtils.getSumThresholdModel(center, epsilon)

Next we need to define a goal. If our model is $f(\mathbf{x'})=\mathbf{y'}$ we are then defining our $\mathbf{y'}$ and the counterfactual result will be the $\mathbf{x'}$ which satisfies $f(\mathbf{x'})=\mathbf{y'}$.

We will define our goal as true, that is, the sum is withing the vicinity of a (to be defined) point $\mathbf{C}$. The goal is a list of Output which take the following parameters

The feature name
The feature type
The feature value (wrapped in Value)
A confidence threshold, which we will leave at zero (no threshold)

In [3]:

from trustyai.model import output

decision = "inside"
goal = [output(name=decision, dtype="bool", value=True, score=0.0)]

We will now define our initial features, $\mathbf{x}$. Each feature can be instantiated by using FeatureFactory and in this case we want to use numerical features, so we'll use FeatureFactory.newNumericalFeature.

In [4]:

import random
from trustyai.model import feature

features = [feature(name=f"x{i+1}", dtype="number", value=random.random()*10.0) for i in range(3)]

As we can see, the sum of of the features will not be within $\epsilon$ (1.0) of $\mathbf{C}$ (500.0). As such the model prediction will be false:

In [5]:

feature_sum = 0.0
for f in features:
    value = f.value.as_number()
    print(f"Feature {f.name} has value {value}")
    feature_sum += value
print(f"\nFeatures sum is {feature_sum}")

Feature x1 has value 8.824930054798465
Feature x2 has value 4.246070534263824
Feature x3 has value 5.512117711918773

Features sum is 18.583118300981063

We execute the model on the generated input and collect the output

In [6]:

from org.kie.kogito.explainability.model import PredictionInput, PredictionOutput

goals = model.predictAsync([PredictionInput(features)]).get()

We wrap these quantities in a SimplePrediction:

In [7]:

from trustyai.model import simple_prediction

prediction = simple_prediction(input_features=features, outputs=goals[0].outputs)

We can now instantiate the explainer itself.

In [8]:

from trustyai.explainers import LimeExplainer

explainer = LimeExplainer(samples=10)

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

We generate the explanation as a dict : decision --> saliency.

In [9]:

explanation = explainer.explain(prediction, model)

In [11]:

explanation.as_dataframe()

Out[11]:

	inside_features	inside_score	inside_value
0	x1	0.583312	8.824930
1	x2	0.000000	4.246071
2	x3	0.804062	5.512118

We inspect the saliency scores assigned by LIME to each feature

We generate the saliency graph with the builtin method plot(decision):

In [13]:

explanation.plot(decision)

Using Python models¶

We will now show how to use a custom Python model with TrustyAI LIME implementation.

The model will be an XGBoost one trained with the credit-bias dataset.

For convenience, the model is pre-trained and serialised with joblib so that for this example we simply need to deserialised it.

In [14]:

import joblib

xg_model = joblib.load("models/credit-bias-xgboost.joblib")
print(xg_model)

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,
              importance_type='gain', interaction_constraints='',
              learning_rate=0.07, max_delta_step=0, max_depth=8,
              min_child_weight=1, missing=nan, monotone_constraints='()',
              n_estimators=200, n_jobs=12, num_parallel_tree=1, random_state=27,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=0.9861206227457426,
              seed=27, subsample=1, tree_method='exact', validate_parameters=1,
              verbosity=None)

This model has as a single output a boolean PaidLoan, which will predict whether a certain loan applicant will repay the loan in time or not. The model is slightly more complex than the previous examples, with input features:

Input feature	Type	Note
`NewCreditCustomer`	boolean
`Amount`	numerical
`Interest`	numerical
`LoanDuration`	numerical	In months
`Education`	numerical	Level (1, 2, 3..)
`NrOfDependants`	numerical	Integer
`EmploymentDurationCurrentEmployer`	numerical	Integer (years)
`IncomeFromPrincipalEmployer`	numerical
`IncomeFromPension`	numerical
`IncomeFromFamilyAllowance`	numerical
`IncomeFromSocialWelfare`	numerical
`IncomeFromLeavePay`	numerical
`IncomeFromChildSupport`	numerical
`IncomeOther`	numerical
`ExistingLiabilities`	numerical	integer
`RefinanceLiabilities`	numerical	integer
`DebtToIncome`	numerical
`FreeCash`	numerical
`CreditScoreEeMini`	numerical	integer
`NoOfPreviousLoansBeforeLoan`	numerical	integer
`AmountOfPreviousLoansBeforeLoan`	numerical
`PreviousRepaymentsBeforeLoan`	numerical
`PreviousEarlyRepaymentsBefoleLoan`	numerical
`PreviousEarlyRepaymentsCountBeforeLoan`	numerical	integer
`Council_house`	boolean
`Homeless`	boolean
`Joint_ownership`	boolean
`Joint_tenant`	boolean
`Living_with_parents`	boolean
`Mortgage`	boolean
`Other`	boolean
`Owner`	boolean
`Owner_with_encumbrance`	boolean
`Tenant`	boolean
`Entrepreneur`	boolean
`Fully`	boolean
`Partially`	boolean
`Retiree`	boolean
`Self_employed`	boolean

We will start by testing the model with an input we are quite sure (from the original data) that will be predicted as false:

In [15]:

x = [[False,2125.0,20.97,60,4.0,0.0,6.0,0.0,301.0,0.0,53.0,0.0,0.0,0.0,8,6,26.29,10.92,1000.0,1.0,500.0,590.95,0.0,0.0,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0]]

We can see that this application will be rejected with a probability of $\sim77\%$:

In [16]:

import numpy as np

print(xg_model.predict_proba(np.array(x)))
print(f"Paid loan is predicted as: {xg_model.predict(np.array(x))}")

[[0.7770493  0.22295067]]
Paid loan is predicted as: [False]

We will now prepare the XGBoost model to be used from the TrustyAI counterfactual engine.

To do so, we simply need to first create a prediction function which takes:

A list of PredictionInput as inputs
A list of PredictionOutput as outputs

If these two conditions are met, the actual inner working of this method can be anything (including calling a XGBoost Python model for prediction as in our case):

In [17]:

from org.kie.kogito.explainability.model import PredictionInput, PredictionOutput

def predict(inputs):
    values = [_feature.value.as_obj() for _feature in inputs[0].features]
    result = xg_model.predict_proba(np.array([values]))
    false_prob, true_prob = result[0]
    if false_prob > true_prob:
        _prediction = (False, false_prob)
    else:
        _prediction = (True, true_prob)
    _output = output(name="PaidLoan", dtype="bool", value=_prediction[0], score=_prediction[1])
    return [PredictionOutput([_output])]

Once the prediction method is created, we wrap in a PredictionProvider class.

This class takes care of all the JVM's asynchronous plumbing for us.

In [18]:

from trustyai.model import Model

cb_model = Model(predict)

We will now express the previous inputs (x) in terms of Features, so that we might use it for the counterfactual search:

In [19]:

def make_feature(name, _value):
    if isinstance(_value, bool):
        return feature(name=name, dtype="bool", value=_value)
    else:
        return feature(name=name, dtype="number", value=_value)

features = [make_feature(p[0], p[1]) for p in [("NewCreditCustomer", False),
        ("Amount", 2125.0),
        ("Interest", 20.97),
        ("LoanDuration", 60.0),
        ("Education", 4.0),
        ("NrOfDependants", 0.0),
        ("EmploymentDurationCurrentEmployer", 6.0),
        ("IncomeFromPrincipalEmployer", 0.0),
        ("IncomeFromPension", 301.0),
        ("IncomeFromFamilyAllowance", 0.0),
        ("IncomeFromSocialWelfare", 53.0),
        ("IncomeFromLeavePay", 0.0),
        ("IncomeFromChildSupport", 0.0),
        ("IncomeOther", 0.0),
        ("ExistingLiabilities", 8.0),
        ("RefinanceLiabilities", 6.0),
        ("DebtToIncome", 26.29),
        ("FreeCash", 10.92),
        ("CreditScoreEeMini", 1000.0),
        ("NoOfPreviousLoansBeforeLoan", 1.0),
        ("AmountOfPreviousLoansBeforeLoan", 500.0),
        ("PreviousRepaymentsBeforeLoan", 590.95),
        ("PreviousEarlyRepaymentsBefoleLoan", 0.0),
        ("PreviousEarlyRepaymentsCountBeforeLoan", 0.0),
        ("Council_house", False),
        ("Homeless", False),
        ("Joint_ownership", False),
        ("Joint_tenant", False),
        ("Living_with_parents", False),
        ("Mortgage", False),
        ("Other", False),
        ("Owner", False),
        ("Owner_with_encumbrance", True),
        ("Tenant", True),
        ("Entrepreneur", False),
        ("Fully", False),
        ("Partially", False),
        ("Retiree", True),
        ("Self_employed", False)]]

We can confirm now, with the newly created PredictionProvider model that this input will lead to a false PaidLoan prediction:

In [20]:

prediction = cb_model.predictAsync([PredictionInput(features)]).get()
prediction[0].outputs[0].toString()

Out[20]:

'Output{value=false, type=boolean, score=0.7835956811904907, name='PaidLoan'}'

We generate a prediction to be passed to the LIME explainer

In [21]:

prediction_obj = simple_prediction(input_features=features, outputs=prediction[0].outputs)

We execute the LIME explainer on the XGBoost model and prediction

In [22]:

cb_explainer = LimeExplainer(samples=100, perturbations=2, seed=23, normalise_weights=False)
cb_explanation = cb_explainer.explain(prediction_obj, cb_model)

We output the top 2 most important features for the prediction outcome

In [23]:

for f in cb_explanation.map()['PaidLoan'].getTopFeatures(2):
    print(f)

FeatureImportance{feature=Feature{name='Education', type=number, value=4.0}, score=3.282586466080571, confidence= +/-0.0}
FeatureImportance{feature=Feature{name='NrOfDependants', type=number, value=0.0}, score=-1.864816175288724, confidence= +/-0.0}

In [25]:

cb_explanation.as_dataframe()

Out[25]:

	PaidLoan_features	PaidLoan_score	PaidLoan_value
0	NewCreditCustomer	-1.605224	0.00
1	Amount	0.484565	2125.00
2	Interest	0.165306	20.97
3	LoanDuration	-0.768934	60.00
4	Education	3.282586	4.00
5	NrOfDependants	-1.864816	0.00
6	EmploymentDurationCurrentEmployer	1.196610	6.00
7	IncomeFromPrincipalEmployer	0.812697	0.00
8	IncomeFromPension	-0.300951	301.00
9	IncomeFromFamilyAllowance	-1.560764	0.00
10	IncomeFromSocialWelfare	-0.010848	53.00
11	IncomeFromLeavePay	-0.129406	0.00
12	IncomeFromChildSupport	1.094789	0.00
13	IncomeOther	0.945468	0.00
14	ExistingLiabilities	-0.147790	8.00
15	RefinanceLiabilities	0.578735	6.00
16	DebtToIncome	0.562596	26.29
17	FreeCash	-1.241136	10.92
18	CreditScoreEeMini	1.528180	1000.00
19	NoOfPreviousLoansBeforeLoan	0.443888	1.00
20	AmountOfPreviousLoansBeforeLoan	-0.881605	500.00
21	PreviousRepaymentsBeforeLoan	-1.143805	590.95
22	PreviousEarlyRepaymentsBefoleLoan	0.162419	0.00
23	PreviousEarlyRepaymentsCountBeforeLoan	1.095744	0.00
24	Council_house	-0.724558	0.00
25	Homeless	-0.492904	0.00
26	Joint_ownership	1.584465	0.00
27	Joint_tenant	0.420845	0.00
28	Living_with_parents	-0.875883	0.00
29	Mortgage	-1.583980	0.00
30	Other	-0.253131	0.00
31	Owner	0.159048	0.00
32	Owner_with_encumbrance	-0.080871	1.00
33	Tenant	0.844265	1.00
34	Entrepreneur	1.417670	0.00
35	Fully	-0.036086	0.00
36	Partially	-0.815799	0.00
37	Retiree	-0.386546	1.00
38	Self_employed	-1.528299	0.00

In [26]:

cb_explanation.plot('PaidLoan')

In [ ]: