%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)
from sklearn.compose import make_column_transformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.preprocessing import OneHotEncoder
from aif360.sklearn.preprocessing import ReweighingMeta
from aif360.sklearn.inprocessing import AdversarialDebiasing
from aif360.sklearn.postprocessing import CalibratedEqualizedOdds, PostProcessingMeta
from aif360.sklearn.datasets import fetch_adult
from aif360.sklearn.metrics import disparate_impact_ratio, average_odds_error, generalized_fpr
from aif360.sklearn.metrics import generalized_fnr, difference
Datasets are formatted as separate X
(# samples x # features) and y
(# samples x # labels) DataFrames. The index of each DataFrame contains protected attribute values per sample. Datasets may also load a sample_weight
object to be used with certain algorithms/metrics. All of this makes it so that aif360 is compatible with scikit-learn objects.
For example, we can easily load the Adult dataset from UCI with the following line:
X, y, sample_weight = fetch_adult()
X.head()
age | workclass | education | education-num | marital-status | occupation | relationship | race | sex | capital-gain | capital-loss | hours-per-week | native-country | |||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
race | sex | ||||||||||||||
0 | Non-white | Male | 25.0 | Private | 11th | 7.0 | Never-married | Machine-op-inspct | Own-child | Non-white | Male | 0.0 | 0.0 | 40.0 | United-States |
1 | White | Male | 38.0 | Private | HS-grad | 9.0 | Married-civ-spouse | Farming-fishing | Husband | White | Male | 0.0 | 0.0 | 50.0 | United-States |
2 | White | Male | 28.0 | Local-gov | Assoc-acdm | 12.0 | Married-civ-spouse | Protective-serv | Husband | White | Male | 0.0 | 0.0 | 40.0 | United-States |
3 | Non-white | Male | 44.0 | Private | Some-college | 10.0 | Married-civ-spouse | Machine-op-inspct | Husband | Non-white | Male | 7688.0 | 0.0 | 40.0 | United-States |
5 | White | Male | 34.0 | Private | 10th | 6.0 | Never-married | Other-service | Not-in-family | White | Male | 0.0 | 0.0 | 30.0 | United-States |
We can then map the protected attributes to integers,
X.index = pd.MultiIndex.from_arrays(X.index.codes, names=X.index.names)
y.index = pd.MultiIndex.from_arrays(y.index.codes, names=y.index.names)
and the target classes to 0/1,
y = pd.Series(y.factorize(sort=True)[0], index=y.index)
split the dataset,
(X_train, X_test,
y_train, y_test) = train_test_split(X, y, train_size=0.7, random_state=1234567)
and finally, one-hot encode the categorical features:
ohe = make_column_transformer(
(OneHotEncoder(sparse=False), X_train.dtypes == 'category'),
remainder='passthrough')
X_train = pd.DataFrame(ohe.fit_transform(X_train), index=X_train.index)
X_test = pd.DataFrame(ohe.transform(X_test), index=X_test.index)
X_train.head()
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | |||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
race | sex | ||||||||||||||||||||||
30149 | 1 | 1 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 58.0 | 11.0 | 0.0 | 0.0 | 42.0 |
12028 | 1 | 0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 51.0 | 12.0 | 0.0 | 0.0 | 30.0 |
36374 | 1 | 1 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 26.0 | 14.0 | 0.0 | 1887.0 | 40.0 |
8055 | 1 | 1 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 44.0 | 3.0 | 0.0 | 0.0 | 40.0 |
38108 | 1 | 1 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 33.0 | 6.0 | 0.0 | 0.0 | 40.0 |
5 rows × 100 columns
Note: the column names are lost in this transformation. The same encoding can be done with Pandas, but this cannot be combined with other preprocessing in a Pipeline.
# there is one unused category ('Never-worked') that was dropped during dropna
X.workclass.cat.remove_unused_categories(inplace=True)
pd.get_dummies(X).head()
age | education-num | capital-gain | capital-loss | hours-per-week | workclass_Federal-gov | workclass_Local-gov | workclass_Private | workclass_Self-emp-inc | workclass_Self-emp-not-inc | ... | native-country_Portugal | native-country_Puerto-Rico | native-country_Scotland | native-country_South | native-country_Taiwan | native-country_Thailand | native-country_Trinadad&Tobago | native-country_United-States | native-country_Vietnam | native-country_Yugoslavia | |||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
race | sex | ||||||||||||||||||||||
0 | 0 | 1 | 25.0 | 7.0 | 0.0 | 0.0 | 40.0 | 0 | 0 | 1 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 |
1 | 1 | 1 | 38.0 | 9.0 | 0.0 | 0.0 | 50.0 | 0 | 0 | 1 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 |
2 | 1 | 1 | 28.0 | 12.0 | 0.0 | 0.0 | 40.0 | 0 | 1 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 |
3 | 0 | 1 | 44.0 | 10.0 | 7688.0 | 0.0 | 40.0 | 0 | 0 | 1 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 |
5 | 1 | 1 | 34.0 | 6.0 | 0.0 | 0.0 | 30.0 | 0 | 0 | 1 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 |
5 rows × 100 columns
The protected attribute information is also replicated in the labels:
y_train.head()
race sex 30149 1 1 0 12028 1 0 1 36374 1 1 1 8055 1 1 0 38108 1 1 0 dtype: int64
With the data in this format, we can easily train a scikit-learn model and get predictions for the test data:
y_pred = LogisticRegression(solver='lbfgs').fit(X_train, y_train).predict(X_test)
accuracy_score(y_test, y_pred)
0.8375469890174688
Now, we can analyze our predictions and quickly calucate the disparate impact for females vs. males:
disparate_impact_ratio(y_test, y_pred, prot_attr='sex')
0.2905425926727236
And similarly, we can assess how close the predictions are to equality of odds.
average_odds_error()
computes the (unweighted) average of the absolute values of the true positive rate (TPR) difference and false positive rate (FPR) difference, i.e.:
average_odds_error(y_test, y_pred, prot_attr='sex')
0.09372170954260936
ReweighingMeta
is a workaround until changing sample weights can be handled properly in Pipeline
/GridSearchCV
rew = ReweighingMeta(estimator=LogisticRegression(solver='lbfgs'))
params = {'estimator__C': [1, 10], 'reweigher__prot_attr': ['sex']}
clf = GridSearchCV(rew, params, scoring='accuracy', cv=5)
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))
print(clf.best_params_)
0.8279649148669566 {'estimator__C': 10, 'reweigher__prot_attr': 'sex'}
disparate_impact_ratio(y_test, clf.predict(X_test), prot_attr='sex')
0.5676803237673037
Rather than trying to weight accuracy and fairness, we can try a fair in-processing algorithm:
adv_deb = AdversarialDebiasing(prot_attr='sex', random_state=1234567)
adv_deb.fit(X_train, y_train)
adv_deb.score(X_test, y_test)
0.8399056534237488
average_odds_error(y_test, adv_deb.predict(X_test), prot_attr='sex')
0.060623189820735834
Note that AdversarialDebiasing
creates a TensorFlow session which we should close when we're finished to free up resources:
adv_deb.sess_.close()
Finally, let's try a post-processor, CalibratedEqualizedOdds
.
Since the post-processor needs to be trained on data unseen by the original estimator, we will use the PostProcessingMeta
class which splits the data and trains the estimator and post-processor with their own split.
cal_eq_odds = CalibratedEqualizedOdds('sex', cost_constraint='fnr', random_state=1234567)
log_reg = LogisticRegression(solver='lbfgs')
postproc = PostProcessingMeta(estimator=log_reg, postprocessor=cal_eq_odds, random_state=1234567)
postproc.fit(X_train, y_train)
accuracy_score(y_test, postproc.predict(X_test))
0.8163190093609494
y_pred = postproc.predict_proba(X_test)[:, 1]
y_lr = postproc.estimator_.predict_proba(X_test)[:, 1]
br = postproc.postprocessor_.base_rates_
i = X_test.index.get_level_values('sex') == 1
plt.plot([0, br[0]], [0, 1-br[0]], '-b', label='All calibrated classifiers (Females)')
plt.plot([0, br[1]], [0, 1-br[1]], '-r', label='All calibrated classifiers (Males)')
plt.scatter(generalized_fpr(y_test[~i], y_lr[~i]),
generalized_fnr(y_test[~i], y_lr[~i]),
300, c='b', marker='.', label='Original classifier (Females)')
plt.scatter(generalized_fpr(y_test[i], y_lr[i]),
generalized_fnr(y_test[i], y_lr[i]),
300, c='r', marker='.', label='Original classifier (Males)')
plt.scatter(generalized_fpr(y_test[~i], y_pred[~i]),
generalized_fnr(y_test[~i], y_pred[~i]),
100, c='b', marker='d', label='Post-processed classifier (Females)')
plt.scatter(generalized_fpr(y_test[i], y_pred[i]),
generalized_fnr(y_test[i], y_pred[i]),
100, c='r', marker='d', label='Post-processed classifier (Males)')
plt.plot([0, 1], [generalized_fnr(y_test, y_pred)]*2, '--', c='0.5')
plt.axis('square')
plt.xlim([0.0, 0.4])
plt.ylim([0.3, 0.7])
plt.xlabel('generalized fpr');
plt.ylabel('generalized fnr');
plt.legend(bbox_to_anchor=(1.04,1), loc='upper left');
We can see the generalized false negative rate is approximately equalized and the classifiers remain close to the calibration lines.
We can quanitify the discrepancy between protected groups using the difference
operator:
difference(generalized_fnr, y_test, y_pred, prot_attr='sex')
0.0027891187222710556