In [1]:

%%html
<script>
  function code_toggle() {
    if (code_shown){
      $('div.input').hide('500');
      $('#toggleButton').val('Show Code')
    } else {
      $('div.input').show('500');
      $('#toggleButton').val('Hide Code')
    }
    code_shown = !code_shown
  }

  $( document ).ready(function(){
    code_shown=false;
    $('div.input').hide()
  });
</script>
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

In [2]:

%load_ext tikzmagic

Interpretability¶

or: opening the black box of NLP models

Visualisation
Probing
Adversarial evaluation

Visualisation¶

(from Belinkov, Gehrmann and Pavlick, 2020

Inspecting weights¶

Easy with n-gram feature-based lienar text classifiers:

unigram_positive_weights unigram_negative_weights

Not so simple for neural models.

transformers

Visualising embeddings¶

t-SNE for dimensionality reduction (van der Maaten and Hinton, 2009)

word2vec:

Visualising embeddings¶

t-SNE for dimensionality reduction (van der Maaten and Hinton, 2009)

BERT:

Visualising attention¶

Visualising self-attention¶

(from Vig, 2019)

Visualising self-attention¶

(from Vig, 2019)

Caveats¶

Attention is not explanation (Jain and Wallace, 2019)
Attention is not not explanation (Wiegreffe and Pinter, 2019)

Close inspection¶

(from John Hewitt; slides)

Probing¶

To find out whether a representation R encodes a property P
- train a model to predict P from the R
- or cast the problem directly according to R's training objective