In [1]:

%%html
<script>
  function code_toggle() {
    if (code_shown){
      $('div.input').hide('500');
      $('#toggleButton').val('Show Code')
    } else {
      $('div.input').show('500');
      $('#toggleButton').val('Hide Code')
    }
    code_shown = !code_shown
  }

  $( document ).ready(function(){
    code_shown=false;
    $('div.input').hide()
  });
</script>
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>
<style>
.rendered_html td {
    font-size: xx-large;
    text-align: left; !important
}
.rendered_html th {
    font-size: xx-large;
    text-align: left; !important
}
</style>

In [2]:

%%capture
%load_ext autoreload
%autoreload 2
import sys
sys.path.append("../statnlpbook/")

#util.execute_notebook('relation_extraction.ipynb')

$$ \newcommand{\Xs}{\mathcal{X}} \newcommand{\Ys}{\mathcal{Y}} \newcommand{\y}{\mathbf{y}} \newcommand{\balpha}{\boldsymbol{\alpha}} \newcommand{\bbeta}{\boldsymbol{\beta}} \newcommand{\aligns}{\mathbf{a}} \newcommand{\align}{a} \newcommand{\source}{\mathbf{s}} \newcommand{\target}{\mathbf{t}} \newcommand{\ssource}{s} \newcommand{\starget}{t} \newcommand{\repr}{\mathbf{f}} \newcommand{\repry}{\mathbf{g}} \newcommand{\x}{\mathbf{x}} \newcommand{\prob}{p} \newcommand{\a}{\alpha} \newcommand{\b}{\beta} \newcommand{\vocab}{V} \newcommand{\params}{\boldsymbol{\theta}} \newcommand{\param}{\theta} \DeclareMathOperator{\perplexity}{PP} \DeclareMathOperator{\argmax}{argmax} \DeclareMathOperator{\argmin}{argmin} \newcommand{\train}{\mathcal{D}} \newcommand{\counts}[2]{\#_{#1}(#2) } \newcommand{\length}[1]{\text{length}(#1) } \newcommand{\indi}{\mathbb{I}} $$

In [3]:

%load_ext tikzmagic

Question Answering¶

Flavours of question answering
Information retrieval
Machine reading comprehension
Question answering from structured knowledge
Executable semantic parsing
Relation extraction via reading comprehension

Question:

Which university did Turing go to?

Answer:

Princeton

Passage:

In 1938, he obtained his PhD from the Department of Mathematics at Princeton University.

Knowledge base:

https://www.wikidata.org/wiki/Q7251#P69

ucph.page.link/qa ¶

(Responses)

Flavours of Question answering (QA)¶

Factoid questions:

Information retrieval (IR)-based QA on unstructured data (text)
Knowledge-based QA on structured data (DB/KB)

Non-factoid questions:

"How" questions

How do I delete my Instagram account?

"Why" questions

Why is the sky blue?

Math problems

(from Lu et al., 2021)

Formats¶

Extractive (span selection)
Cloze
Boolean
Multi-choice
Abstractive

QA on collections of documents¶

Which politician won the Nobel Peace Prize in 2009?

Multi-hop QA¶

What is the middle name of the player with the second most National Football League career rushing yards?

QA on structured knowledge¶

The 1999 film '10 Things I Hate About You' is based on which Shakespeare play?

Extractive QA¶

(from Rajpurkar et al., 2016)

Information retrieval (IR)-based QA¶

General approach:

Retrieve relevant passage(s)
Machine reading comprehension: extract the answer, which can be
- A text span from the passage
- Yes/no
- NULL (unanswerable)

Information retrieval¶

Obtain relevant documents/passages given a query
Example: web search
Learn more in NDAK20002U Neural Information Retrieval (NIR)

Machine reading comprehension (MRC)¶

Input: (Passage, Question)
Output: Answer span

(from Rajpurkar et al., 2016)

demo ¶

MRC modeling¶

How to model span selection?

As sequence labeling (for each token, is it part of the answer?)
- With IOB encoding

(from Yoon et al., 2022)

Or as span selection (find start and end of the answer span)

			start	end
...	termed	the	interferon	signature	.	Here	...

What are the advantages and disadvantages of each?

MRC evaluation¶

Metrics for binary (yes/no) QA:

Accuracy
F-score

Metrics for ranking:

MRR (mean reciprocal rank)

Test questions have $k$ gold answers by different human annotators (e.g., $k=3$ for SQuAD and TyDI-QA, $k=5$ for NQ). Metrics for span selection:

Exact match (EM): text is exactly any of the $k$
Word-based f-score (like in some sequence labeling tasks) averaged over the $k$ gold answers
- Often ignoring punctuation and articles, i.e., a, an, the
- As bag-of-words, not exact positions (because the same answer may appear multiple times)
- Macro-averaged: calculate f-score for each question and average the f-scores

SQuAD¶

(from SQuAD leaderboard)

MRC Models¶

model

(from Li et al., 2020)

Knowledge-based (KB) question answering¶

Information is already organized in tables, databases and knowledge bases!

(Executable) semantic parsing: translate natural language question to SQL/SPARQL/logical form program (query).
Execute the program on a database/knowledge-base and return the answer.