ummeabiha/squad-eqa-runs

Extractive question answering on SQuAD 1.1

Each model reads a question and a paragraph and predicts the start and end token of the answer in the paragraph. All were trained on the SQuAD 1.1 training set (without 22 articles kept for validation) and tested on the official dev set (10,570 questions).

Model	Folder in models/	Params	Test EM	Test F1
1a Scratch Transformer	scratch_f9ed439f	22.5M	34.90	47.97
1b Scratch + SQuAD MLM	scratch_mlm_d5ed7f75	22.5M	55.98	67.37
1c Scratch + WikiText MLM	scratch_wiki_mlm	22.5M	51.17	63.41
2 ELECTRA-base, input cut at 384 tokens	electra_56b9af30	108.9M	85.33	91.89
3 RoBERTa-base, overlapping windows	roberta_26dc0834	124.1M	86.04	92.16

The scratch models are a 6-layer Transformer encoder (hidden size 384, 6 heads) that we wrote in PyTorch, with the bert-base-uncased vocabulary but no pretrained weights. 1b and 1c were first pretrained by us with masked language modelling, 1b on the SQuAD training text and 1c on WikiText-103 plus the SQuAD paragraphs. ELECTRA and RoBERTa were fine-tuned from google/electra-base-discriminator and roberta-base for 2 epochs.

How to use

ELECTRA and RoBERTa:

from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline

repo, folder = "ummeabiha/squad-eqa-runs", "models/roberta_26dc0834"
qa = pipeline("question-answering",
              model=AutoModelForQuestionAnswering.from_pretrained(repo, subfolder=folder),
              tokenizer=AutoTokenizer.from_pretrained("roberta-base"))
qa(question="In which year did Alan Turing introduce the Turing machine?",
   context="Alan Turing was a British mathematician and computer scientist. He introduced the concept of the Turing machine in 1936.")

Any model, including the scratch ones, with inference.py from this repository (download the model folder first):

python inference.py --model-dir models/scratch_wiki_mlm \
    --question "In which year did Alan Turing introduce the Turing machine?" \
    --context "Alan Turing was a British mathematician and computer scientist. He introduced the concept of the Turing machine in 1936."

Training

AdamW, weight decay 0.01, linear warmup and decay, fp16, seed 42, one T4 GPU.

• Scratch models: batch 64, early stopping on validation F1. Learning rate 5e-4 (3e-4 for fine-tuning 1c). MLM pretraining: 20 epochs for 1b, 3 epochs for 1c.
• ELECTRA: learning rate 5e-5, batch 16, input cut at 384 tokens.
• RoBERTa: learning rate 3e-5, batch 16 with 2 accumulation steps, windows of 384 tokens with stride 128.

Limitations

• Only tested on SQuAD-style English Wikipedia paragraphs.
• SQuAD 1.1 has no unanswerable questions, so the models always return an answer, even if the paragraph does not contain one.
• Most errors are answers with a word too many or too few, or another date, number or name close to words from the question.
• Each model was trained once (one seed).

Data and base models

SQuAD 1.1 (Rajpurkar et al., 2016, CC BY-SA 4.0), WikiText-103 (Merity et al., 2016), ELECTRA (Clark et al., 2020, Apache-2.0), RoBERTa (Liu et al., 2019, MIT).