TASK DEFINITION: In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.
PROBLEM: We evaluate performance of our models on the SQUAD BIBREF16 dataset (denoted $\mathcal {S}$ ). We use the same split as that of BIBREF4 , where a random subset of 70,484 instances from $\mathcal {S}\ $ are used for training ( ${\mathcal {S}}^{tr}$ ), 10,570 instances for validation ( ${\mathcal {S}}^{val}$ ), and 11,877 instances for testing ( ${\mathcal {S}}^{te}$ ).

SOLUTION: Which datasets are used to train this model?

PROBLEM: To test the difficulty of our dataset, we checked the majority class label and the accuracies of five state-of-the-art NLI models adopting different approaches: BiMPM (Bilateral Multi-Perspective Matching Model; BIBREF31 , BIBREF31 ), ESIM (Enhanced Sequential Inference Model; BIBREF32 , BIBREF32 ), Decomposable Attention Model BIBREF33 , KIM (Knowledge-based Inference Model; BIBREF34 , BIBREF34 ), and BERT (Bidirectional Encoder Representations from Transformers model; BIBREF35 , BIBREF35 ). Regarding BERT, we checked the performance of a model pretrained on Wikipedia and BookCorpus for language modeling and trained with SNLI and MultiNLI.

SOLUTION: What NLI models do they analyze?

PROBLEM: We investigate the following six idiom resources:

Wiktionary;

the Oxford Dictionary of English Idioms (ODEI, BIBREF31);

UsingEnglish.com (UE);

the Sporleder corpus BIBREF10;

the VNC dataset BIBREF9;

and the SemEval-2013 Task 5 dataset BIBREF15.

SOLUTION:
What dictionaries are used for automatic extraction of PIEs?