SCDE : Sentence Cloze Dataset with High Quality Distractors From Examinations (2020)

Abstract: We introduce SCDE, a dataset to evaluate the performance of computational models through sentence prediction. SCDE is a human created sentence cloze dataset, collected from public school English examinations. Our task requires a model to fill up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. Experimental results demonstrate that this task requires the use of non-local, discourse-level context beyond the immediate sentence neighborhood. The blanks require joint solving and significantly impair each other’s context. Furthermore, through ablations, we show that the distractors are of high quality and make the task more challenging. Our experiments show that there is a significant performance gap between advanced models (72%) and humans (87%), encouraging future models to bridge this gap.

## introduction cloze questions were first proposed by @xcite as a readability test, motivated by gestalt psychology. they become an efficient way of testing reading for public exams, overtaking the dominant paradigm of subjective questions @xcite . cloze datasets @xcite @xcite @xcite became prevalent as questionanswering (qa) benchmarks since they are convenient either to be generated automatically or by annotators. these datasets could be split into two clear types: herein, we focus on the 2nd category. a common property of these datasets is that they have gaps at the level of words, entities or short syntactic spans. the entity and span-based clozes may sometimes be multi-token, but they do not extend beyond a few tokens. nevertheless, none of these datasets have cloze gaps at the level of full sentences. since many syntactic and semantic cues are present in the same sentence, this makes the gap easier to fill compared to the sentence level cloze case where models would have to rely on "discourse" cues beyond the same sentence. besides lack of intra-sentence cues, sentencelevel cloze may require comparing candidates of very different lengths. for instance, the example in another salient aspect of our dataset is that more than 40% of blanks belong to the reasoning category "inference" (more on this in §3.3 and several cloze test datasets are collected to measure reading comprehension ability of machines. cnn/dailymail (hermann et al., 2015), an early dataset of current qa research, constructs cloze questions from article summaries, with article spans as answers. their cloze gaps are entities and hence one or few tokens long at best. the lam-bada dataset (paperno et al., 2016) constructs a corpus of word level cloze gaps, such that each gap is in the last passage sentence. cbt (hill and simha, 2016) creates word level cloze questions by removing a word in the last sentence of every consecutive 21 sentences, with the first 20 sentences being the context. onishi et al. (2016) curate a dataset of who-did-what type sentences with dataset sl mb distractors candidates position context w scde human shared anywhere 319 rocstories (2016) × human -end 25 cloth (2018) × human separated anywhere 243 lambada (2016) × × exhaustive -end 76 cbt (2015) × × automatic -end 465 mrscc (2011) × × human -anywhere 20 in contrast, scde is by design independent of artifacts, since a) given a blank, only some of our candidates are distractors, the rest being answers for other blanks. even if one were to learn a classifier to distinguish distractors without context, the non-distractor candidates would be unresolvable without context. b) we further check how distinguishable our distractors are from non-distractors without context by training a strong classifier in this setting, as described in §5.6. the classifier obtains a reasonably low f1 score of 0.38. in public school examinations have been used as a data source by many earlier qa works, two prominent examples being the clef qa tracks @xcite and race @xcite . ## conclusion we introduce scde, a sentence cloze dataset with high quality distractors carefully designed by english teachers. scde requires use of discourselevel context and different reasoning types. more importantly, the high quality distractors make this task more challenging. human performance is found to exceed advanced contextual embedding and language models by a significant margin. through scde, we aim to encourage the development of more advanced language understanding models. the demand for ways to improve memory is higher in students than it is in adults. students often come across new knowledge in different areas that they need to store for exams.