L ARGE - SCALE C LOZE T EST D ATASET D ESIGNED BY T EACHERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Cloze test is widely adopted in language exams to evaluate students’ language proﬁciency.
In this paper, we propose the ﬁrst large-scale human-designed cloze test dataset CLOTH 1 , in which the questions were used in middle-school and highschool language exams.
With the missing blanks carefully created by teachers and candidate choices purposely designed to be confusing, CLOTH requires a deeper language understanding and a wider attention span than previous automatically generated cloze datasets.
We show humans outperform dedicated designed baseline models by a signiﬁcant margin, even when the model is trained on sufﬁciently large external data.
We investigate the source of the performance gap, trace model deﬁciencies to some distinct properties of CLOTH, and identify the limited ability of comprehending a long-term context to be the key bottleneck.
I NTRODUCTION
Being a classic language exercise, the cloze test (Taylor, 1953) is an accurate assessment of language proﬁciency (Fotos, 1991; Jonz, 1991; Tremblay, 2011) and has been widely employed in language examinations.
Under standard setting, a cloze test requires examinees to ﬁll in the missing word (or sentence) that best ﬁts the surrounding context.
To facilitate natural language understanding, automatically generated cloze datasets were introduced to measure the ability of machines in reading comprehension (Hermann et al., 2015; Hill et al., 2016; Onishi et al., 2016).
In these datasets, each cloze question typically consists of a context paragraph and a question sentence.
By randomly replacing a particular word in the question sentence with a blank symbol, a single test case is created.
For instance, the CNN/Daily Mail (Hermann et al., 2015) take news articles as the context and the summary bullet points as the question sentence.
Only named entities are considered when creating the blanks.
Similarly, in Children’s Books test (CBT) (Hill et al., 2016), the cloze question is obtained by removing a word in the last sentence of every consecutive 21 sentences, with the ﬁrst 20 sentences being the context.
Different from the CNN/Daily Mail datasets, CBT also provides each question with a candidate answer set, consisting of randomly sampled words with the same part-of-speech tag from the context as that of the ground truth.
Thanks to the automatic generation process, these datasets can be very large in size, leading to signiﬁcant research progress.
However, compared to how humans would create cloze questions, the automatic generation process bears some inevitable issues.
Firstly, the blanks are chosen uniformly without considering which aspect of the language phenomenon the question will test.
Hence, quite a portion of automatically generated questions can be purposeless or even trivial to answer.
Another issue involves the ambiguity of the answer.
Given a context and a blanked sentence, there can be multiple words that ﬁt almost equally well into the blank.
A possible solution is to include a candidate option set, as done by CBT, to get rid of the ambiguity.
However, automatically generating the candidate option set can be problematic since it cannot guarantee the ambiguity is removed.
More importantly, automatically generated candidates can be totally irrelevant or simply grammatically unsuitable for the blank, resulting in again trivial questions.
Probably due to these unsatisfactory issues, it has been shown neural models have achieved comparable performance with human within very short time (Chen et al., 2016;
Dhingra et al., 2016; Seo et al., 2016).
While there has been work trying to incorporate human design into cloze question generation (Zweig & Burges, 2011), the MSR Sentence Completion Challenge created by this effort is quite small in size, limiting the possibility of developing powerful neural models on it.
Motivated by the aforementioned drawbacks, we propose CLOTH, a large-scale cloze test dataset collected from English exams.
Questions in the dataset are designed by middle-school and highschool teachers to prepare Chinese students for entrance exams.
To design a cloze test, teachers ﬁrstly determine the words that can test students’ knowledge in vocabulary, reasoning or grammar; then replace those words with blanks and provide three candidate options for each blank.
If a question does not speciﬁcally test grammar usage, all of the candidate options would complete the sentence with correct grammar, leading to highly confusing questions.
As a result, human-designed questions are usually harder and are a better assessment of language proﬁciency.
To verify if human-designed cloze questions are difﬁcult for current models, we train dedicated models as well as the state-of-the-art language model and evaluate their performance on this dataset.
We ﬁnd that the state-of-the-art model lags behind human performance even if the model is trained on a large external corpus.
We analyze where the model fails compared to human.
After conducting error analysis, we assume the performance gap results from the model’s inability to use long-term context.
To verify this assumption, we evaluate humans’ performance when they are only allowed to see one sentence as the context.
Our assumption is conﬁrmed by the matched performances of model and human when given only one sentence.
In addition, we demonstrate that human-designed data is more informative and more difﬁcult than automatically generated data.
Speciﬁcally, when the same amount of training data is given, human-designed training data leads to better performance.
Additionally, it is much easier for the same model to perform well on automatically generated data.
CLOTH D ATASET
In this section, we introduce the CLOTH dataset that is collected from English examinations, and study the assessed abilities of this dataset.
2. D ATA C OLLECTION AND S TATISTICS
We collected the raw data from three free websites in China 2 that gather exams designed by English teachers.
These exams are used to prepare students for college/high school entrance exams.
Before cleaning, there are 20 , 605 passages and 332 , 755 questions.
We perform the following processes to ensure the validity of the data: 1.
We remove questions with inconsistent format such as questions with more than four options; 2.
We ﬁlter all questions whose validity relies on external information such as pictures or tables; 3.
Further, we delete duplicated passages; 4.
On one of the websites, the answers are stored as images.
We use two OCR softwares, tesseract 3 and ABBYY FineReader 4 , to extract the answers from images.
We discard the question when results from the two softwares are different.
After the cleaning process, we obtain a dataset of 7 , 131 passages and 99 , 433 questions.
Since high school questions are more difﬁcult than middle school questions, we divided the datasets into CLOTH-M and CLOTH-H, which stand for the middle school part and the high school part.
We split 11% of the data for both the test set and the dev set.
The detailed statistics of the whole dataset and two subsets are presented in Table 1.
[Table]
[Table]
[Table]
[Table]
https:/ github -ocrhttps://
2. Q UESTION T YPE A NALYSIS
In order to evaluate students’ mastery of a language, teachers usually design tests so that questions cover different aspects of a language.
Speciﬁcally, they ﬁrst identity words in the passage that can examine students knowledge in vocabulary, logic or grammar.
Then, they replace the words with blanks and prepare three incorrect but confusing candidate options to make the test non-trivial.
A sample passage is presented in Table 2.
Passage: Nancy had just got a job as a secretary in a company.
Monday was the ﬁrst day she went to work, so she was very 1 and arrived early.
She 2 the door open and found nobody there. ”
I am the 3 to arrive.”
She thought and came to her desk.
She was surprised to ﬁnd a bunch of 4 on it.
They were fresh.
She 5 them
and they were sweet.
She looked around for a 6 to put them in. ”
Somebody has sent me ﬂowers the very ﬁrst day!”
she thought 7 . ”
But who could it be?”
she began to 8 .
The day passed quickly and Nancy did everything with 9 interest.
For the following days of the 10 , the ﬁrst thing Nancy did was to change water for the followers and then set about her work.
Then came another Monday.
11 she came near her desk she was overjoyed to see a(n) 12 bunch of ﬂowers there.
She quickly put them in the vase, 13 the old ones.
The same thing happened again the next Monday.
Nancy began to think of ways to ﬁnd out the 14 .
On Tuesday afternoon, she was sent to hand in a plan to the 15 .
She waited for his directives at his secretary’s 16 .
She happened to see on the desk a half-opened notebook, which 17 : ”
In order to keep the secretaries in high spirits, the company has decided that every Monday morning a bunch of fresh ﬂowers should be put on each secretarys desk.”
Later, she was told that their general manager was a business management psychologist.
Questions:
1. A. depressed B. encouraged C. excited D. surprised 2.
A. turned B. pushed C. knocked D. forced 3.
A. last B. second C. third D
[Table]
To understand the assessed abilities on this dataset, we divide questions into several types and label the proportion of each type of questions.
We ﬁnd that the questions can be divided into the following types:
• Grammar:
The question is about grammar usage, involving tense, preposition usage, active/passive voices, subjunctive mood and
so on
• Short-term-reasoning:
The question is about content words and can be answered based on the information within the same sentence.
• Matching/paraphrasing: The question is answered by copying/paraphrasing a word.
• Long-term-reasoning:
The answer must be inferred from synthesizing information distributed across multiple sentences.
We sample 100 passages in the high school category and the middle school category respectively.
Each high school passage has 20 questions and each middle school passage has 10 questions.
The types of the 3000 question are labeled on Amazon Turk.
We pay $1 and $0.5 for high school passage and middle school passage respectively.
The proportion of different questions is shown in Table 3.
We ﬁnd that the majority of questions are short-term-reasoning questions, in which the examinee needs to utilize vocabulary knowledge and simple reasoning to answer the questions.
Note that a non-trivial proportion of questions is about grammar, which is understandable since the data is collected from exams for non-native speakers.
Finally, only approximately 22 .
4% of data needs long-term information, in which the long-termreasoning questions constitute a large proportion.
[Table]
CLOTH
[Table] [Table]
CLOTH-H
[Table]
E XPLORING M ODELS ’ L IMITS
In this section, we study if human-designed cloze test is a challenging problem for state-of-the-art models.
We ﬁnd that the language model trained on large enough external corpus could not solve the cloze test.
After conducting error analysis, we hypothesize that the model is not able to deal with long-term dependencies.
We verify the hypothesis by evaluating human’s performance when human only sees one sentence as the context.
3. H UMAN AND M ODEL P ERFORMANCE
Supervised LSTM To test the performance of RNN based supervised models, we train a bidirectional LSTM (Hochreiter & Schmidhuber, 1997) to predict the missing word given the context, with only labeled data.
The implementation details are in Appendix A.1.
Language model Language modeling and cloze test are similar as, in both tasks, a word is predicted based on the context.
In cloze test, the context on both sides may determine the correct answer.
Suppose x i is the missing word and x 1 , · · · , x i − 1 , x i +1 , · · · , x n are the context.
Although language model is trained to predict the next word only using the left context, to utilize the surrounding context, we could choose x i that maximizes the joint probability p ( x 1 , · · · , x n ) , which essentially maximizes the conditional likelihood p ( x i − 1 | x 1 , · · · , x i − 1 , x i , · · · , x n ) .
Therefore, language model can be naturally adapted to cloze test.
In essence, language model treats each word as a possible blank and learns to predict it.
As a result, it actually receives more supervision than the supervised model trained on human-labeled questions.
Additionally, it can be trained on a very large unlabeled corpus.
Interested in whether the state-ofthe-art language model can solve cloze test, we ﬁrst train a neural language model on the training set of our corpus, then we test the language model trained on One Billion Word Benchmark (Chelba et al., 2013) (referred as 1-billion-language-model) that achieves a perplexity of 30 .
(Jozefowicz et al., 2016) 5 .
To make the evaluation time tractable, we limit the context length to one sentence or three sentences.
Human performance We measure the performance of Amazon Turkers on 3 , 000 sampled questions when the whole passage is given.
The comparison is shown in Table 4.
The language model trained on our dataset achieves an accuracy of 0 .
548 while the supervised model’s accuracy is 0 . 484 , indicating that more training data results in better generalization.
When only one sentence is given as context, the accuracy of 1billion-language-model is 0 . 695 , which shows that the amount of data is an essential factor affecting the model’s performance.
If we increase the context length to three sentences, the accuracy of 1billion-language-model only improves to 0 .
In contrast, human outperforms 1-billion-language- lm 1b model by a signiﬁcant margin, which demonstrate that deliberately designed questions in CLOTH are not completely solved even for state-of-the-art models.
[Table]
3. A NALYZING MODEL ’ S PERFORMANCE BY HUMAN STUDY
In this section
[Table]
we would like to understand why the state-of-the-art model lags behind human performance.
We ﬁnd that most of errors made by the large language model involve long-term reasoning.
Additionally, in a lot of cases, the dependency is within the context of three sentences.
Several errors made by the large language model are shown in Table 5.
In the ﬁrst example, the model does not know that Nancy found nobody in the company means that Nancy was the ﬁrst one to arrive at the company.
In the second and third example, the model fails probably because of the coreference from “they” to “ﬂowers”.
The dependency in the last case is longer.
It depends on the fact that “Nancy” was alone in the company,
[Table]
Based on the case study, we hypothesize that the language model is not able to take long-term information into account, possibly due to the difﬁculty of long-term reasoning.
Moreover, the 1billion-language-model is trained on sentence level, which might also result in paying more attention to short-term information.
However, we do not have enough computational resources to train a large model on 1 Billion Word Benchmark to investigate the differences of training on sentence level or on paragraph level.
An available comparison is to test the model’s performance on different types of questions.
We ﬁnd that the model’s accuracy is 0 .
570 on long-term-reasoning while achieving 0 .
699 on shortterm-reasoning, which partially conﬁrms that long-term-reasoning is harder.
However, we could not completely rely on the performance on speciﬁc questions types, partly due to the small sample size.
A more fundamental reason is that the question type labels are subjective and their reliability depends on whether turkers are careful enough.
For example, in the error analysis show in Table 5, a careless turker would label the second example as short-term-reasoning without noticing that the meaning of “they” relies on a long context span.
To objectively verify if the language model’s strengths are in dealing with short-term information, we obtain the ceiling performance of only utilizing short-term information.
Showing only one sentence as the context, we ask the turkers to label all possible options that they deem to be correct given the insufﬁcient information.
We also ask them to select a single option based on their best guesses.
By limiting the context span manually, the ceiling performance with only the access to short context is estimated accurately.
The performances of turkers and 1-billion-language-model are shown in Table 6.
The performance of 1-billion-language-model using one sentence as the context can almost match the ceiling performance of only using short-term information.
Hence we conclude that the language model can almost perfectly solve all short-term cloze questions.
However, the performance of language model is not improved signiﬁcantly when the needed long-term context is given, indicating that the performance gap is due to the inability of long-term reasoning.
[Table]
The human study on short-term ceiling performance also reveals that the options are carefully picked.
Speciﬁcally, when a turker thinks that a question has multiple answers, 3 .
41 out of 4 options are deemed to be possibly correct, which means that teachers design the options so that three or four options all make sense if we only look at the local context.
C OMPARING H UMAN - DESIGNED DATA AND A UTOMATICALLY G ENERATED D ATA
In this section, we compare human-designed data and automatically generated data through extensive experiments.
We demonstrate that human-designed data is more informative, i.e., it provides more valuable supervision signals.
We also show that human-designed data is more difﬁcult since the deleted words and candidate options are carefully chosen by teachers.
4. I NFORMATIVENESS C OMPARISON
At a casual observation, a cloze test can be created by randomly deleting words and randomly sampling candidate options.
In fact, to leverage large-scale data, similar generation processes have been introduced and widely used in machine comprehension (Hermann et al., 2015; Hill et al., 2016;
Onishi et al., 2016).
However, researches on cloze test design (Sachs et al., 1997) show that tests created by deliberately deleting words are more reliable than tests created by randomly or periodically deleting words.
To design accurate language proﬁciency assessment, teachers usually selects words in order to examine students’ mastery of grammar, vocabulary and reasoning.
Moreover, in order to make the question non-trivial, the other three incorrect options provided by teachers are usually grammatically correct and relevant to the context.
For instance, in the fourth problem of the sample passage shown in Table 2, “grapes”, “ﬂowers” and “bananas” all ﬁt the description of freshness.
We know “ﬂowers” is the correct answer after seeing the sentence “Somebody has sent me ﬂowers the very ﬁrst day!”.
Naturally, we hypothesize that human-generated data is more informative than randomly generated data.
In other words, human-generated data provides more valuable supervision signals for a system to understand the complexity of human language, which is reﬂected by the carefully chosen deleted words and candidate options.
To verify this assumption, we train a model on the following generated data while keeping the amount of training data the same:
• Random-options: We replace the candidate options picked by teachers with random words sampled by the unigram distribution.
• Random-blanks: We further replace the deleted words chosen by teachers with random words in the passage, while keeping the number of blanks the same.
The candidate options are also automatically generated.
We train an LSTM based supervised model with different training data, while keeping the same dev set and test set.
The comparison is shown in Table 7.
When trained with human-designed data, the accuracy is 0 .
If we replace the human-generated options with random options, the accuracy drops signiﬁcantly to 0 .
When blanks are also selected randomly, the overall performance
[Table]
further drops to 0 .
Hence, the deleted words and options designed by human provide more knowledge of the language.
Interestingly, it leads to better performance on CLOTH-M to train on “random-blanks” than to train on “random-options”.
It reﬂects the knowledge difference between middle school questions and high school questions, since middle school exams have more grammar questions involving simple words such as “in”, “at” and “on”.
We also ﬁnd that the training accuracy converges much faster when trained on automatically generated data, which might be due to easier questions.
4. C OMBINING H UMAN - DESIGNED D ATA WITH AUTOMATICALLY - GENERATED DATA
In Section 3.1, we show that language model trained on unlabeled data leads to much better performance.
At the same time, we also show the beneﬁts of employing human-designed informative data in Section 4.1.
Motivated by the belief that advantages of high quality and large quantity are usually complementary, we combine these two types of data to achieve better performance.
Notice that discriminative models can also take advantage of unlabeled data just like a language model.
Speciﬁcally, every word in the passage can be a regarded as a question given the corresponding context.
The bidirectional context representation at each word can be obtained with just one pass of the passage (Please see the Appendix A.2 for more details).
We study two methods of leveraging unlabeled data and human-designed data:
Equally averaging Let J h be the average loss for all human-designed questions
and J u be the average loss for all questions that are generated by all words in the passage.
A simple method is to optimize J h + λJ u so that the model learn to predict words deleted by human and all other words in the passage. This model treats each question as equally important.
We set λ to 1 in our experiments.
Informativeness-based weighted averaging A possible avenue towards having large-scale highquality data is to automatically pick out informative questions in a large corpus.
With the belief that human-designed data is informative, the informativeness prediction network is trained to mimic the design behavior of language teachers.
The performance of informativeness prediction network and an example is shown in Appendix A.3.
Let J i denote the negative log likelihood loss for the i − th question and let l i be the outputted informativeness of the i -th question (The detailed deﬁnition of l i is in Appendix A.2).
We utilize the informativeness of questions in a soft way.
The informativeness weighted loss function is deﬁned as J f = (cid:80) i (cid:54)∈ H Softmax i ( l 1 α , · · · , l n α )
J i where H is the set of all human-generated questions and α is the temperature of the Softmax function.
Intuitively, the weighted loss leads to stronger gradients for informative questions.
We present the results in Table 8.
When all other words are treated as equally important, the accuracy is 0 .
543 , similar to the performance of language model.
Informativeness-based weighted averaging leads to an accuracy of 0 . 565 , better than 0 .
543 achieved by equally averaging.
When combined with human-designed data, the performance can be improved to 0 .
4. D IFFICULTY C OMPARISON
Lastly, we verify if human-designed data is more difﬁcult compared to automatically generated data.
We employ the same generation process used in Section 4.1 to replace options and blanks and test the best model’s performance.
As shown in Table 9, On automatically generated questions, our model achieves an accuracy of 0 .
808 and 0 . 812 , while its accuracy is 0 . 583 on human-designed questions, which shows that automatically generated questions are much easier.
[Table] [Table]
R ELATED W ORK
Large-scale automatically generated cloze test (Hermann et al., 2015; Hill et al., 2016; Onishi et al., 2016) leaded to signiﬁcant research advancement.
However, the generated questions do not consider the language phenomenon to be tested and are relatively easy to solve.
Recently proposed reading comprehension datasets are all labeled by human to ensure their quality (Rajpurkar et al., 2016;
Joshi et al., 2017; Trischler et al., 2016; Nguyen et al., 2016).
Aiming to evaluate machines under the same conditions human is evaluated, there are more and more interests in obtaining data from examinations.
NTCIR QA Lab (Shibuki et al., 2014) contains a set of real-world university entrance exam questions.
The Entrance Exams task at CLEF QA Track
(Pe ˜ nas et al., 2014; Rodrigo et al., 2015) evaluates machine’s reading comprehension ability.
The AI2 Elementary School Science Questions dataset 6 provides 5 , 060 scientiﬁc questions used in elementary and middle schools.
Lai et al.
(2017) proposes the ﬁrst large-scale machine comprehension dataset obtained from exams.
They show that questions designed by teachers have a signiﬁcant larger proportion of reasoning questions.
Our dataset focuses on evaluating language proﬁciency while the focus of reading comprehension is reasoning.
C ONCLUSION
In this paper, we propose a large-scale cloze test dataset CLOTH that is designed by teachers.
We show that CLOTH better captures the complexity of human language than automatically designed data, since the deleted words and candidate options are carefully selected by teachers.
We ﬁnd that human outperforms state-of-the-art language model by a signiﬁcant margin, indicating that
CLOTH is a challenging dataset for language understanding.
After detailed analysis, we ﬁnd that the performance gap is due to model’s inability to perform long-term reasoning.
We also verify that human-designed questions are more informative and more difﬁcult comparing to automaticallygenerated questions.
A A PPENDIX A. I MPLEMENTATION D ETAILS
We implement our models by PyTorch 7 .
The code of language model is adapted from PyTorch example projects 8 .
We use Adam (Kingma & Ba, 2014) with the learning rate of 0 .
The hidden dimension is set to 650 and we initialize the word embedding by 300 -dimensional Glove word vector (Pennington et al., 2014).
The temperature α is set to 2 .
We train our model on the combined dataset in the middle school and high school group and test it on two groups separately.
The code will be made public after cleaning.
A. I NFORMATIVENESS G UIDED S UPERVISED M ODEL
Let x denote the passage and z denote whether a word is selected as a question by human, i.e., z is 1 if this word is selected to be ﬁlled in the original passage or 0 otherwise.
Suppose h i is the representation of i -th word given by a bidirectional LSTM.
The network computes the probability of x i being a question in the cloze test as follows:
[Equation]
where l i is the logit or unnormalized energy, indicating whether this word is likely to be a problem selected by human instructors.
We train the network to minimize the binary cross entropy between p and ground-truth labels at each token.
Another network is designed to leverage both human-designed data and all other words.
Similarly, the passage x is encoded by a bidirectional LSTM.
Let s i be the context representation at i -th word.
To mask the word i , h i is deﬁned as [ −→ h i − 1 , ←− h i +
] T where −→ h
i is the hidden representation of forward LSTM and ←− h i is of backward LSTM.
Suppose there are m candidate words w i,j at position i , we denote the cross entropy loss J i between the model prediction q i and the ground-truth
y i as
[Equation]
where e w j is the embedding of the word w j and W is a weight matrix.
In human-designed questions, there is a list of 4 candidate answers for each question.
For all other words, the candidate answers include the whole vocabulary, in which case m is equal to the vocabulary size.
A. T HE I NFORMATIVENESS P REDICTION N ETWORK
Our prediction model achieves an F1 score of 36 . 5 on the test set, which is understandable since there are many plausible questions within a passage.
A predicted sample is shown in Figure 1.
Clearly, words that are too obvious have low scores, such as punctuation marks, simple words ”a” and ”the”.
In contrast, content words whose semantics are directly related to the context have a higher score, e.g., ”same”, ”similar”, ”difference” have a high score when the difference between diary and blog is discussed and ”secrets” has a high score since the subsequent sentence is ”does not want to share with others”.