Paper:
Cloze test is widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-designed cloze test dataset CLOTH 1 , in which the questions were used in middle-school and high-school language exams. With the missing blanks carefully created by teachers and candidate choices purposely designed to be confusing, CLOTH requires a deeper language understanding and a wider attention span than previous automatically generated cloze datasets. We show humans outperform dedicated designed baseline models by a significant margin, even when the model is trained on sufficiently large external data. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending a long-term context to be the key bottleneck. In addition, we find that human-designed data leads to a larger gap between the model's performance and human performance when compared to automatically generated data.

Review:
This paper presents a new dataset for cloze style question-answering. The paper starts with a very valid premise that many of the automatically generated cloze datasets for testing reading comprehension suffer from many shortcomings. The paper collects data from a novel source: reading comprehension data for English exams in China. The authors collect data for middle school and high school exams and clean it to obtain passages and corresponding questions and candidate answers for each question.
The rest of the paper is about analyzing this data and performance of various models on this dataset. 
1) The authors divide the questions into various types based on the type of reasoning needed to answer the question, noticeably short-term reasoning and long-term reasoning. 
2) The authors then show that human performance on this dataset is much higher than the performance of LSTM-based and language model-based baselines; this is in contrast to existing cloze style datasets where neural models achieve close to human performance. 
3) The authors hypothesize that this is partially explained by the fact that neural models do not make use of long-distance information. The authors verify their claim by running human eval where they show annotators only 1 sentence near the empty slot and find that the human performance is basically matched by a language model trained on 1 billion words. This part is very cool.
4) The authors then hypothesize that human-generated data provides more information. They even train an informativeness prediction network to (re-)weight randomly generated examples which can then be used to train a reading comprehension model.
Pros of this work:
1) This work contributes a nice dataset that addresses a real problem faced by automatically generated datasets.
2) The breakdown of characteristics of questions is quite nice as well.
3) The paper is clear, well-written, and is easy to read.
Cons:
1) Overall, some of the claims made by the paper are not fully supported by the experiments. E.g., the paper claims that neural approaches are much worse than humans on CLOTH data -- however, they do not use state-of-the-art neural reading comprehension techniques but only a standard LSTM baseline. It might be the case that the best available neural techniques are still much worse than humans on CLOTH data, but that remains to be seen. 
2) Informativeness prediction: The authors claim that the human-generated data provides more information than automatically/randomly generated data by showing that the models trained on the former achieve better performance than the latter on test data generated by humans. The claim here is problematic for two reasons:
   a) The notion of "informativeness" is not clearly defined. What does it mean here exactly?
   b) The claim does not seem fully justified by the experiments -- the results could just as well be explained by distributional mismatch without appealing to the amount of information per se. The authors should show comparisons when evaluating on randomly generated data.
Overall, this paper contributes a useful dataset; the analysis can be improved in some places.