A Sentence Cloze Dataset for C hinese Machine Reading Comprehension (2020)

Abstract: Owing to the continuous efforts by the Chinese NLP community, more and more Chinese machine reading comprehension datasets become available. To add diversity in this area, in this paper, we propose a new task called Sentence Cloze-style Machine Reading Comprehension (SC-MRC). The proposed task aims to fill the right candidate sentence into the passage that has several blanks. We built a Chinese dataset called CMRC 2019 to evaluate the difficulty of the SC-MRC task. Moreover, to add more difficulties, we also made fake candidates that are similar to the correct ones, which requires the machine to judge their correctness in the context. The proposed dataset contains over 100K blanks (questions) within over 10K passages, which was originated from Chinese narrative stories. To evaluate the dataset, we implement several baseline systems based on the pre-trained models, and the results show that the state-of-the-art model still underperforms human performance by a large margin. We release the dataset and baseline system to further facilitate our community. Resources available through https://github.com/ymcui/cmrc2019

## introduction machine reading comprehension (mrc) is a task to comprehend given articles and answer the questions based on them, which is an important ability for artificial intelligence. the recent mrc research was originated from the cloze-style reading comprehension @xcite @xcite , which requires to fill in the blank with a word or named entity, and following works on these datasets have laid the foundations of this research @xcite @xcite . later on, squad @xcite was proposed, and the answer transformed from a single word to a span, which has become a representative span-extraction dataset and massive neural network approaches @xcite @xcite @xcite have been proposed which further accelerated the mrc research. besides the mrc in english text, we have also seen rapid progress on chinese mrc research. @xcite proposed the first chinese cloze-style reading comprehension dataset: people daily & children's fairy tale (pd&cft). later, @xcite proposed another dataset for cmrc 2017, which is gathered from children's reading books, consisting of both cloze and natural questions. @xcite al. (2018) though various efforts have been made, most of these datasets stop at token-level or span-level inference, which neglect the importance of long-range reasoning of the context. moreover, powerful pretrained models such as bert @xcite , xlnet @xcite , roberta @xcite [passage] a long time ago, there was a queen. [blank1] soon after the child was born, the queen died. figure 1: examples of the proposed cmrc 2019 dataset. the candidate with underline means it is a fake candidate (does not belong to any blank). for clarity, we also provide an english example. to further test the machine comprehension ability, in this paper, we propose a new task called sentence cloze-style machine reading comprehension (sc-mrc). the proposed task preserves the simplicity of cloze-style reading comprehension but requires sentence-level inference when filling the blanks. figure 1 shows an example of the proposed dataset. we conclude our contributions in three aspects. ## baseline system in this paper, we mainly adopt bert and its related variants for our baseline systems. ## conclusion we proposed a new task called sentence cloze-style machine reading comprehension (sc-mrc) and released a chinese dataset for this purpose. the proposed dataset contains both real and fake candidate sentences for filling the clozes, which not only requires the machine to choose the correct sentence but also distinguishes the real sentence from fake sentences. we built up baseline models based on the popular pre-trained language models, and the results show that the state-of-the-art models still underperform the human performance, especially on pac evaluation metric. we hope the release of this dataset could bring language diversity in machine reading comprehension tasks and accelerate further investigation on solving the questions that need comprehensive reasoning over multiple clues.