Investigating Prior Knowledge for Challenging C hinese Machine Reading Comprehension (2020)

Abstract: Machine reading comprehension tasks require a machine reader to answer questions relevant to the given document. In this paper, we present the first free-form multiple-Choice Chinese machine reading Comprehension dataset (C3), containing 13,369 documents (dialogues or more formally written mixed-genre texts) and their associated 19,577 multiple-choice free-form questions collected from Chinese-as-a-second-language examinations. We present a comprehensive analysis of the prior knowledge (i.e., linguistic, domain-specific, and general world knowledge) needed for these real-world problems. We implement rule-based and popular neural methods and find that there is still a significant performance gap between the best performing model (68.5%) and human readers (96.0%), especiallyon problems that require prior knowledge. We further study the effects of distractor plausibility and data augmentation based on translated relevant datasets for English on model performance. We expect C3 to present great challenges to existing systems as answering 86.8% of questions requires both knowledge within and beyond the accompanying document, and we hope that C3 can serve as a platform to study how to leverage various kinds of prior knowledge to better understand a given written or orally oriented text. C3 is available at https://dataset.org/c3/ .

## introduction machine reading comprehension (mrc) tasks have attracted substantial attention from both academia and industry. these tasks require a machine reader to answer questions relevant to a given document provided as input @xcite . in this paper, we focus on free-form multiple-choice mrc tasks-given a document, select the correct answer option from all options associated with a freeform question, which is not limited to a single question type such as cloze-style questions formed by removing a span or a sentence in a text @xcite @xcite @xcite or close-ended questions that can be answered with a minimal answer (e.g., yes or no; @xcite . researchers have developed a variety of freeform multiple-choice mrc datasets that contain a significant percentage of questions focusing on the implicitly expressed facts, events, opinions, or emotions in the given text @xcite @xcite @xcite . generally, we require the integration of our own prior knowledge and the information presented in the given text to answer these questions, posing new challenges for mrc systems. however, until recently, progress in the development of techniques for addressing this kind of mrc task for chinese has lagged behind their english counterparts. a primary reason is that most previous work focuses on constructing mrc datasets for chinese in which most answers are either spans @xcite @xcite or abstractive texts @xcite merely based on the information explicitly expressed in the provided text. with a goal of developing similarly challenging, but free-form multiple-choice datasets, and promoting the development of mrc techniques for chinese, we introduce the first free-form multiple-choice chinese machine reading comprehension dataset (c 3 ) that not only contains multiple types of questions but also requires both the information in the given document and prior knowledge to answer questions. in particular, for assessing model generalizability across different domains, c 3 includes a dialogue-based task c 3 d in which the given document is a dialogue, and a mixed-genre task c 3 m in which the given document is a mixed-genre text that is relatively formally written. all problems are collected from real-world chinese-as-a-secondlanguage examinations carefully designed by experts to test the reading comprehension abilities of language learners of chinese. we perform an in-depth analysis of what kinds of prior knowledge are needed for answering questions correctly in c 3 and two representative freeform multiple-choice mrc datasets for english @xcite , and to what extent. we find that solving these general-domain problems requires linguistic knowledge, domainspecific knowledge, and general world knowledge, the latter of which can be further broken down into eight types such as arithmetic, connotation, cause-effect, and implication. these freeform mrc datasets exhibit similar characteristics in that (i) they contain a high percentage (e.g., 86.8% in c 3 ) of questions requiring knowledge gained from the accompanying document as well as at least one type of prior knowledge and (ii) regardless of language, dialogue-based mrc tasks tend to require more general world knowledge and less linguistic knowledge compared with tasks accompanied with relatively formally written texts. specifically, compared with existing mrc datasets for chinese @xcite ), c 3 requires more general world knowledge (57.3% of questions) to arrive at the correct answer options. we implement rule-based and popular neural approaches to the mrc task and find that there is still a significant performance gap between the best-performing model (68.5%) and human readers (96.0%), especially on problems that require prior knowledge. we find that the existence of wrong answer options that highly superficially match the given text plays a critical role in increasing the difficulty level of questions and the demand for prior knowledge. furthermore, additionally introducing 94k training instances based on translated free-form multiple-choice datasets for english can only lead to a 4.6% improvement in accuracy, still far from closing the gap to human performance. our hope is that c 3 can serve as a platform for researchers interested in studying how to leverage different types of prior knowledge for in-depth text comprehension and facilitate future work on crosslingual and multilingual machine reading comprehension. english. much of the early mrc work focuses on designing questions whose answers are spans from the given documents @xcite @xcite @xcite . as a question and its answer are usually in the same sentence, stateof-the-art methods @xcite have outperformed human performance on many such tasks. to increase task difficulty, researchers have explored a number of options including adding unanswerable @xcite or conversational @xcite questions that might require reasoning @xcite , and designing abstractive answers @xcite @xcite or (question, answer) pairs that involve cross-sentence or crossdocument content @xcite . in general, most questions concern the facts that are explicitly expressed in the text, making these tasks possible to measure the level of fundamental reading skills of machine readers. another research line has studied mrc tasks, usually in a free-form multiple-choice form, containing a significant percentage of questions that focus on the understanding of the implicitly expressed facts, events, opinions, or emotions in the given text @xcite @xcite @xcite . therefore, these benchmarks may allow a relatively comprehensive evaluation of different reading skills and require a machine reader to integrate prior knowledge with information presented in a text. in particular, real-world language exams are ideal sources for constructing this kind of mrc dataset as they are designed with a similar goal of measuring different reading comprehension abilities of human language learners primarily based on a given text. representative datasets in this category include race @xcite and dream @xcite , both collected from english-asa-foreign-language exams designed for chinese learners of english. c 3 m and c 3 d can be regarded as a chinese counterpart of race and dream, respectively, and we will discuss their similarities in detail in section 3.3. ## data in this section, we describe the construction of c 3 (section 3.1). we also analyze the data (section 3.2) and the types of prior knowledge needed for the mrc tasks (section 3.3). ## approaches we implement a classical rule-based method and recent state-of-the-art neural models. ## conclusion we present the first free-form multiple-choice chinese machine reading comprehension dataset (c 3 ), collected from real-world language exams, requiring linguistic, domain-specific, or general world knowledge to answer questions based on the given written or orally oriented texts. we study the prior knowledge needed in this challenging machine reading comprehension dataset and carefully investigate the impacts of distractor plausibility and data augmentation (based on similar resources for english) on the performance of state-of-the-art neural models. experimental results demonstrate the there is still a significant performance gap between the best-performing model (68.5%) and human readers (96.0%) and a need for better ways for exploiting rich resources in other languages.