Extract, Integrate, Compete: Towards Verification Style Reading Comprehension (2021)

Abstract: In this paper, we present a new verification style reading comprehension dataset named VGaokao from Chinese Language tests of Gaokao. Different from existing efforts, the new dataset is originally designed for native speakers’ evaluation, thus requiring more advanced language understanding skills. To address the challenges in VGaokao, we propose a novel Extract-Integrate-Compete approach, which iteratively selects complementary evidence with a novel query updating mechanism and adaptively distills supportive evidence, followed by a pairwise competition to push models to learn the subtle difference among similar text pieces. Experiments show that our methods outperform various baselines on VGaokao with retrieved complementary evidence, while having the merits of efficiency and explainability. Our dataset and code are released for further research.

## introduction reading comprehension has been frequently used in various standardized examinations to evaluate one's language understanding skills, where testtakers are expected to read a long article, answer a series of questions, or verify a given statement according to the passage. for example, in the chinese language tests of gaokao (also known as china national college entrance examination), approximately half of the reading comprehension questions are in a verification style. as shown in while the question answering styled tasks have been intensively studied in the nlp community @xcite @xcite , the verification styled query passage geographic elements are marked with map symbols, enriching the readability of the map. ## vgaokao: verification style reading comprehension dataset standardized language tests have been considered as a test-bed to harvest machine reading comprehension datasets. while most existing efforts focus on squad-like qa datasets @xcite , or cloze style questions @xcite , few efforts are made to verification style questions. in the chinese language test of gaokao, approximately half of the reading comprehension questions instruct students to select a statement (i.e., an option from four choices) that is the most consistent or contradicting with the given passage. these questions are designed to evaluate students' ability in extracting and integrating information from long passages, and analyzing certain linguistic phenomena or semantic relations among several similar sentences 3 . according to the target language skills, we call these questions verification style questions and convert them into the task of verifying given statements against the articles. our task is similar in spirit to the fact checking task, fever @xcite . but, unlike the claims tagged with a definite label in fever (supported, refuted or not-enough-info), the statements in gaokao questions are designed to be not absolutely correct nor absolutely incorrect. in other words, the options of a question in vgaokao differ in how much they are supported by the passage, which is difficult to be quantified with a value between 0 and 1. the design of relative correctness is in the purpose of evaluating test-takers' ability to disambiguate very similar options through considering the subtle difference among them. in this work, we construct a verification style reading comprehension dataset named vgaokao from the chinese language test of gaokao to as-sess a model's skills of extracting evidence from articles and verifying statements against the retrieved evidence pieces and other options. ## our extract-integrate-compete approach we propose a novel extract-integrate-compete approach to address the challenges in vgaokao. as illustrated in figure 2 , our approach includes three stages, iterative evidence extraction, adaptive evi-dence integration and pairwise option competition. ## experiments we conduct experiments on our proposed vgaokao dataset and compare our extract-integrate-compete approach with several baselines. roberta-large-chunk @xcite ) is an end-to-end method without explicit evidence retrieval. this model splits the long passages into fix-length chunks of 200 tokens. candidate answers are obtained from each chunk using existing mrc models, which are further aggregated over all chunks. we use the pre-trained chinese robertawwm-ext-large @xcite . bm25 @xcite is a bagof-words retrieval method, which uses sparse features to retrieve evidence sentences. we use the version implemented by pyserini foot_4 . sent-bert (reimers and gurevych, 2019) uses bert to obtain contextualized dense representations for the texts and retrieve evidence sentences via cosine similarity. beamdr @xcite ) is an iterative evidence selection technique with beam search and dense retrieval. it updates the query by appending the newly-extracted evidence in each iteration. the bm25, sent-bert, and beamdr are evidence extraction methods, which are combined with our proposed pair-wise option competition method to obtain the final question-level results. for bm25 and sent-bert, which cannot address the problem of multiple evidence, we report their ## related works several previous datasets are constructed from different subjects of gaokao, including geography @xcite , english @xcite , and history @xcite . different from these efforts, we focus on chinese subject to introduce native-speaker level reading comprehension challenges. evidence extraction based methods have been studied in mrc @xcite and fact verification @xcite . in this work, we introduce evidence extraction to our method for solving the long article challenge in vgaokao. iterative evidence extraction can be seen as a sort of question decomposition method, a technique widely used in qa tasks with complex questions @xcite . however, in vgaokao, the queries may interweave by implicit semantic relationship, so that models could not explicitly separate the queries into independent sub-queries. we thus adopt an iteative extractor with an adaptive integrator to decompose queries in an implicit way. another stream of works adopt an iterative framework by updating the queries. @xcite iteratively generate new queries by selecting a span from the question and retrieved evidence while @xcite ; @xcite ; @xcite directly append retrieved evidence to the query. compared with these works, we introduce two novel query updating techniques, hard masking and soft masking, together with an evidence integration module to avoid too much overlap between evidence and to dynamically determine the number of required evidence sentences. ## conclusion in this paper, we present a novel verification style reading comprehension dataset named vgaokao from the chinese language tests of gaokao for chinese native speakers, which embed multiple advanced language understanding skills. to address the challenges in vgaokao, we propose a new extract-integrate-compare approach for complementary evidence retrieval/integration and option discrimination. experiments show that our approach outperforms several strong baselines, with additional merits of efficiency and explainability. we believe vgaokao is a challenging test-bed for natural language understanding in chinese and encourage further research in verification style reading comprehensionn.