Reading Comprehension as Natural Language Inference:A Semantic Analysis (2020)

Abstract: In the recent past, Natural language Inference (NLI) has gained significant attention, particularly given its promise for downstream NLP tasks. However, its true impact is limited and has not been well studied. Therefore, in this paper, we explore the utility of NLI for one of the most prominent downstream tasks, viz. Question Answering (QA). We transform one of the largest available MRC dataset (RACE) to an NLI form, and compare the performances of a state-of-the-art model (RoBERTa) on both these forms. We propose new characterizations of questions, and evaluate the performance of QA and NLI models on these categories. We highlight clear categories for which the model is able to perform better when the data is presented in a coherent entailment form, and a structured question-answer concatenation form, respectively.

## introduction given two sentences, a premise and a hypothesis, the task of natural language inference (nli) is to determine whether the premise entails the hypothesis or not. † the concept of semantic entailment is central to natural language understanding @xcite and therefore, nli models have been used to help with various downstream tasks like reading comprehension @xcite , summarization @xcite , and dialog systems @xcite . however, the performance of an nli system on these down-stream tasks has not been studied with respect to semantic or reasoning categories. in this work, we use nli to perform the task of multiple choice reading comprehension @xcite . we analyse the performance of an nli model on this task through the lens of semantics by identifying the reasoning categories (type of questions) where it is beneficial to use an nli model. drawing inspiration from the prior work in the area @xcite @xcite , we use rule-based conversion to create an nli version of the largest available rc dataset -race @xcite . we train a roberta based rc model on the original dataset, and a similar roberta based nli model on the nli version of the dataset. we evaluate and analyse the performance of both these models by characterizing the question types that are better suited for an nli model and a qa model. ## related work reading comprehension (rc) is one of many potential downstream tasks that can benefit from nli @xcite . it is easy to see that rc naturally reduces to a two-class nli problem; specifically, it can be cast as the task of identifying if a given piece of text entails the statement formed by converting a question and a potential answer to an assertive statement (hypothesis). given the intuitive conversion between rc and nli, @xcite designed both a rulebased conversion system as well as a trained neural model to convert question-answering datasets such as squad @xcite and race @xcite to an nli form. however, it is unclear what these converted nli datasets offer compared to the original question-answering datasets w.r.t semantics. we show that converting a rc task to an nli task helps in answering certain types of questions. this establishes the usefulness of the converted datasets. @xcite show that despite the different form of nli and qa tasks, performing coarse pretraining of models on nli datasets like snli @xcite and @xcite not only improves the performance of these models on downstream reading comprehension tasks, but also helps with faster convergence. we show that -for certain types of questions in reading comprehension datasets -simply transforming the task to nli can show improvement in performance, even without pretraining on any nli dataset. @xcite introduced a learnt weightand-combine architecture to effectively re-purpose pretrained entailment models (trained on snli and multinli) to solve the task of multi-hop reading comprehension. they show that for certain datasets, this strategy can produce good results. however, their study focuses mainly on improving model performance using a pre-trained nli model, and lacks an analysis of the reasoning differences arising due to the different form of the nli and qa tasks. we focus our analysis on this aspect. ## nli for reading comprehension this section describes our experimental setup for comparing a qa based approach and an nli based approach for the task of reading comprehension. we first obtain a parallel nli and qa dataset by converting existing rc dataset into an nli dataset. we then train two models, one on each form of the data, and analyse their performance. ## analysis in order to analyse the performance difference from a semantic perspective, we characterize question into 7 semantic categories by identifying the kind of reasoning required to answer the question. ## conclusion there is limited work providing a comprehensive analysis of how nli can be used for qa. in our work, we show that nli can be used for the task of reading comprehension simply by converting the data into nli form. we convert a large rc dataset into nli form and perform a comparative study of the performance of the roberta model trained on qa and nli settings. we propose a categorization of questions that allows for effective comparison of models trained on nli and qa forms of data. our analysis clearly shows that using the nli-based approach is at par with a qa-based approach for most reasoning categories, and it is even better for some. specifically, we find that questions involving deductive reasoning, dialogue interpretation and math are better handled by a model trained on the nli form of data than the qa form. however, questions involving negation favor the qa form. our work allows for careful selection of modeling strategy based on the type of data at hand. sean welleck, jason weston, arthur szlam, and kyunghyun cho. 2019. dialogue natural language inference. in acl. adina williams, nikita nangia, and samuel bowman. 2018. a broad-coverage challenge corpus for sentence understanding through inference. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1112-1122. association for computational linguistics.