Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization (2021)

Abstract: Natural Language Inference (NLI) has garnered significant attention in recent years; however, the promise of applying NLI breakthroughs to other downstream NLP tasks has remained unfulfilled. In this work, we use the multiple-choice reading comprehension (MCRC) and checking factual correctness of textual summarization (CFCS) tasks to investigate potential reasons for this. Our findings show that: (1) the relatively shorter length of premises in traditional NLI datasets is the primary challenge prohibiting usage in downstream applications (which do better with longer contexts); (2) this challenge can be addressed by automatically converting resource-rich reading comprehension datasets into longer-premise NLI datasets; and (3) models trained on the converted, longer-premise datasets outperform those trained using short-premise traditional NLI datasets on downstream tasks primarily due to the difference in premise lengths.

## introduction large-scale, open natural language inference (nli) datasets @xcite have catalyzed the recent development of nli models that exhibit close to human-level performance. however, the use of these nli models for other downstream natural language processing (nlp) tasks has met with limited success. two of the most popular downstream tasks where nli models' use has been explored are multiple-choice question answering (mcrc) and checking factual correctness of summaries (cfcs) @xcite @xcite -both of which can easily be cast into the nli form, as shown in figure 1 . looking closely at the composition of these datasets, it is evident that there is a stark difference in the lengths of the contexts/premises when compared to nli datasets. as seen in ## related work performance on the nli task has improved significantly due to the availability of large scale datasets @xcite that can be used to train data-hungry deep learning models @xcite , including transformer-based architectures @xcite . however, there has been very limited success in translating this performance to downstream nlp tasks. work relevant to the use of these nli models for downstream tasks can be categorized into two categories: (1) work focusing on using models trained on shortpremise nli datasets with fixed or learned aggregations over segmented premises to perform a target downstream task with long contexts @xcite ; and (2) work addressing the need for task-specific nli datasets @xcite @xcite . despite several attempts, efforts to apply models trained on available nli datasets to downstream nlp tasks such as mcrc and cfcs have had limited success. @xcite use hand-crafted rules to first cast mcrc to nli; and subsequently divide a long passage into smaller sentence-level premises. they use a pre-trained nli model to evaluate per-sentence relevance scores concerning one particular hypothesis, and then combine the resulting scores using a learned representation aggregation module to assess the answer given the long passage. @xcite apply a similar approach for the cfcs task, and divide both the provided summary as well as the source documents into single-sentence premises and hypotheses. they use a max pooling operation over the entailment scores of all sentence-level premise-hypothesis pairs to obtain the factual correctness score for each provided summary. both these works note that models trained on sentence-level nli datasets do not transfer well to the mcrc and cfcs tasks. we argue that this divide and conquer approach is not ideal for the problem, and highlight the need for an nli dataset with longer premises. another line of research focuses on re-casting datasets from other tasks into an nli form to facilitate the direct use of nli models on downstream tasks like mcrc and cfcs. @xcite use manual annotation to re-cast sciq (a qa dataset) to scitail -an nli dataset. however, @xcite show that an nli model trained on sci-tail does not perform well on the task of mcrc. similarly, @xcite create an automatically generated training dataset for cfcs. even though the generated data has relatively long contexts, analysis in @xcite demonstrated that a model trained on the aforementioned data showed performance improvement only when the token overlap with the source is high. besides, @xcite derive an nli dataset by converting subsets of various qa datasets. they try two approaches for the conversion -rule-based and neural. for the rule-based approach, they extract pos tags from the question-answer pair and apply hand-crafted rules on them to convert the pair to a hypothesis sentence. their neural approach uses a trained seq2seq bilstm-with-copy model @xcite to convert each question, answer pair into a hypothesis sentence (the corresponding passage being the premise). while their approach looks promising, they do not show the utility of these converted datasets by training an nli model on them. thus, it remains unclear whether the nli datasets generated by the conversion are beneficial for nlp tasks. we posit that this direction of research is promising and largely unexplored. in our work, we attempt to leverage the abundance of large and diverse mcrc datasets to generate long-premise nli datasets, and show that such datasets are useful towards addressing downstream nlp tasks such as mcrc and cfcs which have inherently long contexts. ## nli for downstream tasks typically, nli is cast as a multi-class classification problem, where given a premise and a hypothesis, the model classifies the relation between them as entails, contradicts, or neutral. for the two downstream tasks under consideration: (1) mcrc: multiple choice reading comprehension, and (2) cfcs: checking factual correctness of text-summarization; differentiating between the neutral and contradicts class is often unnecessary. the task is thus reduced to a two-class problem; where the contradicts and neutral classes are clubbed into a not-entails class. mcrc can be cast as an nli task by viewing the given context as the premise and the transformed question-answer combinations as different hypotheses @xcite . the multiple answeroption setting can then be approached as: (a) an individual option entailment task, where more than one answer-option can be correct; or (b) a multiclass classification task across all the answer options, when only a single correct answer exists. cfcs can also be reduced to a two-class nli problem. a factually correct summary should be entailed by the given source text -it should not contain hallucinated facts, and it should also not contradict facts present in the source text. ## conversion of mcrc to nli as shown in figure 1 , we can convert mcrc datasets into two-class nli datasets by reusing the passage as a premise, and paraphrasing the question along with each answer option as individual hypothesis options. we begin by using a rule-based conversion method. a dependency parse of both the question and answer option is generated using the stanford corenlp package @xcite . this is followed by the application of conversion rules proposed by @xcite to generate a hypothesis sentence. however, due to the limited coverage of rules and errors in the dependency parse, some of the generated hypotheses sound unnatural (e.g. the first example in rule-based neural hybrid q: what building were the four captives inside on tuesday? a: cnn headquarters the four captives inside on tuesday were cnn headquarters. the four captives were inside cnn headquarters on tuesday. the four captives were inside cnn headquarters on tuesday. q: how many people were hurt when overhanging metalwork crashed onto a stage in a toronto park saturday afternoon. a: four four were hurt when overhanging metalwork crashed onto a stage in a toronto park saturday afternoon. four people were hurt when overhanging metalwork crashed onto a stage in a toronto park saturday afternoon.. "." # # # # "." was the number of peo four were hurt when overhanging metalwork crashed onto a stage in a toronto park saturday afternoon. due to the recent success of transformer-based text generation models, we train a bart @xcite model to generate a grammatically coherent hypothesis from question + answer option (word/phrase) as input. we use a sequence of datasets as a curriculum to finetune the bart conversion model: (1) starting with cnn/daily mail summarization dataset @xcite , which makes the generated sentences coherent; (2) followed by google's sentence compression dataset @xcite , which limits the generated sequence to a single sentence; and (3) finally the annotated dataset provided by @xcite which has around 71, 000 question-answer, hypothesis pairs from various qa datasets. based on manual inspection, we find that the hypotheses generated by this method indeed sound more natural and diverse than the ones produced by the rule-based conversion † . in some cases, however, the generated hypotheses either discard crucial information, or contain hallucinated facts that do not convey the exact information in the source questionanswer pair ( we design a heuristic to compose a hybrid dataset to overcome the caveats in the neural conversion. we use the number of words in the question-answer concatenation as a proxy for the expected length of the hypothesis. we target the problems of hallucination and missing information in the neural conversions by accepting only those neural-generated hypotheses that lie in the range of 0.8 and 1.2 times the length of the questionanswer concatenation. we replace the rejected neural hypotheses with the rule-based hypothesis, if rule-based conversion is feasible; or with the † more examples of conversion results are presented in appendix d. question-answer concatenation otherwise; as seen in ## experimental setup our experiments involve zero-shot evaluations of pre-trained nli models on downstream nlp tasks. in this section, we describe the transfer learning setup and the datasets used in our experiments. ## results and discussion our experiments aim to answer the following questions: (1) are long premise nli datasets more usefigure 3 : multiple choice classification model. we contrast the performance of nli models trained on the short-premise nli datasets (mnli, anli) with one that is trained on a long-premise nli dataset (race converted ). the models trained on short-premise nli datasets are evaluated in two ways: (1) by treating the entire premise as input; and (2) by segmenting the premise into shorter segments and using a max aggregation over the entailment scores of all the segments @xcite . since the model architecture remains the same, we use the name of the training dataset to refer to the model trained on it. ## conclusion the difficulty of transferring entailment (nli) knowledge to downstream nlp tasks can be largely attributed to the difference in data distributions, specifically the premise lengths. models trained on short-premise nli datasets are not very good at performing inference over longer texts, which is a central feature of important downstream tasks such as qa and text summarization. we leverage the abundance of large and diverse mcrc datasets and the ease of conversion from mcrc into the nli format to automatically and scalably create a long-premise nli dataset to test this long-premise conjecture. we show that the long-premise nature of the converted dataset indeed helps achieve better performance on the downstream tasks of mcrc and cfcs when compared against models trained on traditional short-premise nli datasets. we further discuss and eliminate possible confounding factors in our experiments to ensure the validity of our results. our work highlights a major shortcoming in popular nli datasets that limits their usefulness to downstream nlp applications; and emphasizes the need for long-premise nli datasets. future work in this direction can take us closer to realizing the full potential of nli as a fundamental task in natural language understanding.