Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters (2022)

Abstract: Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. While early work identified certain biases in NLI models, recent advancements in modeling and datasets demonstrated promising performance.In this work, we further explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on. First, we analyze the robustness of these models to longer and out-of-domain inputs. Then, we develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset. Interestingly, we find NLI scores to provide strong retrieval signals, leading to more relevant evidence extractions compared to common similarity-based methods. Finally, we go further and investigate whole document clusters to identify both discrepancies and consensus among sources. In a test case, we find real inconsistencies between Wikipedia pages in different languages about the same topic.

## introduction natural language inference (nli) involves automatically determining whether the meaning of one piece of text (i.e., hypothesis) can be inferred from another (i.e., the premise) @xcite . this formulation is relatively simple, enabling large-scale data annotation (e.g., @xcite , yet imposes complex semantic reasoning challenges (e.g., background knowledge, commonsense), leading to a useful training and evaluation nlp framework @xcite @xcite . formally, in nli, we are interested in learning a function f : (x hyp × x prem ) → y that pre-dicts the relation y between the provided premise x prem and the examined hypothesis x hyp , where @xmath0, neutral, contradiction}. 2 in most nli datasets, x hyp and x prem are short texts consisting of one or few sentences, allowing current large language models (llms) with limited input length to process the two with cross-attention. in practice, however, many systems require operating over long texts such as full documents or even collections of documents without knowing a priori which parts are most relevant. consider the example in figure 1 . the system is trying to reason over a collection of documents and find statements that they all agree upon (consesnus), or alternatively, find potential disagreements across documents. instead of having a clear hypothesispremise pair, each statement across all documents is an hypothesis of interest that should be evaluated against all other documents as the premise. in this paper, we focus on these realistic scenarios and present retrieve-and-classify methods for inferring over long and out-of-distribution (ood) inputs in a zero-shot fashion. as this setting emphasizes the need for a robust sentence-pair nli model as a backbone, we train on multiple datasets, including adversarial and contrastive ones @xcite to increase the model's robustness and avoid dataset-specific biases @xcite @xcite . our proposed pipelines go beyond the length and format supported by most lms and include new retrieval, aggregation and classification solutionsall based on the same classifier. thereby, we continue the line of work on evaluating the robustness of such models and their ability to truly capture semantic relations. moreover, long inputs highlight the commonly overlooked yet practically important ## model and definitions our pipelines build on entailment scores for hypothesis-premise pairs. to predict these scores, we train a sentence-pair nli model that we use as the backbone for all methods. specifically, we pick the t5 encoder-decoder architecture as it has been shown to perform well in multi-task and transfer settings @xcite . see appendix a for more technical details. definitions. as t5 is a seq-to-seq model, we train it over the training set (x hyp , x prem , y ) ∼ d train by feeding the following format to the encoder: "entailment: x hyp [sep] x prem ." the decoder's goal is to generate a single character 'e', 'n', or 'c', representing the three classes in y: entailment, neutral, and contradiction, respectively. thereafter, when making a prediction on a new pair (x hyp , x prem ) ∈ d test , we encode the input and measure the decoder's score s y for each of the three classes. finally, we normalize the three scores with a softmax operator: p @xmath1, s n , s c ) @xcite . note that these scores should not be directly treated as class probabilities as they are not calibrated, especially when evaluating out-of-domain inputs. however, as we will show in our experiments, they can be readily tuned or leveraged as a valuable signal for textual semantic relations. the sentli model. to train our model, we use the following sentence-pair datasets: snli (bowman et al., 2015), mnli @xcite , anli @xcite , fever @xcite , and vitaminc @xcite . for fever, we use the sentence-pair version from vi-taminc with retrieved evidence to neutral claims. this multi-task training is important for improving the robustness of the model by leveraging a large amount of diverse data. also, including adversarial (anli) and contrastive (vitaminc) examples was shown to prevent the model from relying on hypothesis-only biases. we denote our t5 mutli-task nli model trained on all 5 datasets sentli. our application-specific pipelines will differ in the way that they make use of sentli's predictions. we also experiment with models trained only on mnli without multitasking (-m.t), only on the three *nli datasets (-f.v), or with contractnli @xcite examples. ## beyond sentence-level inference we now assume that our target application requires the evaluation of hypotheses against long texts. we also assume that we have limited or no training data for this domain, and focus on zero-shot transfer. formally, we assume that x prem is a document consisting of n sentences s 1:n and we don't know which part of the document is most relevant for verifying or rejecting x hyp . in this case, it is common to not only classify the truthfulness of the given statement, but to also point to the exact evidence in the document that led to this conclusion. this is a crucial requirement, both benefiting the interpretability and trustworthiness of the model (e.g., avoiding hypothesis-only bias), and saving human time needed for manual prediction verification. ## evaluation tasks and datasets we evaluate our methods on the following 3 benchmarks that contain 9 datasets from different domains. in addition, we create a new of its kind dataset with clusters of related documents ( §4.1). the statistics are summarized in docnli uses only two classes, "entail" vs. "not entail". we discuss different zero-shot conversion techniques from 3-way models to binary classifica-tion in appendix b. contractnli @xcite has nli examples in the legal domain. each hypothesis is short and focused, but the premise is a long document (80 sentences on average). a model with input limit of 512 tokens performs poorly here. summac @xcite is a benchmark for predicting the factual consistency of summaries with their source. we follow the zero-shot setting here, but for fair comparison with @xcite , we also tune the threshold on the validation set for each dataset, and report the results on the test set. in early exploration we found naive threshold settings to be competitive as well. ## experiments we evaluate our inference pipelines against supervised models from docnli and contractnli, and the zero-shot summac model, adopting the main evaluation metrics from each paper. we also train t5-large supervised models to directly compare with sentli's zero-shot performance. docnli @xcite used a robertalarge @xcite model with input limit of 512 tokens and a longformer-base @xcite model. contractnli introduced the spannli @xcite model to process long documents that they train to jointly identify key spans and to make the final verdict. summac @xcite used a bertlarge @xcite model that was also trained on multiple nli datasets. for short hypotheses, the performance stays high even beyond the training distribution, but sharply drops around the input length limit of the model. ## related work as mentioned in the introduction, the nli task @xcite , sometimes called recognizing textual entailment (rte), was extensively studied by the nlp community over the past several years as a semantic reasoning benchmark (see @xcite . the field of fact verification @xcite also recently gained increased attention @xcite @xcite , sharing similar pair-wise semantic inference challenges, together with evidence retrieval. while both tasks were found to be vulnerable to idiosyncrasies @xcite @xcite , methods and datasets for reducing the bias were proposed @xcite @xcite @xcite . recently, nli-style models were expanded for concrete purposes beyond benchmarking. for example, showing promising potential in verifying the factual correctness of dialog @xcite , summarization @xcite @xcite , and qa @xcite @xcite systems. the nli format was also found helpful for general selftraining @xcite . here, we focus on realworld direct applications such as automatic con-tract analysis @xcite and identifying discrepancies in document collections. in parallel work, @xcite improve nli models for evaluating summaries by generating in-domain data with automatic perturbations to simulate contradictions. we observe similar improvements when training the backbone nli models on in-domain data for contractnli (supervised vs. zero-shot setting). any additional improvements to the nli model, aimed towards the target domain, are likely to further improve the reasoning capabilities of the retrieve-and-classify pipeline. ## conclusion we present a comprehensive study on the performance of sentence-pair nli models in real world applications that often involve both a shift in domain and long texts. our findings indicate the readiness of these models to provide meaningful signal on the semantic relation between texts that can be easily aggregated towards practical gains. to demonstrate this, we also defined a new zero-shot entailment-focused task over clusters of related documents. our multi-task sentence-pair nli model (sentli) successfully flags spans that stand out due to their claims. ultimately, this study should help practitioners interested in applying nli-style inference in realworld applications to design the best model for their target task. our results suggest that if the hypothesis is short and the premise fully fits in the input limit of the model, a regular cross-attention classifier is likely to perform well in terms of accuracy as it is able to contextualize sentences in the premise. however, if we want to interpret the prediction by identifying the exact piece from the premise that led to the predicted conclusion, breaking the premise into segments could be useful. furthermore, if the premise is too long to fit in the model's receptive field, then breaking the premise into segments and aggregating with our proposed techniques (retrieve-and-classify and reranking) is beneficial. finally, when we don't have a well defined hypothesis, one might still want to automatically reason over a pair or collection of documents and identify any statements that stand out. in this case, our reasoning over clusters methodology shows how to use strong sentence-pair classifiers to obtain useful signals that are then aggregated to highlight specific claims. it's important to note that when designing the methods for this work, we preferred simplicity over performance in order to directly study the quality of the sentli's scores. yet, we achieve high zeroshot performance and even reach state-of-the-art on contractnli. we hope that this work will motivate future research on further expanding these methods, for example by decontextualizing the premise, supporting multi-hop reasoning, expanding the context with sliding windows instead of sentence splitting, or hypothesis fragmentation.