R 2 F : A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference (2022)

Abstract: Document-level natural language inference (DOCNLI) is a new challenging task in natural language processing, aiming at judging the entailment relationship between a pair of hypothesis and premise documents. Current datasets and baselines largely follow sentence-level settings, but fail to address the issues raised by longer documents. In this paper, we establish a general solution, named Retrieval, Reading and Fusion (R2F) framework, and a new setting, by analyzing the main challenges of DOCNLI: interpretability, long-range dependency, and cross-sentence inference. The basic idea of the framework is to simplify document-level task into a set of sentence-level tasks, and improve both performance and interpretability with the power of evidence. For each hypothesis sentence, the framework retrieves evidence sentences from the premise, and reads to estimate its credibility. Then the sentence-level results are fused to judge the relationship between the documents. For the setting, we contribute complementary evidence and entailment label annotation on hypothesis sentences, for interpretability study. Our experimental results show that R2F framework can obtain state-of-the-art performance and is robust for diverse evidence retrieval methods. Moreover, it can give more interpretable prediction results. Our model and code are released at https://github.com/phoenixsecularbird/R2F.

## introduction natural language inference (nli) is the task of determining whether a hypothesis is entailed or not in a premise. while earlier works @xcite @xcite assume that both hypothesis and premise are single sentences, recent research pays more attention on document-level task, namely document-level nli (docnli) @xcite . the task can enlarge the task scope to judge the * corresponding author. compared with sentence-level nli, docnli poses many new challenges, while there are only a few datasets and models. in terms of datasets, @xcite reformat some mainstream nlp tasks, e.g., text summarization and question answering, and build the first large scale dataset docnli with over 1 million document pairs foot_0 . however, the dataset does not provide evidence annotation about how the labels are inferred, i.e., which hypothesis sentences lead to semantic inconsistency, or which premise sentences help to decide the entailment relationship. as shown in figure 1 , although the sample is annotated as not entailment, most of the hypothesis sentences are actually entailed. by contrast, the detailed disinformation of "in 1989" in the third hypothesis sentence eventually decides the entailment relationship between the documents. for each hypothesis sentence, only several premise sentences are enough to serve as the exact evidence to judge its own sentence-level entailment label. in this paper, we argue that evidence discovery is important and challenging for docnli. our pilot experiments in section 4.3 and 4.5 show that randomly selected evidences can still contribute to comparable performance. thus, only the black-box models may be not so convincing. however, to annotate evidence for evaluation is non-trivial. for each hypothesis sentence, on one hand, it may refer to multiple premise sentences. on the other hand, there may be several evidence groups, where each group can independently serve the label prediction. we highlight this as interpretability challenge. label: not_entailment hypothesis: xus cities along the gulf of mexico from alabama to eastern texas were on alert last night as hurricane andrew headed west after hitting southern florida leaving at least eight dead, causing severe property damage, and leaving 1.2 million homes without electricity. ygusts of up to 165 mph were recorded. zit is the fiercest hurricane to hit the us in 1989. {as andrew moved across the gulf there was concern that it might hit new orleans, which would be particularly susceptible to flooding, or smash into the concentrated offshore oil facilities. |president bush authorized federal disaster assistance for the affected areas. premise: xus cities along the gulf of mexico from alabama to eastern texas were on storm watch last night as hurricane andrew headed west after sweeping across southern florida, causing at least eight deaths and severe property damage. zthe hurricane was one of the fiercest in the us in decades and the first to hit miami directly in a quarter of a century. for each sample, only the entailment label between the documents is annotated. for display, we mark each hypothesis sentence and its corresponding premise sentences (namely the evidences, not annotated in the original dataset) with the same number and color. the sample is annotated as not entailment due to the disinformation of "in 1989" in the third hypothesis sentence. the premise is partly omitted. in term of models, current baselines @xcite still largely follow sentence-level nli. they either concatenate two documents for mutual information interaction for classification, or encode them separately for semantic match with document-level representations. however, except for the interpretability issue, they will leave the following challenges unexplored: in this paper, we establish a general solution, named retrieval, reading and fusion (r 2 f) framework, and a new setting for the task. the basic idea of the framework is to simplify documentlevel classification task into a set of sentence-level tasks, and then improve both performance and interpretability with the power of evidence. specifically, the framework splits the hypothesis document into sentences. then for each hypothesis sentence, it retrieves evidence sentences from the premise, and reads to estimate its credibility score upon the evidences. finally, it fuses the sentence-level results and judge the entailment relationship between the two documents. for the setting, we contribute complementary fine-grained annotations for interpretability study. for each hypothesis sentence, we manually annotate entailment label and several evidence groups, where each group is enough to independently infer the label. in summary, our contributions are as follows: ## r f framework our r 2 f framework aims at a general solution for docnli task with interpretability, i.e., to obtain corresponding evidence and predict entailment label for each hypothesis sentence. as shown in figure 2 , the framework consists of 3 components, namely evidence retrieval, reading for credibility estimation, and credibility fusion. for efficiency, the retrieval component is an independent unit to provide evidence input for the other two components, which are optimized jointly. ## related work natural language inference is a fundamental yet important task in natural language processing. for sentence-level nli, several benchmarks @xcite @xcite have been proposed and they have been attracting research attention. moreover, @xcite propose contract nli targeting the legal and business domain, which is a small scale benchmark. the premises are long contract documents while the hypotheses are actually single sentences. besides, @xcite suggest to debias natural language inference and understanding models through causal intervention and counterfactual reasoning. @xcite adopt commonsense inference to enhance future event generation. for document-level nli, @xcite propose the first large scale benchmark docnli based on a set of early benchmarks. current models for the task still largely follow sentence-level nli. differently, in this paper, we emphasize the importance of evidence discovery and aim at a general solution for the task. ## conclusion in this paper, we propose r 2 f framework as a general solution for docnli task and contribute complementary annotation on docnli dataset. our experimental results show that our framework can obtain state-of-the-art performance. besides, the framework is robust for diverse retrieval methods, and consistently obtains higher performance on samples with varying length, especially longer samples. moreover, the framework can give more interpretable prediction results and help to locate the semantic inconsistency. in the future, we will explore an end-to-end framework for the task.