D oc I nfer: Document-level Natural Language Inference using Optimal Evidence Selection (2022)

Abstract: We present DocInfer - a novel, end-to-end Document-level Natural Language Inference model that builds a hierarchical document graph enriched through inter-sentence relations (topical, entity-based, concept-based), performs paragraph pruning using the novel SubGraph Pooling layer, followed by optimal evidence selection based on REINFORCE algorithm to identify the most important context sentences for a given hypothesis. Our evidence selection mechanism allows it to transcend the input length limitation of modern BERT-like Transformer models while presenting the entire evidence together for inferential reasoning. We show this is an important property needed to reason on large documents where the evidence may be fragmented and located arbitrarily far from each other. Extensive experiments on popular corpora - DocNLI, ContractNLI, and ConTRoL datasets, and our new proposed dataset called CaseHoldNLI on the task of legal judicial reasoning, demonstrate significant performance gains of 8-12% over SOTA methods. Our ablation studies validate the impact of our model. Performance improvement of 3-6% on annotation-scarce downstream tasks of fact verification, multiple-choice QA, and contract clause retrieval demonstrates the usefulness of DocInfer beyond primary NLI tasks.

## introduction natural language inference (nli) is a fundamental textual reasoning task seeking to classify a presented hypothesis as entailed by, contradictory to or neutral to a premise @xcite . prior nli datasets and studies have focused on sentence-level inference where both the premises and hypotheses are single sentences (snli @xcite , multinli @xcite , qnli and wnli @xcite ) documentlevel nli extends the reasoning of nli beyond * *corresponding author:puneetm@umd.edu sentence granularity where the premises are in the document granularity, whereas the hypotheses can vary in length from single sentences to passages with hundreds of words @xcite . document level nli is an important problem for many tasks including verification of factual correctness of document summaries, fact-checking assertions against articles, qa on long texts, legal compliance of contracts, etc. even so, it challenges modern approaches due to the limited input bottleneck of modern transformer models. consider that the universally used bert model @xcite can only encode 512 input sub-tokens due to its quadratic self-attention complexity. consequently, evidence in the document premise relevant to the hypothesis can potentially be distributed in several textual spans located arbitrarily far away from each other in long documents, and may not be simultaneously available to draw inference. recent approaches, notably spannli @xcite , hesm @xcite ) and others, have shown that chunking the premise into multiple document spans, scoring them, and aggregating the scores helps mitigate the limited input length problem. such approaches do not allow the inference module to reason over the complete evidence. in contrast to encoding the document as a set of sentences fed into a transformer for inferential reasoning, a recent line of work, e.g. evidencenet @xcite , gear @xcite and hgrga @xcite ), encodes documents as graphs and uses graph reasoning to perform textual inference. graphs allow encoding of various morphological and semantic relationships at various granularities. however, these approaches use graph-based processing subsequent to evidence selection. we address the above challenge with a reasonable assumption that the portion of the premise (the ground truth evidence) necessary and sufficient for inference can fit entirely into the length limit of language model for effective representation learning. our proposed system achieves this by selecting sentences in the document that are contextually relevant for a given hypothesis through pruning irrelevant paragraphs and reinforce learning based optimal sentence selection. our main contributions: prior works in this domain address the length limitation for claim verification by relevant evidence identification and its chunking which are individually scored and probabilistically aggregated (subramanian and @xcite . hierarchical graph modeling may be used to handle the large scale of the premise @xcite @xcite @xcite @xcite . context selection for document-level nlp: recent works have investigated selection of relevant context for document-level nlp tasks such as neural machine translation @xcite , event detection @xcite , relation extraction @xcite . recently, some of the work on document-level nlp has looked at temporal relation extraction @xcite , temporal dependency parsing @xcite , and speech synthesis @xcite using graphs and sequence learning. however, none of them have considered an end-to-end trainable approach for graph learning with to identify the relevant evidence extraction. ## results ablation study of docinfer: impact of reward function: