C ontract NLI : A Dataset for Document-level Natural Language Inference for Contracts (2021)

Abstract: Reviewing contracts is a time-consuming procedure that incurs large expenses to companies and social inequality to those who cannot afford it. In this work, we propose “document-level natural language inference (NLI) for contracts”, a novel, real-world application of NLI that addresses such problems. In this task, a system is given a set of hypotheses (such as “Some obligations of Agreement may survive termination.”) and a contract, and it is asked to classify whether each hypothesis is “entailed by”, “contradicting to” or “not mentioned by” (neutral to) the contract as well as identifying “evidence” for the decision as spans in the contract. We annotated and release the largest corpus to date consisting of 607 annotated contracts. We then show that existing models fail badly on our task and introduce a strong baseline, which (a) models evidence identification as multi-label classification over spans instead of trying to predict start and end tokens, and (b) employs more sophisticated context segmentation for dealing with long documents. We also show that linguistic characteristics of contracts, such as negations by exceptions, are contributing to the difficulty of this task and that there is much room for improvement.

## introduction reviewing a contract is a time-consuming procedure. a study @xcite revealed that "60-80% of all business-to-business transactions are governed by some form of written agreement, with a typical fortune 1000 company maintaining 20,000 to 40,000 active contracts at any given time". contract review is carried out manually by professionals, costing companies a huge amount of money each year. even worse, smaller companies or individuals may opt for signing contracts without access to such professional services. (evidence does not exist when the hypothesis is not mentioned) // denotes a span border figure 1 : an overview of document-level nli for contracts. given a contract, a system must classify whether each hypothesis is entailed by, contradicting to or not mentioned by the contract and identify evidence for the decision as spans in the contract. to address this need, there is a growing interest in contract review automation. recently, @xcite and @xcite introduced datasets for extracting certain terms in contracts, which can help a user comprehend a contract by providing a consistent legend for what sort of terms are discussed in the contract. however, these works only aim to find what sort of terms are present, not what each of such terms exactly states. for example, @xcite involves extracting a span in a contract that discusses about a question "is there a restriction on a party's soliciting or hiring employees ...?". being able to answer such questions can further benefit users by automatically detecting terms that are against the user's policy without having have to read each of the extracted terms. in this paper, we argue that contract review is also a compelling real-world use case for natural language inference (nli). however, rather than evaluating a hypothesis versus a short passage, evaluation is against a whole document. concretely, given a contract and a set of hypotheses (such as "some obligations of agreement may survive ter-mination."), we would like to classify whether each hypothesis is entailed by, contradicting to or not mentioned by (neutral to) the contract as well as identifying evidence for the decision as spans in the contract (figure 1 ). therefore, the problem involves similar evidence identification problems as open domain question answering, a problem less studied in the nli context, and practical usefulness also involves identifying the evidence spans justifying an nli judgment. our work presents a novel, real-world application of nli. we further argue that contractswhich occupy a substantial amount of the text we produce today -exhibit interesting linguistic characteristics that are worth exploring. our contributions are as follows: 1. we annotated and release foot_0 a dataset consisting of 607 contracts. this is the first dataset to utilize nli for contracts and is also the largest corpus of annotated contracts. 2. we introduce a strong baseline for our task, span nli bert, which (a) makes the problem of evidence identification easier by modeling the problem as multi-label classification over spans instead of trying to predict the start and end tokens, and (b) introduces more sophisticated context segmentation to deal with long documents. we show that span nli bert significantly outperforms the existing models. 3. we investigate interesting linguistic characteristics in contracts that make this task challenging even for span nli bert. ## related works helped by their accessibility, there exist multiple prior works on "legal nli" for case and statute laws. one of the subtasks in coliee-2020 shared task @xcite was, given a court decision q and relevant cases, to extract relevant paramodel overall. as discussed in section 1, our task has practical and scientific significance compared to information extraction for contracts @xcite . we showed in our experiments that the nli part of our task is much more challenging than the evidence identification task. furthermore, we gave observations to linguistic characteristics of our dataset that are lacking in these prior works. @xcite presented a dataset where certain types of contract clauses are identified and annotated with "clearly fair", "potentially unfair" or "clearly unfair". while the format of the task input and output is quite similar, our task requires reasoning over a much diverse set of hypotheses than just fair or unfair. similarly, fact extraction and claim verification tasks @xcite , where the task is to extract facts from wikipedia articles and to classify whether the claim is entailed by the facts, have similar input and output formats. such claims and our hypotheses are quite different in nature and working on contracts poses unique challenges as discussed in section 5.2. ## conclusion in this work, we introduced a novel, real-world application of nli, document-level nli for contracts which aim to assist contract review. we annotated a dataset consisting of 607 contracts and showed that linguistic characteristics of contracts, particularly negations by exceptions, make the problem difficult. we introduced span nli bert that incorporates more natural solution to evidence identification by modeling the problem as multi-label classification over spans instead of trying to predict the start and the end token as in previous works. span nli bert performed significantly better than existing transformer-based models. notwithstanding the performance gain by span nli bert, there exists much room for improvement. span nli bert still has poor performance on rare labels, as well as being easily impacted by negations by exceptions. for future works, we will also explore systems that can generalize to different types of contracts and hypotheses. we believe that studying how hypothesis phrasing can affect performance and developing a better way to utilize hypothesis text can be the key to such goal. we hope that the dataset and span nli bert will serve as a starting point for tackling the interesting challenges in our contractnli task.