C on R eader: Exploring Implicit Relations in Contracts for Contract Clause Extraction (2022)

Abstract: We study automatic Contract Clause Extraction (CCE) by modeling implicit relations in legal contracts. Existing CCE methods mostly treat contracts as plain text, creating a substantial barrier to understanding contracts of high complexity. In this work, we first comprehensively analyze the complexity issues of contracts and distill out three implicit relations commonly found in contracts, namely, 1) Long-range Context Relation that captures the correlations of distant clauses; 2) Term-Definition Relation that captures the relation between important terms with their corresponding definitions, and 3) Similar Clause Relation that captures the similarities between clauses of the same type. Then we propose a novel framework ConReader to exploit the above three relations for better contract understanding and improving CCE. Experimental results show that ConReader makes the prediction more interpretable and achieves new state-of-the-art on two CCE tasks in both conventional and zero-shot settings.

## introduction legal contract review is a process of thoroughly examining a legal contract before it is signed to ensure that the content stated in the contract is clear, accurate, complete and free from risks. a key component to this application is the contract clause extraction (cce), which aims to identify key clauses from the contract for further in-depth review and risk assessment. typically, cce consists of two major tasks targeting different query granularities for real-life usages. they are clause analysis (ca) and clause discovery (cd) 2 , where ca aims to identify clauses that belong to a general clause type, while cd aims to identify clauses similar to a specific clause (depicted in figure 1 ). cce is both expensive and time-consuming as it requires legal professionals to manually identify a small number of key clauses from contracts with hundreds of pages in length @xcite . therefore, there is a pressing need for automating cce, which assists legal professionals to analyze long and tedious documents and provides non-professionals with immediate legal guidance. the biggest challenge to automating cce is the complexities of contracts. in the literature, simply treating contracts as plain text, most pretrained language models perform poorly on cce @xcite . some works try to simplify cce from the perspective of contract structure. for example, @xcite assign a fixed extraction zone for each clause type and limit the clauses to be extracted only from their corresponding extraction zones. @xcite use visual cues of document layout and placement as additional features to understand contracts. however, their local context assumption is not flexible and, more seriously, neglects more complicated relations inherent in the contracts. in fact, as shown in figure 1 , contracts are formal documents that typically follow a semistructured organization. the body of a contract is usually organized into some predefined articles such as "definitions" and "terminations", where relevant clauses are orderly described inside. different articles may hold different levels of importance. for example, the "definitions" article is globally important because it clearly defines all important terms that would be frequently referenced, while other articles are sparsely correlated, holding local importance. we attempt to decompose the complexities into a set of implicit relations, which can be exploited to better understand contracts. therefore, as shown in figure 1 , we identify this agreement shall commence on the commencement date and shall continue for a term of ten (10) years, unless previously terminated in accordance with clause 15. ## experimental settings we conduct experiments on two cce tasks, namely ca and cd, in two settings: (1) the conventional setting where the clauses in the training and test sets share the same clause types; and (2) a more difficult zero-shot setting where the clause types differ substantially for training and test set. datasets to implement conreader on ca and cd in both settings, we combine two datasets which originally only tackle one of the tasks: for ca, we use the training set of cuad to train a conreader model. we evaluate it on the test set of cuad for the conventional setting and on the development and test sets of contract discovery for the zero-shot setting. for cd, since we now have a training set from cuad, we apply the same supervised extractive qa setting, where one clause is supposed to be extracted conditioned on the query clause instead of original unsupervised sentence matching formulation. similar to borchmann et al. (2020), we sub-sample k (@xmath0. then, we pair each of the seed clauses with the contract containing the target clause to form k -1 cd examples. by repeating the above process, we can finally get the cd datasets for both training and evaluation. similar to ca, we train another model for cd and evaluate it in two settings. details of data statistics can be found in appendix a.1. ## results conreader-base 14.8 15.9 11.9 12.4 notably, conreader-large also exceeds debertaxlarge by 1.3 aupr with less than half of its parameters (364m vs 750m), demonstrating the effectiveness of our framework. additionally, there are several notable observations: 1) as the queries in cd are clauses, they are more diverse than the 41 queries of ca, making it a more difficult cce task. 2) we find that conreader-base outperforms roberta+pt-base. this implies that explicitly modeling the complexities of the contracts is more valuable than learning from the in-domain data in an unsupervised manner. 3) the improvements of the models designed for long text (longformer and hi-transformer) are less significant than conreader. it suggests that there are more sophisticated issues in contracts other than long text. in addition, longformer favors precision than recall, causing p@0.8r to be 0 in ca and low performance in cd. such a characteristic is not suitable for cce as it has lower tolerance to miss important clauses. ## conclusion we tackle contract clause extraction by exploring three implicit relations in contracts. we comprehensively analyze the complexities of contracts and distill out three implicit relations. then we propose a framework conreader to effectively exploit these relations for solving cce in complex contracts. extensive experiments show that conreader makes considerable improvements over existing methods on two cce tasks in both conventional and zeroshot settings. moreover, our analysis towards interpretability also demonstrates that conreader is capable of identifying the supporting knowledge that aids in clause extraction.