C lause R ec: A Clause Recommendation Framework for AI -aided Contract Authoring (2021)

Abstract: Contracts are a common type of legal document that frequent in several day-to-day business workflows. However, there has been very limited NLP research in processing such documents, and even lesser in generating them. These contracts are made up of clauses, and the unique nature of these clauses calls for specific methods to understand and generate such documents. In this paper, we introduce the task of clause recommendation, as a first step to aid and accelerate the authoring of contract documents. We propose a two-staged pipeline to first predict if a specific clause type is relevant to be added in a contract, and then recommend the top clauses for the given type based on the contract context. We pre-train BERT on an existing library of clauses with two additional tasks and use it for our prediction and recommendation. We experiment with classification methods and similarity-based heuristics for clause relevance prediction, and generation-based methods for clause recommendation, and evaluate the results from various methods on several clause types. We provide analyses on the results, and further outline the limitations and future directions of this line of research.

## introduction a contract is a legal document between at least two parties that outlines the terms and conditions of the parties to an agreement. contracts are typically in textual format, thus providing a huge potential for nlp applications in the space of legal documents. however, unlike most natural language corpora that are typically used in nlp research, contract language is repetitive with high inter-sentence similarities and sentence matches @xcite , calling for new methods specific to legal language to understand and generate contract documents. a contract is essentially made up of clauses, which are provisions to address specific terms of the agreement, and which form the legal essence of the contract. drafting a contract involves selecting an appropriate template (with skeletal set of clauses), and customizing it for the specific purpose, typically via adding, removing, or modifying the various clauses in it. both these stages involve manual effort and domain knowledge, and hence can benefit from assistance from nlp methods that are trained on large collections of contract documents. in this paper, we attempt to take the first step towards ai-assisted contract authoring, and introduce the task of clause recommendation, and propose a two-staged approach to solve it. there have been some recent works on itembased and content-based recommendations. wang and fu (2020) reformulated the next sentence prediction task in bert @xcite as next purchase prediction task to make a collaborative filtering based recommendation system for ecommerce setting. @xcite introduced recobert leveraging textual description of items such as titles to build an item-to-item recommendation system for wine and fashion domains. in the space of text-based content recommendations, @xcite proposed a method to recommend citations in academic paper drafts without using metadata. however, legal documents remain unexplored, and it is not straightforward to extend these methods to recommend clauses in contracts, as these documents are heavily domain-specific and recommending content in them requires specific understanding of their language. in this paper, clause recommendation is defined as the process of automatically providing recommendations of clauses that may be added to a given contract while authoring it. we propose a twostaged approach: first, we predict if a given clause type is relevant to be added to the given input contract; examples of clause types include governing laws, confidentiality, etc. next, if a given clause type is predicted as relevant, we provide contextaware recommendations of clauses belonging to the given type for the input contract. we develop contractbert, by further pre-training bert using two additional tasks, and use it as the underlying language model in both the stages to adapt it to contracts. to the best of our knowledge, this is the first effort towards developing ai assistants for authoring and generating long domain-specific legal contracts. ## methodology a contract can be viewed as a collection of clauses with each clause comprising of: (a) the clause label that represents the type of the clause and (b) the clause content. our approach consists of two stages: (1) clause type relevance prediction: predicting if a given clause type that is not present in the given contract may be relevant to it, and (2) clause recommendation: recommending clauses corresponding to the given type that may be relevant to the contract. figure 1 shows an overview of our proposed pipeline. first, we build a model to effectively represent a contract by further pre-training bert, a pre-trained transformer-based encoder @xcite , on contracts to bias it towards legal language. we refer to the resulting model as contractbert. in addition to masked language modelling and next sentence prediction, contractbert is trained to predict (i) if the words in a clause label belong to a specific clause, and (ii) if two sentences belong to the same clause, enabling the embeddings of similar clauses to cluster together. figure 2 and 3 show the difference in the performance of bert and contractbert to get a meaningful clause embedding. bert is unable to differentiate between the clauses of different types as it is unfamiliar with legal language. on the other figure 3: clustering of clauses using contractbert embedding hand, contractbert is able to cluster similar clause types closely while ensuring the separation between clauses of two different types. ## experiments and evaluation we evaluate three methods for clause type relevance prediction + clause recommendation: (1) binary classification + clause generation, which is our proposed approach; (2) collaborating filtering + similarity-based retrieval; and (3) document similarity + similarity-based retrieval. collaborating filtering (cf) + similarity-based retrieval. clause type relevance prediction can be seen as an item-item based cf task @xcite with contracts as users and clause types as items. we construct a contract-clause type matrix, equivalent to the user-item matrix. if contract u contains clause type i, the cell (u, i) gets the value 1, otherwise 0. we then compute the similarity between all the clause type pairs (i, j), using an adjusted cosine similarity, given by, we obtain the item similarity matrix using this cosine score, and use it to predict if a target clause type t is relevant to a given contract. we compute the score for t using the weighted sum of the score of the other similar clause types, given by, if t gets a high score and is not already present in the contract, it is recommended. we experiment with multiple thresholds above which a clause type may be recommended. given a clause library containing all possible clause types and their corresponding clauses, clause content recommendation can be seen as a similarity-based retrieval task. for a given contract and a target clause type t, we use ct_rep and trgt_cls_rep, and find cosine similarities with each of the clauses belonging to t to find the most similar clauses that may be relevant to the given contract. we do so by computing the similarity of either (i) ct_rep or (ii) (ct_rep + trgt_cls_rep)/2, with individual clause representations. document similarity + similarity-based retrieval. this is based on using similar documents to determine if a target clause type t can be recommended for a given contract. the hypothesis is that similar contracts tend to have similar clause types. to find similar documents, we compute cosine similarities between the given contract's representations ct_rep with those of all the contracts in the (training) dataset to identify the top k similar contracts. if t is present in any of the k similar contracts and is not present in the given contract, it is recommended as a relevant clause type to be added to the contract. we experiment with @xmath0, 5}. similarity-based retrieval for clause content recommendation is same as above. metrics. we evaluate the performance of clause type relevance prediction using precision, recall, accuracy and f1-score metrics, and that of the clause content recommendation using rouge @xcite score. data. we use the ledgar dataset introduced by @xcite . it contains contracts from the u.s. securities and exchange commission (sec) filings website, and includes material contracts (exhibit-10), such as shareholder agreements, employment agreements, etc. the dataset contains 12,608 clause types and 846,274 clauses from around 60,000 contracts. further details on the dataset are provided in the appendix. since this dataset can not be used for our work readily, we preprocess it to create proxy datasets for clause type relevance prediction and clause recommendation tasks. for the former, for a target clause type t, we consider the labels relevant and not relevant for binary classification. for relevant class, we obtain contracts that contain a clause corresponding to t, and remove this clause; given such a contract as input in which t is not present, the classifier is trained to predict t as relevant to be added to the contract. for the not relevant class, we randomly sample an equal number of contracts that do not contain t in them. for recommendation, we use the contracts that contain t (i.e., the relevant class contracts); the inputs consist of the contract with the specific clause removed and t, and the output is the clause that is removed. for both the tasks, we partition these proxy datasets into train (60%), validation (20%) and test (20%) sets. these ground truth labels ({relevant, not relevant} for the first task and the clause content for the second task) that we removed are used for evaluation. the implementation details are provided in appendix. ## results and discussion the cf-based method gives the best recall values for all the clause types, while the precision, accuracy and f1 scores are worse compared to the other two methods. this method does not incorporate any contextual information of the contract clause content and relies only on the presence or absence of clause types to predict if a target type is relevant, thus resulting in high recall and low precision and f1 scores. while the results of document similarity-based and classification methods are comparable, both have merits and demerits. while the document similarity-based method is simpler and more extensible than classification which requires training a new classifier for each new clause type, the former requires a large collection of possible contracts to obtain decent results (particularly the recall values), which may not be available always. further, the performance of document similarity method is dependent on k. this can be seen in the lower recall values for the document similarity method compared to those of classification. the storage costs associated with the contract collection can also become a bottleneck for the document similarity method. also, currently there is no way to rank the clauses in the similar contracts, and hence its recommendations cannot be scoped, while in classification, the probability scores can be used to rank the clause types for relevance. on an average, the f1 scores for binary classification are highest compared to the other methods, while the accuracies are comparable with the document similarity method. for clause content recommendation, we focused primarily on relevance (in terms of rouge). in general, retrieval-based frameworks, like the one we proposed, are mostly extractive in nature, and hence might be perceived as "safer" (or factual) to avoid any noise and vocabulary change in clauses that may be incorporated by generation methods, particularly in domains like legal. however, they can also end up retrieving clauses irrelevant to the contract context at times, as we note from their lower rouge scores, as retrieval is based on similarity heuristics which may not always capture relevance, while generation is trained to generate the specific missing clause in each contract. we also notice that generated clauses have lower linguistic variations in them, i.e., generated clauses belonging to one type often look alike. however, this is expected as most clauses look very similar with only a few linguistic and content variations. we believe because clauses have this repetitive nature, there is a large untapped opportunity to leverage nlp methods for legal text generation while accounting for the nuances and factuality in them, to build more accurate clause recommendation frameworks. we believe our work can provide a starting point for future works to build powerful models to capture the essence of legal text and aid in authoring them. in the future, we aim to focus on balancing the relevance and factuality of clauses recommended by our system. ## conclusions we addressed ai-assisted authoring of contracts via clause recommendation. we proposed clauserec pipeline to predict clause types relevant to a contract and generate appropriate content for them based on the contract content. the results we get on comparing our approach with similaritybased heuristics and traditional filtering-based techniques are promising, indicating the viability of ai solutions to automate tasks for legal domain. efforts in generating long contracts are still in their infancy and we hope our work can pave way for more research in this area. don tuggener, pius von däniken, thomas peetz, and mark cieliebak. 2020. ledgar: a large-scale multi-label corpus for text classification of legal provisions in contracts. in proceedings of the 12th language resources and evaluation conference, pages 1235-1241, marseille, france. european language resources association. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n. gomez, lukasz kaiser, and illia polosukhin. 2017. attention is all you need. tian wang and yuyangzi fu. 2020. item-based collaborative filtering with bert. in proceedings of the 3rd workshop on e-commerce and nlp, pages 54-58, seattle, wa, usa. association for computational linguistics.