Enhancing Contract Negotiations with LLM -Based Legal Document Comparison (2024)

Abstract: We present a large language model (LLM) based approach for comparing legal contracts with their corresponding template documents. Legal professionals use commonly observed deviations between templates and contracts to help with contract negotiations, and also to refine the template documents. Our comparison approach, based on the well-studied natural language inference (NLI) task, first splits a template into key concepts and then uses LLMs to decide if the concepts are entailed by the contract document. We also repeat this procedure in the opposite direction - contract clauses are tested for entailment against the template clause to see if they contain additional information. The non-entailed concepts are labelled, organized and filtered by frequency, and placed into a clause library, which is used to suggest changes to the template documents. We first show that our LLM-based approach outperforms all previous work on a publicly available dataset designed for NLI in the legal domain. We then apply it to a private real-world legal dataset, achieve an accuracy of 96.46%. Our approach is the first in the literature to produce a natural language comparison between legal contracts and their template documents.

## introduction in the dynamic landscape of contract management, the ability to efficiently negotiate, draft, and manage contracts is paramount for organizations seeking to mitigate risks and streamline operations. this paper explores a comprehensive approach to enhancing contract management processes through the implementation of systematic clause variation analysis, which can be further used to create prenegotiated master service agreements (msas), advanced contract classification and summarization techniques. by leveraging historical contract data and automating key aspects of contract management, organizations can significantly reduce ne-gotiation time frames and improve the consistency and quality of their contractual agreements. our work includes several key components aimed at improving contract management through the use of advanced language models: demonstrating the performance of large language models for natural language inference tasks: we investigate the efficacy of large language models (llms) such as mixtral and gpt-4 in performing natural language inference (nli) tasks on the contractnli dataset @xcite . this involves not only assessing the models' ability to understand and infer contractual language but also identifying evidence for each nli task. by demonstrating the superior performance of these models when compared to @xcite , we aim to highlight their potential in automating complex contract analysis tasks, thereby enhancing the efficiency and accuracy of contract management processes. the ability of these models to accurately perform nli tasks is crucial for understanding the nuances and implications of various contract clauses, which in turn supports more informed decision-making during contract negotiations. ## related work legal contracts are characterized by their intricate logical structures, specialized vocabulary, and the necessity for precise interpretation. the ability to perform document-level natural language inference (nli) in this context is crucial for various applications, including contract review, compliance checking, and automated legal reasoning. however, existing nli datasets and models are not well-suited for these tasks, as they are primarily designed for sentence-level inference and lack the context and complexity of full documents. reviewing a contract is a time-consuming and complex process that incurs large expenses for companies. to address this gap, @xcite ) ) introduced contractnli: a dataset for document-level natural language inference for contracts. this is further discussed in section 3.1. the task involves using a span nli bert model to classify whether each hypothesis (a sentence) is entailed by, contradicts, or is not mentioned by (neutral to) the contract, and to identify evidence for the decision as spans in the contract. the span nli bert performed significantly better than existing transformer-based models in terms of nli. our task closely parallels their problem statement, as we aim to determine whether each clause in the template agreement is covered (entails or contradicts) or not covered (neutral) in the contract agreement. the application of large language models (llms) in the context of legal contracts has been extensively explored by @xcite . their problem setup involves legal questions with several answer options, focusing on structured answers rather than generating free text. they employ an embedding-based approach to predict the answer option with the highest similarity to the question text and develop question-specific prompts, eventually landing on a smaller set of reusable prompt templates. our work differs from the aforementioned studies in several key aspects. while previous research has focused on sentence-level nli, structured question answering, clause drafting, and benchmarking legal reasoning tasks, our approach is the first to leverage llms for the direct comparison of legal contracts with their corresponding template documents. by splitting both templates and contracts into sub-clauses and using llms to determine entailment in both directions, we create a comprehensive clause library that aids in refining template documents and assisting in contract negotiations. our method not only outperforms existing models on a publicly available nli dataset in the legal domain but also demonstrates high accuracy on a private real-world legal dataset, showcasing its practical applicability and effectiveness. ## motivation contract review is a very labor-intensive process and there is a growing need to streamline and automate the process of contract review, which is critical in legal and business environments. traditional methods of contract analysis are time-consuming, prone to human error, and often require significant expertise. contract review involves meticulously reading through lengthy and complex documents to identify key clauses, obligations, exceptions, and potential risks. this process demands a deep understanding of legal language and the ability to interpret nuanced terms and conditions, which can vary significantly between contracts. additionally, the need to cross-reference multiple documents and ensure compliance with relevant laws and regulations further complicates the task. by leveraging advanced natural language processing (nlp) techniques, specifically large language models (llms), we aim to enhance the efficiency and accuracy of contract review. our initial experiments with the contractnli dataset provide a valuable opportunity to assess the capabilities of llms in handling complex legal language and inference tasks. this research not only contributes to the field of nlp by addressing the unique challenges posed by legal documents but also has practical implications for improving contract management processes in various industries. ## experiments on contractnli dataset in our experiments, we explored the application of large language models to the contractnli dataset, focusing on two primary tasks: (1) classifying the relationship between a given contract and a set of hypotheses, and (2) identifying evidence within the contract that supports the classification decision. to guide the models' responses, we employed specific prompts tailored to each task. we tested the performance of both commercial and open-source models, including the gpt-4 model, which is accessed via a commercial api. gpt-4 (openai et al., 2024) is a large-scale, multimodal model that exhibits human-level performance on various professional and academic benchmarks. additionally, we fine-tuned the mixtral 8x7b @xcite , a sparse mixture of experts (smoe) language model, which combines multiple expert networks to improve performance while maintaining efficiency. we chose this mixtral model as it was one of the open-source models available at that time with demonstrated superior performance and reduced inference costs. to fine-tune the mixtral model, we employed low-rank adaptation (lora) @xcite , which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. we maintained consistent training parameters for both tasks. specifically, we used the following settings that included a per-device training batch size to 1 and used gradient accumulation to effectively manage memory usage, with accumulation steps also set to 1. gradient check-pointing was enabled to further conserve memory during training. the total number of training steps was capped at 4000. a small learning rate of 2.5e-5 was selected to ensure stable and gradual fine-tuning of the model. training was conducted using bf16 precision to optimize computational resources. here are the explanations for the two tasks within the contractnli dataset. ## proposed solution for internal dataset in our internal problem setting, we are tasked with comparing a negotiated contract against a preestablished template for the contract. these contracts frequently undergo several amendments that add, delete, or modify the original clauses. this scenario closely resembles a natural language inference (nli) task, wherein we seek to determine whether each concept (hypothesis) in the template clauses is either covered (contradicted or entailed) or not covered (neutral) in relation to the contract agreement. additionally, since the documents are often available as scanned pdfs, we must explore ocr solutions to accurately convert them into text for further analysis. one of the main challenges we faced was that many of the documents were images embedded in pdf files, making it difficult to extract and segment the text based on sections. our initial experiments using tesseract-ocr were unsuccessful due to errors introduced during ocr and the difficulty of segmenting free-flowing text without clear delimiters. to address this, we used a document image transformer model capable of identifying sections using boundaries and then performing ocr on the bounded boxes. once the text from each section was extracted, we used gpt-4 model us-ing tailored prompts to extract the correct clauses and compare them with template clauses. this approach allowed us to effectively process and analyze the complex legal language and structure of the contracts, demonstrating the potential of llms in automating and enhancing the contract review process. ## results the clause variations generated by the gpt-4 model using the pipeline in figure 1 on the internal dataset were annotated by the annotation team at jp morgan. the annotators are experienced with handling legal documentation, but may not be able to judge the output at the level of a trained lawyer. the quality of the annotations is deemed sufficient for practical applications. on our internal dataset, the model achieved an accuracy of 96.46%. the accuracy is determined by dividing the total number of correctly identified concepts within each clause by the model, based on their classification as entailed, contradicted, or neutral with respect to the contract document. refer to figure 2 for performace of the model on each clause across the dataset. ## discussion our experiments and results on the contractnli dataset reveal that both the gpt-4 and mixtral models outperform the spannli bert model introduced by @xcite on natural language inference (nli) tasks. for nli, gpt-4 achieved an f1 score of 0.91 on the entailment label, compared to 0.834 by the spannli bert model. additionally, gpt-4 attained an f1 score of 0.70 on the contradiction label, significantly higher than the 0.357 achieved by the spannli bert model. for evidence identification, gpt-4 achieved slightly superior performance than the span nli bert model and demonstrated a mean average precision of 92.68%. these results demonstrate the potential of leveraging state-of-the-art language models for enhancing the accuracy and reliability of nli tasks and evidence identification in legal contexts. the results on the internal dataset demonstrate the efficacy of large language models (gpt-4) in generating outputs that are highly accurate when applied to our internal dataset. the model achieved an impressive accuracy of 96.46%, indicating a high level of precision in its outputs and can be used as a very effective tool by legal professionals in comparing contracts and simplifying the contract negotiation process. the insights provided by the model can help in informed decision-making, further improving the quality of contract review. additionally, the analysis of clause variations enabled the creation of a clause library, focusing on the most frequent additions and deletions of subclauses/concepts. due to the highly confidential nature of the data, we are unable to disclose the specific modified clauses. the integration of the gpt-4 model into the contract negotiation process has the potential to significantly improve outcomes and streamline operations for legal professionals. however, one potential risk in our study is that, although the annotators are experienced in handling legal documentation, their evaluations may lack the nuanced judgment and expertise of a trained lawyer. this could potentially impact the accuracy and reliability of the annotations, particularly in complex legal scenarios. ## conclusion in this study, we explore the efficacy of our novel approach on the contract-nli dataset, a publicly available dataset designed for natural language inference and evidence identification within legal concept from template the deliverables will not contain any malware, .. malicious programs and will not store any data on computers, systems, or network. gpt-4 output from comparing template concept to contract "the document does not explicitly state that the deliverables will not contain any malware, .. will not store any data on computers, systems, or network. however, it does mention that the supplier will comply with certain security and risk management policies, and that the supplier is responsible for assessing and remediating security vulnerabilities." sources from the contract • "5.10 application security." we introduce the first approach that leverages large language models (llms) to generate natural language comparisons between legal contracts and their corresponding templates, conceptualized similarly to a natural language inference (nli) problem on the internal dataset, where we have achieved high accuracy. additionally, we illustrate the capability of llms to perform comparative analysis against both the source text and the text of citations cross-referenced elsewhere in the document. our approach involves framing the comparisons as an nli problem, thereby enabling a more structured and interpretable analysis. the results indicate that our approach not only outperforms existing methods on the contract-nli dataset but also provides a robust framework for the natural language comparison of legal documents. the implications of these findings suggest significant advancements in the automation of legal document analysis and the potential for broader applications in the legal domain.