Party Extraction from Legal Contract Using Contextualized Span Representations of Parties (2023)

Abstract: Extracting legal entities from legal documents, particularly legal parties in contract documents, poses a significant challenge for legal assistive software. Many existing party extraction systems tend to generate numerous false positives due to the complex structure of the legal text. In this study, we present a novel and accurate method for extracting parties from legal contract documents by leveraging contextual span representations. To facilitate our approach, we have curated a large-scale dataset comprising 1000 contract documents with party annotations. Our method incorporates several enhancements to the SQuAD 2.0 question-answering system, specifically tailored to handle the intricate nature of the legal text. These enhancements include modifications to the activation function, an increased number of encoder layers, and the addition of normalization and dropout layers stacked on top of the output encoder layer. Baseline experiments reveal that our model, fine-tuned on our dataset, outperforms the current state-of-the-art model. Furthermore, we explore various combinations of the aforementioned techniques to further enhance the accuracy of our method. By employing a hybrid approach that combines 24 encoder layers with normalization and dropout layers, we achieve the best results, exhibiting an exact match score of 0.942 (+6.2% improvement).

## introduction extracting legal entities from legal documents is an essential challenge for legal assistive software @xcite . its goal is to extract structured information -"what are the legal attributes (agreement date, party, license, etc) that are involved" -from unstructured text. a contract document is a legally binding agreement between two or more parties. it outlines the terms and conditions of the relationship and sets forth the rights and obligations of each party @xcite . here, the party is a person or entity who takes part in a legal transaction, for example, a person with an immediate interest in an agreement or deed, or a plaintiff or a defendant in a lawsuit. a "third party" is a person who is a stranger to a transaction, contract, or proceeding. extracting parties' information from legal contracts can provide numerous benefits to legal assistant software such as concord foot_0 , contractworks foot_1 , and hellosign foot_2 . first and foremost, this can aid legal professionals in identifying parties involved in similar cases, trends, and patterns in legal disputes, and assist in more efficient legal research. secondly, the extraction of parties allows reviewing multiple documents efficiently with the quick document search feature, and it leads to focusing on relevant information and documents through assistant software. moreover, the manual extraction of parties is prone to human errors, which can lead to inaccuracies and inconsistencies @xcite . by automating the process of party extraction, legal professionals can save time and improve the accuracy of their work. extracting parties can be a challenging task despite its usefulness @xcite . one of the most challenging parts of the contract is it may contain numerous names of persons and organizations throughout its pages. these names can refer to various entities other than the parties, such as third-party beneficiaries, agents, assignees, guarantors, witnesses, and experts @xcite . as a result, it can be difficult to distinguish the parties from all the other types of individuals and entities mentioned in the contract. secondly, legal contracts can be lengthy and complex, with various technical terms and clauses that require in-depth legal knowledge to understand fully. ad-1086 ditionally, contracts can be written in a convoluted manner, leading to ambiguity in determining the parties' intent, which can create difficulties in extracting the relevant information accurately. moreover, reviewing a large volume of contracts within a limited time frame can be a labor-intensive task for lawyers, often leading to errors and inconsistencies. finally, the manual review may also be prone to errors due to human oversight or misinterpretation, which can result in inaccuracies in the extracted information @xcite . therefore, automated legal assistant software can be beneficial in addressing this challenge. there has been extensive research on extracting various information from legal documents (legislation, court cases, contracts) @xcite , but there were only three studies found on extracting parties from legal contracts. the first system used a rule-based system to extract parties @xcite , but it cannot scale out since law firms generate a plethora of contract documents, making it impossible to add processing rules for each new contract type. the second attempt focused on only one type of contract (leases) and was not applicable to other types of contracts @xcite . the third and final system solved this problem using a questionanswering method @xcite . however, the extraction system results in a large number of false positives and less reliable outcomes. in addition, there exist only two datasets that facilitate the extraction of parties' information from legal contract documents @xcite , out of which only one is publicly accessible. the initial dataset is annotated with extraction zones that are appropriate for rulebased extraction @xcite . on the other hand, the second dataset (cuad) is annotated for a question-answering system but is unsuitable for precise party detection @xcite . in this research, we try to develop a relatively accurate dataset with a precise match of the parties and doubled the size of the cuad dataset (contribution 1). we attempt to develop a scalable and accurate party extraction system with minimal overheads. to address this, we use roberta (a pre-trained transformer-based language model @xcite ) @xcite as the baseline model and empirically optimized this architecture to best learning capability and improve contextualized representations of the contracts in the parties extraction task. this optimized architecture includes transformer encoder layers, layer normalization, and dropout layers (contribution 2). the remaining sections of the paper are organized as follows: firstly, in section 2, we conduct a comprehensive literature review focusing on party extraction systems and party annotated datasets. subsequently, in section 3, we outline our proposed dataflow and the techniques applied to roberta. we then present the experiments in section 4, which encompasses the dataset description, environmental setup, and evaluation methodology. moving forward, in section 5, we present the experimental findings and compare the accuracy of our approach with different models derived from various techniques. following that, in section 6, we discuss the limitations of our approach and provide insights for future works. finally, we conclude the paper with remarks in section 7. ## related work party extraction is currently being explored through various efforts. the primary methods used for text extraction are rule-based, machine learning, and transformers. however, to the best of our knowledge, only three researchers have attempted to extract parties from legal contracts, and only two have developed datasets for parties. this literature first examines previous research on dataset creation and highlights its advantages and limitations. then, it delves into the details of the three party extraction systems. ## methodology the aim of this research is to create a questionanswering model @xcite to identify parties involved in legal contracts as shown in figure 1. to accomplish this, we have opted to use roberta @xcite as our foundational model. there were several reasons behind our selection of roberta @xcite for this research. firstly, it is a comparatively smaller model compared to other large language models @xcite . secondly, roberta @xcite is equipped with a fast tokenizer, enabling us to process large volumes of text data quickly and efficiently. lastly, this model has been shown to have performance improvements compared to bert @xcite due to its dynamic masking technique. ## experiments this section outlines our proposed solution for identifying parties mentioned in contract documents, which we approached as a question-answering task. our methodology for training and evaluating the model is described, along with the results from experiments using various configurations. ## results and analysis the baseline experiment (d) achieved an em of 0.934, which indicates the quality of the combined dataset compared to the existing dataset (cuad). even though our baseline model achieved a comparatively higher score, we found some mistakenly identified parties with non-formal forms and partial forms during our error analysis. by further studies, we concluded that this is due to the inability of the models to learn the complex structure of the legal text. therefore, we explored additional experiments to increase the learning capability of our model to learn legal text's complex structure. finally, we found that passing the output features from the 12th encoder layer of the original roberta through additional layers will increase the further learning of the model on the training dataset. this will help to learn the complex structure of the input space and the association between the different sub-tokens of a party. for example, from the above analysis and studies, we conducted several experiments by varying the number of encoder layers (l) from 8 to 32. but, as you can see in figure 4 , the average exact match is increased from 0.905 (experiment g) to 0.942 along with the number of layers until @xmath0. during the experiments, we also kept the normalization layer and dropout (0.2) layers on top of the final encoder layer to prevent overfitting. even after, our model reached a lower exact match (0.905) than that of @xmath1. this shows our model is getting overfitted even using normalization and dropout layers. finally, we concluded our experiments and fix our best model from experiment i (@xmath2. through our analysis presented in ## discussions the results of our experiments demonstrate that increasing the number of additional encoder layers indeed leads to improved outcomes. previous research models often struggled to identify and learn these complex structures when the number of encoder layers was less. by increasing the number of encoder layers in our approach, we were able to address this limitation and achieve significant improvements in exact matches. the augmentation of encoder layers allowed our model to better capture and represent the nuanced relationships and patterns present within legal contracts. this, in turn, facilitated the identification and extraction of relevant information pertaining to the legal parties involved. the increased depth of the model architecture enabled it to learn and comprehend intricate complexities, which were previously challenging to capture effectively. furthermore, we introduced some modifications to the activation function and implemented additional normalization techniques on top of the final encoder layer. these adjustments were designed to complement the increased depth of the model and further enhance its ability to learn complex structures within legal contracts. the combined impact of increasing the encoder layers, replacing the activation function, and incorporating additional normalization techniques proved to be highly effective in our research as indicated in one of the main limitations of our model is the potential loss of context during the chunking process of input documents. legal documents often contain intricate language and nuanced details that are crucial for accurately identifying parties. chunking the input documents may lead to the loss of this contextual information, which can adversely affect the model's legal understanding. to mitigate this limitation, future research could explore the use of longformer @xcite ) architectures. longformer models are specifically designed to handle long-range dependencies in a text. ## conclusion we propose a novel method to accurately predict the parties from a legal contract document. we mainly divided the approach into two phases according to the literature review as follows: (1) dataset creation: we introduced a large-scale highquality dataset that includes 1000 contract documents annotated for parties by legal experts; (2) modeling: our dataset underwent evaluation using various techniques to assess the performance of roberta. ultimately, our most successful model exhibited a noteworthy improvement of 6.2% in exact match performance compared to cuad's best model (deberta). our research revealed that the availability of data is a critical bottleneck, as nonessential annotations and a lower amount of data will significantly drop the performance, highlighting the importance of our dataset's extensive annotations. moreover, we demonstrated that the performance of the models is greatly affected by their architecture, indicating that advancements in algorithms by the nlp community could aid in tackling this issue. in conclusion, our dataset not only acts as a benchmark for evaluating nlp models in the legal domain but also accelerates research toward resolving a significant real-world problem in the legal firm.