Leveraging Natural Language Processing and Large Language Models for Assisting Due Diligence in the Legal Domain (2024)

Abstract: Due diligence is a crucial legal process that mitigates potential risks of mergers and acquisitions (M&A). However, despite its prominent importance, there has been a lack of research regarding leveraging NLP techniques for due diligence. In this study, our aim is to explore the most efficient deep-learning model architecture for due diligence in terms of performance and latency, and evaluate the potential of large language models (LLMs) as an efficient due diligence assistant. To our knowledge, this is the first study that employs pre-trained language models (PLMs) and LLMs for the due diligence problem. Our experimental results suggest that methodologies that have demonstrated promising performance in the general domain encounter challenges when applied in due diligence due to the inherent lengthy nature of legal documents. We also ascertain that LLMs can be a useful tool for helping lawyers who perform due diligence.

## introduction due diligence, one component of mergers and acquisitions (m&a), involves identifying multiple factors that indicate successful outcomes produced by a target organisation @xcite . the primary objective of this process is to minimise risks associated with the organisation. like other legal retrieval tasks, such as contract analysis and cross-jurisdictional analysis, it has been conducted manually by legal professionals. due diligence is often regarded as a tedious, expensive, and timeconsuming job, as the buyer must digest a colossal amount of information within a limited time, often without complete access to relevant information sources @xcite . however, it is an exceptionally important task, as deficient due diligence can result in significant detrimental outcomes for the buyer 1 . for this reason, there has been a grow-1 13 huge due diligence disasters. @xcite ing demand for automated and precise techniques for due diligence. the recent remarkable advancements in natural language processing (nlp) field have expanded the potential for developing such techniques. the success of pre-trained language models (plms) based on transformer structure @xcite has led to their application in the legal domain, giving rise to legal-specific plms @xcite @xcite and datasets for pre-training @xcite and downstream tasks, such as contractnli @xcite and lexglue @xcite . furthermore, the recent emergence of large language models (llms) gained significant attention due to their impressive performance in examinations in legal @xcite and other professional domains @xcite , sparking the possibility of the advent of ai assistants in industrial fields. however, despite its importance, applying nlp techniques to the due diligence problem has received limited attention. a leading cause would be the lack of publicly available datasets. due to the nature of m&a, documents for due diligence often contain sensitive information, making it challenging to collect a large-scale dataset. to our knowledge, the kira dataset @xcite , where the task is designed to detect crucial information in legal contract documents, is currently the only publicly available dataset for due diligence, but it is firmly restricted only to academic usage and obtaining permission to access the dataset requires time and effort. also, the inherent lengthiness of legal documents poses an additional obstacle. legal documents often substantially exceed the maximum length that state-of-the-art nlp models can accommodate @xcite , making the models unable to process longer text properly. as a result, most downstream tasks de-signed to evaluate the performance of legal-specific plms have primarily focused on relatively short paragraphs, such as classification @xcite and question answering @xcite . this paper explores the feasibility of applying modern nlp techniques to the due diligence problem. we first examine the performance of three different architectures on due diligence. subsequently, we conducted a few-shot experiments on gpt-4 to ascertain whether llms could be a useful tool to help the due diligence problem. to the best of our knowledge, this is the first work that leverages plms and llms for due diligence. our contributions can be summarised as follows: ## kira dataset for due diligence due diligence is a legal process to effectively mitigate the potential risks associated with a company during mergers and acquisitions (m&a). the due diligence problem can be divided into two primary processes: 1) the identification of relevant passages presented in legal documents based on the required information and 2) the utilisation of these passages to predict any potential risks to the acquiring company. @xcite collected and released the dataset for the first process exclusively for academic purposes. the dataset contains real-world legal documents across 50 topics, such as "evidence of loans" and "administrative agent fees". each document is transformed into text using optical character recognition (ocr) and other preprocessing techniques. each sentence within the documents is annotated by kira's in-house annotators, including law students, contract lawyers, and in-house senior lawyers. this annotation aims to determine the presence of relevant information in a sentence. the basic statistics of the dataset are presented in ## experiments design the kira dataset @xcite , which serves as the primary dataset in our study, is collected for the first process. it formulates due diligence as a binary sequential classification task, where the relevant sentences in legal documents are labelled by human annotators. the noteworthy characteristic of the kira dataset is that the label distribution is highly skewed, where, on average, a document consists of 3300 sentences, but only 4.8 sentences are labelled as "relevant". here, we explore the due diligence performance of three distinct architectures: 1) single-sentence classification, 2) context-aware sentence classification, and 3) hierarchical sentence extraction. the brief illustrations of these models can be found in figure 1 . single-sentence classification. this is the simplest-level architecture that considers each sentence independently. the model takes a list of tokens and predicts its label, i.e., "relevant" or "nonrelevant". we fine-tune two plms: bert @xcite and legalbert @xcite . ## experiments and results single-sentence classification result. we first fine-tuned single-sentence classification models based on bert and legalbert. for the experiment, we chose two topics in the kira dataset due to the extensive time and resources needed to conduct experiments on all 50 topics. specifically, we chose topics 1086 and 1244, where the kirabaseline model performed the best and worst, respectively. the experimental results are presented in the second row of the results revealed two important findings. firstly, both bert and legalbert produced comparable or lower f1 scores than the kira baseline, a simple crf employing human-crafted features. the results indicate that sentence-level sequential information is a crucial factor in the due diligence problem rather than increasing the model complexity. secondly, legalbert did not exhibit a substantial performance advantage over bert, implying that legal plms do not necessarily ensure improved performance in legal-domain downstream tasks. this finding also aligns with the findings of @xcite . context-aware classification result. next, we fine-tuned bert-base with the context-aware architecture @xcite on topics 1086 and 1244. legalbert was not included in this experiment because no significant performance difference was observed with bert-base in singlesentence classification experiments. the performance of the context-aware classification model is presented in the second row of hierarchical sentence extraction result. subsequently, we trained a hierarchical sentence extraction model. on top of the two topics used in preceding experiments, we added three more topics: 1243, 1247, and 1469, where the kira-baseline models demonstrated the poorest performance. the other two architectures were not evaluated for these three topics, as they already generated inferior performance than the hierarchical sentence extraction model in topics 1086 and 1244. when it comes to the sentence-level encoder, we used sentence-bert @xcite all-minilm-l6-v2 model foot_0 . for the document-level encoder, we employed two models: bi-lstm and bert-base. regarding the bi-lstm document-level decoder, we introduced four variations based on the cut-off confidence score (0.5 and 0.9) and single/ensemble methods. the ensemble method made decisions based on majority voting by using the predictions of five models for each test scenario. the experimental results are presented in the third row of in topic 1086, the kira-baseline model performed the best, but our bi-lstm models also produced a decent performance. for the other four topics, while there was no huge difference in terms of the f1 score, our approaches consistently produced substantially higher recall values across all four topics. the high recall model is more efficient than the high precision model from a practical viewpoint in due diligence, where the "relevant" sentences account for an extremely small portion foot_1 , which can greatly reduce the effort for extensive manual review to detect false negatives. let us assume that we have 100k sentences and only 100 sentences are relevant. in-context learning with gpt-4. recently, llms has gained huge attention for passing legal examinations, such as the university of minnesota law school exam @xcite and the us bar exam @xcite . hence, we explored how llms can be employed to assist with prompt question: the definition of collateral/transaction security topic is as follows. 'lenders will typically require some form of security/collateral to be provided by the borrower or other obligors as a precondition to lending to ensure that if the borrower does not repay the loan or defaults under the credit agreement in any other way, the lenders will have recourse to such security to ensure repayment of the loan. this topic assists in identifying which forms of security/collateral are applicable to a particular transaction.' your task is to determine whether given document contains relevant information regarding collateral/transaction security. here are samples for this task: document: {sample_doc_1} answer: {sample_answer_1} the due diligence problem. to conduct experiments, we simplified the task into a binary classification that predicts whether a given paragraph contains relevant sentences or not. we tested gpt-4 on topic 1243 by providing a paragraph consisting of 16 sentences. we sampled 100 examples for each experiment, as conducting experiments on the whole dataset is an extensive resource-consuming work. regarding the prompt design, we first demonstrated the topic definition and task description, followed by two samples. the model was then asked to make a prediction of a new paragraph. the example of the prompt design we used is presented in the experimental results are shown in also, gpt-4 exhibited a very high recall rate and a decent level of precision rate, which can greatly reduce lawyers' workload in the due diligence problem, as described above. implementing a combined system that identifies paragraphs containing relevant sentences through llms and then using a high-precision model to detect relevant sentences automatically could further diminish the workload. ## related works the progress in the field of nlp has been a driving force of the vigorous advancements in legal nlp, leading to a substantial volume of published papers each year since 2017 @xcite . many legal nlp studies involve predicting judgement decisions @xcite @xcite , collecting legal datasets @xcite @xcite and training legal plms @xcite @xcite @xcite . however, the application of nlp in due diligence for m&a has not received attention despite its promising importance. @xcite collected large corpora to train an automated due diligence model and developed a crf model to assess the presence of relevant information in each sentence of a legal document. @xcite developed a question answering (qa) system for the due diligence problem, which operates in two phases: 1) identifying evidence from a contract that contains the answer to the given question and 2) providing an answer based on the detected evidence. the crf model developed by @xcite is used to find evidence in the first phase. don-nelly and roegiest (2020) employed the same crf model for named entity recognition (ner) in legal documents, assuming that named entities would exist in sentences containing important information. the crf model is also utilised by @xcite for ner in legal documents. they assumed that named entities in legal documents would exist in sentences containing important information. therefore, they first used the crf model to extract candidate sentences, and subsequently trained a named entity detection model using the extracted candidates. this two-step approach demonstrated superior performance in terms of both time and accuracy compared to the state-of-the-art deeplearning ner model of that period @xcite . the wide adoption of the crf model suggests that implementing a more accurate relevant sentence extraction model can greatly benefit various legal nlp tasks. ## summary and outlook due diligence plays a crucial role in ensuring a successful m&a. implementing an automated due diligence system will offer significant benefits considering the resources required for due diligence. this paper illuminates the unhighlighted legal nlp topic: the due diligence problem. in this paper, we first explored three neural model architectures: 1) sentence-level classification, 2) context-aware classification, and 3) hierarchical sentence extraction. subsequently, we examined how gpt-4 can be utilised to assist the due diligence problem. we confirmed that the hierarchical sentence extraction model best suits due diligence and is practically more efficient than the previous approach. our experimental results indicate that previous traditional approaches should not be underestimated, as they possess valuable merits that can be employed in practical applications to enhance productivity. we also verified llms' potential as a useful assistant for lawyers who conduct due diligence.