Exploration of Open Large Language Models for e D iscovery (2023)

Abstract: The rapid advancement of Generative Artificial Intelligence (AI), particularly Large Language Models (LLMs), has led to their widespread adoption for various natural language processing (NLP) tasks. One crucial domain ripe for innovation is the Technology-Assisted Review (TAR) process in Electronic discovery (eDiscovery). Traditionally, TAR involves manual review and classification of documents for relevance over large document collections for litigations and investigations. This process is aided by machine learning and NLP tools which require extensive training and fine-tuning. In this paper, we explore the application of LLMs to TAR, specifically for predictive coding. We experiment with out-of-the-box prompting and fine-tuning of LLMs using parameter-efficient techniques. We conduct experiments using open LLMs and compare them to commercially-licensed ones. Our experiments demonstrate that open LLMs lag behind commercially-licensed models in relevance classification using out-of-the-box prompting. However, topic-specific instruction tuning of open LLMs not only improve their effectiveness but can often outperform their commercially-licensed counterparts in performance evaluations. Additionally, we conduct a user study to gauge the preferences of our eDiscovery Subject Matter Specialists (SMS) regarding human-authored versus model-generated reasoning. We demonstrate that instruction-tuned open LLMs can generate high quality reasonings that are comparable to commercial LLMs.

## introduction electronic discovery (ediscovery) @xcite refers to the process of identifying, collecting, and preserving electronic documents and data for the purpose of legal proceedings, investigations, or regulatory compliance. during legal * these authors contributed equally to this work procedures, including litigation proceedings and corporate mergers and acquisitions, the court often issues a production request. the request mandates the parties involved to produce documents pertinent to the case. a large team of legal practitioners meticulously examines millions of digital records to identify the ones that are relevant to the production request. this step is a crucial phase of the ediscovery process, commonly referred to as responsiveness determination. during this early step, legal practitioners often employ artificial intelligence (ai)-based tools to help them identify and prioritize the documents for review. this essential component of the ediscovery process is referred to as technology-assisted review (tar) @xcite . another related domain focused on management and extraction of relevant information from electronic documents for legal purposes is legal information retrieval (lir). however, the important distinctions between ediscovery and the general domain of lir @xcite are in their purpose, and nature of documents under consideration. ediscovery is related to request for production of documents and data that are often in the form of email correspondences, text messages, articles, and financial declarations. in contrast, lir is related to finding legal information in response to specific queries from legal databases, case law repositories, legislative archives, and other legal resources. the language used in the documents for these two tasks differ with the latter being richer in legal terminology. in this paper, our primary focus is on documents pertinent to ediscovery. traditional approaches to tar have focused on boolean querying @xcite , information @xcite and active learning methodologies @xcite . in recent years, the use of supervised learning on labeled documents, referred to as predictive cod-ing @xcite , has gained more popularity within tar workflows. predictive coding often employs binary text classification based on textual, syntactic, semantic, and other data-driven features. motivated by the application of llms in zero-shot and few-shot learning in various domains, including recommender systems and document annotation @xcite @xcite , we propose a novel approach to predictive coding using llms. furthermore, we go beyond the predictive capabilities of llms, and leverage them to provide reasonings for the predictions. this capability can assist human reviewers in making more informed decisions. the use of llms for predictive coding and reasoning represents a paradigm shift in a domain burdened by the ever-increasing volume of documents and escalating review costs @xcite . in this paper, we utilize open llms, notably large language model meta ai v2 (llama2 -13b and 70b versions) @xcite and falcon-7b 1 , as well as the commercially licensed generative pre-trained transformer (gpt)-3.5-turbo model 2 , for both predictive coding and reasoning. we explore various methods to use these llms effectively, including zero-shot prompting, fine-tuning for instruction following (i.e., instruction tuning), and reasoning generation. additionally, we conduct a user study involving ediscovery subject matter specialists (smss) to evaluate the quality of reasoning generated by these models, and those authored by a legal sms. the publicly available enron email dataset @xcite is used in our experiments. this dataset comprises of a large corpus of emails and attachments that had to be reviewed for their responsiveness to production requests. we present experimental results of the various llm-based predictive coding approaches on this dataset. the paper presents related research in section 2, describes the datasets used in section 3, outlines the experimental approaches in section 4, and reports the results of classification models as well as user preferences for generated reasonings in section 5. the application of ai in ediscovery for the purposes of document prioritization, reasoning, and decision-making are not a recent development @xcite @xcite . notably, the text retrieval conference (trec) had consistently featured a dedicated @xcite @xcite @xcite @xcite , where ediscovery was given prime importance, with emphasis on retrieval and ranking-based approaches. several shared tasks related to ediscovery, including tar and privilege review, were organized to stimulate and advance research in this specialized domain. typical tar solutions can be decomposed into retrieval-based @xcite and classification-based approaches @xcite . both of these approaches are instrumental in facilitating the prioritization of documents for subsequent review processes. alongside these methods, active learning strategies (cormack and @xcite have been extensively explored to boost the operational efficiency of these models. these strategies incorporate relevance feedback provided by human reviewers to enhance model performance. however, it's imperative to acknowledge that certain feedback-based processes can be inherently slow, expensive and inconsistent. in this paper, we primarily focus on the classification-based approach, also referred to as predictive coding, which has gained prominence in recent years. conventional tar approaches are often perceived as black box systems, making it challenging to trust their decisions. the generation of reasoning and explanation for tar models has received limited attention. previous approaches @xcite focused on training models to identify snippets (at sentence level) within documents that are classified as either responsive or non-responsive. however, these approaches necessitate training on annotated data, which is both time-consuming and resourceintensive. in this work, we focus on leveraging the capabilities of generative llms for both classification and reasoning. these architectures, rooted in the transformer framework @xcite , are characterized by their auto-regressive decoders. typi-cally, these models are trained on vast and diverse corpora of textual data, demonstrating remarkable proficiency in comprehending natural language instructions across diverse domains. these llms are able to generate coherent natural language responses when provided with appropriate prompts @xcite . they can interpret the topics specified in the prompts and execute them effectively. in this study, we experiment with some open llms, such as llama2 (13b and 70b versions) and falcon-7b, as well as commercially licensed models like gpt-3.5-turbo. ## dataset in this section, we describe the dataset used for our experiments. we used the electronic discovery reference model (edrm) enron email data set version 2, which is the post-processed version employed in the trec 2011 legal ediscovery track @xcite . this dataset serves as a rich resource for exploring various facets of ediscovery, particularly in the context of email communication. ## methodology in this study, we primarily focus on examining several generative models from the decoder-only autoregressive family, a subset of the broader category of llms. these models have gained significant attention and adoption following the introduction of chatgpt foot_0 . legal departments and law firms have expressed interest in utilizing these models to enhance their document review processes, achieving both scalability and cost-efficiency. we explore four models: falcon (7b version), llama2 (both 13b and 70b versions), and gpt (3.5 turbo version). our experimentation involves two distinct methodologies: out-of-the-box (oob) prompting approach and fine-tuning of open llms, with a special emphasis on falcon 7b and llama2 13b, which have been shown to outperform other open llms of similar size across various benchmarks foot_1 . we detail these methodologies in the subsequent sections. ## results in this section, we present the results of applying llms to the predictive coding process. the training and evaluation were conducted on the seed data and qrels for topics 401, 402, and 403 from the trec 2011 legal track. model performance was evaluated on the qrels using standard classification metrics, such as precision, recall, and macro f1 scores. ## conclusion this study is among of the first works to assess the efficacy of generative llms in the ediscovery document review process. we compare the classification performance of various open and commercially-licensed llms, and demonstrate that although oob performance of open llms are worse compared to gpt-3.5, these models can be finetuned to achieve comparable results to gpt-3.5. moreover, we conduct a user study and show that the smss favored ai-generated reasoning over human reasoning, underscoring the viability of these approaches in ediscovery. in terms of scalability, we suggest that there is no inherent need to deploy large models such as gpt-3.5 or llama2-70b for production purposes. the llama2-13b model, which fits on just two 40gb a100 gpus, can be fine-tuned to yield similar classification performance and reasoning that rivals human reasoning -as supported by our user study. this approach offers significant savings in infrastructure costs associated with model deployment.