Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning (2024)

Abstract: The integration of generative Large Language Models (LLMs) into various applications, including the legal domain, has been accelerated by their expansive and versatile nature. However, when facing a legal case, users without a legal background often struggle to formulate professional queries and may inadvertently overlook critical legal factors when presenting their case narrative to LLMs. To address this issue, we propose the Diagnostic Legal Large Language Model (D3LM), which utilizes adaptive lawyer-like diagnostic questions to collect additional case information and then provides high-quality feedback. D3LM incorporates an innovative graph-based Positive-Unlabeled Reinforcement Learning (PURL) algorithm, enabling the generation of critical questions and enhancing user-LLM interactions. Moreover, an integrated LLM-based stopping criterion facilitates precise Court Views Generation (CVG). Our research also introduces a new English-language CVG dataset based on the US case law database, enriching the realm of LLM research and deployment with a vital dimension. D3LM surpasses classical LLMs by delivering outstanding performance and a remarkable user experience in the legal domain.

## introduction recent years have witnessed a substantial increase in the demand for legal services @xcite . despite this growing need, legal resources remain scarce, especially for individuals with modest means @xcite . this scarcity has led to a significant rise in the cost of legal assistance @xcite , making it difficult for ordinary people to afford the necessary legal support. furthermore, this situation exacerbates the inherent inequities in legal proceedings between the affluent and the underprivileged. affluent individuals often benefit from dedicated legal teams to represent them in disputes, whereas the economically disadvantaged lack similar access to legal aid @xcite . this discrepancy not only hinders the ability of the poor to seek justice but also perpetuates a system of latent unfairness in legal cases across socio-economic divides. with the advent of llms, ordinary individuals have gained increased access to legal resources and services. however, the current mode of interaction between users and llms predominantly remains user-led @xcite , where users input queries and receive responses from llms. this interaction mode presents significant limitations, particularly as many users lack a legal background @xcite . their queries often lack the essential factual details about their legal issues, which in turn, hampers the ability of llms to provide effective assistance. recognizing this challenge, we have developed d3lm, a novel framework designed to act as a legal assistant, as illustrated in fig. 1 . in this toy example, we introduce the d3lm model through a real-life scenario where a client, facing legal charges from a minor altercation, seeks assistance. initially, the client provides a vague description, prompting a comparison among three different approaches. traditional llms offer a broad perspective of the court's view, but often overlook critical details due to insufficient information. in contrast, consultations with a lawyer provide a thorough investigation through targeted questions, albeit at a significant cost. d3lm distinguishes itself by autonomously generating specific, case-relevant questions, thereby gaining a deeper understanding of the situation and enhancing the accuracy of legal outcome predictions, all without incurring the high expenses associated with legal counsel. by engaging users in a dialogue to extract detailed information about their legal predicaments, d3lm mirrors the proactive approach of human lawyers in gathering pertinent data to better assist were you under the influence of alcohol at the time? are there any surveillance cameras at the bar that might have captured the incident? yes, video evidence exists that depicts the altercation. yes, somewhat intoxicated. given the reconciliation with the involved party and the absence of intent to cause intentional harm, the prosecutor has opted to either reduce the charges or withdraw them entirely. llm ## methodology in this section, we begin by introducing the definition of the cvg problem in section 3.1. we then proceed to present the overall d3lm framework in fig. 2 in section 3.2. finally, we demonstrate the purl algorithm in alg. 1 in detail in section 3.3. to facilitate understanding and clarity, we provide a summary of the notations used in this paper in tab. 1. ## conclusion and future work our study introduces d3lm, an innovative approach in legal assistant systems for court view generation, diverging from traditional methods by simulating real lawyer-client interactions. by innovatively employing diagnostic analysis through continuous user engagement, we enable the generation of more accurate and case-specific court views. the introduction of our purl algorithm significantly enhances the precision of information extraction. additionally, we contribute a novel english-language dataset for american legal cases, meticulously validated by legal professionals, addressing the scarcity in legal analysis datasets and setting new benchmarks in legal informatics. future work will explore extending this model to other domains, such as medical and consultancy, leveraging its capabilities for broader application.