ELLA : Empowering LLM s for Interpretable, Accurate and Informative Legal Advice (2024)

Abstract: Despite remarkable performance in legal consultation exhibited by legal Large Language Models(LLMs) combined with legal article retrieval components, there are still cases when the advice given is incorrect or baseless. To alleviate these problems, we propose ELLA , a tool for E mpowering L LMs for interpretable, accurate, and informative L egal A dvice. ELLA visually presents the correlation between legal articles and LLM’s response by calculating their similarities, providing users with an intuitive legal basis for the responses. Besides, based on the users’ queries, ELLA retrieves relevant legal articles and displays them to users. Users can interactively select legal articles for LLM to generate more accurate responses. ELLA also retrieves relevant legal cases for user reference. Our user study shows that presenting the legal basis for the response helps users understand better. The accuracy of LLM’s responses also improves when users intervene in selecting legal articles for LLM. Providing relevant legal cases also aids individuals in obtaining comprehensive information. Our github repo is: https://github.com/Huyt00/ELLA .

## introduction large language models (llms), such as llama @xcite , chatglm @xcite and @xcite , have shown impressive performance in various tasks, showing great potential for specific domains, such as law @xcite and finance @xcite . in the legal domain, many attempts have been made @xcite @xcite @xcite , which acquire legal knowledge through continual training and performing a supervised fine-tuning stage with a large-scale legal dataset. these models can offer various services including legal consultations, explaining legal terminology, analyzing legal cases, and preparing legal documents. despite the remarkable performance of llms within the legal domain, they are not exempt from the occurrence of hallucination @xcite . to alleviate this, previous studies @xcite @xcite have proposed retrieval-augmented generation(rag) @xcite frameworks to retrieve legal articles from an external datastore. by leveraging retrieved legal articles, hallucination is reduced and llms can generate more faithful answers. in the legal domain, llms' responses are required to have high accuracy and be supported by reasonable legal bases. therefore, the retrieval component plays an important role as it provides correct and related legal articles for llms. while llms could be augmented with retrieved legal articles to generate faithful responses, when irrelevant ones are retrieved, they inevitably bring noise to llms, leading llms to produce responses with incomplete, incorrect or inconsistent information. for instance, as shown in figure 1 , when a user asks q 1 , the legal article retrieval model retrieves articles 1098, 1101, and 1105 of the civil code 2 according to the query, while fails to retrieve another three relevant ones: article 1100, 1102 and 1093 of the civil code. therefore, llm only suggests that adopters need to meet the conditions c 1 , c 2 and c 3 mentioned in the retrieved legal articles, resulting in incomplete suggestions. then the user continues to ask q 2 . although the related article is retrieved, the irrelevant ones are also retrieved. such irrelevant articles bring noise to llm, leading to the incorrect response r 2 (in fact, the case 𝑸𝑸 𝟏𝟏 : are there any requirements for adopting a child? 𝑹𝑹 𝟏𝟏 : according to the civil code, adopters need to meet the following 3 conditions: 𝑐𝑐 1 , 𝑐𝑐 2 , 𝑐𝑐 3 related legal article: 𝐴𝐴 1098 : 𝑐𝑐 1 𝐴𝐴 1101 : 𝑐𝑐 2 𝐴𝐴 1105 : 𝑐𝑐 3 𝐴𝐴 1100 : 𝑐𝑐 4 𝐴𝐴 1102 : 𝑐𝑐 5 𝐴𝐴 1093 : 𝑐𝑐 6 𝑸𝑸 𝟐𝟐 : (continue to ask) my girlfriend got married, but she has been separated from her husband for several years. does this count as automatic divorce? 𝑹𝑹 𝟐𝟐 : …if your girlfriend has been separated from her husband for several years…it can be considered as automatic divorce… related legal article: 𝐴𝐴 1080 𝐴𝐴 1091 𝐴𝐴 1043 𝑸𝑸 𝟑𝟑 : (new chat) my girlfriend got married, but she has been separated from her husband for several years. does this count as automatic divorce? 𝑹𝑹 𝟑𝟑 : according to the civil code… they have not completed the divorce procedures, so it cannot be considered as automatic divorce… related legal article: 𝐴𝐴 1080 𝐴𝐴 1091 𝐴𝐴 1043 incomplete response incorrect response inconsistent response blue articles mean they are relevant to the query, while orange ones are irrelevant. the blue star means the article is retrieved for llm. we only show the key information in the figure . for the complete conversations, please refer to appendix b mentioned in q 2 should not be considered as automatic divorce). besides, llms may be sensible to the input perturbation @xcite . responses can be contradictory when inputs only differ slightly. for example, when the user begins a new chat and asks q 3 , which is identical to q 2 , the response r 3 is contradictory to r 2 . this inconsistency can potentially bring confusion to users, resulting in a lower-quality consultation. when llms fail to produce coherent and complete responses, relevant legal cases can offer users more in-depth reference information @xcite . however, a legal case retrieval module has rarely been integrated into the existing legal domain llms in civil law systems. additionally, legal terminology may sometimes be embedded in the responses lacking sufficient explanations, posing potential understanding difficulties for users without domain knowledge @xcite . to address the issues mentioned above, we propose ella, a tool empowering llms for interpretable, accurate, and informative legal advice. firstly, we fine-tune bge @xcite , an embedding model for retrieval, to retrieve the legal basis for each sentence in the response. by visually presenting the legal basis to users, users can trust the advice provided by llms. when there is no legal basis for a sentence, it can be viewed as a warning that the sentence may be incorrect. secondly, ella retrieves several legal articles based on the user's query and presents them to users. users can interactively select the relevant legal articles for llms to generate accurate and complete responses while disregarding irrelevant ones to avoid noise. thirdly, we incorporate a legal case retrieval model in ella, intending to present supplementary information for users to reference. considering the long context in legal cases, we find all the key sentences in the article through similarity matching between the query and each sentence in the legal case. we highlight all key sentences in the legal cases for users to improve their reading efficiency. the response interpretation aids users in understanding and placing trust in the advice given by llms. the user study shows that our model can generate more accurate responses when users interactively select relevant legal articles. the legal case retrieval module also offers users more resourceful reference information. ## framework and usage example ella is composed of four parts: 1) chat interface: visually displays the conversation between the user and the llm. 2) interactive legal article selection: provides retrieved legal articles for users to choose from, letting the llm generate new responses based on the user's selected legal articles. 3) response interpretation: provides legal article and judicial interpretations to interpret each sentence of the llm's response. 4) legal case retrieval: displays relevant legal cases for the user to refer to. ## system overview in this section, we detail the implementation of all back-end models of ella. ## evaluation in this section, we automatically evaluate our case retrieval model. we also conduct a user study to evaluate whether ella helps users obtain more accurate, interpretable and informative information during the consultation. ## conclusion we present a novel tool, ella, for legal consultation. ella provides the legal basis and judicial interpretations that supplement the legal advice generated by llms, increasing users' understanding and trust in llm responses. it also displays retrieval results from the retrieval model and allows users to actively select relevant legal articles, thereby assisting the llms in generating more accurate responses. additionally, equipped with a legal case retrieval model, users can refer to relevant legal cases for more comprehensive information. ella enables llms to provide legal advice that is easier to interpret, more precise, and more informative.