Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaboration (2024)

Abstract: Large Language Models (LLMs) could struggle to fully understand legal theories and perform complex legal reasoning tasks. In this study, we introduce a challenging task (confusing charge prediction) to better evaluate LLMs’ understanding of legal theories and reasoning capabilities. We also propose a novel framework: Multi-Agent framework for improving complex Legal Reasoning capability (MALR). MALR employs non-parametric learning, encouraging LLMs to automatically decompose complex legal tasks and mimic human learning process to extract insights from legal rules, helping LLMs better understand legal theories and enhance their legal reasoning abilities. Extensive experiments on multiple real-world datasets demonstrate that the proposed framework effectively addresses complex reasoning issues in practical scenarios, paving the way for more reliable applications in the legal domain.

## introduction large language models (llms) have shown remarkable generalization ability across diverse range of tasks and applications @xcite @xcite . but, current benchmarks may not adequately reflect the reasoning capabilities of llms @xcite and do not accurately reflect real-world situations @xcite . the validation of llms in more realistic and meaningful applications, such as legal reasoning, still requires extensive exploration. in the legal domain, the core competency of legal professionals is to apply legal rules to facts and draw conclusions, as described by the irac (issue, rule, application, conclusion) framework. as shown in figure 2 , a legal professional can determine whether a case fact conforms to specific criminal charges based on legal rules. they critically * corresponding author. assess a case against potential charges, focusing on the key points of relevant legal rules, to accurately identify the appropriate charge and distinguish inapplicable charges. legal rules, which manifest legal theories, determine the legal consequences of factual situations @xcite . therefore, properly applying legal rules reflects the grasp of legal theories. however, powerful llms may struggle to fully understand legal theories and perform basic legal reasoning tasks. existing study @xcite has found that when llms are given criminal facts and legal rules, then asked whether cases constitute a certain charge, they tend to answer "yes", regardless of whether the charge is correct (golden charge) or a closely related one (confusing charge). our empirical experiments also confirmed this issue. we sampled real-world criminal cases involving the charge of misappropriation of public fund, inputting the criminal facts and legal rules into llms, and asked whether the case constituted the golden charge. meanwhile, we created a control group where we input the same criminal facts and related legal rules, asking whether the case constituted a confusing charge (fund misappropriation). these two charges are very similar, with the key difference being whether the defendant's subject position is that of a state functionary. as shown in figure 1 , when performing legal reasoning, regardless of the prompt method or the version of gpt used, llms exhibit significant declines in performance when predicting confusing charges. generally, llms could face following challenges in legal reasoning: inconsistent reasoning. legal reasoning involves multi-step, compositional logic processes @xcite . llms can be easily distracted by the interaction when generating reasoning steps @xcite and may not be trustworthy by the tendency to give affirmative answers @xcite . missing key details. legal rules and criminal facts are often ## preliminary we propose confusing charge prediction task to evaluate the llms' ability to identify correct legal charges based on fact descriptions and legal rules. this task highlights subtle distinctions in legal rules. only llms that capture these nuances can demonstrate their understanding of legal theory. fact description: a concise description of a legal case, represented as a word sequence @xmath0, w 2 , ..., w l }. legal rule: the definition of a specific criminal charge from law articles, also a word sequence r @xmath1, w 2 , ..., w n }, where c is the criminal charge. golden charge: the true crime label of a case. confusing charge: a charge similar to the golden charge but differing in one element @xcite . to ensure llms' trustworthiness in applying legal rules, we require them to confirm the golden charge as true and reject the confusing charge as false. the task can be formalized as: llms should correctly identify the golden charge and explain why the fact description doesn't match the confusing charge, demonstrating understanding of legal theories. ## conclusion in the study, we introduce a challenging task to better evaluate llms' capability to comprehend legal theories. the proposed malr framework can automatically decomposes complex legal tasks and extracts insights from legal rules, enhancing llms' legal reasoning abilities. extensive experiments demonstrate malr's effectiveness in equipping llms with a robust understanding of legal rules.