MAINDZ at S em E val-2024 Task 5: CLUEDO - Choosing Legal o U tcome by Explaining Decision through Oversight (2024)

Abstract: Large language models (LLMs) have recently obtained strong performance on complex reasoning tasks. However, their capabilities in specialized domains like law remain relatively unexplored. We present CLUEDO, a system to tackle a novel legal reasoning task that involves determining if a provided answer correctly addresses a legal question derived from U.S. civil procedure cases. CLUEDO utilizes multiple collaborator models that are trained using multiple-choice prompting to choose the right label and generate explanations. These collaborators are overseen by a final “detective” model that identifies the most accurate answer in a zero-shot manner. Our approach achieves an F1 macro score of 0.74 on the development set and 0.76 on the test set, outperforming individual models. Unlike the powerful GPT-4, CLUEDO provides more stable predictions thanks to the ensemble approach. Our results showcase the promise of tailored frameworks to enhance legal reasoning capabilities in LLMs.

## introduction recent improvements in large language models are leading to a rethinking of legal practices, particularly in the united states @xcite @xcite . this can potentially transform timeconsuming tasks such as brief writing and corporate compliance @xcite . this could also contribute to alleviating the access-to-justice crisis @xcite . the unique properties of llms, including their ability to learn from limited labeled data and proficiency in complex reasoning tasks, make them appealing for legal applications @xcite @xcite . however, enthusiasm is tempered by concerns about the risks associated with llms, such as generating offensive, misleading, or factually incorrect content @xcite . these issues could have significant consequences, particularly affecting marginalized or under-resourced populations @xcite @xcite . to address safety implications, there is a pressing need to evolve and enhance legal reasoning capabilities in llms. despite this urgency, practitioners face challenges in assessing llms' legal reasoning capabilities, as existing legal benchmarks are limited and often fail to capture the diverse aspects of legal tasks @xcite . in this direction, the organizers of semeval-2024 task 5 introduce a novel natural language processing (nlp) task and dataset derived from the u.s. civil procedure domain @xcite . each dataset instance comprises a case introduction, a specific question, and a potential solution argument, along with an in-depth analysis justifying the argument's applicability to the case. when provided with a topic introduction, a question, and a potential answer, the objective of the proposed task is to determine whether the given answer is accurate or not. to tackle this task, we initially transform the dataset into a multiple-choice question answering problem using the multiple-choice prompting (mcp) approach @xcite . we experimented with various open-source language models on this modified dataset, including flan t5 xxl @xcite , llama 7b and 13b @xcite , zephyr 7b @xcite , and mistral 7b @xcite . specifically, we trained these models to solve legal problems while also providing an explanation for the predicted outcome, leveraging the analysis provided. we thus introduce the cluedo approach, which stands for "choosing legal outcome by explaining decisions through oversight". this framework utilizes multiple collaborative models to synthesize the final outcome based on each model's predictions. each individual model is trained to predict the label of the correct candidate answer and generate an explanation accordingly. the final "detective" model operates in a zero-shot manner, relying upon the outputs of the collaborators. the model processes the answers and the explanations of all collaborators and deduces the ultimate answer. the results on the challenge dataset demonstrate that our proposed methodology surpasses the performance of single models trained with standard fine-tuning. furthermore, our approach achieved the second-place position in the public competition, achieving a final test f1 macro score of 0.77 foot_0 . ## system overview this section provides a comprehensive overview of the proposed methodology. firstly, we outline the approach to the multiple-choice questionanswering problem and how we adapt it to our scenario. secondly, we introduce the cluedo framework, along with details about the competitors incorporated into our study. ## results to illustrate the efficacy of the multiple-choice setting and model selection criteria, we conduct individual tests for each configuration and present the obtained results on the development set. the following paragraphs address the research questions previously presented. rq1: impact of the multiple-choice setting. ## conclusion this paper presents a novel solution to the semeval 2024 -legal reasoning task, which introduced a challenge for evaluating contemporary legal language models. we transform the original dataset into a multiple-choice question-answering problem using the multiple-choice prompting approach and propose an original system, namely cluedo, that utilizes multiple collaborative llms and employs a final "detective" model to predict the outcome. results show that our framework outperforms individual models in the public competition while returning more stable predictions, securing second place in the public competition.