NLP at UC S anta C ruz at S em E val-2024 Task 5: Legal Answer Validation using Few-Shot Multi-Choice QA (2024)

Abstract: This paper presents our submission to the SemEval 2024 Task 5: The Legal Argument Reasoning Task in Civil Procedure. We present two approaches to solving the task of legal answer validation, given an introduction to the case, a question and an answer candidate. Firstly, we fine-tuned pre-trained BERT-based models and found that models trained on domain knowledge perform better. Secondly, we performed few-shot prompting on GPT models and found that reformulating the answer validation task to be a multiple-choice QA task remarkably improves the performance of the model. Our best submission is a BERT-based model that achieved the 7th place out of 20.

## introduction the field of natural language processing (nlp) has made significant strides in understanding and generating human language. yet, specialized fields such as legal reasoning within the sphere of civil procedure pose distinct challenges. these challenges stem from the intricate nature of legal texts and the requisite domain-specific knowledge. in this paper, we present a solution to the problem posed in semeval 2024 task 5, which introduces a new nlp task and dataset focused on the u.s. civil procedure. our approach in this task aims to evaluate the ability of large language models (llms) in interpreting and applying legal principles and laws to specific case questions. to support this paper, we have made our codebase publicly available as a github repository. * this repository contains all our code and instructions how to run it. at the request of the task organizers, we have not included the dataset splits as they are meant to be private. the dataset for this task is curated from "the glannon guide to civil procedure" @xcite . it comprises a series of legal cases, each with a general introduction, a specific question related to u.s. civil procedure, and a possible answer candidate. for every answer choice, a comprehensive analysis is provided, rationalizing why it is correct or incorrect. a correct answer is labeled with 1, and an incorrect answer is labeled with 0. the dataset is available in english, and its training, validation, and test splits contain 666, 84, and 98 examples respectively. ## related work the domain of legal question-answering systems has witnessed substantial progress, utilizing cutting-edge computational techniques to address the complexities of legal discourses. a prime example of innovation in this field is the legal-bert system, introduced by @xcite , illustrating the enhanced efficacy of models tailored specifically for legal content through the pretraining of bert @xcite on legal documents @xcite . in addition, @xcite achieved a notable breakthrough by developing a flexible legal question-answering system that goes beyond conventional query patterns, incorporating sparse vector search with a bert-based re-ranking process. the expansion of legal corpora, notably with the casehold corpus by @xcite , marked a significant stride forward. this work employed language models for legal analysis, setting a comprehensive standard for measuring model effectiveness in legal reasoning tasks using a dataset based on u.s. court cases. furthermore, the creation of targeted nlp tasks has played a crucial role in the assessment of models and systems in this field. an important example is the task introduced by @xcite , which is the focus of this paper. these works collectively underscore the diverse methodologies and technological advancements employed in legal question answering. each of these contributions brings unique insights and solutions, paving the way for more sophisticated and efficient legal question-answering systems in the future. ## system overview our approach at creating a system entailed utilizing few-shot prompting on openai's gpt-3.5 (ope-nai, 2023a) and @xcite . we devised a script with a prompt, two examples of the expected model input and desired output, and used an altered version of the task dataset as our input queries for this system. we experimented with a few prompts for both gpt models, trying to figure out what gave us the best results. following the guidance outlined in @xcite and @xcite , we decided to structure our prompt in the following manner: additionally, we also altered the dataset from a binary classification format to a multi-choice qa format. rather than presenting individual questionanswer pairs, each question was now accompanied by the entire set of potential answer choices. figure 1 demonstrates a visual example of this restructuring. as can be inferred from figure 1 , the binary classification format of the dataset requires the system to label each answer choice as 0 or 1, whereas the multi-choice format requires the system to return one single answer prediction for each question. we chose to convert the dataset in this way to address an issue we found in the experimental phase of our work. this issue is elaborated upon in section 4. a small note on our multi-choice qa format, we added an additional option "none of the above" because in some cases, the training or the validation data had all incorrect answer choices listed for a given question. this was done to make sure that the model would not be forced to pick between multiple incorrect answers. binary classification format question: <@xmath0: <@xmath1: <answer candidate 1> question: <@xmath2: <@xmath3: <answer candidate 2> multi-choice qa format question: <@xmath4: <@xmath5: {0: <answer candidate 1>, 1: <answer candidate 2>, 2: <none of the @xmath6. we ran experiments on both dataset formats to compare system performance, and section 5 contains our results for evaluation metrics. ## results our submission to the semeval task ranked 7th out of 20 on the competition leaderboard. what we submitted to the task competition was our best performing finetuned bert model after applying the rule-based algorithm. this was not our best method, as we were able to achieve higher metrics through subsequent experimentation. our best results overall can be seen in our best results without utilizing the rule based algorithm came from few-shot prompting with gpt models using the multi-choice qa format of the dataset.