eagerlearners at S em E val2024 Task 5: The Legal Argument Reasoning Task in Civil Procedure (2024)

Abstract: This study investigates the performance of the zero-shot method in classifying data using three large language models, alongside two models with large input token sizes and the two pre-trained models on legal data. Our main dataset comes from the domain of U.S. civil procedure. It includes summaries of legal cases, specific questions, potential answers, and detailed explanations for why each solution is relevant, all sourced from a book aimed at law students. By comparing different methods, we aimed to understand how effectively they handle the complexities found in legal datasets. Our findings show how well the zero-shot method of large language models can understand complicated data. We achieved our highest F1 score of 64% in these experiments.

## introduction becoming skilled at presenting a legal case is essential for aspiring lawyers. it requires understanding not only the relevant legal areas but also using advanced reasoning tactics like making analogies and spotting hidden contradictions. @xcite . despite efforts to set standards for modern nlp models in legal language understanding, there still aren't complex tasks focusing on argumentation in legal matters. @xcite the dataset utilized in this study was gathered from the glannon guide to civil procedure by joseph glannon @xcite in english. each sample within the dataset comprises a question, a solution, and an introduction elaborating on the provided solution. the objective is to identify whether the given answer, derived from the introduction text, accurately addresses the question. this paper explores various approaches to address the challenge of handling lengthy and intricate data, which can be challenging for human comprehension. initially, we evaluated two models-longformer @xcite and big bird @xcite -known for their effectiveness in classifying data with large input tokens. subsequently, we assessed the performance of two pre-trained models, legal-roberta @xcite and legal-xlm-roberta @xcite for legal data using the original code. finally, we compared the performance of three large language models -gpt 3.5, gemini, and copilot-using the zero-shot method on the test dataset. we recognized the significant impact of leveraging the capabilities and extensive capacity of large language models on analyzing data, especially those focusing on specific topics or lengthy content. looking ahead, our goal is to improve prompts further to achieve superior results not only for this task but also for similar works. further details regarding the implementation can be found in this github repository. ## experimental setup we utilized a dataset of 848 data points, dividing it into 80% for the training dataset, 10% for the validation dataset, and 10% for the test dataset. our experimental approach involved exploring three main inquiries: in the initial phase, we employed the adamw optimizer function with a learning rate set at 5e-5 for both models, conducting training over 3 epochs. the specific values chosen for each hyperparameter are listed in then, in the second phase, we adapted the original code from the article with the necessary modifications. 4 the original code featured the utilization of sliding window simple (sws) and sliding window complex (swc) methods with the legal-bert model. we made adjustments to certain sections of the code to ensure compatibility with the test input data. throughout our evaluation, we utilized the macro f 1 score as the primary evaluation metric. in the final phase of our experiments, we used the api keys provided by openai and google. unfortunately, despite our efforts, we encountered challenges locating the official api for bing. given its superior accuracy compared to other models, we decided to manually record the results of the test dataset. throughout this phase, we employed a variety of prompts and iteratively refined them. our investigation revealed that incorporating terms such as "step by step" or asking for explanations of the inference steps to achieve the desired outcome positively impacted the model's performance. furthermore, we encountered limitations in prompt completeness due to the token size constraints inherent in the input models. for instance, bing's copilot supported a maximum input size of 4000 characters, which proved insufficient for processing our long samples. here is an example of an input prompt: i will provide a question, an answer, and an explanation. your task is to determine if the answer is correct based on the explanation provided. after reading the explanation, please respond with 'yes' if the answer is correct, or 'no' if it is incorrect. ## results after completing the aforementioned three phases, our investigation revealed that despite fine-tuning, existing models struggled to effectively analyze lengthy data within challenging legal contexts, encountering training process issues and yielding suboptimal outputs. moreover, the most promising result emerged from fine-tuning the legal-bert model, serving as the baseline. while we continue to analyze the underlying reasons for this outcome, initial observations suggest that the learning challenge may be linked to the specific characteristics of the dataset employed. when it comes to evaluating the results of the zero-shot method, we identified its considerable potential and bing's copilot model emerged as the top performer, surpassing expectations. following suit, the gpt 3.5 model presented moderate performance, while the gemini model fell short of expected levels. the success of the copilot model lies in its ability to address previous challenges associated with gpt models by leveraging real-time information accessible through the internet. ## conclusion in this paper, we present methods designed for classifying lengthy legal cases. we divided our exploration into three main parts: firstly, we looked into models with large input token sizes such as longformer and big bird. secondly, we examined pre-trained models specifically fine-tuned for legal data, such as legal-roberta and legal-xlm-roberta. lastly, we tested the zero-shot method across three major language models. among these methods, we found that the zeroshot technique and bing's copilot model showed the most promising performance. as for future works, we can explore techniques like data summarization, collaborative approaches such as the round table technique, trying various hyperparameters, and refine prompts to further enhance model performance. these efforts have the potential to advance the effectiveness of classification tasks in legal contexts. as a future work, it would be valuable to explore additional large language models. these models offer extensive capabilities, especially in summarizing lengthy datasets, which could help evaluate various models' performance. however, it is necessary to note that during summarization, some important details might be overlooked. another avenue for future research involves testing the effectiveness of a multi-model approach. @xcite this method entails bringing together different large language model agents in a round table conference format. this setup encourages diverse perspectives and discussions to foster consensus. by adopting this approach, researchers can tap into the combined intelligence of multiple models, potentially enriching analysis across various tasks and domains. in this competition, our team achieved the 17th rank out of 21 groups. our only submission during the evaluation phase utilized the basic prompt and the gpt model. however, significant improvements were made during the post-evaluation phase, resulting in a much higher level of accuracy.