SU - FMI at S em E val-2024 Task 5: From BERT Fine-Tuning to LLM Prompt Engineering - Approaches in Legal Argument Reasoning (2024)

Abstract: This paper presents our approach and findings for SemEval-2024 Task 5, focusing on legal argument reasoning. We explored the effectiveness of fine-tuning pre-trained BERT models and the innovative application of large language models (LLMs) through prompt engineering in the context of legal texts. Our methodology involved a combination of techniques to address the challenges posed by legal language processing, including handling long texts and optimizing natural language understanding (NLU) capabilities for the legal domain. Our contributions were validated by achieving a third-place ranking on the SemEval 2024 Task 5 Leaderboard. The results underscore the potential of LLMs and prompt engineering in enhancing legal reasoning tasks, offering insights into the evolving landscape of NLU technologies within the legal field.

## introduction legal texts, including laws, interpretations, arguments, and agreements, are commonly conveyed through writing, resulting in great amount of legal documents. analyzing these documents, a core aspect of legal work, becomes more intricate as these collections expand. natural language understanding (nlu) technologies offer potential assistance to legal professionals in this regard. however, their effectiveness hinges on the ability of current stateof-the-art models to adapt to diverse tasks within the legal field. the legal argument reasoning task @xcite of semeval-2024 represents a significant challenge in the domain of natural language processing (nlp) and an informal addition to the currently existing model evaluation benchmarks such as lexglue @xcite . our approach involves fine-tuning pre-trained bert models and exploring the innovative use of large language models (llms) through prompt engineering to address this task. as a result of our work, we are ranked 3-rd in the semeval 2024 task 5 foot_0 leaderboard out of 20 participating teams. the implementations of the different approaches is available on github foot_1 and the fine-tuned models could be accessed in huggingface foot_2 . ## background task 5 of semeval 2024 is novel nlp problem focused on legal argument reasoning within the context of u.s. civil procedure. it contributes a dataset comprised of instances each containing a general introduction to a case, a specific legal question, a proposed solution argument, and a detailed analysis explaining the applicability of the argument. this dataset aims to benchmark the performance of legal language models, posing a significant challenge due to the complexity and nuanced understanding required for legal reasoning. instances are organized to support a binary classification task: determining the correctness of a given answer to a legal question, aimed at facilitating research on legal argument reasoning. in the domain of text classification, conventional methodologies often employ "short encoders" such as bert @xcite , roberta @xcite , and xlnet @xcite , which have demonstrated commendable efficacy in diverse contexts, ranging from news topic classification to sentiment analysis in movie reviews. nevertheless, these encoders are constrained by their 512token processing limit, rendering them less effective for analyzing extensive documents like court judgments. to circumvent this limitation, more advanced approaches, including the hierarchical attention network (han) @xcite , a synergy of bert and cnn, and the combination of xlnet with bigru @xcite , have been developed, enhancing the semantic understanding of longer texts. despite these technological strides, the pursuit of an optimal algorithm that can adeptly navigate the complexities of extended documents persists. the application of automated systems in the legal domain encounters distinct challenges, arising from the specialized language employed and the necessity for intricate multi-step reasoning over extensive texts. furthermore, the potential of leveraging recent advancements in prompting techniques for legal domain-specific tasks remains largely unexplored. typically, effective prompting in general nlp tasks has been noted with concise inputs, often limited to a single sentence or a small collection of sentences, accompanied by a restricted array of target labels. this underscores the ongoing quest to adapt and refine nlp techniques to meet the unique demands and intricacies of legal reasoning. ## system overview after analyzing the dataset, we identified that the final system should be capable of handling relatively lengthy contexts and to perform well on reasoning and fact-checking tasks. in this section we separately introduce the different approaches we have experimented with on solving the legal argument reasoning task by dividing them into methods for handling long texts and such for optimizing the nlu capabilities for the legal domain. ## results notably, the integration of large language models (llms) with legal prompt engineering (lpe) techniques, particularly with gpt-3.5-turbo and gpt-4, led to a significant leap in performance metrics. these models outperformed traditional transformer models, highlighting the effectiveness of lpe in enhancing the model's ability to interpret and reason over legal texts. the comparative analysis of model performances on both development and test datasets revealed consistent patterns. models utilizing llms with lpe not only achieved the highest macro-f1 scores but also demonstrated robustness across different data sets, underscoring their potential for real-world applications in legal reasoning and argu-mentation. ## conclusion our participation in semeval-2024's legal argument reasoning task has yielded valuable insights into the capabilities of transformer-based models and llms in processing and reasoning over legal texts. while our methods have shown promise, particularly in leveraging llms and prompt engineering, the complexity of legal reasoning poses ongoing challenges. further investigation can be done in a solution based on a hierarchical transformer variant such as hier-bert @xcite , @xcite . our initial experiments with that model architecture did not lead to very high results (0.47 f1-macro on the dev partition) and because of the setup complexity, we decided to leave it for future research opportunities.