DUT h at S em E val 2024 Task 5: A multi-task learning approach for the Legal Argument Reasoning Task in Civil Procedure (2024)

Abstract: Text-generative models have proven to be good reasoners. Although reasoning abilities are mostly observed in larger language models, a number of strategies try to transfer this skill to smaller language models. This paper presents our approach to SemEval 2024 Task-5: The Legal Argument Reasoning Task in Civil Procedure. This shared task aims to develop a system that efficiently handles a multiple-choice question-answering task in the context of the US civil procedure domain. The dataset provides a human-generated rationale for each answer. Given the complexity of legal issues, this task certainly challenges the reasoning abilities of LLMs and AI systems in general. Our work explores fine-tuning an LLM as a correct/incorrect answer classifier. In this context, we are making use of multi-task learning toincorporate the rationales into the fine-tuning process.

## introduction in recent years, large language models (llm) development has witnessed unprecedented advancements, with large language models such as gpt-3 demonstrating remarkable capabilities in understanding and generating human-like text. however, the effectiveness of these models in reasoning tasks remains an area of ongoing exploration and enhancement. while llms excel in linguistic fluency and context understanding, their capacity for reasoning often falls short of human-level comprehension @xcite . in this paper, we describe the duth participation in semeval 2024 task 5: the legal argument reasoning task in civil procedure @xcite 1 , on leveraging the reasoning capabilities of large language models for multiple-choice question-answering in the context of us civil procedure. the task can be formulated as follows: the training set has a size of 666 entries, which is smaller compared to other similar datasets aiming to examine the capabilities of llms using human-generated rationales @xcite . although the complexity of legal domain text and the number of details engulfed in real legal cases are large. the cognitive skills required to understand and handle legal cases make this task an interesting challenge for llms' reasoning abilities. our proposed system is a legalbert @xcite classifier, fine-tuned on a downstream task incorporating rationales for each answer. our code implementation builds on the organizers' and is publicly available. 2 additionally, we experimented with a multi-task flan-t5 model. this strategy involves a different way to use rationales in the training process. the model is trained to predict the correct labels and, at the same time, generate relevant rationales. its performance is evaluated through a custom loss function that accounts for the loss of the label prediction task and the loss of the rationale generation task separately. although it did not surpass the performance of the legalbert classifier, it is an interesting approach that can be further examined on the current task. ## results the comprehensive scores of our systems across the utilised models are presented on firstly, regarding legalbert, it is possible that simply adding the rationale to the input along with the introduction, question and answer will not helping the model learn the logic behind justifying each answer. in fact, it makes the model perform worse compared to setups where only introduction, question and answer is used @xcite . additionally, our multi-task system, although incorporating a more complex training mechanism, it does not seem to be able to distinguish answers efficiently. the small version of flan t5 is only of 80 million parameters. at this scale, it might be difficult for language models to grasp complex concepts laying on rationales. this, in fact, can be confirmed by prompting the multi-task model to generate rationales based on the input. the generated rationales barely makes any sense. ## conclusion through our experiments, we could not find a significantly performing system. even the multi-task approach, which makes good use of the rationales to better establish a connection between input and label, could not perform well. but we demonstrated the possible limitations and difficulties of such tasks, where logical reasoning is needed in order for a model to perform well. the primary benefit of multi-task learning lies in the use of rationales, enabling the model to perceive the reasons behind the correctness or incorrectness of every answer. in this work, our capabilities were constrained by hardware limitations, leading us to experiment with a smaller language model. however, this model's capacity to comprehend longer content is limited by its size. next steps could involve experimentation with bigger language models regarding the multi-task approach. we believe that a larger model could better grasp the context of the rationales and draw better associations between a question and possible answers. another approach regarding the multitask strategy is to incorporate rationales through a more efficient loss function. another weighing strategy could be used, for example.