Mistral at S em E val-2024 Task 5: Mistral 7 B for argument reasoning in Civil Procedure (2024)

Abstract: At the SemEval-2024 Task 5, the organizers introduce a novel natural language processing (NLP) challenge and dataset within the realm of the United States civil procedure. Each datum within the dataset comprises a comprehensive overview of a legal case, a specific inquiry associated with it, and a potential argument in support of a solution, supplemented with an in-depth rationale elucidating the applicability of the argument within the given context. Derived from a text designed for legal education purposes, this dataset presents a multifaceted benchmarking task for contemporary legal language models. Our manuscript delineates the approach we adopted for participation in this competition. Specifically, we detail the use of a Mistral 7B model to answer the question provided. Our only and best submission reach an F1-score equal to 0.5597 and an Accuracy of 0.5714, outperforming the baseline provided for the task.

## introduction the content of the task 5 hosted at semeval-2024 @xcite , was originally introduced in @xcite . asserting a legal argument represents a fundamental proficiency necessary for aspiring legal professionals to acquire. this proficiency demands not only a comprehension of pertinent legal domains but also advanced reasoning skills, including the utilization of analogy-based arguments and the identification of implicit contradictions. despite recent strides in establishing objective metrics for contemporary natural language processing (nlp) models across diverse facets of legal language comprehension, the absence of a sophisticated task addressing argumentative reasoning within legal contexts persists. in this article, is discussed a novel task alongside a corresponding benchmark dataset. the introduction of a genuinely challenging task, sourced from legal educational resources, will serve to elucidate strengths and weaknesses inherent in contemporary legal transformer models, including but not limited to legal-bert @xcite . specifically, at the semeval-2024 task 5 is unveiled a novel, openly accessible legal dataset tailored for the binary text classification of issues within u.s. civil procedure. the primary objective is to ascertain whether a proposed solution to a given inquiry is deemed accurate or erroneous. the corpus draws inspiration from "the glannon guide to civil procedure" authored by joseph glannon @xcite , which caters to law students by offering a comprehensive examination of fundamental u.s. civil procedure topics, inclusive of multiple-choice queries designed to assess reader comprehension. through the inception of this freshly minted corpus, the intent extends to scrutinizing the efficacy of various methodological approaches while establishing performance benchmarks. to address these objectives, there is an ongoing demand for automated tools capable of extracting and categorizing data, facilitating the classification with recent nlp models. recent advancements in the area of the machine and deep learning architectures have spurred heightened interest in natural language processing (nlp). substantial endeavours have been directed towards devising techniques for the automated identification and categorization of textual content accessible on the internet today. in the literature, to perform text classification tasks, several strategies have already been proposed. in the last fifteen years, some of the most successful strategies have been based on svm @xcite , on convolutional neural network (cnn) @xcite , on graph neural network (gnn) @xcite , on ensemble models @xcite and, recently, on transformers @xcite . participants in semeval-2024 task 5 were tasked as follows. the task at hand involves evaluating the accuracy of an answer candidate provided in response to a question, accompanied by a brief introductory passage pertaining to the subject of the question. the objective is to ascertain whether the candidate answer is indeed incorrect or correct. to face with the task, we propose a transformer-based approach which made use of mistral 7b @xcite . we used the model in a zero-shot setup described in the rest of this paper. specifically, we prompted the latest pretrained version of mistral with each sample in the dataset. specifically, we provided a candidate answer to a question, asking the model if the answer to the legal question was correct or not. the model replied with a yes or no, eventually providing some further explanation. the subsequent sections of this work are structured as follows: section 2 offers background information on task 5, held at semeval-2024. in section 3, we outline the approach introduced in this study. section 4 delves into the specifics of the experimental setup employed to reproduce our findings. the outcomes of the official task and relevant discussions are presented in section 5. finally, section 6 concludes our study and suggests avenues for future research. we make all the code publicly available and reusable on github foot_0 . ## background for the task 5 at semeval-2024 is proposed a legal corpus, publicly accessible for binary text classification tasks focusing on issues within u.s. civil procedure. the primary objective is to determine the correctness of solutions provided in response to specific questions. this corpus draws its content from "the glannon guide to civil procedure" authored by joseph glannon (glannon, 2023), tailored for law students. the book encompasses fundamental u.s. civil procedure topics and includes multiple-choice questions aimed at evaluating reader comprehension. through collaboration with the author and publisher, task organizers secured permission to utilize the content of "the glannon guide to civil procedure" for constructing this dataset, which is freely available to the research community. the book comprises 25 chapters, each containing multiple-choice questions pertaining to a particular topic, prefaced by an introduction. every question is followed by 3 to 5 answer candidates, among which one is deemed correct. these answer candidates serve as hypotheses, necessitating an examination of their respective prerequisites for accuracy. the correctness or incorrectness of an answer is subsequently expounded upon in the accompanying analysis. the dataset construction process involved automated parsing of the book's content, leveraging its structured format to extract individual components of each instance (i.e., introduction, question, answers, and analysis). additional parsing rules were employed to detect anomalies in the structure, such as instances where the same introduction was shared across multiple questions. however, certain sections of the book required manual extraction, particularly regarding the correctness of answer candidates, as this information was typically embedded within the free-text analysis section. the analysis segments were organized to address each answer candidate separately, classifying them as true or false. to achieve this, the organizers adopted a strategy of isolating the relevant aspects for each answer, despite the absence of explicit keywords or structural indicators guiding the segmentation process. despite efforts to maintain consistency, some structural inconsistencies were noted throughout the dataset. two samples from provided datasets are available online foot_1 and reported in the the organizers adhere to the schedule for se-meval24, which means the following dates: ## system overview even if it has already been proved that the transformers are not necessarily the best option for any text classification task @xcite , depending on the goal, some strategies like domainspecific fine-tuning @xcite , or data augmentation @xcite @xcite ) can be beneficial for the considered task. so far, several large language models (llms) have proved to be able to address a plethora of different nlp tasks. for example, in the recent literature, there has been mention of llama, as presented by @xcite . llama stands out as a collection of publicly available large language models (llms) that rival the capabilities of closed-source counterparts like gpt-3. however, to address the task 5 hosted at semeval-2024 we made use of a zero-shot learning approach @xcite , making use of mistral 7b @xcite . mistral 7b, a language model boasting 7 billion parameters, is engineered to excel in both performance and efficiency. in comparison to the leading open 13b model (llama 2), mistral 7b demonstrates superior performance across all assessed benchmarks. moreover, it outperforms the leading publicly available 34b model (llama 1) across various tasks involving code generation, mathematical operations, and reasoning. the model capitalizes on groupedquery attention (gqa) to expedite inference, complemented by sliding window attention (swa) to effectively process sequences of varying lengths while minimizing inference costs. additionally, a fine-tuned variant, mistral 7b -instruct, is tailored for adhering to instructions. this version, outperforms llama 2 13b -chat model across both automated and human benchmarks. the introduction of mistral 7b instruct underscores the ease with which the base model can be fine-tuned to achieve notable performance enhancements. notably, this variant lacks any moderation mechanisms. our approach is few-shot (littenberg-tobias et al., 2022) and make use of the above-mentioned mistral 7b. more specifically, given the task hosted at semeval-2024, we asked the model: "is the answer to the question above true or false? answer using only true or false:". to this request, the model replied with one or more words -usually starting with a true or false -that we parsed to extract one of the two labels (i.e., 0 for false and 1 for true). for example, given the introduction: ## experimental setup we implemented our model on google colab. the library we used come from huggingface and as already mentioned is mistral 7b foot_2 . we employed the v0.2 iteration of mistral 7b, which represents an enhanced version of the mistral-7b-instruct-v0.1 model. to harness the capabilities of instruction fine-tuning, prompts must be enclosed within @xcite and [/inst] tokens. additionally, the initial instruction should commence with a sentence identifier. the next instructions should not. the assistant generation will be ended by the end-ofsentence token id. we also imported the llama library @xcite from llama_cpp. the library is fully described on github foot_3 . the dataset provided for all the phases are available on the official competition page. we did not perform any additional fine-tuning on the model. to run the experiment, a t4 gpu from google has been used. after the generation of predictions, we exported the results on the format required by the organizers. as already mentioned, all of our code is available on github. ## results given the binary nature of the classification task, the organizers proposed f1 score and accuracy as the two evaluation metrics to be considered for the final ranking. the f1 score is defined in the equation 1. where tp stands for the number of correctly predicted right answers, fp stands for the f1 accuracy mistral 7b 0.5597 0.5714 given the previous definitions, the accuracy is defined as stated in the equation 2 . in in the ## conclusion this paper presents the application of a mistral 7b-model for addressing the task 5 at semeval-2024. for our submission, we decided to follow a zero-shot learning approach, employing as-is, an in-domain pre-trained transformer. after several experiments, we found beneficial to build a prompt containing the question for the model. then we provide as a prompt: the introduction, the question and an answer candidate. the model is asked to decide whether the candidate answer is correct or not. the task is challenging, and there is still opportunity for improvement, as can be noted looking at the final ranking. possible alternative approaches include utilizing the few-shot capabilities or also the use of other models like gpt and t5, eventually using further data, or directly integrating other samples from the training and from the development sets. further improvements could be obtained with a fine-tuning and modelling the problem as a text classification task. furthermore, given the interesting results recently provided on a plethora of tasks, also other few-shot learning @xcite @xcite or data augmentation strategies @xcite ; tapia-téllez and escalante, 2020; siino and tinnirello, 2023) could be employed to improve the results. looking at the final ranking, our simple approach exhibits some room for improvements. however, it is worth notice that required no further pre-training and the computational cost to address the task is manageable with the free online resources offered by google colab.