Transformers at S em E val-2024 Task 5: Legal Argument Reasoning Task in Civil Procedure using R o BERT a (2024)

Abstract: Legal argument reasoning task in civil procedure is a new NLP task utilizing a dataset from the domain of the U.S. civil procedure. The task aims at identifying whether the solution to a question in the legal domain is correct or not. This paper describes the team “Transformers” submission to the Legal Argument Reasoning Task in Civil Procedure shared task at SemEval-2024 Task 5. We use a BERT-based architecture for the shared task. The highest F1-score score and accuracy achieved was 0.6172 and 0.6531 respectively. We secured the 13th rank in the Legal Argument Reasoning Task in Civil Procedure shared task.

## introduction mastering the art of arguing a legal case is essential for lawyers. this necessitates deep knowledge of the particular area of law along with advanced reasoning capabilities, including drawing similarities and differences. researchers have made significant efforts towards setting the benchmark models for the new natural language processing (nlp) problems in the domain of legal language understanding @xcite . the task, legal argument reasoning task in civil procedure 1 @xcite , organized at semeval-2024 aimed at classifying the solution to a given problem as right or wrong. classifying an answer to a given question as correct or incorrect is a new nlp task. in particular, in the legal domain limited number of publicly available corpora exist. this contributes to added difficulty of this task @xcite . recent advances in the field of nlp have addressed various issues, such as long texts and under-resourced domains. these include long short term memory @xcite , and gated recurrent units (chung et al., 1 https://github.com/trusthlt/semeval24 2014). but transformers @xcite have taken the performance to new heights which were not possible earlier. in the past, various efforts have been made to perform domain-specific adaption of different existing techniques and models. some of these adaptions include scibert which was pre-trained for scientific texts, specifically in the bio-medical domain @xcite . similarly, biobert was created with special emphasis on the bio-medical area @xcite . in this paper, we discuss our use of a transformerbased model, roberta, in the shared task of legal argument reasoning task in civil procedure at semeval-2024. ## related work researchers have used and explored various techniques in the past. in the work done by @xcite ; @xcite , it was found that bertbased architectures did not perform very well on problems that required specialized domain knowledge. two possible solutions were found to address this issue. the first was to further pre-train bert on domain-specific corpora, and the second possible solution was to pre-train bert from scratch on domain-specific corpora @xcite . @xcite performed domain-adaption of bert in the bio-medical domain. the experiment explored the effect of further pre-training bert base for 470,000 steps on biomedical articles. the performance of the resulting model, biobert, was evaluated on biomedical datasets. this led to an improvement in performance when compared to bert base. @xcite proposed a family of bertbased models, scibert, for scientific texts with a special focus on the bio-medical domain. two approaches were followed for scibert, the first was further pre-training bert base, and the second figure 2: label generation for unseen data in the work carried out by chalkidis et al. (2020), bert domain adaption was performed for the legal domain. a systematic analysis was performed for the three available techniques. the first technique was to use bert out of box, the second technique was to perform additional pre-training on bert using domain-specific corpora, and the third approach was to perform pre-training from the start using domain-specific corpora. ## dataset description the dataset provided by the organizers was selected from the domain of the u.s. civil procedure and is based on a book aimed at law students. in the training set, there are 666 instances, out of which 505 are labeled as 0 and 161 are labeled as 1. for each instance in the training data, there is a general introduction to a case, a question from that case, a possible argument solution along with a detailed analysis of why the argument is valid for that case. the test set, on the other hand, contains a question, answer and an explanation on the basis of which a label needs to be assigned to each instance. the assigned label will indicate whether or not the answer to the question is right or not. ## methodology it was observed that in the dataset provided by the organizers, the number of instances in class 0 was 505, while the number of instances labeled as 1 was 161. hence, in order to address the data imbalance, minority sampling was performed by randomly picking 161 instances from those labeled as class 0. this ensured that no bias existed in the trained model. for identifying whether the answer to a given problem was correct or not, the roberta large model was employed. the roberta model was designed by facebook ai in 2019 @xcite . roberta is a pre-trained transformer model which was trained in a self-supervised manner, i.e. only raw texts were used to train it without the involvement of human labeling. while training the model, all the fields present in the training data, namely, question, answer, and analysis, were used to predict the provided label. the weighted adam optimizer along with crossentropy loss was used as the optimizer and the loss function respectively. the learning of the optimizer was set at 1e-5. the roberta model was trained for 100 epochs with the aforementioned parameters with a batch size of 8. the training procedure has been summarised in figure 1 . the fine-tuned transformer was used to then predict the label for the unseen data as shown in figure 2 . ## results and discussion a bert-based transformer, roberta was discussed to perform categorization of an answer as right or wrong given a case, question, and a possible answer. the data imbalance was handled by performing under sampling on the majority class instances in a random fashion. this was followed by fine-tuning the roberta large model for 100 epochs. after fine-tuning, the model achieved an f1 score of 0.5511 and an accuracy of 0.6020. as shown in overall, we achieved the 13 th rank in the legal argument reasoning task in civil procedure shared task at semeval-2024 out of the 21 participating teams. ## conclusion and future work legal argument reasoning is a new nlp task, aimed at classifying a candidate answer as correct or incorrect given an introduction to the topic, a question and a candidate answer. in this work, we describe our use of a bertbased architecture, roberta in the legal argument reasoning task in civil procedure shared task at semeval-2024. ensembling techniques have shown promising results on various nlp tasks in different domains. using an ensemble approach of different transformers may hence improve the performance. transformers trained specifically with a focus on legal transformation such as legal-bert @xcite can improve the performance further.