LTRC at S em E val-2023 Task 6: Experiments with Ensemble Embeddings (2023)

Abstract: In this paper, we present our team’s involvement in Task 6: LegalEval: Understanding Legal Texts. The task comprised three subtasks, and we focus on subtask A: Rhetorical Roles prediction. Our approach included experimenting with pre-trained embeddings and refining them with statistical and neural classifiers. We provide a thorough examination ofour experiments, solutions, and analysis, culminating in our best-performing model and current progress. We achieved a micro F1 score of 0.6133 on the test data using fine-tuned LegalBERT embeddings.

## introduction rhetorical role labeling is a critical aspect of analyzing and interpreting legal documents, particularly in the field of judgment. the labeling of rhetorical roles involves identifying the semantic purpose of each sentence and categorizing the different roles played by each part of a judgment, including the parties involved, the arguments presented, and the decision made. identifying the different rhetorical roles of sentences in a legal case document can enhance the document's comprehension @xcite and aid in various downstream tasks, such as classification @xcite ), text summarization @xcite , case law analysis, and entity extraction @xcite @xcite . despite its potential benefits, the lack of structure, technical jargon, multiple themes, and high specificity makes it difficult even for legal experts to identify the rhetorical roles effectively. therefore, the task of rhetorical labeling in legal documents is a taxing and challenging one that requires advanced nlp techniques. by understanding the various rhetorical roles at play, legal professionals can more effectively analyze and interpret the content of legal documents, and ultimately, make informed legal decisions @xcite . this process can help lawyers to understand the implications of the judgment, and ensure that their clients' interests are protected in the legal system. while it may be challenging to automate the entire judicial process, automating intermediate tasks can be valuable in assisting legal practitioners and accelerating the system. nevertheless, legal documents possess unique characteristics, and the application of existing nlp models and techniques can be difficult due to the difference between legal texts and commonly occurring texts used to train such models. consequently, the development of legal domain-specific techniques is essential to enhance the application of nlp in the legal domain. as a step in that direction, task 6 (legaleval: understanding legal texts) of semeval @xcite proposed 3 shared subtasks which will act as building blocks in developing legal ai applications. in this paper, we describe our efforts on sub-task a: rhetorical roles (rr) prediction foot_0 . ## related work in recent years, there has been a surge in research in the field of legal text processing, leading to the creation of several datasets, tasks, and applications. examples include but are not limited to prior case retrieval @xcite , summarization @xcite , events and named entities extraction @xcite , and judgment prediction @xcite @xcite . rhetorical role (rr) labeling is a process of assigning specific labels such as fact, argument, final judgment, and more to individual sentences within a court case document. this task is a critical component of legal analytics, as it can improve the readability of lengthy case documents and aid in various downstream tasks. moreover, it can add a structural framework to the document, thereby enabling an organized and systematic analysis of the legal document. data mining and machine learning approaches @xcite were employed for classifying a sentence to its respective rhetorical role. however, the dataset size and the number of classes or roles are less. several neural models @xcite were also built using lstms @xcite and crf @xcite . legal judgments from the supreme court of india and the supreme court of the united kingdom (u.k.) were considered and seven rhetorical roles were identified. transformerbased models (kohli et al., 2021) like roberta @xcite with attention mechanism were employed to capture long-range relations. it has been demonstrated that incorporating both span segmentation and sequence classification (santosh et al., 2023) can enhance accuracy in various tasks. a new corpus of legal documents from diverse legal domains has been created @xcite , which includes more detailed and fine-grained annotations of 13 rhetorical role labels. this corpus forms the baseline for the shared task. ## conclusion automated intermediate tasks have the potential to enhance the capabilities of legal practitioners and accelerate the legal system, particularly in populous countries like india where the number of legal cases is increasing at an almost exponential rate. our solution for the sub-task involved experimenting with ensemble embeddings and indicated that richer embeddings help in better accuracy and overall task results. however, given the class imbalance observed for a few rhetorical labels, the models misclassify samples belonging to classes that have lower representation. enriching the embeddings and adding a crf layer for the classifier would improve the results.