T eam U nibo at S em E val-2023 Task 6: A transformer based approach to Rhetorical Roles prediction and NER in Legal Texts (2023)

Abstract: This study aims to tackle some challenges posed by legal texts in the field of NLP. The LegalEval challenge proposes three tasks, based on Indial Legal documents: Rhetorical Roles Prediction, Legal Named Entity Recognition, and Court Judgement Prediction with Explanation. Our work focuses on the first two tasks. For the first task we present a context-aware approach to enhance sentence information. With the help of this approach, the classification model utilizing InLegalBert as a transformer achieved 81.12% Micro-F1. For the second task we present a NER approach to extract and classify entities like names of petitioner, respondent, court or statute of a given document. The model utilizing XLNet as transformer and a dependency parser on top achieved 87.43% Macro-F1.

## introduction the exponential growth of pending legal cases in highly populated countries, such as india, has highlighted the need for automation in the judicial process. although a full automation of the legal domain may not be feasible, intermediate tasks can be automated to assist legal practitioners and improve efficiency. nevertheless, the unique nature of legal texts presents challenges for direct application of nlp models and techniques developed for common texts, thus requiring the development of specialized nlp approaches specifically tailored to the legal domain. the objective of the first task is to segment a given legal document by predicting the rhetorical role label for each sentence such as a preamble, fact, ratio, arguments, etc. these are referred to as rhetorical roles (rr). this segmentation is a fundamental building block for many legal ai applications like judgment summarizing, judgment outcome prediction, precedent search, etc. legal documents are typically long (tens of pages), unstructured, noisy (e.g., grammatical and spelling mistakes due to manual typing in courts), and use different lexicons (legal jargon), which makes end-to-end transformer-based pre-trained models for sentence classification ineffective. in this paper we propose an approach based on the usage of transformer-based models pre-trained on legal text and fine-tuned on a rhetorical roles dataset in a context aware way, to make them able to accurately segment even long documents. we first experimented with simple single sentence classification transformers (both pretrained on general texts and legal texts) and then, leveraging on the fact that in a document nearby sentences classes tend to influence each other, we experimented enriching the sentence representations with information coming from the context. first attaching 2 bilstm layers to the transformer embedding model and then simply extending the input tokens of the trasformer by adding the tokens from a context window around the sentence. the second task, legal ner, allows to effectively generate metadata information that can be exploited for many legal application like knowledge graph creation, co-reference resolution and in general to build any query-able knowledge base that would allow faster information access. to address this task, we've experimented two transformers-based approaches with increasing model complexity. we tried to reproduce both the baseline's methods proposed by p. @xcite . the first method, a simple linear layer as head and fine-tuning of the whole model was easy to reproduce and we were also able to reach a higher score. for the second one, which uses a transition-based parser, we tried to reproduce it stacking on top of the transformer a bilstm and a crf layer as in @xcite and in particular as in the work of r. @xcite , by the way in this case we were not able to reach the same performance. we experimented different transformers, namely roberta, inlegal-bert @xcite , which is a variant of bert trained on indian legal documents, and xl-net given it's ability to process longer sequences of tokens. after having tested all of them for the baseline model, the most performing ones (roberta and xlnet) were then used as base for our custom models. all the experiments have been conducted on the datasets provided by the challenge @xcite . only the best models were trained on the whole dataset (including the test set) in order to produce the best results possible for the challenge's submission. with this retraining procedure, the context aware model using inlegalbert reached a micro-f1 score of 81.12% on the submission data of task a, while our custom model reached 87,4% f1 score on the submission data of task b. ## related works rhetorical role prediction is a task in nlp aimed at identifying the role played by individual sentences or clauses in a larger discourse structure, such as a news article or scientific paper. this task is important for tasks such as summarization, text generation, and information retrieval. traditional approaches to rhetorical role prediction have relied on hand-crafted features and statistical models such as conditional random fields (crfs) @xcite , and support vector machines (svms) @xcite . recently, deep learning models such as transformers have shown significant promise in rhetorical role prediction. transformers, introduced by @xcite , have shown impressive results in various nlp tasks, including language modeling, machine translation, and sentiment analysis. in rhetorical role prediction, the transformer model is typically used as an encoder to produce contextualized representations of the input text. these representations are then fed into a downstream classification layer to predict the rhetorical role of each sentence or clause. several studies have shown that using pre-trained transformer models, such as bert @xcite and roberta @xcite , can significantly improve rhetorical role prediction performance over traditional approaches. named entity recognition (ner) is a fundamental task in nlp, aimed at identifying and extracting named entities, such as person names, locations, organizations, and other named entities from textual data. several deep learning models have achieved state-of-the-art performance in ner, with the introduction of neural network architectures such as recurrent neural networks (rnns), convolutional neural networks (cnns), and more recently, transformers. in ner, the transformer model is typically used as an encoder to produce contextualized representations of the input tokens. these representations are then fed into a downstream classification layer to predict the named entity tags. furthermore, recent works have explored combining the transformer model with bilstm and crf to improve the sequence labeling performance. crf is a probabilistic model that takes into account the sequence of labels and their dependencies, which can help address the label bias problem that arises in token-level classification, while bidirectional long short-term memory (bilstm) is a neural network architecture that uses two layers of long short-term memory (lstm) cells @xcite to capture both past and future context of each input token. several studies have demonstrated that incorporating crf into the transformer-based ner model can lead to further performance gains. ## conclusion given the significance of automating intermediate tasks within the judicial process in the face of growing numbers of pending legal cases in populous countries, the legaleval challenge calls for the development of nlp techniques tailored to the legal domain. for task a, we proposed a transformer-based approach for classifying rhetorical roles in legal documents. our experiment showed that fine-tuning transformer-based models pre-trained on legal text in a context-aware manner led to improved performance in this task. this was found to be more effective compared to the simple single sentence classification transformers pre-trained on either general or legal texts. thanks to this approach, the classification model utilizing inlegalbert as a transformer achieved 81.12% micro-f1 on the hidden test set. however, the model shows limited discriminating capacity with respect of some minority classes. the imbalance in the class distribution is likely due to the nature of legal documents, which contain long discussions and statements by the court and parties in the core sections of the document. a potential solution to this challenge is to use a multilabel setup, which would enable the model to predict multiple classes per sample. this would allow for further investigation of the predicted classes. for task b, we proposed a transformer-based approach with a dependency parser to detect and classify entities in legal documents. our experiments showed that fine-tuning a transformer-based model with a simple classification head is already a good solution for this ner task, but adding a dependency parser on top can be useful to increase the performances. indeed our custom model reached an higher score, but as expected, considering the long training and computational power availability for the work of @xcite , our approach was not able to get too close to the results of the best proposed baseline. beside that, a main limitation of our model was that it doesn't process an entire document but only a single sentence. in their paper the organizers proposed some post-processing steps to consider document level context, nonetheless we didn't work on that since the true goal of this competition was to develop an end-to-end neural architecture. as future improvement for our model we could experiment some techniques to inject document level information when processing a single sentence. to conclude, an ideal evaluation pipeline for legal documents could entail a sequential execution of sentence classification and named entity recognition, utilizing the predicted rhetorical role to enhance the performance of the latter.