A nt C ontent T ech at S em E val-2023 Task 6: Domain-adaptive Pretraining and Auxiliary-task Learning for Understanding I ndian Legal Texts (2023)

Abstract: The objective of this shared task is to gain an understanding of legal texts, and it is beset with difficulties such as the comprehension of lengthy noisy legal documents, domain specificity as well as the scarcity of annotated data. To address these challenges, we propose a system that employs a hierarchical model and integrates domain-adaptive pretraining, data augmentation, and auxiliary-task learning techniques. Moreover, to enhance generalization and robustness, we ensemble the models that utilize these diverse techniques. Our system ranked first on the RR sub-task and in the middle for the other two sub-tasks.

## introduction certain heavily populated nations, such as india, are faced with a surfeit of unresolved legal cases, and the development of automated ai tools can potentially aid legal practitioners in expediting the judicial system. due to the intricate nature of the judicial process, the semeval-2023 task 6 @xcite proposes three sub-tasks that can serve as foundational components for creating legal ai applications, i.e. sub-task a: rhetorical roles prediction (rr), sub-task b: legal named entities extraction (l-ner) and sub-task c: court judgement prediction with explanation (cjpe). the rr sub-task is to organize unstructured legal documents into semantically cohesive units (classification at the sentence level), while the second l-ner sub-task involves identifying relevant entities within a legal document. the third cjpe sub-task focuses on predicting the outcome of a judgment (classification at the document level) and providing a corresponding explanation. the task at hand focusing on indian legal documents is inherently challenging. firstly, legal documents exhibit considerably greater length, and a higher degree of unstructured content and noise when compared to shorter documents used for the training of traditional natural language processing models. moreover, the presence of legal-specific lexicons within the documents makes general pretrained neural models ineffective. further, legal provisions vary from one nation to another, and the legal domain also has several sub-domains that align with different legal provisions. it follows that a pretrained model equipped with legal knowledge specific to one sub-domain may not be applicable to another distinct sub-domain. another key challenge in this task is the limited size of annotated data due to the time-consuming and expensive annotation process @xcite . to address the aforementioned challenges, we propose a system based on the baseline model called scibert-hsln in @xcite and equipped with domainadaptive pretraining, data augmentation strategies, as well as auxiliary-task learning techniques. in summary, we list here our findings that are derived from the experimentation: ## system overview given a legal case, d, containing the sentences [s 1 , s 2 , ...s n ], the rr sub-task is to predict the label y i for each sentence s @xmath0. we consider it as a sequence labeling task and take the scibert-hsln architecture @xcite as the backbone of our system, where each sentence is fed into a pretrained model to get word embeddings, these embeddings are further passed through a word bi-lstm layer @xcite followed by an attention-based pooling layer to get sentence representations s ′ 1 , s ′ 2 , ...s ′ n . for rr sub-task, these sentence representations are subsequently processed by another bi-lstm, resulting in contextualized sentence representations, c 1 , c 2 , ...c n , with a final crf @xcite layer utilized to give the prediction of rr. the non-colored part of figure 1 illustrates the backbone of this architecture. we enhance our system with domain-adaptive pretraining, data augmentation, and auxiliary-task learning techniques. further details of these techniques are discussed in the subsequent subsections. in the case of l-ner sub-task, sentence representations are directly fed into a crf layer to do the span classification. and for cjpe sub-task, we prepend a global token to the input sequence and use a self-attention layer instead of the bi-lstm to generate contextualized sentence representations. the global token attends to all other sentences and is utilized to generate the final prediction of the given judgement case, while the attention scores are used to extract explanations. ## training setup for fine-tuning our system, we nearly follow the hyperparameter configurations utilized in scibert-hsln @xcite , with the exception of setting the learning rate to 5e-04, dropout rate to 0.1, bi-lstm hidden size to 768, and accumulation steps to 2. we use these hyperparameters in all of our experiments, with the maximum number of epochs and early stopping epochs set to 70 and 10, respectively. the training process is executed on nvidia v100 gpus, and all the performance is evaluated on the development set released by the organizer, except for the final resembling. ## main results finally, our system got an f1 of score 85.93 on the leaderboard and ranked 1st in rr sub-task. in ## conclusion in conclusion, the paper proposes a system for the semeval-2023 task 6, that addresses the challenges of domain-specificity legal documents, and limited annotated data. the system employs a hierarchical model, enhanced by domain-adaptive pretraining, data augmentation, and auxiliary-task learning techniques. the models that utilize these techniques are ensembled to further improve the system's performance. the system ranked first in rr sub-task and achieved a moderate performance for the other two sub-tasks.