T eam S hakespeare at S em E val-2023 Task 6: Understand Legal Documents with Contextualized Large Language Models (2023)

Abstract: The growth of pending legal cases in populouscountries, such as India, has become a major is-sue. Developing effective techniques to processand understand legal documents is extremelyuseful in resolving this problem. In this pa-per, we present our systems for SemEval-2023Task 6: understanding legal texts (Modi et al., 2023). Specifically, we first develop the Legal-BERT-HSLN model that considers the com-prehensive context information in both intra-and inter-sentence levels to predict rhetoricalroles (subtask A) and then train a Legal-LUKEmodel, which is legal-contextualized and entity-aware, to recognize legal entities (subtask B).Our evaluations demonstrate that our designedmodels are more accurate than baselines, e.g.,with an up to 15.0% better F1 score in subtaskB. We achieved notable performance in the taskleaderboard, e.g., 0.834 micro F1 score, andranked No.5 out of 27 teams in subtask A.

## introduction the growing amount of legal cases and documents requires more and more human efforts to process them. @xcite in some countries, such as india, legal cases have accumulated in an incredible number. for example, india has more than 47 million cases pending in the courts @xcite . this has created a need for automated methods to help judges efficiently understand and process relevant and reliable information. in addition, these methods can also help students, legal scholars, and court officials who deal with legal documents daily. one way to assist them is to automatically understand and highlight the key information and context of long legal documents. however, understanding legal documents by machines is not an easy task. first, legal documents are often full of technical terminologies, which * these authors contributed equally to this work. can span different legal divisions @xcite . furthermore, legal documents can be specific to special cases, such as health @xcite , it technology @xcite , and cyber security @xcite @xcite , which will involve domainspecific words. in addition, legal documents can be extremely long @xcite , which makes dependency-based techniques and models, such as rnn models, fail to extract the context information due to gradient vanishes. finally, typos and unstructured documents introduce noises @xcite , which makes automated natural language processing challenging. despite these challenges, predicting rhetorical roles and recognizing named entities in legal documents are very useful for automating the processing and understanding of legal documents. rhetorical role prediction segments texts and structures noisy legal documents into topically and semantically coherent units @xcite . named entity recognition helps identify key legal entities in long documents @xcite , which can not only help judges process cases in a more efficient way, but also benefit the next automation steps. these two tasks can serve as key steps in these methods @xcite . in this paper, we propose to solve the rhetorical role prediction and named entity recognition problems in the legal document domain with contextualized large-language models. we first systematically build models with well-known design choices based on popular pre-trained models (e.g., bert and xlm-roberta) @xcite , then systematically evaluate the performance of different models and identify the key limitations, and eventually propose our legal contextualized models as our final solutions. for rhetorical role prediction, we model this task as the sequential sentence classification problem and build the legal-bert-hsln model, which considers the comprehensive context semantics in both intra-and inter-sentence levels. for named entity recognition, we propose to build a legal-luke model that is both sensitive to context and entity-aware. our evaluation results show that our proposed models are more accurate than baselines, e.g., legal-luke is 15.0% better than our baseline bert-crf in f1 score (more details in §4.3). furthermore, we also achieved the top performance on the rhetorical role prediction task leaderboard, i.e., ranked no.5 out of 27 teams and achieved the 0.8343 micro f1 score (see §4.2). we briefly summarize our primary contributions as follows. ## evaluations we conduct extensive experiments using the rhetorical role classification and legal named entity recognition tasks. we have implemented the legal-best-hsln model and the legal-luke model based on pytorch @xcite . ## conclusion in this paper, we propose to understand legal documents with context-sensitive language models. for the sub-task of rhetorical role classification, we design the legal-bert-hsln model, which learns the hierarchical context information to solve the sequential sentence classification problem. for legal named entity recognition, we implemented the legal-luke model which is both contextualized and entity-aware. our evaluation results reveal the outperformance of our models compared to baselines and we are the top-5 teams for the rhetorical role classification task on the leaderboard.