u O ttawa at S em E val-2023 Task 6: Deep Learning for Legal Text Understanding (2023)

Abstract: We describe the methods we used for legal text understanding, specifically Task 6 Legal-Eval at SemEval 2023. The outcomes could assist law practitioners and help automate the working process of judicial systems. The shared task defined three main sub-tasks: sub-task A, Rhetorical Roles Prediction (RR); sub-task B, Legal Named Entities Extraction (L-NER); and sub-task C, Court Judgement Prediction with Explanation (CJPE). Our team addressed all three sub-tasks by exploring various Deep Learning (DL) based models. Overall, our team’s approaches achieved promising results on all three sub-tasks, demonstrating the potential of deep learning-based models in the judicial domain.

## introduction the legal sector generates an overwhelming volume of information from many sources, including law firms, law courts, independent attorneys, and legislators. as a result, tools to help legal professionals manage large volumes of legal documents are becoming increasingly necessary. the legaleval 2023 shared task @xcite foot_0 aims to build a legal research community by addressing three specific problems in the legal domain. first, sub-task a, rhetorical roles prediction (rr), involves identifying the rhetorical roles played by various sentences in a given text. subtask b, legal named entities extraction (l-ner), involves identifying distinct named entities that appear in legal texts. finally, sub-task c, court judgement prediction with explanation (cjpe), involves predicting the outcome of a given court case, along with an explanation of the reasoning behind the prediction. our team participated in all three sub-tasks of the shared task. in sub-task a, we created a hierarchical bilstm-crf model with randomly-initiated embeddings. in sub-task b, we created a bertbased named entity recognition model. in sub-task c, we used caselawbert to create judgement prediction and explanation models. the rest of this paper is organized as follows: in section 2, we briefly review the literature background on each sub-task. the detailed descriptions of our approaches to the three sub-tasks are presented in sections 3, 4, and 5, respectively. section 6 concludes the paper and comments on future work directions. ## related work rhetorical role labeling was first automated by @xcite , where conditional random fields (crf) developed a generic approach to perform segmentation using seven rhetorical roles. @xcite developed a method for the identification of factual and nonfactual sentences using fasttext. later, the automatic machine learning approaches and rule-based scripts for rhetorical role identification evolved which was compared to @xcite . @xcite created a large corpus of rrs and propose transformerbased baseline models for rr prediction. the use of the bi-lstm-crf model with sent2vec features to label rhetorical roles in supreme court documents in @xcite serves as a precursor for transformer models. in this subtask, we extend the work in @xcite and explore transformer models to see which performs better in the prediction of rhetorical roles. named entity recognition for other domains has been a significant area of research. in the biomedical field, @xcite and @xcite both describe ner models for the biomedical domain. @xcite and @xcite both describe a multilingual named entity recognition system using a pre-trained bert model in several languages. @xcite describes a procedure for the creation of a customized named entity recognition model using bert. the described procedure is adapted and used for this subtask. in the previous few years, researchers have successfully attempted legal judgment prediction (ljp) tasks for the text of judicial cases in several languages and within different legal systems, such as the european, chinese, and indian systems. a new english ljp dataset containing 11,478 cases from the european court of human rights (echr) was created by @xcite , and the results of various dl architectures were reported on three tasks: binary classification, multi-class classification, and case importance prediction. moreover, a hierarchical version of bert @xcite has been proposed to overcome bert's input token count limitation for the ljp task. a lot of work in the field of predicting law articles or charges is currently being done in china. in @xcite , the chinese ai and law challenge dataset (cail 2018) was released for legal judgment prediction. the dataset includes annotations for the judgments of over 2.6 million criminal cases. it consists of detailed annotations of related law articles to cases, the prison terms, and the charges. for the indian legal system, @xcite introduced the indian legal document corpus (ildc) dataset and used it to experiment with the case judgment prediction and explanation (cjpe) task, providing various models and proposing a hierarchical occlusion-based model for explainability, which create the baseline for the current sub-task c. ## sub-task a: rhetorical roles prediction legal documents are usually found to be lengthy, unstructured, and quite difficult to process. reading and understanding legal text is arduous as they usually span from tens to hundreds of pages. in this subtask, we automatically annotate/classify sentences into semantically coherent units called rhetorical roles. ## sub-task c: court judgement prediction with explanation the aim of this sub-task is to automatically predict the outcome (accepted or rejected) of a legal case given the case document (sub-task c-1) and also to provide explanations for the prediction (subtask c-2) by selecting the relevant sentences in the document that contribute to the decision. ## conclusion and future work our work performed well on the legaleval 2023 shared tasks, placing 15th, 6th, 3rd, and 10th on subtasks a, b, c-1, and c-2, respectively. in subtask a, we created a hierarchical bilstm-crf model with randomly initiated embeddings. in sub-task b, we created a bert-based named entity recognition model. finally, in sub-task c-1 and c-2, we use caselawbert to create judgement prediction and explanation models. the results demonstrate the feasibility of using deep learning models and domain-specific contextualized language models to understand legal texts and develop legal nlp applications. as part of future research, we plan to investigate the usefulness of combining some legal tasks, such as legal-ner and cjpe. the identification of entities within the legal text may facilitate a better interpretation of legal judgments. in addition, we will investigate the effectiveness of our models by considering new emerging ai-powered solutions such as legal prompting.