S em E val-2023 Task 6: L egal E val - Understanding Legal Texts (2023)

Abstract: In populous countries, pending legal cases have been growing exponentially. There is a need for developing NLP-based techniques for processing and automatically understanding legal documents. To promote research in the area of Legal NLP we organized the shared task LegalEval - Understanding Legal Texts at SemEval 2023. LegalEval task has three sub-tasks: Task-A (Rhetorical Roles Labeling) is about automatically structuring legal documents into semantically coherent units, Task-B (Legal Named Entity Recognition) deals with identifying relevant entities in a legal document and Task-C (Court Judgement Prediction with Explanation) explores the possibility of automatically predicting the outcome of a legal case along with providing an explanation for the prediction. In total 26 teams (approx. 100 participants spread across the world) submitted systems paper. In each of the sub-tasks, the proposed systems outperformed the baselines; however, there is a lot of scope for improvement. This paper describes the tasks, and analyzes techniques proposed by various teams.

## introduction in populous countries (e.g., india), pending legal cases have grown exponentially. due to the nature of the legal domain, it may not be possible to automate the entire judicial pipeline completely; nevertheless, many intermediate tasks can be automated to augment legal practitioners and hence expedite the system. for example, legal documents can be processed with the help of natural language processing (nlp) techniques to organize and structure the data to be amenable to automatic search and retrieval. however, legal texts are different from commonly occurring texts typically used to train nlp models. legal documents are quite long (tens and sometimes hundreds of pages), making it challenging to process @xcite ). another challenge with legal documents is that they use different lexicons compared to standard texts. moreover, legal documents are manually typed in countries like india and are highly unstructured and noisy (e.g., spelling and grammatical mistakes). above mentioned challenges make it difficult to apply existing nlp models and techniques directly, which calls for the development of legal domainspecific techniques. to promote research and development in the area of legal-ai we proposed shared tasks via "legaleval -understanding legal texts". this paper gives details about the shared task and our experience in organizing legaleval. in total 26 teams submitted systems papers in the shared task. though the documents pertained to the indian legal system, teams across the globe participated showing an interest in developing legal-nlp techniques that could later be generalized to other legal systems as well. the paper is organized as follows. in section 2, we outline work done in the legal-nlp domain. section 3 describes the tasks in detail. this is followed by details of the corpora (used for various sub-tasks) in section 4. details of models developed by each of the teams are discussed in section 5. section 6 analyzes the results furnished by various techniques for each of the sub-task. finally, section 7 summarizes and concludes the paper. ## related work in recent years, legal nlp has become an active area for research. researchers have worked on various research problems pertaining to the legal domain such as legal judgment prediction @xcite @xcite @xcite @xcite @xcite , rhetorical role labeling @xcite @xcite , summarization @xcite , prior case retrieval and statute retrieval @xcite @xcite . due to space limitations, we do not go into details of various tasks and allude to surveys on legal-ai by @xcite ; @xcite ; @xcite . there have been some works in the area of rhetorical role labeling and legal judgement prediction; however, the research problems remain far from being solved, hence we organize shared tasks on these tasks. moreover, legal-ner (named entity recognition) is an unexplored area, and we would like to draw the community's attention to it. ## tasks description in legaleval 2023, we proposed the following three shared tasks, which focus on automatically understanding legal documents. the documents were in english. task a: rhetorical roles (rr) prediction: given that legal documents are long and unstructured, we proposed a task for automatically segmenting legal judgment documents into semantically coherent text segments, and each such segment is assigned a label such as a preamble, fact, ratio, arguments, etc. these are referred to as rhetorical roles (rr). concretely, we propose the task of rhetorical role prediction: the task is to segment a given legal document by predicting (figure 1 ) the rhetorical role label for each sentence @xcite . we target 13 rrs as outlined in our previous work @xcite al. (2022b) please refer to @xcite . rr task relevance: the purpose of creating a rhetorical role corpus is to enable an automated understanding of legal documents by segmenting them into topically coherent units (figure 1 ). this segmentation is a fundamental building block for many legal ai applications like judgment summarization, judgment outcome prediction, precedent search, etc. task b: legal named entity recognition (l-ner): named entity recognition is a widely studied problem in natural language processing @xcite . however, legal documents have peculiar entities like names of the petitioner, respondent, court, statute, provision, precedents, etc. these entity types are not recognized by the standard named entity recognizer. hence there is a need to develop a legal ner system; consequently, we proposed task b as legal named entity recognition (l-ner). in this task, we target 14 legal named entities as described in our previous work @xcite and also shown in task relevance: similar to the importance of ner in the general nlp domain, legal ner is equally essential in the legal domain. for example, l-ner is the first step in extracting relevant entities for information extraction and retrievalbased tasks. blackstone ( https://github. com/iclrandd/blackstone ) provides a li-state of kerala petitioner -versus-raneef respondent ...appellant ...respondent judgment markandey katju, j. judge leave granted. heard learned counsel for the parties. the appellant has filed this appeal challenging the impugned order of the kerala high court. dated 1.9.2010 granting bail to the respondent, dr. raneef who is a medical practioner (dentist) in ernakulam district in kerala, and is accused in crime no. 704 of 2010 of p.s. muvattupuzha for 2 offences under various provisions of the i.p.c., the explosive substances act , and the unlawful activities (prevention) act. 1. 2. 3. court respondent case number statute statute figure 2: example of named entities in a court judgment brary for l-ner; however, it is limited to only uk cases and does not generalize well to case documents from other countries; moreover, the set of entities covered is quite limited. task c: court judgment prediction with explanation (cjpe): in recent years, in order to augment a judge in predicting the outcome of a case, the task of legal judgment prediction has been an active area of research @xcite . however, given the nature of the legal process, judgment prediction without providing an explanation for the judgments is not of much practical relevance to legal practitioners. we proposed the task of legal judgement prediction with explanation, where, given a legal judgment document, the task involves automatically predicting the outcome (binary: accepted or denied) of the case and also providing an explanation for the prediction. the explanations are in the form of relevant sentences in the document that contribute to the decision. this task is an extension of one of our previous works @xcite . we divided the task into two subtasks: (1) court judgment prediction and (2) explanations for the prediction. as done in our previous work, we take all the steps to remove any bias in the dataset and address the ethical concerns associated with the task. task relevance: the volume of legal cases has been growing exponentially in many countries; if there is a system to augment judges in the process, it would be of immense help and expedite the process. moreover, if the system provides an explanation for the decision, it would help a judge make an informed decision about the final outcome of the case. ## corpora we focus on freely and publicly available indian legal documents in english. the documents are obtained from indian legal search portal indi-ankanoon ( https://indiankanoon.org/ ). india follows a common-law system, which is similar to the one in the u.s. and the u.k. the tasks organizers with the help of legal experts in the past, have created the annotated datasets for rr @xcite , l-ner @xcite and cjpe @xcite . the data is annotated by senior law students, professors, and legal practitioners, thus ensuring the high quality of the data. we annotated around 250 judgments which are likely to have 850k tokens and roughly 10k entities. the rr data has about 265 legal documents with 841, 648 tokens and 26, 304 sentences, with an average of 3176 tokens per document. it is split into the train, validation, and test sets with an approximate 70-10-20 split. l-ner corpus has about 14, 444 judgment documents annotated with 17, 485 named entities belonging to 14 categories. the cjpe corpus has about 34k documents, and these are split into the train, validation, and test sets. ## participating systems we organized the shared task using the codalab platform. in total, 40 teams participated in various sub-tasks and finally 26 teams (approx. 100 participants) submitted systems description papers. overall, there were 17 submissions for task-a (rr) and 11 submissions each for task-b (ner) and task-c (cjpe). teams other than from india also participated in the shared task. the demographic distribution of teams is shown in figure 3 . next, we summarize the proposed approaches by various teams for each of the tasks. ## results and discussion baselines: we used transformer-based models as a baseline for each of the tasks. the baseline model for rr tasks was scibert-hsln @xcite ( https: //github.com/legal-nlp-ekstep/ rhetorical-role-baseline). we used roberta base with the transitionbased parser as the baseline for the ner task ( https://github.com/ legal-nlp-ekstep/legal_ner ). similarly, the baseline model for cjpe was a hierarchical model with xl-net at the first level and bi-gru at the next level https:// github.com/exploration-lab/cjpe . task a: results for task a are shown in task b: results for task b are shown in ## conclusion in this paper, we present the findings of 'semeval 2023 task 6: legaleval -understanding legal texts', consisting of three subtasks: i) subtask a, rhetorical role identification, aimed to segment a legal document by assigning a rhetorical role to each of the sentence present in the document ii) subtask b, named entity recognition, for identifying ner tags present in a legal text and iii) subtask c, legal judgment prediction with explanation, which aims to predict the final judgment of a legal case and in addition, provides explanations for the prediction. in total, 40 teams participated and 26 teams (approximately 100 participants) submitted systems papers and presented various models for each task. we provide an overview of methods proposed by the teams. analysis of the systems and results points out more scope for improvement. we hope these findings will provide valuable insights to improve approaches for each of the sub-tasks.