Nonet at S em E val-2023 Task 6: Methodologies for Legal Evaluation (2023)

Abstract: This paper describes our submission to the SemEval-2023 for Task 6 on LegalEval: Understanding Legal Texts. Our submission concentrated on three subtasks: Legal Named Entity Recognition (L-NER) for Task-B, Legal Judgment Prediction (LJP) for Task-C1, and Court Judgment Prediction with Explanation (CJPE) for Task-C2. We conducted various experiments on these subtasks and presented the results in detail, including data statistics and methodology. It is worth noting that legal tasks, such as those tackled in this research, have been gaining importance due to the increasing need to automate legal analysis and support. Our team obtained competitive rankings of 15th, 11th, and 1st in Task-B, Task-C1, and Task-C2, respectively, as reported on the leaderboard.

## introduction the semeval task-6 @xcite aims to automate several tasks to streamline the indian legal process, which is often slow and delayed due to the country's large population and a shortage of judicial resources. additionally, people in india are not always fully aware of the country's laws. to make the legal process more accessible to the general public, the semeval task addresses crucial problems that are specific to the indian judicial context. one task is the "legal named entity recognition (l-ner)" system, which identifies named entities in the legal text. legal judgments contain intriguing entities like the names of the petitioner, respondent, judge, lawyer, date, organization, gpe, statute, provision, precedent, case number, witness, and other persons, which are typically not recognized by conventional entity recognition systems. therefore, developing systems that are tailored to the legal domain is crucial. in task b, participants were tasked with identifying the legal entities present in legal judgments. a court judgment is divided into two parts: the preamble, which includes the names of the parties, the court, lawyers, and other details, and the decision (judgment), which follows the preamble. the organizers separately provided the preamble and judgment text datasets. apart from the legal ner system, the semeval task-6 also includes another important task c, that is crucial for automating the indian legal process. the subtask, referred to as task c1, is the legal judgment prediction (ljp) task. this task aims to determine whether the legal judgment favors the appellant or the defendant, which is modeled as a binary classification problem. given a large number of pending legal cases in india, automating the process of predicting legal judgments can significantly reduce the burden on the judicial system. moreover, the task of finding an explanation for the legal decision is equally important. for this, we locate a span from the legal judgment that is highly correlated with the reasoning behind the legal decision. subtask c2 focuses on finding a suitable explanation for the binary classification task in subtask c1, thereby providing an additional layer of transparency to the legal process. to accomplish these tasks, we experiment with several models and techniques. for the legal named entity recognition system, we try out a spacy-based model and a fine-tuned bert model to detect the legal named entities. for the legal judgment prediction task, we pass the last 512 tokens of the legal judgment through transformer models, as well as try out hierarchical transformer models on the entire dataset to train the models for the task of judgment prediction. lastly, for the court judgment prediction with explanation task, we check various span lengths taken from the end of the document. the datasets provided to the participants as a component of the semeval task were in the english language. our main contributions can be summarized as: (1) our contributions in this paper include participating in three subtasks: legal named entity recognition (l-ner), legal judgment prediction (ljp), and court judgment prediction with explanation (cjp). ## background we have participated in task-b named legal named entity recognition (l-ner), task-c1 called legal judgment prediction (ljp), and task-c2, called court judgment prediction with explanation (cjpe). legal named entity recognition: the task of legal named entity recognition (legal ner) @xcite involves identifying named entities in structured legal text. these entities include petitioner, respondent, court, statute, provision, precedent, and more. however, conventional named entity recognizers, like spacy, do not recognize these entity types, highlighting the need for domain-specific legal ner systems. additionally, the indian legal system has its unique processes and terminologies, making it necessary to develop a specific legal ner for indian court decision texts. unfortunately, there are no publicly available annotated datasets for the task of l-ner on indian court cases. to address this gap, a recent study @xcite has developed a legal ner model specifically for indian legal data. this work highlights the need for an annotated dataset to train the legal ner model and proposes a methodology for creating such a dataset. by training their model on the created dataset, they achieved state-of-the-art results on the task of legal ner on indian legal data. legal judgment prediction: legal judgment prediction is an important task in the field of legal informatics. a recent work by malik et al. @xcite focuses on predicting the outcome of indian supreme court cases using state-of-theart transformer models. the task is modeled as a binary classification problem and applied to the entire legal case except for the final judgment. the authors use the manually annotated explanations for every legal case judgment in the indian legal document corpus (ildc) dataset as gold standards for the court judgment explanation task. another work that addresses the task of judgment prediction is the chinese ai and law challenge dataset @xcite . this dataset is the first large-scale chinese legal dataset with 2.6 million cases, which makes it an excellent resource for judgment prediction on chinese data. the human annotation in this work is rich, and the authors compared simple text classification approaches, such as fasttext, tfidf+svm, cnn, etc., on the facts of the legal judgment. the hindi legal document corpus @xcite is a corpus of 900k legal documents annotated in hindi, designed for the task of bail prediction. the work experiments with a range of state-of-the-art models and proposes a multi-task learning framework where the main task is bail prediction, and the auxiliary task is summarization. finally, there is a work on legal judgment prediction @xcite , which provides a dataset of 11k legal judgments. the work compares different models, including bigru-attn, han, lwan, bert, and hier-bert, on the facts of the legal case. ## system overview in this section, we provide an overview of the methodologies used for the tasks we participated in, which include legal named entity recognition (l-ner), legal judgment prediction (ljp), and court judgment prediction with explanation. ## conclusion in conclusion, we participated in three subtasks of the semeval-2023 for task 6 on legaleval: legal named entity recognition (l-ner), legal judgment prediction (ljp), and court judgment prediction with the explanation. for l-ner, we utilized a combination of pre-trained transformer models, domain-specific embeddings, and a modified spacy pipeline to build a high-performing ner model. in ljp, we experimented with various hierarchical transformer-based models and utilized the last 512 tokens of judgments through pre-trained transformer models. for the explanation task, we found that taking the last 300 words from the end of the legal judgment performed best in terms of the rouge-2 f1 score. furthermore, we compared the results of different models for each subtask to analyze their strengths and weaknesses. however, since the gold standard data corresponding to the test set for every subtask was not released by the organizers, we were unable to perform an error analysis on the test dataset. we will do this once the gold standard labels are available publicly. looking ahead, we plan to explore joint optimization frameworks that incorporate both rhetorical role information and sentence position in the document to create more accurate span detection models. overall, our participation in the competition has allowed us to showcase the effectiveness of our proposed models and techniques for tasks in the legal domain.