R esearch T eam_ HCN at S em E val-2023 Task 6: A knowledge enhanced transformers based legal NLP system (2023)

Abstract: This paper presents our work on LegalEval (understanding legal text), one of the tasks in SemEval-2023. It comprises of three sub-tasks namely Rhetorical Roles (RR), Legal Named Entity Recognition (L-NER), and Court Judge- ment Prediction with Explanation (CJPE). We developed different deep-learning models for each sub-tasks. For RR, we developed a multi- task learning model with contextual sequential sentence classification as the main task and non- contextual single sentence prediction as the sec- ondary task. Our model achieved an F1-score of 76.50% on the unseen test set, and we at- tained the 14th position on the leaderboard. For the L-NER problem, we have designed a hybrid model, consisting of a multi-stage knowledge transfer learning framework and a rule-based system. This model achieved an F1-score of 91.20% on the blind test set and attained the top position on the final leaderboard. Finally, for the CJPE task, we used a hierarchical ap- proach and could get around 66.67% F1-score on judgment prediction and 45.83% F1-score on the explainability of the CJPE task, and we attained 8th position on the leaderboard for this sub-task.

## introduction the court pending cases are rapidly raising in populous countries like india. towards this challenge, systems that can automatically process, understand and organize legal documents need to be designed. another major challenge of processing legal documents is that they are unstructured, very lengthy, contain a lot of legal jargon, etc. thus, the existing nlp systems can not be directly applied. the semeval-2023 @xcite provides 3 different problems on the indian legal documents namely: 1. rhetorical roles (rr) @xcite ) @xcite : segmenting a legal document into semantically similar sentences. the rr task can be defined as the classification of sentences in a legal document into pre-defined classes. 2. legal named entity recognition (l-ner) @xcite : legal entity extraction from the legal document. the task is to find out the legal entities like court names, petitioner names, respondents, judge names, and case numbers mentioned in a given legal document. 3. court judgement prediction with explanation (cjpe) @xcite : court judgment output prediction with explanations. given a legal document, the task is to find out the final outcome of the case i.e whether the appeal raised by the petitioner is accepted or not. in case of acceptance, generate the explanation of the case. the datasets for all the above sub-tasks are available in the english language. in this work, we present our proposed solutions for the above three problems. we proposed a multi-task learning neural network model to solve the sub-task rr. our model consists of two classification layers. the first classifier is the sequential sentences classification which is the primary task. next, the second classifier is the single sentence classification which acts as the secondary task. for the legal ner task, we proposed a hybrid model consisting of a multi-stage sequence-to-sequence training framework based on the pre-trained large legal language model and a rule-based system to improve the performance of the neural network model. finally, for the cjpe sub-task, we proposed two separate models for judgment prediction and explanation generation. the work of @xcite has demonstrated that fine-tuning or pre-training of language models on domain-specific data significantly improves the model performance on the downstream task. they have validated the same by pre-training the bert model @xcite (biobert) on bio-medical text data which significantly outperformed the bert model which was pre-trained on general text data. we also developed our own llm based on roberta @xcite . all of our proposed models are based on our legal rorerta model. for cjpe sub-task 1, we used a hierarchical approach to train the model. in the first stage, we used our legal roberta to fine-tune the model on the dataset and this fine-tuned model embedding is used to train a bigru model for the final predictions. in recent years, there has been a lot of research and development work on legal nlp like chinese legal ner and relationship extraction @xcite , legal judgment prediction for chinese @xcite , russian court decisions data analysis using distributed computing and machine learning to improve lawmaking and law enforcement @xcite . however, the application of nlp and ml solutions in the indian legal system is low. the legal documents are very different from the general text, they are unstructured, very long, and contain a large amount of jargon. the existing nlp systems can not be directly applied for solving the above-mentioned problems. our proposed solutions leverage a pretrained large language model trained on indian legal documents. our proposed models achieve an f1-score of 76.50%, 91.20%, and 66.67% on the hidden test set of rr, l-ner, and cjpe respectively. our main contributions are summarized below: the legaleval task of semeval-23 consists of three sub-tasks. 2.1 rhetorical roles (rr): the task is to classify the sentences in a legal judgment into semantically similar classes. the semeval-23 rr task's dataset consists of 13 class labels namely preamble (pream-ble), facts (fac), ruling by lower court (rlc), issues (issue), argument by petitioner (arg_petitioner), argument by respondent (arg_respondent), analysis (analysis), statute (sta), precedent relied (pre_relied), precedent not relied (pre_not_relied), ratio of the decision (ratio), ruling by present court (rpc) and none (none). the detailed definitions of each class and the dataset preparation strategies are provided in kalamkar et al. 2022b. we also propose a baseline system using the deep learning architecture proposed in brack et al. 2021. malik et al. 2021a proposed a multi-task learning (mlt) framework with rhetorical role classification as the primary task and label shift prediction as the auxiliary task. the label shift prediction is a problem that the model has to predict whether the current sentence s t has the same label as the previous sentence s i-1 in a legal document. another work from ghosh and wyner 2019 proposed a bilstm-crf model with the sent2vec model as a feature extraction layer. an example sample of the rr dataset is provided in figure 1 2.2 legal ner: the legal ner is the task of classifying all the tokens present in a sentence in a legal judgment document. the legal entities covered in semeval-23 include 14 entity types, court, pe-titioner, respondent, judge, lawer, date, org, gpe, statute, provision, precedent, case_number, witness, and other_person. detailed definitions of each class and dataset preparation methods are provided in kalamkar et al. 2022a. for this task, we proposed a hybrid model consisting of a multi-stage knowledge transfer framework and a rule-based system. some of the previous work done on simi-lar problem statements are discussed below: the work of leitner et al. 2019 proposed an approach for named entity recognition (ner) in the legal domain of the german language. the dataset consists of 67,000 sentences and 54,000 annotated entities approximately. they proposed a state-ofthe-art model comprising of conditional random fields (crfs) and bidirectional long-short term memory networks (bilstms). in another work, trias et al. 2021 proposed an ensemble language model using a transformer neural network architecture combined with a finite state machine to extract names from the english language legal text. in the work of au et al. 2022, the authors describe a publicly available legal ner dataset called ener, based on legal company filings available from the us securities and exchange commission's edgar dataset. they demonstrated that training a number of different ner algorithms on the general english conll-2003 corpus but testing on their test collection confirmed significant degradation in accuracy, as measured by the f1score. chen et al. 2020 proposed a legal triplet extraction system for drug-related criminal judgment documents. the system extracts the entities and the semantic relationships jointly and benefits from the proposed legal lexicon feature and multitask learning framework. ## system overview in the recent years, because of the emergence of pre-trained large language models (llm), the performance of the nlp models has drastically improved. though there are many pre-train llms for general english and other domains, there are very few llms for indian legal text @xcite . towards this, we also developed our own indian legal roberta language model. the mask language modeling pre-training objective was adopted for pre-training the model. our llm is pre-train on around 1.1m documents which we scrapped from the indian kanoon website 1 . while collecting these documents, we considered the indian court cases from the years 1960 to 2023. also, we covered the cases of major indian courts like the supreme court, high court, and district courts from different states and union territories of india. moreover, we also covered 11 different types of cases of food adulteration, education, industrial disputes, tax, criminal, civil, automobile, land and property, industry and labor, constitutional and financial. the details of the actual dataset released by the legaeval organizers are explained below in the data section. on top of the training datasets, we also generated a synthetic dataset to solve data scarcity as well as the class imbalance problem. ## data: we use the legaeval dataset of the task in semeval-23. the rr dataset consists of 247 documents having 28986 sentences for training and 30 documents having 2879 sentences for validation. on top of this training dataset, we used a synthesis dataset of around 1000 sentences for low occurrences classes i.e. sta, issue, and pre_not_relied. the class distribution for the training and validation dataset is provided in to train our legal roberta based model, we used the huggingface @xcite mask language model training script. we used 5% of the whole dataset as a validation dataset with 15% masking probability. we initialized the model weight from the roberta base model and used the same roberta base tokenizer. the model was trained for 3 epochs i.e. 600k steps with batch size 16 and learning rate of 1e -5. the model was trained on 32 gb v100 gpu for 5 days. we trained multiple models for the different legal-eval sub-tasks. all the models use our legal llm model to extract contextual work representations. ## results our rr models are evaluated on the development dataset using the micro average f1-score. the results are provided in