IRIT _ IRIS _ C at S em E val-2023 Task 6: A Multi-level Encoder-based Architecture for Judgement Prediction of Legal Cases and their Explanation (2023)

Abstract: This paper describes our system used for sub-task C (1 & 2) in Task 6: LegalEval: Understanding Legal Texts. We propose a three-level encoder-based classification architecture that works by fine-tuning a BERT-based pre-trained encoder, and post-processing the embeddings extracted from its last layers, using transformer encoder layers and RNNs. We run ablation studies on the same and analyze itsperformance. To extract the explanations for the predicted class we develop an explanation extraction algorithm, exploiting the idea of a model’s occlusion sensitivity. We explored some training strategies with a detailed analysis of the dataset. Our system ranks 2nd (macro-F1 metric) for its sub-task C-1 and 7th (ROUGE-2 metric) for sub-task C-2.

## introduction with the increasing population, there is also an increase in the number of legal cases, and the shortage of capable judges to attend to them makes it more difficult to process them in time, hence creating backlogs. in such a scenario, a method to assist the judges and suggest an outcome (with an explanation) of a legal case becomes necessary. such methods can also assist a legal professional to help people; unfamiliar with the legal terms and proceedings; to make informed decisions. in the past, there have been many such approaches using machine learning techniques. in sub-task c of task 6: legaleval @xcite ) the organizers tried to tackle this problem in the context of indian legal cases from the supreme court of india (sci). task c is divided into two sub-tasks, c-1 and c-2. c-1 is the judgment prediction of the court cases/documents, and c-2 is the explanation of the said prediction. in this work, we propose a classification framework taking a few ideas from the hierarchical transformer approach by @xcite which builds upon bert @xcite ) as a backbone encoder model. we fine-tune a bert-based pre-trained model and experiment with features extracted from its last layers, and use it for further classification (sub-task c-1) process, through a combination of transformer encoders and rnns. we explored the effect of training this framework with different parts of the training dataset and provide an analysis of its impact. for the explanation task (sub-task c-2) we exploit the idea of "occlusion sensitivity" @xcite ) and develop a mechanism to extract the relevant sentences from a document which serves as the explanation for its prediction using the classification method in sub-task c-1. our model ranked 2nd foot_0 out of 11 participating teams on sub-task c-1 obtaining 0.7228 macro-f1, and 7th 1 out of 11 on sub-task c-2 with rouge-2 = 0.0428 (8th, macro-@xmath0.4). for extra details on the ranking of task c-1 refer to section 2.1.1 and section 5. our code is available at github foot_1 . ## task description because explanations are difficult to annotate, explanations have to be made without explicit training on the annotated explanations. this makes c-2 an extractive explanation task for which no annotated explanation is provided for training with the reasoning that a predictive model should be able to explain its prediction. ## results we uploaded the test predictions from some of the best models ( in the explanation extraction algorithm is model dependent since its backbone is dependent on the model's output sensitivity to the parts of the input. this can be seen from the ## conclusion in this paper, we presented our system for subtask-c of semeval 2023 task-6. we propose a threelayered decision classification architecture for processing large legal case documents that build upon pre-trained encoders, embedding extraction, and post-processing with transformer encoder and rnn layers. which ranked 2nd out of 11 participating teams in sub-task c-1. we explored the effects of training this architecture with ablation and study its impact. we also developed an algorithm for extracting an explanation for a decision prediction and use it for explanation extraction in sub-task c-2, which ranks 7th out of 11 participating teams (rouge-2 metric). we give a detailed analysis of the training and test dataset used in the competition. in the future, we plan to take this work further to develop a more general transformer architecture for decision prediction and their explanation (both abstract and extractive) from general large legal case documents and specifically within the french legal system (project lawbot foot_6 ).