nclu_team at S em E val-2023 Task 6: Attention-based Approaches for Large Court Judgement Prediction with Explanation (2023)

Abstract: Legal documents tend to be large in size. In this paper, we provide an experiment with attention-based approaches complemented by certain document processing techniques for judgment prediction. For the prediction of explanation, we consider this as an extractive text summarization problem based on an output of (1) CNN with attention mechanism and (2) self-attention of language models. Our extensive experiments show that treating document endings at first results in a 2.1% improvement in judgment prediction across all the models. Additional content peeling from non-informative sentences allows an improvement of explanation prediction performance by 4% in the case of attention-based CNN models. The best submissions achieved 8’th and 3’rd ranks on judgment prediction (C1) and prediction with explanation (C2) tasks respectively among 11 participating teams. The results of our experiments are published

## resources we adopt only the datasets provided by organizers: ildc single-train with 5082 documents and ildc valid of 994 documents @xcite . section 4 provides all the details of these datasets. organizers mention the noise of the original texts, so we additionally glew those word couples that are separated by «-» and could be found in the manually composed list of words foot_1 . ## document processing we consequently apply our text-processing mechanism and experiment with its stage separately. ## datasets and experimental setup the competition has two stages: development and evaluation. we consider ildc single-train for the preliminary model training with a further assessment on ildc valid before the evaluation stage described in section 2. as for the evaluation stage, we use both of these resources in model fine-tuning. since ildc single-train is the main resource publicly available, we decided to use it for model parameters selection. figure 1 illustrates the distribution of the ildc single-train document lengths in words for the original text and after processing by v1 (section 3). while peaks of all distributional plots illustrate the significant portion of documents with the length of 1k-3k words, the mean document length of the processed texts is ≈ 4k words shifted from the probability plot peaks due to the presence of 7.5% documents longer than 8k words. removing repetitive sentences from texts (v1) reduces the probability peak from 3% to 2% with standard mean deviation increase by 3.8% (figure 1 ). we consider application of v2 processing towards the joined ildc single-train and ildc valid collections and keep only @xmath0. as for the evaluation stage, task organizers provide ildc test-c1 and ildc test-c2 with 1500 and 50 documents respectively @xcite . statistics of the number of words per document of processed texts are illustrated in for cnn and attcnn models, we consider ildc single-train statistics to limit input size by 4k and cover ≈ 68% of ildc single-train documents completely. we adopt precomputed word2vec model from the nlpl repository foot_4 , based on "english wikipedia dump of february 2017" with vector size of 300, windows size of 5, and 302,866 word entries. as for additional input features, we consider token position feature size of 5, the sliding window size of 3, and experiment with the number of filters f (sec. 1.1) of {300, 600, 1200}. we train models with a batch size of 32 documents and terminate this process once the accuracy on ildc single-train exceeds 98%. we use arenets framework @xcite for training and inferring the results. in the case of language models, we consider roberta base foot_5 , legalbert base foot_6 , and longformer base foot_7 with an attention window size of 512. for the bert-based models, we consider 510 last tokens foot_8 (similar to v2 with @xmath2. as for longformer, we increase this limit up to 1024 last tokens. all these models were finetuned for 10 epochs with a learning rate of 2e-6 and a batch size of 6. for the result explanations, in the case of attcnn, we select n @xmath3. in the case of language models, we set the window size m @xmath4. ## result analysis and discussion we first analyze the findings of the results in c1. in the case of cnn model, we consider the word window size of 3, and experiment with the various amount of filters @xmath5. with @xmath6. the inversion of the text documents and reduction of non-salient sentences (v2) allows an improvement of the past result with extra ≈3-4%. in the case of the language models, our preliminary experiments on ildc valid dataset with roberta illustrate the highest results once using the last 510 tokens as input. excluding the repetitive sentences from documents (v1) improves the results by 0.5%. the best result achieved by roberta on ildc test-c1 goes alongside with cnn model with 58.53 by f1. due to the legal domain of documents adopted in legalbert pretraining, this model improves roberta classification results by ≈ 8% with @xmath7.51, ranked as #8 out of 11 participants. comparing the result f1 difference with the best submissions, the 3rd best submission (uottawa.nlp23 team) improves this re-sult by +4.31; by +8.77 with the 2nd best result (irit_iris_(c/a) team), and by +11.34 with the top result (bluesky team). ## conclusion in this paper, we provide extensive experiments on attention-based approaches applied to a couple of legal document processing tasks: 1) judgment prediction and 2) explanation of the obtained results. the first task is considered as a text classification problem, while the second one is an extractive text summarization with a salient sentence selection approach based on the text parts of the most attentive words. since documents are relatively long and, to the best of our knowledge, could not be completely considered as input of most approaches, we experiment with additional techniques of reduction and mimicking the output class. the findings of our experiments illustrate that treating document endings as input in the models at first results in a 2.1% improvement across all the models. additional content peeling from non-informative sentences allows us to improve explanation performance by 4% in the case of an attention-based cnn model.