Viettel- AI at S em E val-2023 Task 6: Legal Document Understanding with Longformer for Court Judgment Prediction with Explanation (2023)

Abstract: Court Judgement Prediction with Explanation (CJPE) is a task in the field of legal analysis and evaluation, which involves predicting the outcome of a court case based on the available legal text and providing a detailed explanation of the prediction. This is an important task in the legal system as it can aid in decision-making and improve the efficiency of the court process. In this paper, we present a new approach to understanding legal texts, which are normally long documents, based on data-oriented methods. Specifically, we first try to exploit the characteristic of data to understand the legal texts. The output is then used to train the model using the Longformer architecture. Regarding the experiment, the proposed method is evaluated on the sub-task CJPE of the SemEval-2023 Task 6. Accordingly, our method achieves top 1 and top 2 on the classification task and explanation task, respectively. Furthermore, we present several open research issues for further investigations in order to improve the performance in this research field.

## introduction the sub-task of cjpe invites participants to analyze a legal judgment document to predict the outcome of the case and justify their prediction with relevant sentences from the document that contributes to the decision @xcite . the objective of predicting court judgments is to enhance the efficiency and accuracy of the legal system by providing decision-makers with valuable predictions and insights. transformer-based models, such as bert @xcite and roberta @xcite , currently dominate most natural language processing (nlp) tasks, including text classification. however, the quadratic complexity of their attention mechanisms imposes limits on the maximum input length (512 sub-word tokens in bert, roberta), which may not be able to handle all the meaning in the legal domain where longer documents are common @xcite . to address this issue, sparse-attention models, such as longformer @xcite and big-bird @xcite , increase the maximum input length to 4096 sub-word tokens, which may improve the performance in understanding legal documents. this paper focuses on investigating the data and experimenting with various large models, as we believe this sub-task involves processing long documents. using the longformer and tf-idf features @xcite , we achieve superior results and apply them to the explanation task. in the explanation task, we design a linear explanation algorithm and show the mechanism effectively inference the most important sentences leading to the decision of judgment. we also provide extensive experiments to analyse the capability of our proposed model on capturing semantic information in judgments. the main contributions of this study are summarized below: in the semeval 2023 task 6 @xcite , we have achieved the top-1 ranking for the subtask legal judgment prediction, and top-2 ranking for the sub-task court judgment prediction and explanation foot_0 . our implementation is available via github foot_1 . ## related work pretrained language models, which are trained on large-scale unsupervised corpora, have shown their effectiveness in various downstream tasks with limited training samples such as text classification @xcite ) and text generation @xcite . for text classification, traditional methods often apply a "short encoder" such as bert @xcite , roberta @xcite and xl-net @xcite . although these methods have archived promising performance on various domains including news' topic classification and movie reviews analysis @xcite , they can only adapt to maximum 512 tokens, while realworld documents such as court judgments could be extremely large @xcite . therefore, more advanced algorithms including hierarchical models han @xcite , bert + cnn and xlnet + bigru @xcite were invented to capture the semantic of arbitrary long documents. however, these methods are still not optimal due to two reasons. firstly, longer documents also cause more noisy information, the methods have no mechanism to automatically de-noise trivial information of judgments such as case number and introduction which do not contribute to the judgment decision @xcite . secondly, the provided algorithms cannot be trained on an end-to-end fashion. the language models bert and xlnet are not optimized to capture the characteristics of this specific legal domain. while judgment decision prediction could be done automatically by machine learning, providing an explanation of the decision is crucial. this enables lawyers and courts to verify the correctness of machine answers. there has been a vast amount of research in model interpretability including lime @xcite , shap(lundberg and lee, 2017), and breakdown @xcite . however, most of the explainers measure the important scores at word-level or highlight a short phrase only. retrieving key sentences that lead to model decisions like cjpe tasks would require algorithm modification. with the maximum tokens length 512, the percentage of document coverage is only 8.8 and only 14.8% of important sentences are within the learned area. therefore, using a simple encoder such as bert @xcite and roberta @xcite cannot learn to predict the decision of the judgment. using sparse-attention algorithm that supports larger models such as longformer @xcite , bigbird @xcite which support 4096 tokens can capture the semantic of about 82.9% documents. we also see that 99% of the important sentences are covered by about 11,000 tokens, this inspires us to design a hierarchical model using longformer as a primary encoder, as it requires only 3 chunks to cover almost all documents. however, adding more information from the first part could add more noise and cause the learning process harder to optimize. legal case, we employ an encoder model (such as roberta large or longformer) to vectorize the document into an embedding space. this document embedding often well-captures the semantic information of a document thanks to the advancement of the transformer architecture @xcite . for processing large documents, we have the option to select either the first or last chunk to represent the document, and the exceed tokens are truncated. the encoder could be short as it is often limited to the maximum tokens length (e.g. 512, 4096). therefore, we enhance the document representation by concatenating the output of the encoder with tf-idf features. a tf-idf feature is a document vector that captures the information from the whole document. this gives global information about the document and gives more evidence to predict the judgment decision. we assume that word-level counting would also help better predict the judgment decision. for example, if a judgment consists of a significant number of words "kill" and "dead", the decision would very likely to be accepted. because the tf-idf feature dimension could be exceedingly large, we try to build a vocabulary v consisting of meaningful words only. the vocabulary is built from the train data set. in which, we filter the rare words by using a frequency threshold f . all words that have the number of occurrences w less than f are truncated. the most common words are also filtered out since they are mainly stop-words and give zero information to the decision prediction task. the concatenated feature between encoder and tf-idf is thus forwarded to a classifier, in order to generate the probability of a document for each class. ## conclusions in this paper, we fine-tuned the longformer model by incorporating tf-idf features and employed various techniques to explain the cjpe task. along with implementation, we experimented with different input settings and parameters to discover the best possible solution. as for future research, we aim to delve deeper into the shap theory and investigate ways to improve the explanation task beyond the simple linear method. additionally, we are interested in exploring novel strategies to train and represent documents and identify essential features that influence court predictions and also the explanation task. further investigation on discovering a more effective approach to leverage the hierarchy technique for large-scale language models presents an opportunity for enhancement in this endeavor and ought to be acknowledged.