Steno AI at S em E val-2023 Task 6: Rhetorical Role Labelling of Legal Documents using Transformers and Graph Neural Networks (2023)

Abstract: A legal document is usually long and dense requiring human effort to parse it. It also contains significant amounts of jargon which make deriving insights from it using existing models a poor approach. This paper presents the approaches undertaken to perform the task of rhetorical role labelling on Indian Court Judgements. We experiment with graph based approaches like Graph Convolutional Networks and Label Propagation Algorithm, and transformer-based approaches including variants of BERT to improve accuracy scores on text classification of complex legal documents.

## introduction rhetorical role labelling for legal documents refers to the task of classifying sentences from court judgements into various categories depending on their semantic function in the document. this task is important as it not only has direct applications in the legal industry but also has the ability to aid several other tasks on legal documents such as summarization and legal search. this task is still in it's early stages, with huge scope for improvement over the current state-of-the-art. to facilitate automatic interpretation of legal documents by dividing them into topic coherent components, a rhetorical role corpus was created for task 6, sub-task a of the international workshop on semantic evaluation @xcite . several applications of legal ai, including judgment summarizing, judgment outcome prediction, precedent search, etc., depend on this classification. the dataset @xcite used to implement the above approaches is relatively small, consisting only of a few hundred annotated documents and 7 sentence classes. ## dataset the dataset @xcite is made up of publicly available indian supreme court judgements. it consists of 244 train documents, 30 validation documents and 50 test documents making a total of 36023 sentences. for every document, each sentence has been categorized into one of 13 semantic categories as follows: ## proposed techniques and algorithms we try several different approaches for the task at hand. all our models use legal-bert as their base, and use various methods for further processing and refining of results. the legal-bert family of models is a modified pretrained model based on the architecture of bert @xcite . the variant used in this paper is legal-bert-base, a model with 12 layers, 768 hidden units, and 12 attention heads. it has a total of 110m parameters and is pretrained for 40 epochs on a corpus of 12 gb worth of legal texts. this model was fine-tuned on the task dataset for 2 epochs with a learning rate of 1e-5 using the adam optimizer and cross entropy loss ## results we trained the models and tested them on the validation set. the accuracy scores have been reported in we see that the performance of these models is significantly better than the previous attempts at this problem. the improvement of the results of previously studied models can be attributed to the increase in dataset size, along with other changes in the structure of the task. however, our context-based legal-bert approach outperforms the other frameworks by a significant margin. this exhibits that the context of each sentence is critically important in determining its label, and that we are successful in incorporating the context of each sentence into its representation. we saw that graph-based approaches did not significantly improve performance compared to the current state-of-the-art models. however, it is important to note that we were unable to run the graph convolution network using the entire train dataset due to compute constraints. despite such constraints, there might be other reasons for the mediocre performance of graphbased models. one possible reason is that the representation of the sentences used for building the model was not able to capture information necessary to make better predictions. this also explains how the context-based legal-bert performed so much better -it improved the quality of sentence representation, successfully capturing a wider range of features pertaining to the task at hand. ## conclusion and future work in this paper, we tried several different techniques to perform a sentence classification task on legal documents. through our experiments, we show that incorporating context into the cls tokens of model accuracy legal-bert 65.56% legal-bert + classifier 67.15% graph label diffusion 66.34% gcn 67.42% context-based legal-bert 71.02% for future improvements on these models, we could try the graph convolutional network approach on the complete dataset. we could also try the various methods of classification, such as a custom neural network or label diffusion, on the context-based cls tokens. moreover, we could further try to incorporate more sentences for context of each target sentence. this would require the use of a long-former model, since the total number of tokens passed into the model will increases.