LRL _ NC at S em E val-2023 Task 6: Sequential Sentence Classification for Legal Documents Using Topic Modeling Features (2023)

Abstract: Natural Language Processing techniques can be leveraged to process legal proceedings for various downstream applications, such as sum- marization of a given judgement, prediction of the judgement for a given legal case, prece- dent search, among others. These applications will benefit from legal judgement documents already segmented into topically coherent units. The current task, namely, Rhetorical Role Pre- diction, aims at categorising each sentence in the sequence of sentences in a judgement document into different labels. The system proposed in this work combines topic mod- eling and RoBERTa to encode sentences in each document. A BiLSTM layer has been utilised to get contextualised sentence repre- sentations. The Rhetorical Role predictions for each sentence in each document are gen- erated by a final CRF layer of the proposed neuro-computing system. This system secured the rank 12 in the official task ranking, achiev- ing the micro-F1 score 0.7980. The code for the proposed systems has been made available at https://github.com/KushagriT/SemEval23_ LegalEval_TeamLRL_NC

## introduction legal domain hosts a wide range of text documents corresponding to different stages of the legal proceedings. especially in populous countries, such as india, which has a huge and branched judicial system, automation of certain steps in the judicial pipeline can be beneficial to the overall working of the system. legal documents are generally long and unstructured, and typically consist of jargons specific to the legal domain. due to presence of various subdomains, such as criminal law, civil law, income-tax law, among others, the systems developed on one domain may not generalise well to other domains. additionally, the legal domain suffers from lack of availability of annotated corpora @xcite . all the above observations make the task truly challenging. natural language processing techniques can be leveraged to process such corpora for various downstream applications, such as judgement summarising, judgement outcome prediction, precedent search, among others. these applications will benefit from legal judgement documents that are already segmented into topically coherent units. these coherent units are called rhetorical roles. this task of semantic segmentation of a judgement document, into rhetorical roles, aids in the overall processing of legal documents. the aim of the current task @xcite is to categorise sentences in a legal case proceeding (judgement document), to different labels. thus the task effectively boils down to rhetorical role prediction. the dataset @xcite consists of 13 rhetoric role labels given in this paper proposes a system for rhetorical role prediction for legal judgement documents. the proposed approach uses a hierarchical sequence encoder consisting of basic sentence representation and a contextualised sentence representation followed by a crf @xcite layer for final label prediction. experiments were conducted with several variations of the proposed scheme, and the best performing one is discussed in detail in this paper. the novelty of the proposed approach lies in the use of topic modeling augmented with transformerbased (roberta) sentence embeddings to generate sentence representations. this combination of roberta with topic modeling to generate sentence representations has been found to be more effective than just using transformer-based embedding. the paper is organised as follows. the related past works are discussed in section 2. section 3 and section 4 discuss the details of the proposed system and the experimental setup, respectively. the results from the systems are given in section 5. the paper is concluded in section 6. rhetoric role label preamble preamble facts fac ruling by lower court rlc issues issue argument by petitioner arg_petitioner argument by respondent arg_respondent analysis analysis statute sta precedent relied pre_relied precedent not relied pre_not_relied ratio of the decision ratio ruling by present court rpc none none most of the existing works use conditional random fields (crfs) @xcite as an important component in the architecture for sequential sentence labelling. these works primarily use hierarchical sequence encoders to contextualise sentence representations, followed by a crf for the sequential classification in each document since they incorporate dependencies between subsequent labels. in particular, @xcite and @xcite use bilstm-crf as models for rhetorical role prediction. @xcite propose a hierarchical neural network model for sequential sentence classification task, which we call a hierarchical sequential labeling network (hsln). this consists of an rnn or cnn layer and a bilstm layer to get sentence representation and contextualised sentence representation, respectively. this is followed by a single hidden layer feedforward network to transform the sentence representation to the probability vector. the final layer is crf to jointly optimise the predicted label sequence. in recent years there has been a surge in pretrained transformer-based models for most of the natural language processing tasks. in particu-lar, for the task of sequential sentence classification, @xcite use pre-trained language models, such as bert @xcite to capture contextual dependencies without the need for neither hierarchical encoding nor a crf. their approach achieved state-of-the-art results on four datasets exhibited. ## system overview in the first step the system extracts individual sentence representations that are not contextualised with respect to the whole document. this step uses roberta @xcite for generating text embeddings. the roberta model has been fine-tuned for masked language modeling (mlm) task on the training and development subsets of the related ildc (indian legal documents corpus) dataset @xcite . sentence representation is the cls token embedding for that sentence. this fine-tuned roberta model is applied to each sentence of each document to get corresponding sentence embeddings which are the cls embeddings from the roberta model. the overall scheme may be explained as follows: suppose each document d is represented by a s d × l matrix, where s d is the number of sentences in the document and l is the maximum length of all sentences in the document, where each sentence with less number of words than l are padded with a padding token. this step generates an embedding for each sentence in the document, and thereby converts a document from s d × l to s d × robert a_embedding_size. the existing sentence encoding from roberta model accounts for the information from all the words of the sentence and how the words interact. but, to account for topical composition of each sentence with respect to the information learned from the entire corpus, the features are augmented using a topic modeling approach. the idea is to train a nonparametric topic model, namely, hierarchical dirichlet process (hdp) @xcite on the collection of training documents, which learns the number of topics k from the data itself. application of hdp on a sentence produces a k-dimensional vector which represents the topical composition of that sentence. the per-document per-sentence preliminary representation obtained from roberta model in the previous step is concatenated with hdp topic proportions per sentence, to get a combined sentence representation. this results in a s ×(robert a_embedding_size+ k) matrix of feature vectors corresponding to each document. using these combined sentence embeddings as features a document level bilstm of hidden state dimension hidden_size_document is applied to each document to get hidden states corresponding to each sentence. this results in features of dimension s × (2 × hidden_size_document) for each document, consisting of concatenated hidden states from backward and forward lstm. these hidden states are input to a linear neural network layer. the output is pseudo-prediction vectors corresponding to the tag space of size t arget_size which is the number of labels in the target space, resulting in predictions of dimension s × (t arget_size). up to this point the model has taken into account features of neighbouring sentences for predicting the rhetorical role ( figure 1 provides the diagram for the proposed system. two variations of the proposed approach were also submitted to the task. ## experimental setup the experiments were carried on google colaboratory in python 3.8.10 with nvidia tesla p100 gpu. pytorch @xcite , huggingface transformers @xcite , and pytorch-crf foot_0 are used as the key frameworks for the experiments. for fine-tuning roberta for mlm task the following settings were followed. to prepare the text for training the main model, the newline character, ellipsis and comma are replaced by whitespace. the text is tokenized using robertatokenizerfast tokenizer pretrained from roberta-base model. for each sentence, padding and truncation are enabled, with padding to maximum sequence length in the batch and truncation at maximum sequence length of 256. roberta special tokens have been added. to the number of tags, two additional tags, namely, <@xmath0. for batching, documents are sorted according to the number of sentences in the document. top b documents are selected and are represented as s 1 × l 1 , s 2 × l 2 , • • • , s b × l b , where b is the chosen batch size. the maximum number of sentences in the batch are @xmath1, • • • , s b ) and maximum length of sequence in the batch is @xmath2, • • • , l b ). each sentence in each document in the batch is padded with padding token (<@xmath3. in case of roberta, @xmath4. thus the documents are now of the sizes, s 1 × l, • • • , s b × l. each document is padded to the size s × l, by adding sentences consisting only @xmath5, with corresponding labels @xmath6. this prepares a batch of size b × s × l and corresponding labels b × s. corresponding mask vectors of size b × s × l are prepared, with entries 0 or 1, with 0 indicating presence of @xmath7. the model settings are given in the redundant topics (called dead topics) were purged, and the k topics learned from data were obtained. the topic proportions of a sentence using this model represent the additional features. to batch the corresponding hdp feature vectors for a document, the vectors are padded with k-dimensional zero vectors so that there are uniform number of sentences per document in that batch. the correlated topic model (ctm) was trained on 30 topics, with minimum collection frequency 5. the model is trained on the train subset of the given data, and the model checkpoint according to the best performance on validation subset is chosen for testing on the test subset of the dataset. ## results several experiments were conducted and the system for submission was selected based on the performance on the validation subset of the dataset. the proposed system is denoted as, roberta+hdp+bilstm+crf, where the roberta model has been fine-tuned for mlm task on ildc dataset. a variation of the proposed scheme, denoted as, roberta+ctm+bilstm+crf, has also been tried. this uses parametric topic model ctm instead of nonparametric topic model hdp, to extract additional features. similar variations have been tried, using bert @xcite . the results obtained on the test subset of the dataset for the proposed scheme are given in the best performance is observed in case of the proposed system. it is also observed that adding hdp features improve the overall performance of the system. further, the performance is better in case of using nonparametric topic model hdp, than using parametric topic model ctm, for extracting additional features. ## conclusion this work proposes a system for the task of rhetorical role prediction in legal documents which can be reformulated as a sequential sentence classification task. the proposed system combines topic modeling and roberta to get sentence representations for each sentence in a document. this is followed by a bilstm layer to get contextualised sentence representations. the final layer is a crf layer which takes into account the dependencies between labels to generate sequential rhetorical role predictions for each document. this paper discusses in the detail the system description of the proposed scheme, along with the experimental setup for reproducibility. the novelty of the proposed approach lies in the use of topic modeling to augment the sentence representation. this paper discussed results corresponding to different ablations of the proposed approach. one possible future direction of research can be to explore other ways to incorporate the corpus information gained from topic modeling, to improve the task of sequential sentence classification.