Semantic Segmentation of Legal Documents via Rhetorical Roles (2022)

Abstract: Legal documents are unstructured, use legal jargon, and have considerable length, making them difficult to process automatically via conventional text processing techniques. A legal document processing system would benefit substantially if the documents could be segmented into coherent information units. This paper proposes a new corpus of legal documents annotated (with the help of legal experts) with a set of 13 semantically coherent units labels (referred to as Rhetorical Roles), e.g., facts, arguments, statute, issue, precedent, ruling, and ratio. We perform a thorough analysis of the corpus and the annotations. For automatically segmenting the legal documents, we experiment with the task of rhetorical role prediction: given a document, predict the text segments corresponding to various roles. Using the created corpus, we experiment extensively with various deep learning-based baseline models for the task. Further, we develop a multitask learning (MTL) based deep model with document rhetorical role label shift as an auxiliary task for segmenting a legal document. The proposed model shows superior performance over the existing models. We also experiment with model performance in the case of domain transfer and model distillation techniques to see the model performance in limited data conditions.

## introduction the number of legal cases has been growing almost exponentially in populous countries like india. for example, as per the india's national judicial data grid, there are about 41 million cases pending in india @xcite . as per some of recent estimates by a retired supreme court of india judge, it will take about 450 years * equal contributions † corresponding author to clear the backlog of cases @xcite . technology could come to the rescue in dealing with the backlog, for example, if there were a technology (based on nlp techniques) that could help a legal practitioner to extract relevant information from legal documents then it could make the legal process more streamlined and efficient. however, legal documents are quite different from conventional documents used to train nlp systems (e.g., newspaper texts). legal documents are typically long (tens of pages) @xcite , unstructured @xcite , noisy (e.g., grammatical and spelling mistakes due to manual typing in courts) @xcite , and use different lexicon (legal jargon). the use of a specialized lexicon and different semantics of words makes pre-trained neural models (e.g., transformer-based models) ineffective @xcite . the legal domain has several sub-domains (corresponding to different laws, e.g., criminal law, income tax law) within it. although some of the fundamental legal principles are common, the overlap between different sub-domains is low; hence systems developed on one law (e.g., income tax law) may not directly work for another law (e.g., criminal law), so there is the problem of a domain shift @xcite @xcite . in this paper, we target legal case proceedings in the form of judgment documents. to aid the processing of long legal documents, we propose a method of segmenting a legal document into coherent information units referred to as rhetorical roles @xcite . we propose a corpus of legal documents annotated with rhetorical roles (rrs). rrs could be useful for various legal applications. legal documents are fairly long, and dividing these into rhetor-ical role units can help summarize documents effectively. in the task of legal judgment prediction, for example, using rrs, one could extract the relevant portions of the case that contributes towards the final decision. rrs could be useful for legal information extraction, e.g., it can help extract cases with similar facts. similarly, prior cases similar to a given case could be retrieved by comparing different rhetorical role units. in this work, we make the following contributions: 1. we create a new corpus of legal documents annotated with rhetorical role labels. in contrast to previous work (8 rrs) @xcite , we create a more fine-grained set of 13 rrs. further, we create the corpus on different legal domains ( §3). 2. for automatically segmenting the legal documents, we experiment with the task of rhetorical role prediction: given a document, predict the text segments corresponding to various roles. using the created corpus, we experiment with various deep text classification and baseline models for the task. we propose new multi-task learning (mtl) based deep model with document level rhetorical role shift as an auxiliary task for segmenting the document into rhetorical role units ( §4). the proposed model performs better than the existing models for rr prediction. we further show that our method is robust against domain transfer to other legal sub-domains ( §5). we release the corpus, model implementations and experiments code: https://github.com/ exploration-lab/rhetorical-roles 3. given that annotating legal documents with rr is a tedious process, we perform model distillation experiments with the proposed mtl model and attempt to leverage unlabeled data to enhance the performance ( §5). we also show the use-case for rr prediction model. ## related work legal text processing has been an active area of research in recent times. a number of datasets, applications, and tasks have been proposed. for example, argument mining @xcite , information extraction and retrieval @xcite , event extraction @xcite , prior case retrieval @xcite , summarization @xcite , and case prediction @xcite @xcite . re-cently, there has been a rapid growth in the development of nlp and ml technologies for the chinese legal system, inter alia, @xcite al. (2019); hu et al. (2018); jiang et al. (2018); yang et al. (2019) ; @xcite . few works have focused on the creation of annotated corpora and the task of automatic rhetorical role labeling. @xcite developed a corpus, temis of 504 sentences annotated both syntactically and semantically. the work of @xcite focuses on the process of annotation and conducting inter-annotator studies. @xcite conducted document segmentation of u.s. court documents using conditional random fields (crf) with handcrafted features to segment the documents into functional and issue-specific parts. automatic labeling of rhetorical roles was first conducted in @xcite , where crfs were used to label seven rhetorical roles. @xcite developed a method for identification of factual and nonfactual sentences using fasttext. the automatic ml approaches and rule-based scripts for rhetorical role identification were compared in @xcite . @xcite create a large corpus of rrs and propose transformer based baseline models for rr prediction. our work comes close to work by @xcite , where they use the bilstm-crf model with sent2vec features to label rhetorical roles in indian supreme court documents. in contrast, we develop a multitask learning (mtl) based model for rr prediction that outperforms the system of @xcite . ## rhetorical roles corpus corpus acquisition: we focus on indian legal documents in english; however, techniques we develop can be generalized to other legal systems. we consider legal judgments from the supreme court of india, high courts, and tribunal courts crawled from the website of indiankanoon ( https://indiankanoon.org/ ). we also scrape competition law documents from indian tribunal court cases (national company law appellate tribunal (nclat), competition appellate tribunal (compat), competition commission of india (cci)). we focus on two domains of the indian legal system: competition law (cl) (also called as anti-trust law in the us and anti-monopoly law in china) and income tax (it). cl deals with regulating the conduct of companies, particularly concerning competition. with the help of legal experts, we narrowed down the cases pertinent to cl and it from the crawled corpus (also see ethical considerations in app. a). ## rhetorical roles prediction we would like to automate the process of segmenting a legal document, to develop ml models for the automation, we experiment with the task of rhetorical roles prediction. task definition: given a legal document, d, containing the sentences [s 1 , s 2 , ...s n ], the task of rhetorical role prediction is to predict the label (or role) y i for each sentence s @xmath0. baseline models: for the first set of baseline models, the task is modeled as a single sentence prediction task, where given the sentence s, the model predicts the rhetorical role of the sentence. in this case, the context is ignored. we consider pre-trained bert @xcite and legal-bert @xcite ) models for this. as another set of baseline models, we consider the task as a sequence labeling task, where the sequence of all the sentences in the document is given as input, and the model has to predict the rr label for each sentence. we used crf with hand-crafted features @xcite and bilstm network. label shift prediction: rhetorical role labels do not change abruptly across sentences in a document, and the text tends to maintain topical coherence. given the label y for a sentence s i in the document, we hypothesize that the chances of shift (change) in the label for the next sentence s i+1 are low. we manually verified this using the training set and observed that on average in a document, if the label of sentence s i is y, then 88% of the times the label of the next sentence s i+1 is same as y. note that this is true only for consecutive sentences, but in general, label shift inertia fades as we try to predict beyond the second consecutive sentence. since we are performing a sequence prediction task, this alone is not a good model for label prediction. nevertheless, we think that this label shift inertia can provide a signal (via an auxiliary task) to the main sequence prediction model. based on this observation, we define an auxiliary binary classification task: label shift prediction (lsp), that aims to model the relationship between two sentences s i and s i+1 and predict whether the labels y i for s i and y i+1 for s i+1 are different (shift occurs) or not. in particular, for each sentence pair @xmath1, s i+1 } ∈ d, we define the label of lsp task, @xmath2, otherwise @xmath3, here y i is the rhetorical role for sentence s i . note that for the full model at the inference time, the true label of a sentence is not provided; hence predicting a shift in label makes more sense than performing a binary prediction that the next sentence has the same label or not. we model the lsp task via two different models: sbert-shift: we model the label shift via a siamese network. in particular, we use the pretrained sbert model @xcite to encode sentences s i and s i+1 to get representations e i and e i+1 . the combination of these representations (e i ⊕ e i+1 ⊕ (e ie i+1 )) is passed through a feed-forward network to predict the shift. bert-sc: we use the pre-trained bert model and fine-tune it for the task of lsp. we model the input in the form of sentence semantic coherence task, [cls]⊕s i ⊕[sep ]⊕s i+1 ⊕[sep ] to make the final prediction for shift. in general, the bert-sc model performs better than sbert-shift ( the intuition is that a label shift would help the rhetorical role component make the correct prediction based on the prospective shift. the mtl model (figure 3 ) consists of two components: the shift detection component and the rhetorical role prediction component. the shift component predicts if a label shift occurs at i th position. the output of the bilstm layer of shift component is concatenated with the bilstm output of the rhetorical role component. the concatenated output is passed to a crf layer for the final prediction of the rhetorical role. the loss for the model is given by: @xmath4, where, l shif t is the loss corresponding to label shift prediction and l rr is the loss corresponding to rhetorical role prediction, and hyperparameter λ balances the importance of each of the task. if λ is set to zero, we are back with our baseline bilstm-crf model. since there are two components, we experimented with sending the same encodings of sentences to both the components (e 1 = e 2 ), as well as sending different encodings of the same sentence to both components (e 1 ̸ = e 2 ). the proposed model is very different from the previously proposed bilstm-crf by @xcite that does not use any multitasking and label shift information. ## experiments, results and analysis due to the complexity of the task of rr prediction and to be comparable with the existing baseline systems, for experiments, we consider 7 main labels (fac, arg, pre, rod, rpc, rlc, and sta). we plan to explore all fine-grained rr label (13) pre- lstm lstm lstm crf crf crf y'i y'1 y'n lstm lstm lstm crf crf crf lstm lstm lstm crf crf crf yi y1 yn lstm lstm lstm crf crf crf yi y1 yn e1(s1) e1(si) e1(sn) e2(s1) e2(si) e2(sn) s 1 s i s n .... .... .... .... .... .... e1 e2 shift component rr component figure 3: mtl architecture for rhetorical role labelling and shift prediction. dictions in the future. based on recommendations by legal experts, we ignore sentences with non (none) label (about 4% for it and 0.5% for cl) (more details in app. d.1). further, the it domain did not have any instance of dissent (dis) label, and cl has only three documents with very few dis instances. based on consultations with law experts, we discarded dis sentences (more details in app. d.1). we randomly split (at document level) it/cl into 80% train, 10% validation, and 10% test set. in contrast to bhattacharya et al. (2019), we did not perform cross-validation for better comparison across different models. we also experiment with a combined dataset of it and cl (it+cl); the splits are made by combining individual train/val/test split of it and cl. we experimented with a number of baseline models ( model distillation: rr annotation is a tedious process, however, there is an abundance of unlabelled legal documents. we experimented with semi-supervised techniques to leverage the unlabelled data. in particular, we tried a self-training based approach @xcite . the idea is to learn a teacher model θ tea on the labelled data d l . the teacher model is then used to generate hard labels on unlabeled sentences s @xmath5: next, a student model θ stu is learned on labeled and unlabeled sentences, with the loss function for student training given by: here, α u is a weighing hyperparameter between the labelled and unlabelled data (details in app. d). the process can be iterated and the final distilled model is used for prediction. the results of model distillation are shown in ## conclusion we introduce a new corpus annotated with rhetorical roles. we proposed a new mtl model that uses label shift information for predicting labels. we further showed via domain transfer experiments the generalizability of the model. since rr are tedious to annotate, we showed the possibility of using model distillation techniques to improve the system. in the future, we plan to explore cross-domain transfer techniques to perform rr identification in legal documents in other indian languages. nevertheless, we plan to grow the corpus. we also plan to apply rr models for other legal tasks such as summarization and information extraction.