NITK _ LEGAL at S em E val-2023 Task 6: A Hierarchical based system for identification of Rhetorical Roles in legal judgements (2023)

Abstract: The ability to automatically recognise the rhetorical roles of sentences in a legal case judgement is a crucial challenge to tackle since it can be useful for a number of activities that come later, such as summarising legal judgements and doing legal searches. The task is exigent since legal case documents typically lack structure, and their rhetorical roles could be subjective. This paper describes SemEval-2023 Task 6: LegalEval: Understanding Legal Texts, Sub-task A: Rhetorical Roles Prediction (RR). We propose a system to automatically generate rhetorical roles of all the sentences in a legal case document using Hierarchical Bi-LSTM CRF model and RoBERTa transformer. We also showcase different techniques used to manipulate dataset to generate a set of varying embeddings and train the Hierarchical Bi-LSTM CRF model to achieve better performance. Among all, model trained with the sent2vec embeddings concatenated with the handcrafted features perform better with the micro f1-score of 0.74 on test data.

## introduction in a country with a large population like india, the number of legal cases has been increasing rapidly. people are overloaded by the vast amount of information and documents available online, which is a result of the expansion of the internet and big data @xcite . legal documents are one example of this type of online information growth in the subject of law. constitutions, contracts, deeds, orders/judgments/decrees, pleadings, statutes, and wills are some examples of these documents. these documents are very lengthy to read and comprehend and have a structure that is quite elaborate compared to a regular text. legal case documents are difficult to process automatically due to their length and unstructured nature if the documents could be divided up into cohesive information pieces, a legal document processing system would greatly benefit. in a legal case document, rhetorical role labelling of a sentence means identifying the semantic function a sentence of a legal document serves, such as the case's facts, the parties' arguments, ruling from the current court, etc. semantic search, summarization, case law analysis, and other downstream tasks can all benefit from knowing the rhetorical roles that sentences play in a legal case document. however, legal documents lack a clear organizational structure, making it challenging to comprehend the nuanced differences between rhetorical roles. rhetorical roles(rrs) aid in the extraction of legal information, such as cases with similar facts. comparing various rhetorical role units can also be used to find previous cases that are comparable to the one in concern. using rrs, one may, for instance, extract the key information from the case that affects the judgement. the goal of this paper is to showcase the proposed methodology as a part of semeval-2023 task 6: legaleval: understanding legal texts sub-task a: rhetorical roles prediction (rr) competition and also discuss different methods implemented to achieve this task. we secured 16th position out of 27 teams that participated in this task. the task is to segment a given legal document by accurately identifying the rhetorical role of each sentence. this task involves classifying consecutive sentences into many classes using a single label. evaluation metric being micro f1 score is used for evaluation based on the hidden test results. in this paper, we use indian court judgements dataset from semeval-2023 task 6: sub-task a: rhetorical roles prediction (rr). we describe different methods for segmenting a legal document into coherent information units i.e, rhetorical roles to help with the processing of lengthy legal documents @xcite . our corpus has rhetorical roles annotated (rrs). rest of the report organisation is as follows: the following section 2 provides a related work on analysis of the related papers and glimpses about dataset used and kind of input and output to the system. next section 3 shows the proposed work of the project. followed by section 4 analysis and application results are highlighted and in section 5 concludes the work that is discussed at the end of the paper. ## related works a legal document processing system would benefit immensely from being able to segment the documents into coherent informational chunks. authors in @xcite proposed a new corpus of legal documents that have been annotated with a set of 13 labels for semantically coherent units (referred to as rhetorical roles), such as facts, arguments, statutes, issues, precedents, rulings, and ratios. the ability to automatically recognise the rhetorical roles of words in a legal case judgement is an essential topic to work on because it's potentially useful for a broad range of activities that occur afterwards, such as summarising legal judgements and doing legal searches. automatic text summarization helps people save a significant amount of time at work @xcite and stay updated on world events by condensing news stories, technical documentation, books, essays, conferences, and meetings to a much more digestible format with little data loss @xcite . the study in @xcite emphasises the significance of summary in increasing the accessibility of court decisions and presents a helpful comparison of several summarization techniques for this job. authors in @xcite claim that the final presentation of the summary of a legal document is proven to be improved by understanding of fundamental structures and distinct segments of the document. authors in @xcite describes the process of extending bert, a language model that has already been trained, for text classification tasks. assigning a label reflecting the rhetorical state of each sentence in a particular segment of a document, this paper offered a new annotation method for the rhetorical structure of legal decisions. authors in @xcite present a deep learning model for legal text recognition based on the combination of long short-term memory (lstm) and conditional random field (crf) models which could potentially be used in areas like information retrieval and processing legal documents. the creation of appropriate feature sets for the effective usage of crfs in the process of segmenting @xcite , a legal text along different rhetorical functions is also highlighted by these authors. using a conditional random field (crf) layer with (bi-lstm) network, the authors of @xcite proposed a model that captures contextual information and dependencies between the named entities. in the latter phases of document summarization, the segmentation of a legal text into certain rhetorical categories may well be employed to improve the outcomes. when compared to straightforward extraction-based summarization, the updated final summary improves the readability and efficiency. ## results and analysis during the 21 days of evaluation period (january 10 through 31, 2023), 27 codalab users submitted around 174 submissions for rr shared tasks. each team has the eligibility of a maximum 20 submissions with maximum submissions of 2 per ## conclusion and future work in this paper, we describe our knowledge based system for the semeval-2023 task 6: legaleval: understanding legal texts sub-task a: rhetorical roles prediction (rr), that ranked 16 out of 27 teams participated in the task. to predict the rhetorical role of every sentence of legal documents of the supreme court of india, we have used the hierarchical bi-lstm crf model. in addition to the layers in the hierarchical bi-lstm crf, we have added an attention layer to the model to give prior importance for the few parts of the data. the precision, recall and f1 scores for all these embeddings are calculated with the base model (hierarchical bi-lstm crf model) and roberta transformer method is also implemented and are compared to find the best suitable model and the best embedding that gives the best labelling of the rhetorical roles of the sentences in the data. here we have observed that out of all, the sent2vec embeddings with the hand crafted features gave the best micro f1-score compared with the others. furthermore, we can improve the model by adding different layers to the current model with the different embeddings for better performance and result.