T ü R euth Legal at S em E val-2023 Task 6: Modelling Local and Global Structure of Judgements for Rhetorical Role Prediction (2023)

Abstract: This paper describes our system for SemEval-2023 Task 6: LegalEval: Understanding Legal Texts. We only participate in Sub-Task (A), Predicting Rhetorical Roles. Our final submission achieves 73.35 test set F1 score, ranking 17th of 27 participants. The proposed method combines global and local models of label distributions and transitions between labels. Through our analyses, we show that especially modelling the temporal distribution of labels contributes positively to performance.

## introduction this paper describes our system for semeval-2023 task 6: legaleval: understanding legal texts. while the shared task hosts 3 sub-tasks, we only participate in sub-task (a), predicting rhetorical roles (rrs) from pre-segmented judgments. refer to the overview paper @xcite for a detailed description of the task and data. briefly, the task is as follows: given a judgment that is pre-segmented into paragraphs (often containing only 1 sentence) and an inventory of rhetorical roles, predict the appropriate rr for every paragraph. rrs structure the judgment into semantically coherent units and provide information on what is discussed in each unit. for example, the rr preamble denotes formal introductory statements of judgments. in total, there are 13 rrs in this task. the shared task provides 245 segmented and labelled indian judgements (in english) for training, 30 segmented and labelled judgments for development, and 50 segmented but unlabelled test set judgments for evaluation. test set labels are not released by the organisers. the main challenge of this task is how to integrate document level information and paragraph level information. since solving the task requires precise understanding of paragraph content and the context, fine-tuning pre-trained large language models (lms) is the common starting point. however, lms can not process long documents such as judgments on standard hardware at once. therefore, we develop and evaluate a method to use document-level information in order to refine paragraph-level predictions from a fine-tuned lms. in particular, we model the positions in a document where rrs usually occur, structural constraints that indicate whether some rrs may or may not be predicted when other rrs have already been predicted earlier, and lexical constraints motivated by domain knowledge. throughout this paper, we assume the following notation: r is the set of rhetorical roles, @xmath0, a judgement j consists of a sequence of paragraphs s 1 , . . . , s t where t is the number of paragraphs in j. paragraphs are indexed by t (we think of them as a time-series). ## related work closest to our method is previous work by the shared task organisers @xcite . both works propose to first encode each sentence of a judgement by lm. then, another sequence processor, such as lstm or crf, is applied to predict rrs from the sequence of sentence encodings. @xcite focus on how to arrive at sentence encodings from token encodings computed by a bert model. on the other hand, @xcite focus on label shift. they train additional models to predict whether 2 consecutive sentences have the same rr. then, they use this information as feature in their sequence labelling models. in this work, we continue the approach of @xcite and include more structural modelling. however, in our experiments we found models to predict whether consecutive sentences share the same rr to yield insufficient performance on the negative class (when rrs are different). most likely, the reason is that, as noted by @xcite , most consecutive sentences share the same rr, resulting in very imbalanced data. therefore, we choose a binary model of whether certain transitions are possible or not instead, which is less expressive but trivial to learn. furthermore, treating whole judgments as datapoints for supervised sequence labelling yields only 245 train datapoints for this shared task, which may be problematic because of overfitting. also, endto-end training of sentence encoders and sequence labelling models is very expensive. therefore, we do not use any sequence level neural models, but combine scores from individual models during inference. ## method our method combines sentence-level information with document-level information in order to improve performance of models only using limited contexts. concretely, we ## results in this section, we present the final performance of our method and further insights regarding the individual models described in section 3.1. ## conclusion in this work, we propose and evaluate different models of global and local judgment structure. besides using pre-trained language models, the greatest performance gains result from modelling the temporal distribution of labels, i.e. where in the document a label occurs more often. furthermore, this information can be easily integrated with most sequence labelling methods. otherwise, we also show that modelling how the possibility of predicting rrs depend on previously predicted rrs and integrating domain knowledge improve performance to a lesser degree. we do not find relevant improvements from restricting transitions between rrs to such transitions that appear in the training data. as possible next steps, we think it would be interesting to improve performance of lms, e.g. by showing them more context, and to develop better models of local transitions between rrs. finally, we make our code available on github.