IRIT _ IRIS _ A at S em E val-2023 Task 6: Legal Rhetorical Role Labeling Supported by Dynamic-Filled Contextualized Sentence Chunks (2023)

Abstract: This work presents and evaluates an approach to efficiently leverage the context exploitation ability of pre-trained Transformer models as a way of boosting the performance of models tackling the Legal Rhetorical Role Labeling task. The core idea is to feed the model with sentence chunks that are assembled in a way that avoids the insertion of padding tokens and the truncation of sentences and, hence, obtain better sentence embeddings. The achieved results show that our proposal is efficient, despite its simplicity, since models based on it overcome strong baselines by 3.76% in the worst case and by 8.71% in the best case.

## introduction semeval-2023 task 6 (legaleval) focuses on the understanding of texts from the legal domain and it comprises three sub-tasks: legal rhetorical role labeling, legal named entity recognition, and court judgment prediction with explanation @xcite . the legal rhetorical role labeling (legal rrl) task aims to assign semantic roles to the sentences from a legal document. examples of such roles are facts, arguments, and rulings. the exact roles are defined by the documents' nature and by the following utilization of the labeled sentences. legal documents are usually long and complex, so the task is relevant because the labeled sentences assist the document interpretation and the extraction of information. the labeled sentences are useful for downstream tasks, such as summarization and judgment prediction. the task is not trivial even for professionals from the legal domain, mainly because of its inherent subjectivity. the legal rrl sub-task dataset comprises judgment documents written in english from the indian legal system. the documents' sentences were manually labeled by students from indian law universities, whose each sentence was assigned with one of 13 pre-defined rhetorical roles @xcite . the legal rrl can be tackled as a sentence classification task, and this approach requires the utilization of an encoder to convert text sentences into numeric vectors. recent legal rrl works exploit pre-trained transformer models to encode sentences. however, they do not fully leverage the context exploitation ability of transformer models: sentences are usually encoded in isolation and the inter-sentence context is lost @xcite , the inter-sentence context is yielded by a recurrent neural network (rnn) @xcite or by using data augmentation ( @xcite , or only cover facts instead of the full range of rhetorical roles ( @xcite . this work presents the approach exploited by the irit_iris team to tackle the semeval-2023 task 6, sub-task a. we propose a novel strategy to encode sentences that fully leverages the context exploitation ability of transformer and hence yields richer sentence embeddings. instead of encoding sentences in isolation, we exploit sentence chunks that avoid the truncation of core sentences and the utilization of padding tokens. achieved results lead us to deem that our proposal is efficient as the respective models overcome strong baselines by 3.76% in the worst case and by 8.71% in the best case. a secondary finding is that models based on pre-trained roberta and longformer models benefit more from the proposed approach than bert-based models foot_0 . ## background to leverage the context exploitation ability of transformers, we just need to feed the encoder with a sequence of sentences separated by a special token such as the [sep] one from bert @xcite . because most transformer models have a limit regarding the sequence length (e.g., 512 for bert) and legal documents are usually long, feeding the encoder with a whole document is not computationally feasible. a workaround is breaking the documents into sentence chunks and feeding the encoder with such chunks as proposed by @xcite . a downside of this approach is that inter-sentence context is restricted to chunks (sentences in a chunk are not aware of sentences in other chunks). @xcite and van den berg and markert (2020) rely on overlapped chunks to overcome this: consecutive chunks share some of their edge sentences as illustrated in figure 1 . only two works about legal rrl leverage sentence chunks and transformers to encode sentences. in @xcite a chunk comprises up to three sentences: the core sentence (i.e., the sentence to be classified) and its left and right neighboring sentences (remark that the first sentence does not have a left neighbor and the last sentence does not have a right neighbor). nevertheless, this model performs worse than one that encodes sentences in isolation. @xcite exploit chunks without sharing sentences as proposed by @xcite but the model performs significantly inferior to one that leverages inter-sentence context through rnns. ## dynamic-filled contextualized sentence chunks the approaches from @xcite , @xcite , and van den berg and markert (2020) differ in the number of shared sentences, number of core sentences, and how the sentences are distributed among chunks. in @xcite there are no shared sentences and the number of chunks is dynamically set from the number of sentences in the document and from a hyperparameter delimiting the maximum number of sentences in a chunk. the chunks are filled so that they contain almost equal amounts of sentences. to assure that, sentences can be truncated. to establish chunks with the same number of tokens, which allows batch processing, the chunks are often filled with padding tokens. @xcite and van den berg and markert (2020) propose chunks with a fixed number of core sentences and edge sentences that must be set in advance. the drawback of this approach is that it yields chunks with a small number of content tokens when the concerned sentences are short. a large number of core sentences could avoid this, but it may lead to many truncated core sentences. chunks with many padding tokens are contextpoor and truncation of core sentences results in data loss. such facts may result in poor sentence embeddings that harm the performance of classification models. ## conclusions this work presents a novel approach to leverage the context exploitation ability of transformers and employs it to tackle the legal rrl subtask of semeval-2023 task 6. the proposal relies on dfc-scs as a way of achieving better sentence embeddings from transformers' hidden states. models based on dfcscs are compared to baselines based on similar approaches and the results show that our proposal is effective: the best dfcsc model achieves a 0.8139 f1 score whereas the best baseline does 0.7573. in the future, we plan to employ dfcscs over other legal rrl datasets as a way of reinforcing our findings. victor sanh, lysandre debut, julien chaumond, and thomas wolf. 2019. distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. corr, abs/1910.01108. jaromír savelka, hannes westermann, and karim benyekhlef. 2020. cross-domain generalization and knowledge transfer in transformers trained on legal data. in proceedings of the fourth workshop on automated semantic analysis of information in legal text held online in conjunction with the 33rd international conference on legal knowledge and information systems, asail@jurix 2020, december 9, 2020, volume 2764 of ceur workshop proceedings. ceur-ws.org. esther van den berg and katja markert. 2020. context in informational bias detection. in proceedings of the 28th international conference on computational linguistics, coling 2020, barcelona, spain (online), december 8-13, 2020, pages 6315-6326. international committee on computational linguistics.