P oli T o HFI at S em E val-2023 Task 6: Leveraging Entity-Aware and Hierarchical Transformers For Legal Entity Recognition and Court Judgment Prediction (2023)

Abstract: The use of Natural Language Processing techniques in the legal domain has become established for supporting attorneys and domain experts in content retrieval and decision-making. However, understanding the legal text poses relevant challenges in the recognition of domain-specific entities and the adaptation and explanation of predictive models. This paper addresses the Legal Entity Name Recognition (L-NER) and Court judgment Prediction (CPJ) and Explanation (CJPE) tasks. The L-NER solution explores the use of various transformer-based models, including an entity-aware method attending domain-specific entities. The CJPE proposed method relies on hierarchical BERT-based classifiers combined with local input attribution explainers. We propose a broad comparison of eXplainable AI methodologies along with a novel approach based on NER. For the L-NER task, the experimental results remark on the importance of domain-specific pre-training. For CJP our lightweight solution shows performance in line with existing approaches, and our NER-boosted explanations show promising CJPE results in terms of the conciseness of the prediction explanations.

## introduction ai-powered tools for legal ai can analyze vast amounts of documents providing lawyers and judges with relevant insights on case laws @xcite . for example, based on the analysis of past court cases legal ai models can predict the outcome of similar cases, thus offering efficient and effective ways of resolving disputes @xcite . the legal domain poses relevant challenges due to the specificity of the legal language, the variety of scenarios and application contexts, and the temporal evolution of norms and regulations @xcite . this paper addresses two notable legal ai tasks, i.e., legal named entity recognition and court judgment prediction with explanation. legal named entity recognition the l-ner task aims at annotating portions of legal content with domain-specific entities. annotations are particularly helpful to support content retrieval and indexing. for example, the identification of relevant legal provisions can help lawyers and judges make informed decisions based on previous cases. despite ner being a well-known nlp task, the application of existing techniques to legal data is challenged by the inherent complexity and nuances of the language used in the legal domain @xcite . we address l-ner by exploring the use of established transformer-based models, established for many nlp tasks @xcite and tailored to the legal domain @xcite . unlike previous studies, we also explore the use of the entity-aware attention mechanism @xcite , which allows the transformer to also attend to domain-specific entities. the preliminary results achieved on the development set confirm the potential benefits of leveraging the entity-aware attention mechanism in l-ner. ## related works transformer-based models have exhibited remarkable performance in many legal ai domains such as legal question answering @xcite and legal document summarization @xcite . this paper addresses three specific research lines in legal ai: named entity recognition, court judgment prediction, and predictions explanation. ## system overview in this section we separately introduce the approaches to l-ner, cjp (classification only), and cjpe (classification with predictions explanation). ## experimental results in this section, we report the outcomes of our empirical investigations on the three aforementioned tasks. specifically, we participated in the shared task competition for tasks b, c-1, and c-2 @xcite and evaluated the performance of our proposed models on the respective test sets using the official evaluation metrics. we rank 11 th out of 17 participants to the l-ner task, 4 th out of 11 participants to cjp, and 5 th out of 11 to cjpe. these findings offer valuable insights into the potential of the proposed models for enhancing the accuracy and efficiency of legal nlp applications. parameter setting for the l-ner task, the luke-based models are trained using a batch size of 256 with a 10 -4 learning rate, while the bertbased models utilize a batch size of 1 and a learning rate of 10 -4 . both models are trained for a maximum of 5 epochs with an early stopping criterion. additionally, a weight decay of 0.01 is applied for both models, and the warmup ratio is set to 0.06. for the cjp task, we train sentence encoders for a maximum of 15 epochs, using a learning rate of 5 • 10 -5 , a warmup ratio of 0.06, a weight decay of 0.01, and a batch size of 64. the hierarchical transformer-based architecture has a maximum length of 256 tokens and it is trained for a maximum of 100 epochs, with a learning rate of 5•10 -5 , and a batch size of 256. ## conclusions and future research directions this paper addressed two established legal ai task, i.e., the named entity recognition and court judgment prediction. more specifically, in light of the achieved results, the l-ner models fine-tuned on legal text data outperformed those fine-tuned on general text data. specifically, the bert-base model fine-tuned on the european court of human rights (echr) dataset performed best. the hierachical approach to cjp achieved results comparable to the baseline methods. for the cjpe task, ner-boosted explanations show promising results in producing concise yet informative explanations. as future work, we plan to explore the applicability of entity-aware transformers to other legal ai tasks such as relation extraction. entities could help to better capture the inter-dependencies between different entities and their relationships within the legal text.