u O ttawa at L egal L ens-2024: Transformer-based Classification Experiments (2024)

Abstract: This paper presents the methods used for LegalLens-2024, which focused on detecting legal violations within unstructured textual data and associating these violations with potentially affected individuals. The shared task included two subtasks: A) Legal Named Entity Recognition (L-NER) and B) Legal Natural Language Inference (L-NLI). For subtask A, we utilized the spaCy library, while for subtask B, we employed a combined model incorporating RoBERTa and CNN. Our results were 86.3% in the L-NER subtask and 88.25% in the L-NLI subtask. Overall, our paper demonstrates the effectiveness of transformer models in addressing complex tasks in the legal domain.

## introduction the huge amount of information and massive use of the internet has propelled to ignore legal violations, individual rights, cultural values and societal norms. these hidden violations demand serious attention and urgent solution due to serious effects on rights and justice and it requires advanced tools for professionals to effectively manage large amount of paperwork. legal violation identification seeks to automatically detect legal violations within unstructured text and link these violations to potential victims. the legallens 2024 shared task @xcite aims to foster a legal research community by tackling two key challenges in the legal domain. subtask a focuses on identifying legal violations (a.k.a identification setup) using named entity recognition (ner). subtask b focuses on linking these violations to potentially affected individuals (a.k.a identification setup) using natural language inference (nli). our team participated in both subtasks of the shared task. in subtask a, we used the spacy library and a deberta-based model. in subtask b, we developed a roberta-based model combined with a cnn-based model. ## related work there has been extensive research on legal named entity recognition (ner) for german legal documents. @xcite al. (2019 @xcite described methods for the coliee 2023 competition, using a sentence transformer model for case law retrieval and a finetuned deberta model for legal entailment that used snli @xcite and multinli @xcite datasets for training. tang (2023) explored improving legal natural language inference (nli) by employing general nli datasets with supervised fine-tuning and examining the impact of transfer learning from adversarial nli to contractnli. the objective of @xcite is to offer a theoretically grounded characterization of explanation-based natural language inference (nli) by integrating contemporary philosophical accounts of scientific explanation with an analysis of natural language explanation corpora. gubelmann et al. (2023) investigated how large language models (llms) handle different pragmatic sentence types, like questions and commands, in natural language inference (nli), highlighting the insensitivity of mnli and its finetuned models to these sentence types. it developed and publicly released fine-tuning datasets to address this issue and explored chatgpt's approach to entailment. ## subtask a: legal named entity recognition(l-ner) subtask a, which involves finding named entities for specific types that may appear in legal texts, is explained in this section. we developed a bert-based model for this subtask as part of the legallens task, achieving an f1-score(macro f1-score) of 86.3%. ## subtask b: legal natural language inference (l-nli) the goal of this subtask is to automatically classify the relationships between different legal texts. specifically, we aim to determine whether a legal premise, such as a summary of a legal complaint, entails, contradicts, or remains neutral with respect to a given hypothesis, like an online review. this task, termed legal natural language inference (l-nli), involves sentence-pair classification to assess these relationships. by creating an nli corpus tailored for legal documents, we facilitate applications like legal case matching and automated legal reasoning. detailed task definitions and datasets are provided in @xcite and related resources. ## conclusion and future work our experiments demonstrated the success of transformer models, such as roberta and deberta, in handling complex legal tasks, including violation detection and inference. in subtask a (l-ner), incorporating deberta into the spacy pipeline model f1-score 1-800-shared-tasks 0.853 baseline 0.807 semantists 0.785 nowj 0.746 uottawa 0.724 yielded strong results for legal named entity recognition. in subtask b (l-nli), combining roberta with cnn for keyword detection boosted classification accuracy. however, despite using robust models, generalizing to unseen cases proved challenging, particularly with nuanced legal language. while the cnn improved phrase detection, more advanced methods, like attention mechanisms, may further enhance performance. future work should explore architectures finetuned on legal texts or combine transformers with graph models to capture legal relationships. additionally, leveraging llms like gpt-4 could improve legal reasoning.