L egal L ens Shared Task 2024: Legal Violation Identification in Unstructured Text (2024)

Abstract: This paper presents the results of the LegalLens Shared Task, focusing on detecting legal violations within text in the wild across two sub-tasks: LegalLens-NER for identifying legal violation entities and LegalLens-NLI for associating these violations with relevant legal contexts and affected individuals. Using an enhanced LegalLens dataset covering labor, privacy, and consumer protection domains, 38 teams participated in the task. Our analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods. The top-performing team achieved a 7.11% improvement in NER over the baseline, while NLI saw a more marginal improvement of 5.7%. Despite these gains, the complexity of legal texts leaves room for further advancements.

## introduction legal violations are everywhere, but often go unnoticed. in many areas such as privacy, consumer protection, environmental law, and labor regulations, traces of these violations, indicating wrongdoing, are frequently lost in the vast amounts of digital information. as the world becomes increasingly digital, it is inevitable that traces of these legal violations can be found online. this concept is the foundation of the legallens project @xcite . these violations pose significant risks to individuals and institutions, undermining legal and ethical standards in our increasingly digital society. therefore, developing advanced methods to detect and address these violations is crucial. identifying legal violations on the open web presents two primary challenges: first, determining where to search, and second, accurately interpreting whether the information indicates a legal violation. the first challenge involves going through massive amounts of online content, selecting sources that are likely to yield relevant information while accounting for varying levels of credibility and relevance. the second challenge lies in applying legal knowledge and to determine the legal grounds for these potential violations, and identify victims who may be entitled to compensation. to advance this field, and to address these challenges, legallens tasks were presented in @xcite . the underlying assumption of legallens is that legal violations often leave digital traces, which can be uncovered through careful analysis. legallens presented a two-step approach to tackle these challenges: the first is legallens-ner (named entity recognition) to extract legal violation entities from online data. the task involves detection and categorization of specific legal violation entities such as laws, violations, violators, and victims within unstructured text. simple ner methods do not focus on these types of entities and fail to capture the ambiguity of legal language. figure 3 shows an example of the ner task. the second step is the legallens-nli (natural language inference) to associate identified violations with relevant legal cases or statutes. more specifically, the task given a premise (allegation summary of a legal case) determine the relationship to a hypothesis (a potential detected violation) and classify their relationship as entailment, contradiction, or neutrality. figure 2 shows an example of the nli task. the datasets for these two sub-tasks were built upon proprietary data by darrow.ai 1 , designed to be as realistic as possible and to capture the nuances and variability of real-world cases. the data was generated in utilizing gpt-4o (openai, 2023) and domain experts, ensuring both realism and com-plexity. the 1st shared task on legallens was organized to encourage new research at the intersection of natural language processing and legal studies and to stimulate interest in legal violation detection within the nlp community. in this paper, we present the results of the shared task, offering a detailed description of the evaluation data and the systems developed by participants. we analyze the performance of the participating systems, evaluating their capabilities in processing legal language and identifying legal violations. the top-performing systems for ner showed a substantial improvement over the baseline, with a 7.11% increase in f1 score for the best team. the nli task saw more marginal progress, with only one team outperforming the baseline by 5.7%. while these improvements highlight progress in legal violation detection, particularly in entity recognition, there remains significant room for further advancements in handling the complexities of natural legal language inference. as a result, this shared task holds value not just for experts in machine learning and nlp, but also for legal professionals, sociologists, and policymakers. this initiative has the potential to foster interdisciplinary collaborations and contribute to advancements in detecting legal violations in the digital era. we are happy to see interdisciplinary teams with participants from cs and nlp alongside legal practitioners, students and social science. the remainder of this paper is structured as follows: in section 2, we provide an overview of the legallens tasks. section 3 describes our data collection process, while section 4 presents the systems and results. section 5 delves into the details of the three winning teams, and section 6 offers an overview of the current research landscape. ## what is legallens to efficiently detect legal violations across various domains in the online digital data, a system must be developed that can scan large datasets, isolate relevant information, and accurately map it to appropriate legal grounds. this involves scanning large amount of online data, contextualizing the findings by linking them to specific legal grounds, and clearly explaining potential violations. additionally, the system must identify the affected individuals or entities who may be entitled to legal recourse, thereby enabling effective enforcement and remediation. legallens is designed to address these challenges by providing a structured approach to detecting legal violations in digital data. it achieves this by breaking down the task into two key components: legallens-ner to identify relevant legal entities and legallens-nli to determine the relationship between data points and legal grounds. in the following section, we will delve deeper into each of these sub-tasks, explaining how they con-tribute to the overall goal of efficiently detecting and contextualizing legal violations. for the full description of both tasks please refer to the original legallens work @xcite . ## dataset curation for legallens shared task building upon the original legallens dataset and addressing its limitations, we have created a more comprehensive and challenging benchmark for legallens sub-tasks. our goal was to create a dataset that not only mimics real-world scenarios but also presents a challenging benchmark for state-of-the-art nlp models in the legal domain. the resulting dataset for the shared task maintains the dual-task structure of the original legallens, focusing on ner for violation identification and nli for matching violations with known cases. with improved prompt practices, better annotators guidelines, human expert practices, and following feedback from the original paper @xcite , we improve the gen-eration process and the resulting annotations, yielding more realistic content, improve data quality and reduce bias. our enhancement process consisted of three primary steps: the first was to clean the dataset from duplicated or almost duplicates examples. in some cases we have found that similar patterns appears in the dataset too often that is not making sense. we have tried to detect these patterns and exclude instances that repeat them. also, to prevent potential biases and ensure broader applicability, we masked all company names within the dataset, including defendants and plaintiff names in the nli dataset. we found that models were prone to overfitting if this masking process was not applied. additionally, we have implemented an improved three-step validation where legal experts conducted a multi-stage validation process, including a review for factual accuracy and legal relevance, crossvalidation of ner annotations, and examination of premise-hypothesis pairs for logical consistency, completeness and correctness in the nli task. all annotation conducted via argilla @xcite available under an apache-2.0 license. ## system descriptions and performance the competition was hosted on codabench 2 @xcite . during the evaluation phase, the leaderboard was hidden, meaning participants did not receive feedback on their submission scores until the phase concluded. each team was allowed one submission per sub-task. both sub-tasks were evaluated as in the original paper: the legallens-ner sub-task was assessed using the weighted f1 score, to account for class imbalance, with each class's f1 score weighted by the number of true instances. evaluation was conducted using the seqeval @xcite ) method, which requires exact matches between predicted and true entity spans-both the boundaries and the entity type must match precisely. we followed the ibo format (inside, beginning, outside), where a correct match requires both the boundaries and tags to be accurate. the legallens-nli sub-task used the standard macro f1 score. participants received the hidden test set only two days before the submission deadline, after submitting the source code of their best architecture. changes to the model were not permitted after the release of the hidden test set. during the evaluation phase, organizers verified that the predictions could be reproduced using the submitted source code. ## deeper analysis in this section, we describe the key methodologies and innovative techniques employed by the topperforming teams in the legallens shared task. ## related work in recent years, there has been increased interest at the intersection of nlp and the legal domain, with work spanning legal judgment prediction @xcite @xcite to information extraction @xcite to document analysis @xcite to text generation agarwal-etal-2022-extractive. in the information extraction field, specifically in named entity recognition (ner), @xcite have focused on evaluating data agreements for compliance with european privacy laws using nlp techniques. in another study, @xcite employed multi-task domain adaptation for ner within the legal domain, showing modest improvements in recall across romanian and german languages. the work by @xcite asked language models to detect legal entity types. additionally, ner has seen increased usage in the legal domain, including efforts to extract entities from court judgment documents in various jurisdictions @xcite . additionally, @xcite ) presented e-ner; an annotated named entity recognition corpus of legal text. however, these entity types, even in legal domain ner tasks, aren't specifically tailored for detecting legal violations and lack the complexity needed for this challenging task. despite these advancements, existing research typically focuses on a standard set of entity types such as plaintiff and defendant, with limited exploration of more diverse or nuanced entities relevant to legal violations. furthermore, these studies are limited in scope, often focusing on specific legal domains or industries. nli in the legal domain has gained significant attention in recent years. @xcite explored nli at the document level for contracts, while @xcite introduced lawngnli from us legal opinions. @xcite presented caseholdnli and a documentlevel nli model using optimal evidence selection. @xcite introduced a legal nli dataset for the validity assessment of legal will statements and @xcite evaluated the validity of legal will statements across states, using three inputs-statement, condition, and law-to classify the relationship as support, refute, or unrelated. despite the increased interest, @xcite is the first to introduce legal violation detection as a general nli task across multiple domains. prior work has focused on domain-specific use cases, such as privacy protection @xcite @xcite , but these models lack the versatility needed to address the broad spectrum of legal violations across different contexts. legallens was the first to establish a cross-domain approach for detecting legal violations. ## conclusion and future work the legallens shared task demonstrated the potential of leveraging nlp techniques to address the challenge of legal violation detection across diverse domains. despite the task's rapid timeline-less than two months from launch to completion-the significant participation of 87 individuals, organized into 38 teams, and the promising results underscore the community's interest and the relevance of this problem. we call on the broader research community, particularly those in interdisciplinary fields, to contribute resources, methodologies, and diverse perspectives. collecting and consolidating these perspectives will deepen our understanding of the complexities within this field. as we refine and build upon the legallens framework, we encourage diverse perspectives and innovative approaches that can address the challenges of this important task. collaboration across disciplines will be crucial in advancing the state of the art in this important area. the top models achieved a 0.416 f1 score in legallens-ner (microsoft/deberta-v3-base) and 0.853 f1 score in legallens-nli (phi3). however, a significant drop was observed in identifying the "violated by" and "violated on" entities, indicating room for improvement. this gap suggests the potential for integrating other information extraction techniques, even possibly from outside the legal domain. key questions remain unresolved: how will the techniques scale with larger language models and adapt to less-resourced languages? can we enhance the granularity of legal entity interactions, particularly in more implicit scenarios? additionally, how will these approaches generalize across broader legal domains and real-world applications? limitations a challenge of identifying cases of legal violation in the open web is information sparsity. in other words, these cases do not present themselves in entirety, and in one place. often times, the salient details of a case are spread across multiple sources on the web, and individually do not offer much insight into the case. it is only when these individual details are stitched together, do they afford themselves to a holistic understanding of the full story, and subsequent evaluation of the case.