L egal L ens: Leveraging LLM s for Legal Violation Identification in Unstructured Text (2024)

Abstract: In this study, we focus on two main tasks, the first for detecting legal violations within unstructured textual data, and the second for associating these violations with potentially affected individuals. We constructed two datasets using Large Language Models (LLMs) which were subsequently validated by domain expert annotators. Both tasks were designed specifically for the context of class-action cases. The experimental design incorporated fine-tuning models from the BERT family and open-source LLMs, and conducting few-shot experiments using closed-source LLMs. Our results, with an F1-score of 62.69% (violation identification) and 81.02% (associating victims), show that our datasets and setups can be used for both tasks. Finally, we publicly release the datasets and the code used for the experiments in order to advance further research in the area of legal natural language processing (NLP).

## introduction the widespread use of the internet has changed how information moves and connects in our society. every day, the digital domain is flooded with a multitude of textual data, spanning from news articles and reviews to social media posts 1 . within this sea of unstructured text, legal violations can often go unnoticed, concealed by the vast amount of surrounding information. these violations not only pose potential harm to individuals and entities but also challenge the very fabric of legal and ethical standards in the digital era. the significance of addressing these hidden violations cannot be overstated; as they have widespread implications for individual rights, societal norms, and the principles of justice. as a result, there is a pressing need to develop sophisticated methods to sift through the noise and identify these breaches. legal violations often leave data trails. to detect these trails for pinpointing the violations, previous studies have often relied on specialized models tailored for specific domain applications @xcite . these models, while effective in their specific domains, lack the versatility needed to address the wide array of legal violations that can occur across different contexts. legal violation identification aims to automatically uncover legal violations from unstructured text sources and assign potential victims to these violations. we designed two setups, one for each task, the first for solving the legal violation identification task (a.k.a identification setup) using named entity recognition (ner), and the other for associating these violations with potentially affected individuals (a.k.a resolution setup) using natural language inference (nli). our dataset for the ner task is not limited to any specific domain, while the nli dataset is focused on four common legal domains. followed by recent research in the field of data generation @xcite @xcite , we chose to employ @xcite for synthetic data generation due to his ability to produce a large, diverse, and high-quality dataset that closely mimics the syntactic complexity of legal language, offering a scalable and ethically sound alternative to manual data crafting. we employed a thorough verification process to validate the data for both its realistic and complexity. our approach involved automated data generation based on real-world event contexts in the english language, complemented by manual reviews conducted by seasoned legal annotators on the generated data. ## curating custom legal datasets: a multi-stage approach to ner and nli tasks existing datasets may not adequately address the diverse range of legal violations and contexts central to our study, which is not in specific areas. to overcome these challenges, we employed a systematic and carefully planned data generation process, consisting of three stages: prompting, labeling, and data validation. this approach aimed at creating two robust datasets for two nlp tasks in the legal domain. we chose to focus on two key tasks: this dual-setup approach was designed to mimic the process of legal violation detection and resolution, generating high-quality data that closely resembles real-world scenarios. based on recent research in prompt-based methods @xcite , our study employs prompts for a variety of reasons. llms have been shown to adapt to specialized tasks through techniques like instruction tuning @xcite , reinforcement learning from human feedback @xcite , and in-context learning @xcite when prompted with natural language instructions. prompts facilitate task-specific optimization, a quality emphasized by dialogprompt @xcite , which aligns with our focus on ner and nli in the legal domain by fine-tuning on the generated dataset. additionally, the sensitivity of prompts in context, as demonstrated in time-aware prompts in text generation @xcite , is crucial for understanding specific legal contexts like resolved class-action cases. as a result, our methodology leverages a prompt-based approach, optimized for the legal domain, to generate high-quality data for ner and nli tasks. ## human expert annotations data validation holds particular importance in our study due to the synthetic nature of the dataset. to ensure that the dataset is both realistic and challenging, we have implemented several validation methods. in this structured process, summaries of complaint documents and tasks for the ner and nli models were generated automatically. legal experts then carefully examined these auto-generated summaries and tasks. their primary role was to meticulously review each output, ensuring that the summaries accurately reflected the key points of the complaints and that the tasks were correctly aligned with the context provided by these summaries. additionally, each record was subjected to examination by several annotators, which serves to reduce potential bias in the evaluation. these annotators were tasked with identifying and suggesting any missing entities, as well as in checking for hallucinations-instances where the generated content might stray from factual accuracy. to maintain a rigorous and unbiased validation, all annotators received identical instructions, and the data presented to them was systematically shuffled. their detailed examination was crucial in pinpointing discrepancies, unclear areas, or potential inaccuracies in both the summaries and the associated tasks. this thorough validation process, attentive to both content accuracy and the prevention of hallucinations and bias through multiple annotators review, ensures the integrity and quality of our synthetic dataset. figure 4 in appendix b presents a screenshot of the annotation platform we used. upon further examination of our data, a comparison between machine-generated and humanauthored content revealed significant similarities. this comparison involved analyzing various linguistic and structural features of the texts. both displayed identical average sentence lengths. moreover, there was not significant difference between the character count between the generated content and the human-authored text. additionally, when comparing the pos tags between the real text and the generated text, by averaging the total counts of each tag occurrences, the average difference was found to be 26% and the median was 16%. a key part of our validation process was the classification task. in this task, three independent annotators had to distinguish between machinegenerated and human-written records, a challenge also noted in recent research @xcite . our annotators' goal was to label each record based on its origin: machine-generated or human-written. the annotators achieved an average f1-score of 44.86%. however, their cohen's kappa scores, which were 0.0821, 0.2149, and 0.0988, showed only minor agreement among them. this low level of agreement, as indicated by cohen's kappa scores, points out the complexity of the task. it also suggests that our machine-generated content closely resembled human writing, making it difficult even for experts to tell them apart. the use of cohen's kappa in our study is supported by its well-known effectiveness in binary classification tasks, especially in data annotation scenarios @xcite . ## experiments in this section, we explore several methods to tackle the challenging and realistic setups that we created. more precisely, we analyzed the performance of language models on these setups by conducting three sets of experiments. (1) we evaluated models that are inspired by the bert architecture through the process of fine-tuning @xcite . ## error analysis to improve our models and enrich our understanding, we conducted a thorough error analysis of top-performing models across tasks. this analysis identifies their limitations, providing a clear roadmap for future refinements.