Enhancing Legal Violation Identification with LLM s and Deep Learning Techniques: Achievements in the L egal L ens 2024 Competition (2024)

Abstract: LegalLens is a competition organized to encourage advancements in automatically detecting legal violations. This paper presents our solutions for two tasks Legal Named Entity Recognition (L-NER) and Legal Natural Language Inference (L-NLI). Our approach involves fine-tuning BERT-based models, designing methods based on data characteristics, and a novel prompting template for data augmentation using LLMs. As a result, we secured first place in L-NER and third place in L-NLI among thirty-six participants. We also perform error analysis to provide valuable insights and pave the way for future enhancements in legal NLP. Our implementation is available at https://github.com/lxbach10012004/legal-lens/tree/main

## introduction a violation of law refers to the actions of breaking rules or regulations set by the legal system and authority. these violations harm individuals, organizations, and the principles of fairness and justice, particularly in the digital age. therefore, developing intelligent systems to detect violations and assist legal experts is essential. thanks to the exploration of advanced techniques in nlp, prior studies developed specialized models to address the problems of detecting violations automatically @xcite @xcite . this year, legallens @xcite is first held with the aim of detecting and monitoring violations in various domains including commercial, privacy, environmental law, and consumer protection. the competition contains two tasks: violation detection via named entity recognition (l-ner) and predicting potential victims of the violation using natural language inference (l-nli). the l-ner task requires a model to determine four types of entities @xcite given a passage. the l-nli task identifies whether the relationship between a complaint (premise) and a review (hypothesis) is entailed, contradicted, or neutral. the paper reports the work of nowj team in both tasks. for the first task, l-ner, independent classification is limited because there are strong dependencies in the output sequence (e.g. b-law cannot follow i-law, details in section 3.1). therefore, we address the problem by following sequence labeling with an architecture of bert and conditional random field (crf) to compute output probability jointly. regarding the second task, one of the main challenges is the lack of a high-quality labeled dataset, whereas general nli data has been highly developed on large datasets. thus, we propose a novel prompt for data augmentation using recent llms to overcome the shortage of labeled data. state-of-the-art language models are then fine-tuned on augmented training data to develop consistent models for the legal domain. the following sections of the paper are organized as follows: section 2 presents prior studies addressing named entity recognition and natural language inference tasks, especially in the legal domain. we describe details of our methodology for two tasks in sections 3 and 4. section 5 concludes the paper and points out some future work. legal natural language inference: nli, also known as textual entailment recognition has gained interest from researchers in recent years. there are a few law-related resources in nli, including contractnli @xcite , lawngnli @xcite , legalnli @xcite , and an annual competition coliee @xcite . however, the cost of constructing high-quality datasets in the legal domain is expensive due to expert-effort requirements in data annotation. thus, prior studies @xcite focused on data augmentation to overcome the limited dataset. @xcite proposed a data augmentation process based on logical structures of original statutory articles to enrich the training set automatically. recently, llms have shown their state-of-the-art in various nlp tasks, including legal nlp. @xcite , the winner of the legal statute entailment task in coliee 2024, leveraged the powers of llms for data augmentation and explore the hidden relations between the premise and hypothesis. particularly, they summarized the legal article (premise) as complementary information and experimented with various prompting techniques on flant5-xxl, an open-source model. ## conclusion this paper presents our work in the legallens competition. for the l-ner task, we leverage the contextual embeddings of bert-based models and compute sequence dependency using a linear-chain crf layer. for the l-nli tasks, we propose a novel prompt to generate synthesis data using llms. the experiments highlight the effectiveness of data augmentation in improving language models' performance. consequently, we secured first place in l-ner and third place in l-nli. we also perform error analysis to offer valuable insights and groundwork for future advancements in legal nlp. future work would focus on improving the robustness and performance of models by exploiting the integration of logical knowledge and llms.