Semantists at L egal L ens-2024: Data-efficient Training of LLM ’s for Legal Violation Identification (2024)

Abstract: In this paper, we describe our system for LegalLens-2024 Shared Task on automatically identifying legal violations from unstructured text sources. We participate in Subtask B, called Legal Natural Language Inference (L-NLI), that aims to predict the relationship between a given premise summarizing a class action complaint and a hypothesis from an online media text, indicating any association between the review and the complaint. This task is challenging as it provides only limited labelled data. In our work, we adopt LLM based methods and explore various data-efficient learning approaches for maximizing performance. In the end, our best model employed an ensemble of LLM’s fine-tuned on the task-specific data, and achieved a Macro F1 score of 78.5% on test data, and ranked 2nd among all teams submissions.

## introduction legal violation identification is an important problem that aims to automatically uncover legal violations from unstructured text sources and assign potential victims to these violations. in the past, several works have addressed this but often relied on specialized models tailored for specific domain applications @xcite . these models, while effective in their specific domains, lack the versatility needed to address the wide array of legal violations that can occur across different contexts. the legallens shared task @xcite proposes to address the legal violation identification task using named entity recognition (ner), and the other for associating these violations with potentially affected individuals using natural language inference (nli). in this paper, we report our system for addressing the nli task @xcite . broadly, our approach is to adopt llm based methods and explore various data-efficient learning approaches for maximizing performance on the nli task. our best model employed an ensemble of llm's fine-tuned on the task-specific data. our final system achieved a macro f1 score of 78.5 and ranked 2nd among all teams submissions. surprisingly, the classical falcon llm's outperformed many other sota llm's. also, our benchmark results highlight the challenge of these tasks and indicate there is ample room for model improvement. we demonstrate the limitation of the general llm based methods and discuss possible future work. ## task and dataset description the legallens challenge @xcite proposes two shared sub-tasks: participants can choose either of the two sub-task or both. we participate in sub-task b, defined as below. ## our approach natural language inference (nli) is the task of detecting inferential relationships between a premise text and a hypothesis text @xcite @xcite , which is considered fundamental in natural language understanding (nlu) research @xcite . in l-nli task, the premise is a summary of a class action complaint and the hypothesis an online media text, and the objective is to determine if the relationship is 'entailed', 'contradicted', or 'neutral'. several nli systems have been proposed in the literature @xcite , and can be adapted for the l-nli task. @xcite investigated this by finetuning popular small language models, such as bert and roberta, and reported that the models struggled with the task. also, using their legal counterparts, like legal-bert @xcite , legal-roberta @xcite , and legal-english-roberta @xcite models also did not lead to much improvements. this can be attributed to the small data, as most of the models typically assume sufficiently large number of labelled data. this is particularly true for nli which is essentially a 3-way sentence pair classification problem. in comparison, llms are reported to learn relatively better in low data situations and generalize well to out-of-distribution (ood) test data sets @xcite . this is in part due to their pre-training on variety of datasets, eg. snli and mnli, as supported by the preliminary results of @xcite using fine-tuned falcon @xcite and llama @xcite models. in our work, we perform a more extensive study by considering more llm's and explore various llm based strategies and techniques, beyond prompt engineering, for maximizing performance in the given task. ## experiments & results in the l-nli challenge dataset, there are 312 instances distributed across 4 legal domains (see the experiments were executed on nvidia-geforce rtx 2080 series with eight cores of gpu machines with 8*12 gb of memory for all our experiments. also, to train t5 large models, we have used nvidia-geforce tesla v100 series sxm2-32gb with 5 cores of gpu machines. models were trained for 3-5 hours for training and reasoning. the pretrained weights for the transformers prior to fine-tuning were from the huggingface nlp library. ## discussion and conclusion this paper described our system for legallens-2024 shared task that aims to automatically uncover legal violations from unstructured text sources and assign potential victims to these violations. we participate in subtask b, called legal natural language inference (l-nli), that aims to predict the relationship between a given premise summarizing a class action complaint and a hypothesis from an online media text, indicating any association between the review and the complaint. this task is challenging in view of the limited labelled data, and hence we explored various approaches for data-efficient learning with llm's, such as peft fine-tuning, data augmentation and llm ensembling. in the end, our ensemble approach performed the best and achieved a macro f1 score of 78.5%, and ranked 2nd among all teams submissions. the key findings are: in summary, our research highlight the challenge of legal violation identification in real-life, and the limitations of sota llm's. this further suggests that there is ample room for model improvement and scope for possible future work, especially under limited data settings.