Bonafide at L egal L ens 2024 Shared Task: Using Lightweight D e BERT a Based Encoder For Legal Violation Detection and Resolution (2024)

Abstract: In this work, we present two systems—Named Entity Resolution (NER) and Natural Language Inference (NLI)—for detecting legal violations within unstructured textual data and for associating these violations with potentially affected individuals, respectively. Both these systems are lightweight DeBERTa based encoders that outperform the LLM baselines. The proposed NER system achieved an F1 score of 60.01% on Subtask A of the LegalLens challenge, which focuses on identifying violations. The proposed NLI system achieved an F1 score of 84.73% on Subtask B of the LegalLens challenge, which focuses on resolving these violations by matching them with pre-existing legal complaints of class action cases. Our NER system ranked sixth and NLI system ranked fifth on the LegalLens leaderboard. We release the trained models and inference scripts.

## introduction social networks and other online platforms are increasingly becoming effective tools to address consumer complaints; however, the vast amount of unstructured textual data makes it challenging to identify valid complaints and if they are associated with any legal violations. there is a pressing need to develop sophisticated methods to identify these hidden breaches, as they have significant implications for individual rights and legal obligations, if any. in this regard, @xcite propose two subtasks -subtask a, legallens ner (named entity recognition), to detect legal violations mentioned in the text and subtask b, legallens nli (natural language inference), to match the detected violations with resolved class action cases. to address these subtasks in this paper, we propose ner and nli models based on training deber-tav3 encoders. we finetune task-specific encoders on our synthetically augmented dataset. in summary, we list our findings here: in section 2, we examine the related works on ner and nli tasks. section 3 provides an overview of the methodologies employed for the tasks. in section 4, we describe the experimental setup. section 5 discusses the results and findings; section 6 discusses the conclusions. nli nli classifies the logical relationship between a premise (a given statement) and a hypothesis (a proposed conclusion) as entailed, contradictory, or neutral. early work on nli focused on rule-based systems and logical inference @xcite . the advent of large-scale datasets, such as the snli @xcite , multinli corpus @xcite , xnli @xcite enabled the development of sophisticated models. transformer-based models such as roberta @xcite and xlnet @xcite have pushed the limits of nli performance by giving human-like scores. nli is a critical task in nlp that serves as a benchmark for natural language understanding. although significant progress has been made, challenges remain in developing systems that can perform robust and generalizable inference across diverse domains and languages. in this work, we use tasksource's nli model and finetune it on the legallens nli dataset. tasksource is a framework that harmonizes data sets for multitask learning and evaluation in nlp by providing a collection of pre-processing methods @xcite . ## methodology in this section, we introduce our approach for each of the subtasks. ## experimental settings ner we finetune the gliner models on the legallens ner dataset using a dropout rate of 0.5 and a batch size of 8. we employ adamw optimizer with a base learning rate of 1e-5 for pretrained layers (the transformer backbone, deber-tav3) and 5e-5 for non-pre-trained layers (ffn layers, span representation). the model is trained to a maximum of 10 epochs, starting with a 10% warm-up phase, followed by a decay phase using a linear scheduler. we save the best checkpoint and, subsequently, reduce the learning rate to 5e-6, and train this checkpoint until convergence. to address class imbalance, we use focal loss, instead of cross-entropy loss, with alpha 0.75 and gamma 2. we experiment with three different architectures proposed by @xcite and knowledgator engineering foot_1 -original gliner, the biencoder and the poly-encoder as shown in nli we train four models and test them on each legal domain. each of these four models is trained on three domains at once and tested on the fourth to prevent data leakage as described by @xcite . for each domain, we finetune tasksource's nli model using a learning rate of 2e-5, a sequence length of 256, and a batch size of 8 for a maximum of 7 epochs using a cosine scheduler. we our experiments reveal that shuffling entity order and randomly dropping entities did not affect the metrics. after identifying the best model, we trained it on a synthetic dataset generated using few-shot learning. however, this approach did not yield any improvement in results. we then applied rule-based entity filtering, which improved the development dataset results by 0.5%, increasing the final f1 score from 59.53% to 60.01%. the system ranked sixth on the leaderboad with an f1 score of 33.00% on the test dataset @xcite . although our model results did not surpass the baselines @xcite , further exploration of medium and large variants of gliner could be beneficial. due to limited computational resources, we were unable to include them in our experiments. nli for each legal type category, we employ four distinct models. during the evaluation process on the unlabeled test set, we consider the collective assessment of all four models. the final label for a premise-hypothesis pair is determined by the model exhibiting the highest confidence score among the four. our findings indicate that data augmentation proved beneficial, albeit to a certain extent. when we expanded the dataset to triple its original size by incorporating an additional set of paraphrases, we observed that the corresponding increase in f1 scores was not proportional to the increase in data volume. this suggests that there may be diminishing returns in terms of performance improvement beyond a certain threshold of data augmentation. we compare our results with the baseline proposed by @xcite . they finetune falcon 7b @xcite and report an f1 score of 81.02% compared to 84.73% for our model. the system ranked fifth on the leaderboad with an f1 score of 65.30% on the test dataset @xcite . in the error analysis of the final model, we see that both the models are capable of handling first class errors-confusions between contradict and entailed. however, our model does better with handling second-class errors-misclassification of contradict or entailed as neutral; and falcon 7b model does better with handling another class of errors-misclassification of neutral as contradict or entailed. the confusion matrix for our model is shown in figure 1 . it is interesting to note that a multitask deberta based encoder surpassed the performance of a 7b parameter by 3.17%. our model is capable of resolving the ambiguities and complexities related to wage norms. finally, it can be stated that paraphrasing can serve as a data augmentation technique to enhance the natural language understanding capabilities of smaller models. ## conclusion in conclusion, we present two systems developed for the legallens 2024 shared task, comprising a zero-shot bidirectional deberta encoder with domain-adaptive pretraining for the ner subtask and a multitask deberta encoder enhanced by data augmentation techniques for the nli subtask. the experiments demonstrate that synthetic data generation can enrich datasets and improve the performance of encoder-based models. however, it is evident that more data does not necessarily translate to better performance. by optimizing on smaller but richer datasets and employing suitable training objectives, smaller models can outperform larger language models. further exploration of different augmentation strategies, with a particular focus on generating more contextually diverse synthetic data, employing adversarial data, or leveraging domain-specific paraphrasing techniques, may yield performance improvements for ner tasks. while rule-based filtering improved the f1 score by 0.5%, the adoption of more sophisticated post-processing strategies, such as probabilistic methods or ensemble techniques, holds the potential to further enhance the results. finally, the proposed systems secured the sixth and fifth ranks in the legallens ner and legal-lens nli tasks, respectively, demonstrating their competitiveness in the shared task.