D e BERT a Beats Behemoths: A Comparative Analysis of Fine-Tuning, Prompting, and PEFT Approaches on L egal L ens NER (2024)

Abstract: This paper summarizes the participation of our team (Flawless Lawgic) in the legal named entity recognition (L-NER) task at LegalLens 2024: Detecting Legal Violations. Given possible unstructured texts (e.g., online media texts), we aim to identify legal violations by extracting legal entities such as “violation”, “violation by”, “violation on”, and “law”. This system-description paper discusses our approaches to address the task, empirically highlighting the performances of fine-tuning models from the Transformers family (e.g., RoBERTa and DeBERTa) against open-sourced LLMs (e.g., Llama, Mistral) with different tuning settings (e.g., LoRA, Supervised Fine-Tuning (SFT) and prompting strategies). Our best results, with a weighted F1 of 0.705 on the test set, show a 30 percentage points increase in F1 compared to the baseline and rank 2 on the leaderboard, leaving a marginal gap of only 0.4 percentage points lower than the top solution. Our solutions are available at github.com/honghanhh/lner.

## introduction the internet has revolutionized how we share and interact with information. every day, we generate an enormous quantity of digital textual data in the form of news articles, blogs, and social media posts. the information we consume and produce, not to mention the platforms we interact on contain a multitude of legal claims, and violations are no exceptions. it is undeniable that these violations pose potential risks to individuals and organizations as well as go against the fabric of legal and ethical standards, including individual rights, societal norms, and the principles of justice. previous studies often trace the legal violations from their data trails by using specialized models tailored for specific domain applications @xcite . while these models can be effective in their narrow domains, they often lack the necessary versatility to address the wide array of legal violations across contexts. to address this, @xcite formulate a new task of automatically identifying legal violations from unstructured text sources in the form of legal named entity recognition (l-ner). while baseline methods have been created to address this task, there remains a gap in developing more advanced methods to sort through this online noise and identify these breaches. inspired by the work of @xcite on legallens consisting of a novel textual dataset for legal violation identification using large-scale language models (llms), we address a comparative analysis of different approaches on this dataset through the legallens 2024: detecting legal violations shared task @xcite . the contributions of this paper are two-fold: the paper is organized as below. section 2 summarizes the previous works for the l-ner task. section 3 describes the architecture, dataset, and evaluation metrics for the comparative analysis. in section 4, we report the performances of our methods on the development set. we also compare our best classifier on the development set with the test set against the baseline. finally, we propose error analysis in section 5, followed by the conclusion with future works in section 6. ## related works the primary works for legal violation identification were mostly on domain-specific topics such as data agreements for compliance @xcite , data privacy breaches @xcite , and industry-specific regulations @xcite . despite their potential, these studies suffered from the limitation of specific types of legal domains or particular sectors. one of the most popular directions for legal violation identification was to consider the task as a named entity recognition @xcite @xcite task, or so-called l-ner. in non-neural approaches, dozier et al. (2010) extracted the named entities (nes) in the us case law and many other legal documents by implementing list lookups, contextual rules, and statistical models. in neural ones, leitner et al. (2019) suggested a bilstm-crf model for their novel manually annotated datasets about german court decisions with 19 nes while others proposed lstm-crf for lener-br foot_0 legal documents in brazilian. chalkidis et al. (2020) presented legal-bert foot_1 with different bert-based model fine-tuned on 12 gb of english legal texts. further works @xcite elaborated the neural architecture for legal identification via ner task by convolutional neural networks (cnn) and multi-layer perceptions (mlp). several other language models (e.g., bert, distilbert, roberta) were also fine-tuned to enhance the performance of legal violation identification @xcite in the same legallens foot_2 corpora. with the advent of large-scale language models (llms), numerous works have been done to take advantage of llms to [1] explain legal terms present in legislative documents @xcite analyze the legal textual data (e.g., court decision analysis, rivalling seasoned law students) in depth @xcite , [3] generate synthetic data in legal domains @xcite , or [4] fine-tune a specialized classifier (e.g., llama-2) for the downstream task @xcite , to mention a few. ## methods in this section, we explore three different setups to tackle the challenge of the l-ner task, including: @xcite we evaluate transformers variants (e.g., roberta @xcite , deberta @xcite , and deberta-lstm) through the process of fine-tuning; [2] we explore prompting llms in zero-shot settings @xcite with different fine-tuned checkpoints (e.g., mistral @xcite , llama-2 @xcite , llama-3.1 (dubey et al., 2024)); and [3] we perform parameter-efficient fine-tuning (peft) using low-rank adaptation (lora) with llms. ## results roberta, deberta); [2] prompting (e.g., mistral, llama); and [3] lora (e.g., qwen2, mistral, llama). in general, all the fine-tuned bert-based language models outperform llms for both lora and instruction-tuning settings by a significant margin. across all models, deberta attains the best performances, achieving an f1 of 64.8% and a precision of 63.3% on the development set. given the best performance on the development set of deberta as a fine-tuned token classifier, we reported the results in weighted f1 of our classifier on the hidden test set in comparison with other competitors and the baseline from the legallens 2024: detecting legal violations task in ## error analysis entity type errors: figure 1 visualizes the comparison in f1 performance for each class among different models reported in additionally, the three datasets exhibit significant variability, as illustrated by the distinct class coverage of models in figure 3 in appendix b, which provides insights into the data distribution. this variability may explain why models trained on the training set may not generalize well to the development and test sets. moreover, analyzing the named entities present in each slot and examining how various models perform about these, could yield additional valuable findings. the disparity in performance: although deberta outperformed other masked language models of smaller size (e.g., roberta), a larger model size does not always lead to better performance, especially when lora fine-tuning is used, which can sometimes lead to poorer results. this is consistent with the results of @xcite , which highlighted the difficulties in fine-tuning the llms compared to the smaller masked language models (e.g., bert), especially when the amount of training data is limited. furthermore, we acknowledged the difference in objective functions between deberta as a fine-tuned token classifier and other llms (e.g., llama-3.1) as a sft-lora classifier. while deberta employed the per-token cross-entropy objective function, llms fine-tuned via causal language modelling, wherein the task is to learn the joint probability distribution of all tokens by maximizing the likelihood of the data. as a result, deberta provided a more fine-grained and stronger gradient signal that well constrained the class space by the number of possible entities in our dataset. this highlights the gap between masked and casual language models in token classification tasks for specific domains like l-ner. additionally, as shown in the findings of @xcite , ls lora provided significant improvement over sft-lora. however, there is still room for improvement when compared to deberta. practical use of llms for legal domain: despite not surpassing the performance of fine-tuned and lora methods, prompt-based methods are still a promising tool for finding the potential violation for legal documents, especially when working with limited data of the same domain or when no annotated data is available for a given domain. while it may not be as good as models trained on dedicated annotated data (fully supervised ones), it can significantly speed up the process by suggesting the violation types later reviewed and refined by human experts. additionally, tools like jsonformer, which enforce structured output constraints, can help significantly in automating these tasks. by ensuring that model outputs conform to predefined formats (e.g., json), these tools simplify post-processing workflows, making the outputs easier to analyze and validate using non-llm methods, as structured formats facilitate clearer interpretation and error-checking mechanisms @xcite . in-domain fine-tuning: we evaluated the performance of fine-tuned deberta checkpoints on several ner datasets relevant to this task foot_6 . surprisingly, no significant improvement was observed compared to the base deberta model. however, based on our analysis of the zero-shot performance capabilities of llms (see figure 3 ), there appears to be greater overlap between the dataset styles of the training set and the hidden test set than between the training and development sets. this suggests that having better distributions of train-dev-test splits can help with improving upon this task. additionally, domain-specific fine-tuning where similar patterns are reflected could also potentially enhance the performance of llms, although further experimentation is required to validate this hypothesis. therefore, future work could explore fine-tuning an llm on a legal domain corpus, which may yield better results for this and similar tasks @xcite . ## conclusion in this study, we presented a comparative analysis of three different approaches to identify the legal violations via the l-ner task at legallens 2024: detecting legal violations, including [1] fine-tuning masked language models as token classifier; [2] zero-shot prompt engineering with llms; [3] finetuning llms with lora as token classifier. overall, the first approach using deberta as the backbone outperformed other settings, demonstrating the gap in performance between masked language models and other causal llms in token classification tasks, especially when the amount of training data is limited. as a result, when a complete training dataset is accessible, opting for a fully-supervised finetuned system remains the optimal choice. however, instruction-tuning llms with well-defined prompting is still a potential technique with competitive results when no annotated data is available.