L egal L ens 2024 Shared Task: Masala-chai Submission (2024)

Abstract: In this paper, we present the masala-chai team’s participation in the LegalLens 2024 shared task and detail our approach to predicting legal entities and performing natural language inference (NLI) in the legal domain. We experimented with various transformer-based models, including BERT, RoBERTa, Llama 3.1, and GPT-4o. Our results show that state-of-the-art models like GPT-4o underperformed in NER and NLI tasks, even when using advanced techniques such as bootstrapping and prompt optimization. The best performance in NER (accuracy: 0.806, F1 macro: 0.701) was achieved with a fine-tuned RoBERTa model, while the highest NLI results (accuracy: 0.825, F1 macro: 0.833) came from a fine-tuned Llama 3.1 8B model. Notably, RoBERTa, despite having significantly fewer parameters than Llama 3.1 8B, delivered comparable results. We discuss key findings and insights from our experiments and provide our results and code for reproducibility and further analysis at https://github.com/rosequ/masala-chai

## introduction information extraction tasks, such as named entity recognition (ner) have been predominantly limited to identifying common entities such as person, location, and organization. as an extension, previous studies using benchmark @xcite , have achieved high metrics for effectiveness. for example, finetuned bert base model achieved an f1 macro of 92.4 on @xcite . however, applying ner to specialized domains, such as legal and medical texts, presents challenges due to their complex terminology, domain-specific language, and limited availability of annotated training data. the legallens 2024 shared task aims to push the research in the areas of legal nlp by inviting participants to work on two tasks: legal named entity * equal contribution recognition (l-ner), and legal natural language inference (l-nli) @xcite . the first subtask involves identifying violation indicators by extracting legal entities such as law, violation, violated by, and violated on. similarly, the motivation behind legal natural language inference is to understand the relationship between a pair of legal texts (hypothesis and premise) as contradiction, entailment, and neutral. in this paper, we present our teammasala-chai's-submission to the legallens shared task. we explore the performance of various transformer models, both open-source and commercial, on ner and nli tasks in the legal domain. while we suggest enhancing performance with dspy and textgrad, the results still fall short compared to fine-tuning smaller models like roberta. our experiments revealed that while models like gpt-4o struggled with legal tasks, even when using advanced techniques like prompt optimization, smaller models like roberta performed competitively, achieving an f1 macro score of 0.701 for ner and 0.833 for nli. this highlights that finetuning smaller, more efficient models can deliver results comparable to larger language models. we present our findings, discuss the nuances of using llms, and share our code to support reproducibility and further exploration. the label set y for our task includes the entity types law (legal statutes or regulations), violation (specific violations), violated by (responsible entities), violated on (victim or affected party), as well as a non-entity label o (outside any named entity). ## system description our approaches for the named entity recognition (ner) and natural language inference (nli) tasks involved i) fine-tuning pre-trained language models, and ii) utilizing prompt engineering techniques. we experimented with pre-trained transformer models, including bert (both uncased and cased versions) @xcite , distil-bert (sanh, 2019), roberta @xcite , flan t5 (chung et al., 2024), llama 3.1 @xcite , and @xcite . each model was fine-tuned on the train split. ## discussion while finetuning gpt-4o for ner, we noticed the primary reason gpt-4o achieves inferior performance is due to i) its difficulty tagging long sequences of text, and ii) hallucination of entities. we notice that for a given sequence of tokens, once the model predicts entities such as b-violation, it goes on to predict i-violation entities until the end of the sequence. as for the hallucinatory nature of llms, we notice that there are entities that are outside the label set y are being predicted (entities such as b-l-i are hallucinated). observing the loss plots corresponding to gpt-4o finetuning from figure 1 also shows that there tends to be high variance during finetuning. the llms also fail to output a sequence consis-tently of bio tags that are the same length as the input sequence of tokens. this presents challenges for evaluating performance, as the truth label sequences are supposed to be of equivalent length to the input sequence. while running inference with gpt-4o configurations, we noticed a mismatch in the length of the list of input tokens and output tags in both validation and test sets. specifically, we saw a mismatch of 92.9% samples in the test set when running inference with gpt-4o raw, 92.1% of samples in the test set when running inference with a finetuned gpt-4o model, and 91.8% of samples in the test set when running inference with gpt-4o + dspy, making gpt-4o unsuitable for legal ner. in the case of nli, while llama 3.1 delivers the best performance, roberta comes close, despite having only 1.5% of llama 3.1's 8b parameters. we anticipate that further research into smaller language models, through methods like continued pretraining, could eventually achieve performance parity if not superior results-a direction we reserve for future exploration. additionally, methods for fine-tuning llama 3.1 8b including full fine-tuning with a larger legal corpus can also be explored to for further improvements. ## conclusion in this report, we outlined our approach to tackling named entity recognition (ner) and natural language inference (nli) tasks in the legal domain as part of the legallens shared task. our experiments highlighted the strengths of fine-tuning transformerbased models such as roberta and llama 3.1, particularly for handling complex legal text. despite the strong performance of these models, especially roberta for ner, we observed limitations in commercially available large language models like gpt-4o, which struggled with sequence length mismatches and hallucinations during ner tasks. additionally, while llama 3.1 achieved the best nli results, roberta demonstrated competitive performance despite having significantly fewer parameters. this suggests that smaller models, when fine-tuned effectively, can rival much larger models in legal nlp tasks. our results indicate that there is still room for improvement in entity extraction and reasoning in the legal domain.