Augmenting Legal Decision Support Systems with LLM -based NLI for Analyzing Social Media Evidence (2024)

Abstract: This paper presents our system description and error analysis of our entry for NLLP 2024 shared task on Legal Natural Language Inference (L-NLI). The task required classifying these relationships as entailed, contradicted, or neutral, indicating any association between the review and the complaint. Our system emerged as the winning submission, significantly outperforming other entries with a substantial margin and demonstrating the effectiveness of our approach in legal text analysis. We provide a detailed analysis of the strengths and limitations of each model and approach tested, along with a thorough error analysis and suggestions for future improvements. This paper aims to contribute to the growing field of legal NLP by offering insights into advanced techniques for natural language inference in legal contexts, making it accessible to both experts and newcomers in the field.

## introduction in today's digital age, vast amounts of information circulate online, creating an overwhelming stream of text that spans news articles, social media, and user-generated content. within this unstructured data, legal violations often remain hidden, blending into the surrounding noise. legal violations frequently leave behind data traces. to identify these traces and detect violations, prior research in legal nli @xcite has typically utilized specialized models designed for particular domain applications @xcite . uncovering these violations is not only * equal contribution crucial for upholding individual rights and ethical standards, but also for maintaining societal justice in an increasingly digital world. addressing this challenge requires more than traditional methods. while existing models have proven effective within their specialized domains, they lack the flexibility needed to tackle the complex and varied nature of legal violations found in diverse online contexts. our work seeks to bridge this gap by leveraging advanced language models for the nuanced task of legal natural language inference (l-nli), as part of the nllp 2024 shared task. the aim was to classify relationships between legal complaints and reviews as either entailed, contradicted, or neutral. in this study, we implemented a range of techniques, including multi-layered fine-tuning and alignment strategies, to enhance text classification. we experimented with several llms, such as gemma (team, 2024), phi3 (abdin, 2024), zephyr @xcite , llama @xcite , openhermes @xcite and qwen @xcite refining each model for optimal performance. these approaches proved highly effective, with our system outperforming other entries by a large margin. beyond technical achievements, we present a thorough error analysis, highlighting where the models excelled / struggled. through our findings, we aim to advance the field of legal nlp, making complex legal analysis accessible to a wider audience, while pushing the boundaries of nli in legal domain. the code and models used in the official submission and the later found best model can be found here. 1 2 2 dataset the dataset for the nli task consists of a legal premise (a summary of resolved class-action cases) and a corresponding hypothesis (an online media text). the training and test splits of the dataset consist of 312 and 84 samples. for the initial finetuning, the test and validation subsets of the snli dataset @xcite were used consisting of 20000 samples. the distributions of each of the training sets and the test set can be seen in train-1 train-2 test entailed 34.0% 32.7% 47.6% neutral 33.1% 33.9% 34.5% contradict 32.9% 33.3% 17.9% ## error analysis we were able to completely avoid type-1 errors i.e classification of 'entailed' as 'contradict' and 2 and figure 3 . it can be observed from both figure 2 and figwe found these to be cases where the hypothesis consisted of multiple sentences which entail the premise followed by a vague / unrelated statement, while some are to be labelled as 'entailed' and rest as 'neutral' based on the perceived tone/feeling of the user, it would be likely that there might not be consensus among human annotators as well in many such cases. it is worth looking into the performance of models trained on not just the labels, but also the reasoning of the annotators on why a certain label was chosen, as it might help the model learn better. ## scope for improvement as seen in ## conclusion compared to the well known snli dataset which consist of premise and hypothesis pair which are usually one or two sentences long, the current dataset has texts (both premise and hypothesis) which are roughly four times longer leading to more complexity. since, the snli dataset has a 98% consensus and 58% unanimous annotation among 5 annotators, it can be expected that a human annotation on the current dataset can lead to even less proportion of texts where a consensus or unanimous vote can be reached. yet, our models were able to provide a reliable performance completely avoiding type-1 errors, performing better than human annotations expected from those with domain knowledge, hinting at a potential of practical applicability.