Validate Your Authority: Benchmarking LLM s on Multi-Label Precedent Treatment Classification (2025)

Abstract: Automating the classification of negative treatment in legal precedent is a critical yet nuanced NLP task where misclassification carries significant risk. To address the shortcomings of standard accuracy, this paper introduces a more robust evaluation framework. We benchmark modern Large Language Models on a new, expert-annotated dataset of 239 real-world legal citations and propose a novel Average Severity Error metric to better measure the practical impact of classification errors. Our experiments reveal a performance split: Google’s Gemini 2.5 Flash achieved the highest accuracy on a high-level classification task (79.1%), while OpenAI’s GPT-5-mini was the top performer on the more complex fine-grained schema (67.7%). This work establishes a crucial baseline, provides a new context-rich dataset, and introduces an evaluation metric tailored to the demands of this complex legal reasoning task.

## introduction in common law jurisdictions, the doctrine of precedent, or stare decisis, is a cornerstone of the legal system @xcite . it compels judges to decide cases by referencing previous decisions with similar factual situations. this makes it imperative for legal practitioners to determine if a judicial decision is still considered "good law," or if its authority has been weakened or nullified by subsequent cases. a case can be "negatively treated" in various ways; for instance, it can be explicitly "overruled" by a higher court, its reasoning can be "criticized," or it can be "distinguished" as not applying to a new set of facts. for a lawyer, building an argument upon a case that is no longer good law can be a critical error, potentially leading to unfavorable judgments and professional repercussions-especially when the error is severe. to address this challenge, the legal industry has historically relied on commercial citator services. leading platforms like shepard's on lexisnexis @xcite , keycite on westlaw @xcite , and bcite on bloomberg law @xcite employ teams of legal editors to analyze how cases are treated in subsequent decisions. these services provide signals, such as color-coded flags, to quickly alert practitioners to potential negative treatment foot_0 . however, these services are not infallible. documented human error in manual citation analysis @xcite have spurred interest in automated solutions. the recent emergence of powerful large language models (llms) has significantly advanced the capabilities of legal natural language processing (nlp), offering the potential to handle the complex, nuanced reasoning inherent in legal texts. this paper builds upon this body of work by evaluating the performance of modern llms on the task of classifying case law citation treatments. the central research question we address is: how accurately can contemporary llms replicate the sophisticated legal reasoning required to classify the various ways a judicial precedent is treated in a subsequent case? early research in this area utilized rule-based approaches @xcite . later work, such as the lexa system by @xcite , demonstrated the feasibility of combining knowledge engineering with baseline machine learning models for this task. more recently, traditional machine learning models @xcite were used to classify how a cited case is treated. while previous studies have highlighted the intrinsic difficulty of this task for earlier neural network architectures, the capabilities of the latest generation of llms remain to be thoroughly benchmarked. this paper contributes to the field by presenting a systematic evaluation of these models on a challenging, multi-label classification task, using an expert-annotated dataset of real-world legal decisions to provide a new baseline for this critical task. ## related work in this section, we are reporting the related work to the research question that we are examining in this paper, in 3 category that surrounds the task that we have defined with our research question. ## experiment to evaluate the capabilities of modern llms on this task, we designed a systematic experimental setup focusing on zero-shot and few-shot learning paradigms. this approach was chosen to simulate a realistic use case where fine-tuning a model on a large, domain-specific dataset is often impractical due to data scarcity and computational cost. we constructed our few-shot prompts by randomly selecting three precedent treatment examples from our dataset. to ensure a rigorous evaluation and prevent data contamination, any example used for in-context learning was excluded from the test set. for all prompts, the context provided to the model was a curated snippet from the citing case, specifically the paragraph(s) where the cited case is analyzed. we did not use the full text of the legal document. this was done for brevity and to isolate the most relevant text for the classification task. ## conclusion in this paper, we conducted a comprehensive evaluation of modern llms for the task of negative precedent treatment classification. our findings show that leading proprietary models have significant potential for automating this crucial aspect of legal research. gemini 2.5 flash was the top performer on the high-level schema with 79.1% accuracy, while gpt-5-mini performed best on the more challenging fine-grained schema at 67.7% accuracy. despite this promise, overall performance remains constrained by the dataset's class imbalance and the complexity of legal reasoning. we identify two primary challenges: a scarcity of context-rich legal data and the task's inherent semantic complexity, which pushes the limits of current models. critically, we find standard accuracy metrics insufficient for this domain. our proposed average severity error provides a more robust evaluation of model reliability, essential for trustworthy legal ai. we establish a vital baseline and provide a new dataset to the community to advance research in this area. the dataset used in this work is publicly available on the hugging face hub at https://huggingface.co/datasets/ mmikaildemir/negative_treatment .