Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification (2024)

Abstract: In legal decisions, split votes (SV) occur when judges cannot reach a unanimous decision, posing a difficulty for lawyers who must navigate diverse legal arguments and opinions. In high-stakes domains, %as human-AI interaction systems become increasingly important, understanding the alignment of perceived difficulty between humans and AI systems is crucial to build trust. However, existing NLP calibration methods focus on a classifier’s awareness of predictive performance, measured against the human majority class, overlooking inherent human label variation (HLV). This paper explores split votes as naturally observable human disagreement and value pluralism. We collect judges’ vote distributions from the European Court of Human Rights (ECHR), and present SV-ECHR, a case outcome classification (COC) dataset with SV information. We build a taxonomy of disagreement with SV-specific subcategories. We further assess the alignment of perceived difficulty between models and humans, as well as confidence- and human-calibration of COC models. We observe limited alignment with the judge vote distribution. To our knowledge, this is the first systematic exploration of calibration to human judgements in legal NLP. Our study underscores the necessity for further research on measuring and enhancing model calibration considering HLV in legal decision tasks.

## introduction the task of case outcome classification (coc) involves the classification of legal case outcomes based on textual descriptions of their facts. while achieving high performance is desirable, in highstakes domains such as legal and medical decisions, the quantification of a model's predictive confidence, or conversely, its uncertainty, is particularly valuable. it allows experts to make more informed decisions, especially when the model may be uncertain or where the consequences of a misdiagnosis are significant. evaluating whether models are aware of their limitations is known as assessing their uncertainty, with popular methods such as calibration @xcite . calibration assesses the extent to which predictive probabilities accurately reflect the likelihood of a prediction being correct. models can opt to abstain when the uncertainty exceeds a predefined threshold -a method commonly referred to as selective classification @xcite . current nlp research focuses on prediction confidence and calibration to assess a classifier's awareness of its predictive performance only. this evaluation is commonly conducted against the human majority class. however, recent developments in nlp research have shed light on the prevalence of inherent human label variation (hlv) @xcite , observing disagreement * across various tasks @xcite . scholars in the field argue for the acknowledgment and acceptance of hlv, as it mirrors the diverse and pluralistic nature of human values @xcite . notably, @xcite has demonstrated that widely-used calibration metrics may not be suitable for datasets characterized by inherent human disagreement. in light of these findings, we posit that models should not only be calibrated to recognize their own uncertainty (performance calibration) but also be equipped to discern instances where humans exhibit uncertainty (human calibration) @xcite . this dual focus aims to foster trust among end-users and mitigates potential harm caused by models. consequently, a critical aspect of this trust involves ensuring the alignment of perceived difficulty between human and models. this motivates us to study split votes (sv) in court decisions (fig 1 ). the judge vote ratio is a naturally occurring human disagreement at the european court of human rights (ecthr). we present sv-echr, a coc dataset with judge split vote information. we study the disagreement sources among judges from a legal, linguistic, and nlp perspective. we adapt and analyse the taskagnostic taxonomy components of @xcite and introduce sv-specific subcategories. we also quantitatively assess the effects of different subcategories on judges' agreement using proxy variables. the results suggest that disagreements are mainly due to the social-political context of cases. in addition, we assess the alignment of perceived difficulty between coc models and humans, as well as confidence-and human-calibration. while we see acceptable performance in perceived difficulty and confidence calibration, our analysis indicates suboptimal human calibration. this underscores the necessity for a more in-depth inquiry into methods to better align models' calibration with human behaviour, highlighting opportunities for further research in this direction. disagreement / human label variation is receiving growing attention in mainstream nlp. various works highlight the presence of hlv, emphasizing the abundance and plausibility of such human disagreements @xcite . researchers advocate for embracing hlv for more trustworthy ai @xcite . as real-world applications are used to assist diverse audiences, it becomes crucial to investigate and include pluralistic human values in nlp systems @xcite . this motivates our study of disagreements among judges' split-votes in the legal decision process. recently, several studies have proposed task-specific taxonomies to identify potential sources of disagreement in various nlp tasks @xcite @xcite . building on the meta-analysis of these existing taxonomies, @xcite generalize them to two layers of task-agnostic categories and introduce task-specific categories for coc rationale annotation. in this work, we study judges' disagreements in case decision votes, presenting a taxonomy of disagreement with sv-specific subcategories. perceived difficulty regarding model difficulty, there is increasing interest in identifying difficult data instances. various techniques, such as influence functions (koh and liang, 2017) and training loss @xcite have been proposed to identify the difficulty of data instances to a certain model. pointwise v-usable information (pvi) is a recently introduced difficulty metric @xcite , which incorporates mutual information and other types of informativeness @xcite . despite its recent introduction, pvi has gained significant attention and proven effective in various tasks, including rationale evaluation @xcite and data selection for augmentation @xcite . in this work, we leverage pvi to evaluate the alignment of perceived difficulty between human and coc models. calibration recently in mainstream nlp, researchers posit that the overall reliability of a model is determined by two sides: 1) trustworthiness, which is addressed through models' confidence measuring, and 2) fairness, which is addressed through their confidence alignment with humans @xcite . existing calibration studies primarily focus on a classifier's confidence of its predictive performance, commonly evaluated against the human majority class @xcite . @xcite instead argue that calibrating against the human majority class is not meaningful in settings with inherent hlv. in our work, we assess the performance and human calibration of coc models in the context of splitvote, a naturally occurring hlv. to the best of our knowledge, this is the first systematic exploration of human calibration in the legal domain. uncertainty evaluation within the deep learning community, uncertainty is often classified into two types: aleatoric and epistemic uncertainty. epistemic (or model) uncertainty arises from a lack of knowledge about the best model, often exacerbated by out-of-distribution examples. aleatoric uncertainty, on the other hand, stems from inherent ambiguity and can be considered as the variability in experiment outcomes @xcite . as sv is due to inherent disagreement among judges, we regard it as aleatoric uncertainty. aleatoric uncertainty is typically quantified through metrics such as entropy @xcite and softmax response @xcite , as recognized in prior works @xcite . however, studies show that these methods are based on predictive entropy, and actually measure total uncertainty, combining both epistemic and aleatoric uncertainty (van amersfoort et al., 2020). only when we have prior knowledge that either aleatoric or epistemic uncertainty is low, can we use predictive entropy as a suitable measure for the other type @xcite . given our small, label-imbalanced, and temporally shifted dataset, we choose to refrain from utilizing these methods to directly measure the aleatoric uncertainty of our cases. instead, we opt to assess difficulty using pvi, which measures predictive entropy and can be considered an equivalent for total uncertainty. ## legal and linguistic backgrounds the legal decision making process at the ec-thr begins with the applicants lodging their accusation, alleging one or more violations of articles of the european convention on human rights (echr). this process aligns with task b (allegation prediction) in the widely used lexglue benchmark @xcite . after receiving the complaint, the court undertakes a review of the case, aiming to determine whether a violation has indeed occurred, corresponding to task a (violation prediction) in lexglue. cases falling under the purview of well-established ecthr caselaw are directed to a three-judge committee, while others may find themselves before a seven-judge chamber, where decisions are reached through a majority vote. in some circumstances, the grand chamber, comprising 17 judges, adjudicates cases referred to it by request. when rendering a judgment, the court typically examines only the specific articles alleged by the applicant. this procedure aligns with task a|b (violation prediction given allegation; @xcite . our study evaluates the uncertainty of coc models for task a|b, mirroring the real legal process. ## the sv-echr dataset dataset collection we extract the judge votes distribution for each alleged article * from the public database hudoc * using regular expressions. the information about judge votes is always present in the conclusion section and generally follows certain patterns (see more details in app a). we did a two-round quality assessment. * the f1 score of our regular expression rules increased from 0.81 in the first round to 0.98 in the second round. ## conclusion we present sv-echr, a new coc dataset enriched with naturally occurring split-votes by judges on the ecthr. we also present a svenriched taxonomy of disagreement sources. our experiments reveal shortcomings when combining ts and ece to improve and measure calibration of models against labels subject to inherent human disagreement. this is due to a distribution mismatch reflected in low distce scores. soft loss training produces only slightly better human calibration scores. we call on the community to explore methods for measuring and improving the alignment of model calibration with human behavior; as well as more research into incorporating hlv in nlp.