The Craft of Selective Prediction: Towards Reliable Case Outcome Classification - An Empirical Study on E uropean Court of Human Rights Cases (2024)

Abstract: In high-stakes decision-making tasks within legal NLP, such as Case Outcome Classification (COC), quantifying a model’s predictive confidence is crucial. Confidence estimation enables humans to make more informed decisions, particularly when the model’s certainty is low, or where the consequences of a mistake are significant. However, most existing COC works prioritize high task performance over model reliability. This paper conducts an empirical investigation into how various design choices—including pre-training corpus, confidence estimator and fine-tuning loss—affect the reliability of COC models within the framework of selective prediction. Our experiments on the multi-label COC task, focusing on European Court of Human Rights (ECtHR) cases, highlight the importance of a diverse yet domain-specific pre-training corpus for better calibration. Additionally, we demonstrate that larger models tend to exhibit overconfidence, Monte Carlo dropout methods produce reliable confidence estimates, and confident error regularization effectively mitigates overconfidence. To our knowledge, this is the first systematic exploration of selective prediction in legal NLP. Our findings underscore the need for further research on enhancing confidence measurement and improving the trustworthiness of models in the legal domain.

## introduction the task of case outcome classification (coc) involves categorizing the outcome of a legal case based on the text of case facts. it has garnered substantial attention not only within the legal but also in the nlp community @xcite @xcite . it is important to acknowledge that these tasks commonly labeled as 'legal judgment prediction' are, in reality, instances of retrospective classification rather than prediction as the fact statements obtained from judgment documents are often not finalized until the decision outcome is known, as emphasized by @xcite , introducing the potential confounding artifacts in data @xcite . the main utility of this task and data lies in understanding the capabilities of nlp models to analyze fact statements for extracting and learning text patterns corresponding to specific convention articles, as drafted by the court. identifying potentially violated human rights provisions from a textual fact description is a task that human experts can do well and that requires substantial domain knowledge along with textual understanding phenomenon @xcite . correlation between text patterns and violations has been a subject of interest in empirical research, particularly in political science @xcite @xcite . though coc has witnessed improvements in performance with use of better pre-trained models @xcite @xcite or innovative modelling strategies with better architectures and loss functions @xcite @xcite or incorporation of legal knowledge @xcite , the accuracy of these models is not guaranteed for all instances. therefore, it is crucial to assess the reliability of model predictions, particularly in high-stakes decision-making scenarios-an aspect that has not received adequate attention in the community. in this paper, we explore the reliability of coc systems using selective prediction setting @xcite . in this setting, the objective is to reduce the error rate by abstaining from predictions when the model is uncertain, while maintaining high coverage. in essence, we consider a model reliable if it possesses the self-awareness capability to acknowledge when it doesn't know enabling it to defer to humans for manual inspection, thereby ensuring its trustworthiness @xcite . under the selective prediction setting, we construct a selective classifier by combining a standard classifier with a confidence estimator. the confidence estimator gauges the model's confidence for a given input instance and based on this confidence, the selective classifier decides whether to abstain from predicting on uncertain cases. an ideal confidence estimator should provide higher confidence for correctly classified examples compared to incorrect ones. in this study, we conduct an empirical investigation on the coc task, focusing on european court of human rights (ecthr) cases which adjudicates complaints by individuals against states regarding alleged violations of their rights as enshrined in the european convention of human rights. our goal is to assess four design choices to obtain more reliable coc models: (i) how does the choice of pre-trained models, such as general bert @xcite or legal domain-specific models like legalbert @xcite or lexlm @xcite , impact reliability? does the size of the pre-trained model, such as base or large, play a role? (ii) is there a universally effective confidence estimator, such as softmax response @xcite or monte carlo dropout based methods @xcite ? (iii) how do additional training loss constraints in the form of regularizers @xcite or learning directly with abstention as an option @xcite affect it? we evaluate these design choices on three coc task variants with varying difficulty, ranging from predicting violations alleged by the claimant to violations decided by the court. additionally, we examine the impact of these choices on different buckets of articles based on frequency of cases with corresponding article violations. based on our empirical exploration, we observe: (i) domain-specific pre-training enhances model calibration, but exclusive focus on downstream task-specific pre-training corpus is detrimental. a domain-related yet diverse corpus is crucial for effective pre-training. larger models, while more accurate, tend to be overconfident. (ii) computationally expensive monte carlo dropout methods provide superior confidence estimates. (iii) adding confident error regularization @xcite improves model calibration. to encourage future work towards better uncertainty quantification in coc task, we release our code, including pipelines to evaluate design choices based on selective prediction and classification performance. while the majority of existing research in coc focuses on improving predictive performance, there is an increasing emphasis on the reliability of models within legal nlp. this includes perspectives on explainability @xcite @xcite and fairness @xcite @xcite . recently, mainstream nlp has seen researchers, such as @xcite pose that the overall reliability of a model is determined by two facets: 1) fairness, which is related to the alignment of model confidence with human expectations, and 2) trustworthiness, which involves accurate confidence measurement by the model. @xcite study models' calibration in alignment with human behavior using split-vote cases from ecthr, pioneering on research on model reliability from a fairness perspective in legal nlp. in contrast, our work systematically explores selective prediction in coc task. to the best of our knowledge, this is the first coc research on model reliability from a trustworthiness perspective in legal nlp. ## ecthr tasks & datasets we chose to work with the ecthr corpus because of its publicly available dataset with detailed articlespecific allegations and violation information, leading to multi-label classification setting, in contrast to the simplified binary classification setting in corpora from other jurisdictions @xcite . we experiment with the following three coc task variants. following valvoda et al. 2023, we use the 14 articles which form the core rights of the convention. task b: allegation identification @xcite we utilize data from lexglue ecthr b @xcite , where the fact description serves as input to identify the set of convention articles that the claimant alleges to have been violated. task a: violation identification @xcite we leverage data from lexglue ecthr a to predict which of the convention's articles has been deemed violated by the court using the facts description as input. task a is more challenging than b as it involves the identification of suitable articles along with prediction of their violations. task a|b: violation identification given allegation information @xcite this involves the identification of violations from the case facts along with allegedly violated articles as the input. this task mirrors the realistic legal process, as the court is aware of the allegations made by the applicants when determining the violations. this task is easier compared to task a, as the first substep of task a (i.e., identifying suitable articles) is provided directly as input in this variant. dataset splits & metrics for prediction performance lexglue consists of of 11k case fact descriptions chronologically split into training (2001-2016, 9k cases), validation (2016-2017, 1k cases), and test sets (2017-2019, 1k cases). following chalkidis et al. 2022a, we report macro-f1 (m-f1) scores for all tasks across the 14 articles. ## selective prediction a standard classifier learns a function f : @xmath0, takes input x and maps it to set of labels y . we pair a standard classifier with a selection function g : @xmath1, 1} to obtain a selective classifier @xmath2, g); h : @xmath3, ⊥ is a special label indicating the abstention of prediction. given an input x, the selective classifier outputs as follows: the selective classifier yields an output from f when the selection function predicts that prediction should be given, or abstains if the selection function predicts that it should not predict. convenient way to formulate the selection function g is relying on a confidence function g and a threshold γ ∈ r as: a lower rpp value indicates better estimator. however, gu and hopkins 2023 highlights that rpp and auc values are influenced by both the prediction and confidence estimation functions. they propose refinement metric, normalizing with the worst-case kendall-tau distance, offering a calibrated interpretable metric where 0, 0.5 and 1 signifies the best, random and the worst case respectively. ## experiments following chalkidis et al. 2022a, we employ a hierarchical extension of a pre-trained transformer model to account for longer input texts as our base model. we use corresponding pre-trained model to encode each paragraph in the input independently to obtain [cls] representation for each paragraph. we then pass these paragraph representations to a transformer layer to learn contextual information from other paragraphs. finally, we max pool over these context-aware paragraph representations to obtain final representation of case facts which is sent to classification layer. in case of task a|b, we concatenate a multi-hot feature vector containing the task b labels to the final representation before passing it to the classifier as in santosh et al. 2022a. detailed hyperparameters can be found in app. a. to assess the reliability of coc models using diverse pre-trained language models as backbone, we employ (i) bert-base (kenton and toutanova, 2019), trained on general english corpora (ii) incaselawbert-base @xcite , pretrained on indian supreme court and high courts case documents by initializing with caselawbert @xcite which is pre-trained on us case law from federal and state courts. (iii) legalbert-base @xcite incorporates pre-training on eu, uk, and cases from the us, european court of justice, and ecthr. (iv) lexlm-base @xcite , pre-trained on lexfiles, a diverse legal corpus spanning us, uk, eu, india, and canada jurisdictions. (v) lexlmlarge @xcite , a larger version pretrained on the same lexfiles corpus. among them, bert lacks specific legal pre-training, while incaselawbert has access to legal corpus but not from the ecthr jurisdiction, relevant to our coc task. legalbert and lexlm have access to ecthr corpus in pre-training, but they differ in the proportion of ecthr corpus to other corpora, where lexlm has less ecthr proportion compared to legalbert's pre-training corpora. we use these 5 pre-trained models as backbone in base model and fine-tune on coc task with 4 training loss functions (i.e task-specific loss, cer, ece, gambler in sec. 4.2). we employ 4 variants (sr, smp, pv, bald in sec. 4.1) to derive the confidence estimate and thus compute the selective prediction metrics (aurcc, rpp, refinement) and classification metric (macro-f1) across these 80 (5*4*4) configurations for each of the three tasks. ## conclusion we introduce the problem of selective prediction for coc task, aiming to enhance model reliability by abstaining in cases of low confidence. through empirical investigation on 3 coc task variants with 5 pre-trained models, 4 confidence estimators, and 4 loss functions, we assess how these design factors contribute to better selective prediction in coc. our findings reveal that legal domain-specific pretrained models outperform in classification-related metrics and are well-calibrated, but an exclusive focus on a specific corpus can prove detrimental. larger models tend to be overconfident compared to their base versions. despite being computationally intense, mc dropout methods provide superior confidence estimates. adding cer regularization helps alleviate overconfidence. we hope this preliminary investigation spurs additional research into the selective prediction of coc models, em-phasizing the critical need for models to be aware of what they don't know in high-stakes domains like legal coc.