Deconfounding Legal Judgment Prediction for E uropean Court of Human Rights Cases Towards Better Alignment with Experts (2022)

Abstract: This work demonstrates that Legal Judgement Prediction systems without expert-informed adjustments can be vulnerable to shallow, distracting surface signals that arise from corpus construction, case distribution, and confounding factors. To mitigate this, we use domain expertise to strategically identify statistically predictive but legally irrelevant information. We adopt adversarial training to prevent the system from relying on it. We evaluate our deconfounded models by employing interpretability techniques and comparing to expert annotations. Quantitative experiments and qualitative analysis show that our deconfounded model consistently aligns better with expert rationales than baselines trained for prediction only. We further contribute a set of reference expert annotations to the validation and testing partitions of an existing benchmark dataset of European Court of Human Rights cases.

## introduction the task of legal judgment prediction (ljp) has recently gained increasing attention in the legal and mainstream nlp communities @xcite @xcite @xcite . legal cases are resolved through the exchange of arguments in front of a decision body by lawyers who represent litigating parties. this typically involves evidential reasoning, the determination of relevant rules from sources of law (e.g., codes, regulations, precedent), their application to the case, and the balancing of legal and societal values. in the nlp context, ljp takes the form of classifying the outcome of a case from some textual representation of its specific facts, effectively skipping legal reasoning. this forms a counterpoint to knowledge-focused approaches to outcome prediction (e.g., brüninghaus and ashley, 2005; branting, 2013; grabmair, * these authors contributed equally to this work 2017) that connect to a lawyer's understanding of the domain but also require substantial knowledge engineering. this carries particular risk in the legal domain, where systems may rely on data elements that are statistically predictive but legally irrelevant, or even forbidden as decision criteria (e.g., the race of an accused person). this can lead to undesirable consequences, ranging from suboptimal litigation strategy decisions, flawed inference about factors predictive for the outcome, to disparate impact of decisions across groups that are to be treated equally. if legal decisions are to be informed by predictive systems processing textual case descriptions, then such systems must strive to be as closely aligned with legally relevant and permissible parts of the input as possible. in this work, we focus on ljp for the european court of human rights (ecthr), which adjudicates complaints by individuals against states about alleged violations of their rights as enshrined in the european convention of human rights. we trained deep neural models on four tasks across two existing, related datasets @xcite around predicting such violations alleged by the claimant and decided by the court. we find that the models substantially base their predictions on aspects of the text that correlate with the outcome but either have no legal bearing or are forbidden nationality-related information that stem from the distribution of cases arising at the court. to improve the alignment of model focus with legal expert understanding, we apply a series of deconfounding measures, including a vocabularybased method which identifies predictive tokens using a simple model. the third author, who is an ecthr expert, then identifies distractors among them. the distracting signal can subsequently be removed from the encodings via adversarial training. this procedure is an effective way of engaging with domain experts and obtaining information about what the model should be steered away from by means of deconfounding, rather than trying to attract the model towards relevant elements via expensive data collection for supervised training. for simplicity, throughout this paper, we use 'deconfounding' in an inclusive sense as the mitigation of distracting effects of (a) confounders in the statistical sense that influence both the dependent and independent variables, (b) reverse causation relationships, and (c) other attributes that spuriously correlate with the target variable. see fig. 1 for an overview of our experiment design. we evaluate our trained and deconfounded models with regard to an alignment of its explanation rationales with (1) a dataset of expert passage relevance assessments we collected and will make available to community as a supplement to @xcite , and (2) on expert relevance assessments published as part of @xcite . our results show that our deconfounding steps succeed in improving the model focus alignment with expert-identified, relevant patterns on both sets of reference annotations. in sum, we make the following contributions: ## related work ljp as an nlp task has been tackled using ngram representations (e.g., @xcite , word embeddings and domain models @xcite , and deep neural networks (e.g., @xcite @xcite . special attention must be given to the origin of the text from which the prediction is to be made. @xcite recharacterize ljp on texts produced before the outcome is known as 'forecasting' and observes that most current works 'classify' judgments based on the data compiled after the outcome has been determined. they also find that forecasting is a harder task. this result is consistent with our finding of confounding effects from text production by the ecthr, resulting in a prediction from fact descriptions that were influenced by the decision. moverover, the relationship between the information ljp models rely on and legal expert analysis of texts remains underexplored. @xcite find that transformer-based models exploit spurious correlations and that simple models, such as xgboost, can achieve similar performance. @xcite extract model rationales for alleged violation prediction and observes limited overlap with expert markup. similarly, a small study in @xcite finds that users do not perceive case prediction-derived highlighting as useful in making predictions themselves. our work contributes to this state of the art by using adversarial deconfounding to improve the overlap between what systems predict from with what legal experts consider relevant. deconfounding a growing number of works have raised awareness that deep neural models may exploit spurious statistical patterns and take erroneous shortcuts @xcite @xcite . a common method of mitigating this is adversarial learning. @xcite use a gradient reversal layer @xcite to deconfound lexicons in text classification. other domains that adopt adversarial training to eliminate confounders include bioinformatics @xcite and political science @xcite . many existing works on identifying shortcuts focus on situations where these patterns are known in advance and may require potentially expensive data collection. in fairness-focused legal nlp, @xcite observe and remedy group disparities in ljp performance on the ec-thr informed by metadata attributes (respondent state, applicant gender, applicant age). we extend this to explainability in ljp by involving a legal expert in a procedure that allows an efficient, incremental identification of distracting information, as well as its removal via adversarial training. interpretability we employ interpretability techniques to evaluate model alignment with expert rationales. @xcite reviews and categorizes the main current interpretability methods. though initial works @xcite used attention scores as explanation for model decisions, @xcite ; @xcite point out that saliency methods, such as gradient based methods @xcite , propagation based methods @xcite , occlusion based methods @xcite , and surrogate model based methods @xcite are better suited for explainability analysis. however, the reliability and informativeness of these methods remains an open research problem. our model uses the currently most commonly used integrated gradients (ig) @xcite , which computes the gradient of the model's output with respect to its input features. ## ecthr tasks & datasets the ecthr has been the subject of substantial prior work in ljp. we use two datasets for model training and evaluation: first, for binary violation we use the dataset by @xcite of approx. 11k case fact statements, where the target is to predict whether the court has found at least one convention article to be violated. to evaluate alignment, we annotate 50 (25 each) expert rationales for cases from both the development and test partitions (see app. c for the annotation process). second, for article-specific violation, we use the lexglue dataset by @xcite , which consists of 11k case fact statements along with information about which convention articles have been alleged to be violated, and which the court has found to be violated. for alignment, we merge this data with the 50 test set rationales from @xcite . while both datasets stem from the ec-thr's public database, they differ in case facts and outcome distribution as we explain in sec. 3.1. the input texts consist of each case's facts section extracted from ecthr judgments. this section is drafted by court staff over the course of the case proceedings. while it does not contain the outcome explicitly, it is not finalized before the final decision has been determined, potentially creating confounding effects. we conduct experiments on four ljp tasks: task j -binary violation for our task j, the model is given a fact statement and is asked to predict whether or not any article of the convention has been violated. we train our models on @xcite and evaluate alignment on the set of expert rationales we collected. task b -article allegation we train and evaluate on lexglue's ecthr b, * where the fact description is the basis to predict the set of convention articles that the claimant alleges to have been violated. it can be conceptualized as topic classification in that the system needs to identify suitable candidate articles (e.g., the right to respect for private and family life) from fact statements (e.g., about government surveillance). we test alignment on the expert rationales by @xcite . ## expert-informed deconfounding we apply an expert-informed deconfounding method designed to mitigate the distracting effects of confounding elements and spurious correlations. as @xcite observe, accounting for confounders is common practice throughout many data analysis tasks to capture the intended signal and facilitate explainable models. in ljp, we understand confounding elements as such that influence both the observed legal outcome (convention violations found by the court as coded in the dataset) and the input text from which this outcome is to be predicted (here: ecthr fact statements). already covered examples are the different distribution of information across sections for admissible and inadmissible cases and the length of the fact descriptions (inadmissible cases tend to require less factual information to be dismissed). * an example of a spurious correlation is the identity of the respondent state (certain article violations will be claimed more often against a small number of governments, leading to a correlation). they each should have no bearing on the probability of an outcome in a given case as a judge will not decide against a violation because the facts are short, or because the case is against a particular government. confounding effects and spurious information in ljp may not be known ahead of time, especially if the legal decision is not made on the basis of an immutable a priori document, but rather on the basis of text that is technically a part of the eventual judgment. our expert-informed method is intended to mitigate such situations where spurious correlations are introduced in the text production but may not be known in advance as explicit confounders. our method consists of two steps: (i) identification of distracting attributes for deconfounding through a combination of simple model training and minimal expert markup, and (ii) mitigation of these effects through adversarial training. ## experiments & discussion in this section we describe our experiments in using our proposed deconfounding methodology to improve the alignment of model focus on the input with expert rationales on our set of ljp tasks. ## conclusion our results show that our deconfounded ljp models are consistently better aligned with expert rationales than a baseline optimized for the target label only, and in many cases can even achieve better prediction performance. however, the improvement is small and the paragraphs focused on by all our models are still quite different from what an expert has annotated as relevant, as indicated by generally low precision@oracle scores (<50%). still, our quantitative results show that expert-informed deconfounding ljp works in principle and can potentially go a long way to train more robust and trustworthy neural ljp models, as well as derive more useful legal insight from them.