Zero-shot Transfer of Article-aware Legal Outcome Classification for E uropean Court of Human Rights Cases (2023)

Abstract: In this paper, we cast Legal Judgment Prediction on European Court of Human Rights cases into an article-aware classification task, where the case outcome is classified from a combined input of case facts and convention articles. This configuration facilitates the model learning some legal reasoning ability in mapping article text to specific case fact text. It also provides an opportunity to evaluate the model’s ability to generalize to zero-shot settings when asked to classify the case outcome with respect to articles not seen during training. We devise zero-shot experiments and apply domain adaptation methods based on domain discrimination and Wasserstein distance. Our results demonstrate that the article-aware architecture outperforms straightforward fact classification. We also find that domain adaptation methods improve zero-shot transfer performance, with article relatedness and encoder pre-training influencing the effect.

## introduction legal judgment prediction (ljp) has recently gained considerable attention in the mainstream nlp community (e.g., @xcite @xcite @xcite . in ljp, the outcome of a case should be classified/predicted based on a textual description of case facts. in actual legal reasoning, legal practitioners (e.g., advocates, judges) determine relevant rules from the sources of law (e.g., statutes, regulations, precedent) that are relevant to the case at hand. they then carry out an analysis to determine which rules apply to the case at hand, and deduce the outcome of the case by applying them. subsuming case facts under elements of rules given in legal sources plays a critical role in this process. many current ljp approaches (e.g., @xcite @xcite tackle this as a straightforward classification problem with the textual descriptions of case fact as the sole input. this reliance on the model learning statistical correspondences from case fact descriptions directly to outcomes neglects the role of legal sources in this relationship. as a consequence, the model may learn sub-optimal fact-outcome patterns that are informed by the case distribution in the data rather than learning to align facts with the legal source text containing applicable rules. the models may also attend to outcome-correlating distractors present in the dataset rather than engage in the legal fact-vs-law reasoning that is required of legal practitioners for a proper justification of the outcome @xcite . this work seeks to remedy this incomplete inference and enable the model to learn more authentic reasoning between rules and case facts by casting ljp into an article-aware classification setting and subjecting it to a zero-shot transfer challenge. article-aware classification has been explored on chinese criminal case corpora @xcite @xcite . similarly, @xcite has modeled statutory reasoning by classifying us tax law provisions concatenated with textual case descriptions. we build on this prior work in two ways. first, we develop and evaluate our model on a public dataset @xcite of cases by the european court of human rights (ecthr), which hears complaints by individuals about possible infringements of their rights enshrined in the european convention on human rights (echr) by states. to the best of our knowledge, this is the first work applying article-aware case outcome prediction setting to human rights adjudication. our approach pairs case fact descriptions with candidate echr articles and assigns a binary target label depending on whether the article has been alleged/deemed to have been violated, or not. our results show that the article-aware classification model outperforms the traditional classification setup by a small but consistent margin. second, we subject the model to a zero-shot transfer task. models trained on case facts alone cannot produce inferences about convention articles they did not observe during training. by contrast, human judges can conduct outcome analysis with new/amended legal provisions because they are trained to understand the rules they contain and apply them to case facts in an expertise-informed way, even in the absence of secondary sources (e.g., commentaries to the rule, etc.). article-aware classification allows an emulation of this process by means of a zero-shot benchmarking task on articles unseen at training time. we compare two conditions where (1) the model either has no access to the target articles, or (2) it is allowed to 'read' the target articles but is not given any prediction outcome labels for case-target article pairs. we experiment with domain adaptation by means of a domain discriminator @xcite and wasserstein distance @xcite . our results show that this improves performance on unseen articles compared to a vanilla model. we study the impact of law-specific pre-trained encoders on this zero-shot transferability compared to the standard language pre-trained one. intuitively, we observe that our models perform better in zeroshot transfer if the target/unseen articles are semantically related to articles seen at training time. it should be noted that, despite these tasks being typically referred to as instances of 'legal judgment prediction', ecthr fact statements are typically not finalized until the decision outcome is known, making the task effectively one of retrospective classification rather than prediction @xcite . while this does lead to distracting and confounding phenomena (see our prior work in @xcite , the dataset remains a useful resource for the development of nlp models that analyze these fact statements for text patterns that correspond to specific convention articles as drafted by the court. consequently, in this paper we hence speak of our models as engaging in case outcome classification (coc). our main contributions in this paper are 1 : legal judgement prediction: ljp/coc as an nlp task has been studied using corpora from different jurisdictions, such as the ecthr @xcite @xcite @xcite @xcite @xcite chinese criminal courts @xcite @xcite @xcite , us supreme court @xcite , indian supreme court @xcite the french court of cassation @xcite , brazilian courts (lage-freitas et al., 2022), the federal supreme court of switzerland @xcite , uk courts @xcite and german courts @xcite early works @xcite @xcite @xcite used bag-of-words features. more recent approaches use deep learning @xcite @xcite . large pre-trained transformer models have since become the dominant model family in coc/ljp @xcite , including legal-domain specific pre-trained variants @xcite that have been employed for the benchmark ecthr corpus we use in this paper @xcite . prior work on chinese criminal case corpora case extends fact-based classification by providing the text of legal source articles as additional input. @xcite used an attention-based neural network which jointly models charge prediction and relevant article extraction in a unified framework whose input includes the text of legal articles. sim-ilarly, @xcite @xcite employ matching mechanism between case facts and article texts. to the best of our knowledge, ours is the first work to adapt articleaware prediction to the ecthr corpus, which is situated in the in human rights litigation domain. going beyond previous works, we further benchmark the zero-shot transfer performance of such models, providing a test bed to evaluate their capability to process article texts they have not seen during training time and applying them to case facts towards classifying allegations/outcomes. ## dataset, tasks & settings we use the lexglue ecthr dataset provided by @xcite , which consists of 11k case fact descriptions along with target label information about which convention articles have been alleged to be violated (task b), and which the court has eventually found to have been violated (task a). the dataset is chronologically split into training @xcite , and test set (2017-2019) with 9k, 1k, and 1k cases, respectively. the label set includes 10 prominent echr articles, which forms a subset of all the rights contained in the convention and its protocols. in both the ecthr a and b benchmarks, it is assumed that the model classified the target from the fact description alone, which we refer to as the fact classification variant. for our article-aware classification settings, we augment the dataset with the texts of the 10 articles copied from the publicly available echr convention document foot_0 . we formulate the article-aware prediction variant for both tasks: given both the case fact statements and a particular article information, the model should classify the binary outcome of whether an article has been alleged to be violated by the claimant (task b) or found to have been violated by the court (task a). our zero-shot transfer task then involves determining violation/allegation from case facts with respect to articles which are not seen during training time. we consider a 'domain' to be a particular convention article (i.e., 10 convention articles form 10 domains). the objective is to train a model on a source domain (seen articles) with the goal of performing well at test-time on a target domain (unseen articles). following @xcite , we propose two settings under zero-shot coc: zero-shot restrictive / unsupervised domain adaptation (uda): in this setting, the model is given a pair of case facts and the text of training set articles (i.e., the source domain) along with their corresponding violation/allegation outcome label. in the target domain, it is provided with case facts and article text pairs as well, but the outcome label is withheld. the goal of uda is to learn an outcome classifier from the outcome labelled source domain which should generalize well on the target domain by leveraging outcome-unlabeled target data. this setting is legally realistic, as the text of new or modified written legal sources is typically known for a given task and available for domain adaptation (e.g., a public administration decision support tool receives an update after relevant legislation has changed). ## method we employ a hierarchical neural model which takes the case fact description x along with the article a as input and outputs a binary outcome (allegation in task b and violation in task a) for case x with respect to article a. our architecture is a modified version of the enhanced sequential inference model (esim) @xcite incorporating conditional encoding @xcite that has been adapted to deal with long input sequences following hierarchical attention networks @xcite . we experiment with two domain adaptation components based on adversarial training: (1) a classificationbased domain discriminator and (2) a wassersteindistance based method which aims to reduce the difference between the source and the target domain distributions. ## conclusion we cast case outcome classification on ecthr data into an article-aware architecture. this configuration is inspired by realistic legal reasoning involving both the case facts and convention articles to determine possible allegations/violations. assuming non-finetuned pre-trained encoders, we observe a performance improvement over a simple fact-only classification model. it also enables us to conduct experiments in zero shot transfer coc with and without access to unlabeled target data during domain adaptation. while we show that domain adaptation techniques are in principle suitable to facilitate generalization, the divergence between source and target domain performance is large and this task variant is very difficult. we further observe that the effectiveness of domain adaptation interacts with law-specific pre-training of transformer-based encoders and with the relatedness of the source and target domains. overall, this zero-shot coc task formulation opens up new research opportunities towards legal nlp models that are more aligned with expert reasoning.