On the Role of Negative Precedent in Legal Outcome Prediction (2023)

Abstract: Every legal case sets a precedent by developing the law in one of the following two ways. It either expands its scope, in which case it sets positive precedent, or it narrows it, in which case it sets negative precedent. Legal outcome prediction, the prediction of positive outcome, is an increasingly popular task in AI. In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction. We discover an asymmetry in existing models’ ability to predict positive and negative outcomes. Where the state-of-the-art outcome prediction model we used predicts positive outcomes at 75.06 F1, it predicts negative outcomes at only 10.09 F1, worse than a random baseline. To address this performance gap, we develop two new models inspired by the dynamics of a court process. Our first model significantly improves positive outcome prediction score to 77.15 F1 and our second model more than doubles the negative outcome prediction performance to 24.01 F1. Despite this improvement, shifting focus to negative outcomes reveals that there is still much room for improvement for outcome prediction models. https://github.com/valvoda/Negative-Precedent-in-Legal-Outcome-Prediction

## introduction the legal system is inherently adversarial. every case pitches two parties against each other: the claimant, who alleges their rights have been breached, and the defendant, who denies breaching those rights. for each claim of the claimant, their lawyer will produce an argument, for which the defendant's lawyer will produce a counterargument. in precedential legal systems @xcite , the decisions in the past judgements are binding on the judges in deciding new cases. 1 therefore, both sides of the dispute will rely on the outcomes of previous cases to support their position @xcite @xcite . the claimant will assert that her circumstances are alike those of previous claimants whose rights have been breached. the defendant, on the other hand, will allege that the circumstances are in fact more alike those of unsuccessful claimants. the judge decides who is right, and by doing so establishes a new precedent. if it is the claimant who is successful in a particular claim, the precedent expands the law by including the new facts in its scope. if it is the defendant who is successful, the law is contracted by rejection of the new facts from its scope. the expansion or contraction is encoded in the case outcome; we will refer to them as positive outcome and negative outcome, respectively. positive and negative outcomes are equally binding, which means that the same reasons that motivate the research of the positive outcome also apply to the negative outcome. both are important for computational legal analysis, a fact that has been known at least since lawlor (1963). 2 however, the de facto interpretation of precedent in today's legal nlp landscape focuses only on positive outcomes. several researchers have shown that a simple model can achieve very high performance for such formulation of the outcome prediction task @xcite @xcite , a finding that has been replicated for a number of jurisdictions @xcite . in this work, we reformulate outcome prediction as the task of predicting both the positive and negative outcome given the facts of the case. our results indicate that while a simple bert-based classification model can predict positive outcomes at an f 1 of 75.06, it predicts negative outcomes at an f 1 of 10.09, falling short of a random baseline which achieves 11.12 f 1 . this naturally raises the question: what causes such asymmetry? in §8, we argue that this disparity is caused by the fact that most legal nlp tasks are formulated without a deep understanding of how the law works. searching for a way to better predict negative outcomes, we hypothesize that building a probabilistic model that is more faithful to the legal process will improve both negative and positive outcome prediction. to test this hypothesis we develop two such models. our first model, which we call the joint model, is trained to jointly predict positive and negative outcome. our second model, which we call the claim-outcome model, enforces the relationship between the claims and outcomes. while the joint model significantly 3 outperforms state-of-the-art models on positive outcome prediction with 77.15 f 1 , the claimoutcome model doubles the f 1 on negative outcome prediction at 24.01 f 1 . we take this result as strong evidence that building neural models of the legal process should incorporate domainspecific knowledge of how the legal process works. ## the judicial process in order to motivate our two models of outcome, it is necessary to first understand the process of how law is formed. broadly speaking, the legal process can be understood as a task of narrowing down the legal space where the breach of law might have taken place. initially, before the legal process begins, the space includes all the law there is, that is, every legal article. 4 it is the job of the lawyer to narrow it down to only a small number of articles, a subset of all law. finally, the judge determines which of the claimed articles, if any, has been violated. we can therefore observe two distinct interactions between the real world and the law: (i) when a lawyer connects the real world 3 throughout the paper we report significance using the two-tailed paired permutation test with @xmath0.05. and law via a claim, and (ii) when a judge connects them via an outcome. in practice, this means that judges are constrained in their decision. they cannot decide that a law has been breached unless a lawyer has claimed it has. a corollary of the above is that lawyers actively shape the outcome by forcing a judge to consider a particular subset of law. in doing so a lawyer defines the set of laws from which the judge decides on the outcome. 5 the power of a lawyer is also constrained. on one hand, lawyers want to claim as many legal articles as possible, on the other there are only so many legal articles that are relevant to their client's needs. thus, there are two principles that arise from the interaction of a lawyer and a judge. first, positive outcome is a subset of claims. second, negative outcome consists of the unsuccessful claims, namely, the claims the judge rejected. there is a close relationship between claims and negative outcomes: if we knew the claims the lawyer had made, we could define negative outcome as exactly those articles that have been claimed, but that the judge found not to be violated. much like how outcomes are a product of judges, claims are a product of lawyers. and, unlike facts, they are not known before human legal experts interact with the case. therefore, to study the relationship of outcomes and facts, one can not rely on claims as an additional input to the model. the only input which is available and known before a case is processed by the court, are the facts. outcome prediction task. legal facts are the transcript of the judge's description of what has happened between the claimant and the defendant. under the current formulation of the outcome prediction task, models are trained to predict whether case facts correspond to a violation of each article, that is, the models are trained to predict a vector in {0, 1} k where 1 indicates a positive outcome and k is the number of legal articles under consideration. what is wrong with current work? in the above formulation, 0 is ambiguous-it can indicate either that the article not claimed or that the judge ruled that that specific article was not breached. existing models, which don't take any information about claims into accounts, implicitly assume that all articles have been claimed, which is almost never the case in practice. under this assumption, the role of the legal claim and of negative outcomes is therefore effectively ignored. reformulating the task. how should negative outcome then be modeled? given the domain specific knowledge about the interaction of a judge and a lawyer, our position is that models that predict outcomes should model the claims and outcomes together. to this end, we first need information about which laws have been claimed. in §6, we discuss the creation of a new corpus, which contains the necessary annotation for this task. in the next section we develop two models that jointly predict outcomes and claims using two basic assumptions about how the law operates. we believe that our reformulation of the task has two advantages. first, considering positive and negative outcomes together is a step towards better evaluation of legal outcome prediction models. second, incorporating the roles of a judge and a lawyer within the models of outcome is a step towards better models of law. ## law-abiding models in this section we formulate our two probabilistic models of law. our law-abiding models are built on top of the two assumptions described below. notation. we define a probability distribution over three random variables. ## baselines we contextualize the performance of the joint and claim-outcome model with a number of baselines. as a starting point we build a simple classification model trained to predict positive or negative outcome separately, see figure 1a . we further want to test whether the advantage of our joint model stems from encoding the relationship between positive and negative outcome, or whether it is down to simply training on more data. we test this by formulating the task as a multi-task learning objective, see figure 1b . while this model is trained on the same amount of data as our joint model, it does not explicitly encode the relationship between positive and negative outcomes. ## experimental setup pre-trained language models. we obtain high-dimensional representations enc(f ) by finetuning one of the following pre-trained language models with f as an input: longformer is built on the same transformer @xcite architecture as bert @xcite and legal-bert @xcite , but it can process up to 4,096 tokens. we select this architecture because the facts of legal documents often exceed 512 tokens; a model that can process longer documents could therefore be better suited to our needs. training details. all our models are trained with a batch size of 16. we conduct hyperparameter optimization across learning rate {3e-4, 3e-5, 3e-6}, dropout {0.2, 0.3, 0.4}, and hidden size {50, 100, 200, 300}. we truncate individual case facts to 512 tokens for bert and legal-bert or 4,096 tokens for the longformer. our models are implemented using the pytorch @xcite and huggingface @xcite libraries. we use adam for optimization @xcite and train all our models on 1 tesla v100 32gib gpus for a maximum of 1 hour. we train for a maximum of 10 epochs. 7 the models are trained on the training set, see we work with the ecthr corpus, foot_3 which contains thousands of instances of case law pertaining to the european convention of human rights (echr). ecthr cases contain a written description of case facts, which is our f , and information about claims and outcomes. since positive outcomes are a subset of all claims, the exclusion set of claims and positive outcomes constitutes the set of negative outcomes. @xcite , we report all results as f 1 micro-averaged. we report significance using the two tailed paired permutation tests with @xmath1.05. the bulk of our results is contained in ## discussion the results reported above raise the question of why models severely underperform on negative outcome prediction. the simplest answer could be the amount of training data that is available for each task. we test this hypothesis by comparing the performance on articles 8 (796 negative vs. to find a better explanation of the performance asymmetry, we now turn in our discussion to the legal perspective. in precedential jurisdictions, of which ecthr is one @xcite @xcite , the decisions of a case are binding on future decisions of the court. two cases with the same facts should therefore arrive at the same outcome. of course, in reality, the facts are never the same. rather, cases with similar circumstances will, broadly speaking, lead to similar outcomes. this is achieved by applying the precedent. in such cases, the judge will in effect say that the new case is not substantially different from an already existing case and therefore the same outcome will be propagated. on the other hand, if the previous precedent is not to be followed, the judge needs to distinguish the case at hand from the precedent. distinguishing the case from the precedent is a more involved task than applying the precedent. it requires identifying what exactly about the new facts sets the new case apart from the previous one. this can of course be done for both cases with positive and negative outcome. both can be applied or distinguished. 10 since judges deal with claims, each of which comes with an argument built around the precedent that favours the claimant's viewpoint, we believe that negative outcomes overwhelmingly rely on distinguishing the case from the precedent. this is evidenced in the yearly reports of the ecthr (2020), which list cases where the judges decided to distinguish the facts of the case at hand. distinguishing almost always leads to a negative outcome. we observe the same trend in our ecthr corpus. it might therefore be the case that while there is such a thing as a prototypical positive precedent, there is no prototypical negative precedent. this could explain why the simpler architectures struggle to learn to predict it. while a simpler model is ill-suited for the task since it is trained to find a similarity between the negative outcome cases, our claim-outcome model does not assume that negative outcome cases are similar in the first place. instead, our model assumes similarity between claims. since claim prediction can be modeled with a high accuracy @xcite , we can reveal the negative outcome as a disagreement between a judge and a lawyer (i.e., claims and the outcomes). by investigating individual cases in the chalkidis et al. corpus, we can identify a further possible explanation for the baseline model performance. for instance, the case of wetjen and others v. germany (wetjen) is concerned with article 8: right to respect for private and family life. in this case, religious parents used caning (among other methods) as a punishment for their children. the german state intervened and placed the children in foster care. the parents claim interference with their right to family life. on a superficial level, the case is similar to two article 8 cases both cited in wetjen: that of shuruk v. switzerland (shuruk), and suss v. germany (suss). in shuruk, religious parents fight for an extradition of a child. the mother of the child argues that it would be an interference with her right to a family life if the child was to be extradited to the husband, who has joined an ultra-orthodox jewish movement. a component of the case is an allegation of domestic violence the husband was supposed to have perpetrated against his wife. in suss, the german state has denied a divorced father access to his daughter due to the frequent quarrels between the parents during the visits. the father alleges breach of article 8. in wetjen and suss, the judges have decided a violation of article 8 has not occurred, they have ruled the opposite in shuruk. on the surface, the facts are alike, especially between shuruk and wetjen-both cases contain elements of abuse, religion, and state intervention. however, to a human lawyer, the distinction between the cases is fairly trivial. in wetjen, the state is allowed to intervene to protect a child from an abusive ultra-religious parents, which is very similar situation to suss, where a state is allowed to intervene to protect the child from quarrels between divorced parents. all our models are exposed to both shuruk and suss in the training set. however, for the positive outcome baseline, the information about suss being related to article 8 is lost. conversely, for the negative outcome baseline, the information about shuruk being related to article 8 is lost. it is therefore not surprising that the best performing negative 11 and positive 12 outcome baseline models both get the wetjen outcome prediction wrong. on the other hand, the best claim-outcome model, which is trained to learn that both shuruk and suss are related to article 8 via the claim prediction objective, makes the correct outcome prediction in the wetjen case. in conclusion, our claim-outcome model is indeed a better way of modeling negative outcome, but its superiority is not due to the fact that it is learning anything about the law itself. it simply leverages the fact that positive outcomes and claims are easy to predict and enforces the relationship between them. to identify the negative outcome with high f 1 will require deeper understanding of law than our models are currently capable of. 9 related work juris-informatics can trace its origins all the way to the late 1950s @xcite . the pioneers used rule-based systems to successfully capture aspects of legal reasoning, using thousands of hand-crafted rules @xcite . yet due to the ever-changing rules of law, these systems were too brittle to be employed in practice. particularly in common law countries, the majority of law is contained in case law, where cases are transcripts of the judicial decisions. this allows the law to constantly change in reaction to each new decision. the advances of natural language processing (nlp) in the past two decades have rejuvinated the interest in developing applications for the legal domain. areas explored include question answering @xcite , legal entity recognition @xcite , text summarization @xcite , judgment prediction @xcite , majority opinion prediction @xcite , and ratio decidendi extraction @xcite . 11 simple baseline longformer. 12 simple baseline legal-bert. our work is similar to the recent study of chinese law judgement prediction by @xcite and @xcite , who break down court judgements into the applicable law, charges, and terms of penalty. operating in the civil law system (which outside of china is also used in germany, and france, inter alia), they argue that predicting applicable law is a fundamental subtask, which will guide the prediction for other subtasks. in the context of echr law, we argue that legal claims are one such guiding element for outcome prediction. while similar, applicable law and claims are different. in the work above, the judge selects the applicable law from the facts as part of reaching the outcome. this is not the case for echr law, or any other precedential legal system known to the authors, where the breach of law is claimed by a lawyer, not a judge. finally, the ecthr dataset has been collected by @xcite , who have predicted outcomes of the echr law and the corresponding articles using neural architectures. our work builds on their research by reinstantiating the outcome prediction task on this dataset to include negative precedent. similar datasets, which one could apply our method to, include caselaw access project and us supreme court caselaw. 13 ## conclusion while positive and negative outcomes are equally important from the legal perspective, the current legal ai research has neglected the latter. our findings suggest that negative outcome is much harder to predict than positive outcome, at least for current deep learning models. this has severe implications for how well the current legal models can model judicial outcome. the same models that predict positive outcome with 75.06 f 1 fall short of a random baseline of 10.09 f 1 on the negative outcome prediction task. we discuss possible reasons why negative outcome prediction is so much harder to learn. specifically, we suspect that negative outcomes are mostly caused by a judge distinguishing the case from its precedent. this lead us to believe that learning to predict negative outcomes requires more legal understanding than the current models are capable of. we believe that negative outcome prediction is therefore a particularly attractive task for evaluating progress in legal ai. our work improves over the existing models by inducing the relationship between the judge and the lawyer in our claim-outcome model architecture. however, the best negative outcome prediction model achieves only a third of the performance of the positive outcome one. in future work we hope to study the phenomenon of precedent more closely, with the aim of building models capable of narrowing this performance gap. one possible avenue would be to relax our assumption 1 to study the potential relationships between the individual legal articles. we leave this direction for future work.