Legal Judgment Prediction: If You Are Going to Do It, Do It Right (2023)

Abstract: The field of Legal Judgment Prediction (LJP) has witnessed significant growth in the past decade, with over 100 papers published in the past three years alone. Our comprehensive survey of over 150 papers reveals a stark reality: only ~7% of published papers are doing what they set out to do - predict court decisions. We delve into the reasons behind the flawed and unreliable nature of the remaining experiments, emphasising their limited utility in the legal domain. We examine the distinctions between predicting court decisions and the practices of legal professionals in their daily work. We explore how a lack of attention to the identity and needs of end-users has fostered the misconception that LJP is a near-solved challenge suitable for practical application, and contributed to the surge in academic research in the field. To address these issues, we examine three different dimensions of ‘doing LJP right’: using data appropriate for the task; tackling explainability; and adopting an application-centric approach to model reporting and evaluation. We formulate a practical checklist of recommendations, delineating the characteristics that are required if a judgment prediction system is to be a valuable addition to the legal field.

## introduction the task of predicting court decisions has been a topic of great interest since at least the 60s @xcite . it has gained traction in recent years due to policies promoting publishing case law around the world, and increased efficiency and popularity of machine learning. today there are more than 150 academic papers claiming to predict court decisions (a.k.a. legal judgment prediction) using natural language processing (nlp) and machine learning, many reporting accuracies and f1-scores of over 90% (see, for instance, @xcite @xcite . the media has speculated about the advent of robojudges and automated lawyers for years @xcite @xcite . many in the justice sector are persuaded that machine learning technologies might usefully be employed to predict the outcomes of cases @xcite . some judges believe these technologies could even make "very minor" decisions. 1 yet, these systems are not widely used. while there are a number of ethical considerations @xcite including issues of bias in the data @xcite that could be the reason for the hesitancy, in this paper we will demonstrate that there may be a different root cause, namely that the setup of the experiments presented in the overwhelming majority of academic papers does not allow the systems to do what they claim to be doing -predicting court decisions. ## related work the field of legal judgment prediction (ljp) has rapidly expanded over the years, with over 150 papers published in the last 10 years. the majority of the papers that we reviewed use the text of historical decisions to make predictions on unseen data. some do not use nlp, but rely on manually extracted variables to make the predictions @xcite . we looked specifically at papers in english that conduct experiments on court decisions, rather than any other legal decisions (excluding, for instance @xcite focused on wipo decisions). of all the countries whose court cases feature among the reviewed papers, china has the highest proportion overall (78 papers). china publishes millions of cases online, many of which are available as part of the cail2018 dataset @xcite with over 2 million cases. bench-mark datasets attract many to develop new methods and improve the scores, with lexglue echr @xcite among the more prominent ones for the european court of human rights (ecthr). in fact, at least 27 papers we reviewed attempt to predict decisions of the ecthr, with one providing access to a regularly updated website (jurisays.com) that predicts decisions in real-time. a work that provides an even more ready-to-use tool is @xcite . the paper includes a link to a website where one can input one's claims and receive a prediction for italian and bulgarian vat cases. 2 with so much published research, predicting court decisions (at least for some courts) may seem like a solved task, e.g. @xcite achieves 99% accuracy for predicting decisions of french courts, @xcite achieves 98% accuracy for the judgments of the ecthr. ## getting your 'facts' right in this paper we focus on issues with the way that research in ljp is conducted today. the main issue, which concerns the experimental setup, was previously addressed in @xcite , (focusing on @xcite paper), and later in @xcite (reviewing a larger corpus of 27 papers). we are going to revisit this issue taking account of 171 papers in the field. presently, predictive systems predominantly rely on case facts extracted from judgments. specifically, these machine learning systems are fed the text of the 'facts' section of the cases (and sometimes also the reasoning of the court) as inputs together with the corresponding decisions (e.g. violation/no violation of an article of the european convention on human rights) as labels. the systems are trained to 'predict' decisions; they learn patterns within the input which correspond to particular labels. they are then evaluated on a test set (i.e. 'unseen' case facts that were not included in the training set). the 'facts' that comprise the test set, like those of the training set, are extracted from published final judgments. the labels used to evaluate the systems' performance on the test set can also be found in those judgments. while such systems might achieve high performance, they are never actually tested on 'real' data, that is data that is genuinely available to potential end-users before 2 https://adele-tool.eu the judgment is reached or published. the actual user of these systems, let's say a lawyer looking to advise a client about the likely outcome of a court hearing, does not have access to the same formulation of the facts as is available and set out in the judgment. that formulation cannot exist until the judgment is made. one might imagine that the 'facts' of the case set out in a judgment are the same as, equivalent to, or a reasonable proxy for the factual information available prior to the court's decision. however, this is incorrect. facts very often emerge in the process of litigation. one side's account of the facts may (and typically will) be countered by the other side's account. each side's account may be iteratively refined and expanded to respond to the other's. factual evidence given by witnesses at a hearing may differ from earlier accounts given by those same witnesses. if, as is generally the case in lower courts, there is a dispute about the facts, the court will have to make a decision about the facts. in addition, courts are only concerned with those facts that are relevant for the decision in the case, taking account of the nature of the dispute or issue. that is, the courts are only concerned with those facts that relate to the legal rules which the court judges to be relevant. as a result the facts described in the judgment are often "highly selective summaries of the original case record, written by the decision-makers themselves and tailored for consistency with the decision." @xcite . for all these reasons the 'facts' set out in the judgment may be very different -in substance and in form -from earlier accounts of the facts. some research confirms that the way in which the 'facts' of a case are formulated has a material effect on the performance of predictive systems @xcite . @xcite also experimented with the ecthr data by comparing the results of training the model on the facts extracted from the final judgments (not available prior to decision-making) and training it on the facts as published by the court just after receiving the submission of the alleged victim of human right violation. their work shows that models that perform well on the 'facts' extracted from final judgments do not perform nearly as well when trained on the 'real' data, available prior to the decisions. the best model (hierarchical bert) achieves an f1-score of 0.92 on final judgments and only 0.64 on facts available prior to the decision. systems trained and tested foot_1 on 'facts' extracted from judgments can only model that the facts described in a judgment correspond to a decision that has already been made; they do not in any way allow users to actually predict any judgments. although it is usual for ljp systems to rely on the 'facts' section of cases, some systems may use different information. for instance, @xcite and @xcite claim to predict decisions of the ecthr by creating an abstract dialectical frameworks decision tree based on factors the court considers when making decisions regarding the right to a fair trial. however, when used to 'predict' the outcome of new cases, the system depends on the user answering complex questions about 'base factors' (the leaf nodes of the decision tree) which can only be authoritatively decided by a judge. it depends, not on pre-existing information, but on interpretative decisions made by the user. the system is therefore unable to do the prediction without having part of what needs to be decided fed into it. we limited our analysis to ljp papers published in the past 10 years (2014 until august 2023). given the amount of papers published on the topic, our list might not contain all of the published research in the field, but we expect that we found most of the available papers. we relied on descriptions of the datasets presented in the papers in order to establish the data that the authors used for the experiments. we present the list of all reviewed papers, including information about their data and performance at https://shorturl.at/pxkr3 . out of 171 reviewed papers, we found only 12 (∼7%) that use appropriate data for predicting decisions, with only 7 using text-based input. additionally we found 3 papers that in principle might provide a way to predict future court decisions, but because their data is private and inaccessible, it is unclear whether all of the data used for the testing was available before the relevant judgment and therefore suitable for prediction of future outcomes. we discuss all 15 papers in section 5. ## prediction in law v. prediction in nlp we hesitate to describe the activity of lawyers in advising about possible outcomes of cases as 'prediction'. lawyers often qualify advice about outcomes by observing (correctly) that the law might change, that there may be room for different interpretations of the relevant law, that their client's account of the facts might not be accepted, that new issues (of fact and/or law) might arise in the course of the court proceedings. lawyers will rarely offer a binary 'win or lose' prediction, a quantified confidence rating, or a prediction which is not hedged about with appropriate caveats @xcite . this hesitation should not be mistaken for failure: the life of the law is its ability to adapt; the processes and procedures through which the facts of a case emerge -gradually and iteratively -are crucial for preserving the ability of the citizen to articulate their case @xcite . lawyers are both less and much more than fortune tellers. they have an active role to play in developing the law. nevertheless their role in advising about the possible outcomes of cases is 'prediction' in the sense that they are concerned with future outcomes of cases. if lawyers want to know the outcome of an already decided case they can read it in the judgment. some few of the papers we reviewed use nlp to output a classification (e.g. violation/no violation) in response to a textual account of the facts which pre-dates the judgment. these systems are designed to predict a future outcome. however, most of the papers use nlp to output a classification in response to a textual account of facts extracted from an existing judgment. they classify facts in a judgment as associated with a specific verdict (i.e. a label), but the actual verdict can be found in the same judgment. there is therefore a mismatch between the usual framing of the task of ljp in nlp and a lawyer's understanding of 'prediction' of judgment. this mismatch might be understood as one of terminology. the machine learning community uses the word 'prediction' as a substitute for 'classification'; 'prediction' in this sense does not imply looking into the future. lawyers, on the other hand, understand 'prediction' to entail prediction of some future event. for this reason @xcite suggested researchers avoid the term 'prediction' and clearly differentiate between 'outcome identification, outcome-based judgement categorisation and outcome forecasting'. however, the mismatch is more deep-seated and significant than one of terminology. it concerns the nature of the task performed by ljp models and the utility of the models for the intended end-user. if these are to have real-world utility for lawyers, citizens or others (e.g. political scientists) who wish to anticipate the outcome of cases they must be able to offer prediction of future judgments. in other words, they must offer a prediction of what the judgment will be, not 'predict' what the judgment was. much current research relating to ljp pays little attention to who, precisely, might use these systems and for which use cases. the majority of papers simply state that these systems could be useful to 'legal professionals' without differentiating between, for example, judges, paralegals, prosecutors, defence lawyers, clerks of court. each of these might have different reasons for using these systems, different obligations and professional responsibilities, and therefore different use cases. the possibility that citizens might use these systems is frequently overlooked. however, as a matter of practicality, if one is to build an ljp system which has utility for some class or classes of user, it is essential to consider why they might want to use the system and what data they might use as inputs to the system. different users may have access to different kinds of information, in different formats and languages, from different sources, reflecting different versions of events, and at different stages before or during court proceedings. an ljp system built for a specific class of end-users needs to be trained and evaluated on the data that would be available to such users. otherwise the results do not tell one anything about how the system would perform in a real-life scenario and the system may have little or no utility. for example, a system built for prospective applicants to the ecthr and intended for use before an application is submitted to the court would have to be trained and tested on data available to the applicant at that stage. such data might include this data is not commonly available, with access to court records varying across jurisdictions. it may be restricted to physical (rather than online) access, to cases in higher courts, to certain kinds of case documents, or provided only for a fee @xcite . the papers that use appropriate data for future predictions consistently exhibit much poorer performance compared to those relying on flawed data. among papers employing text-based input for training, only one achieves over 70% accuracy. specifically, @xcite attain 75% accuracy in predicting ecthr decisions based on summaries from so-called communicated cases, prepared by the court, derived from applicant submissions, and published before the judgments are made. however, a study with a similar albeit smaller dataset subsequently reports a reduced accuracy rate of 65%-68% depending on the year of the test data @xcite . note that these summaries are only available if and when the ecthr accepts an application and only for a subset of cases. consequently, depending on the end-user and their specific requirements, these summaries may still not suffice for making predictions. since applicants do not have the case summary before they submit the applications to the court, it is suitable for prediction only once the legal proceedings begin. then applicants may use the case summary as input to an ljp system to estimate their prospects of success, potentially opting for a settlement to avoid lengthy proceedings. using lower court decisions to predict higher court outcomes as explored by @xcite and jacob de menezes-neto and clementino (2022) offers a way to use pre-judgment data, yet performance has been modest with the papers reporting f1-score of 57% and a matthews correlation coefficient (mcc) of 0.37, respectively. predicting court decisions is an inherently challenging task. the challenge is compounded by the fact that many countries and courts only publish a subset of their cases at each level, making it difficult to access sufficient data from both lower and higher courts. 4 it is crucial to acknowledge, however, that not all cases eligible for appeal to a higher court will ac-tually pursue this course of action, introducing potential selection bias when training and evaluating the system. other papers that use text as input rely on various documents from different stages of legal proceedings. for example, @xcite al. (2022) while these documents may be available before the decision is made, they typically represent the claims of one party, and therefore may not include all relevant information. the papers report accuracy scores of 67%, 61% and an mcc of 0.48, respectively. the papers predicting decisions of the us supreme court (e.g. @xcite often use a manually annotated scdb dataset @xcite , containing 240 expertannotated variables, rather than text from legal documents. such data is rarely available for other courts, and would require significant manual effort to compile. these papers do not use textual input, yet both achieve 70% accuracy. such data could potentially be supplemented with, for example, decisions of the lower court and/or expert opinions. for instance, @xcite use oral argument in combination with the annotated variables, therefore creating a system that can predict decisions once the court case started, reporting 74% accuracy. chen and eagel (2017) incorporate seemingly unrelated data like weather and news trends to improve the performance (up to 79% accuracy). similarly, published judgments, although unsuitable for directly predicting judgments, could serve as a source from which to extract factors or variables that are relevant to the outcomes of those cases. these factors, in turn, might be used for prediction. for instance, hsieh et al. ( 2021 ) extract information about plaintiff and defendant, including their income and debt information, to make predictions about discretionary damages. however, it is not always obvious whether the factors are available in advance or will have to be established by the court. for example, frança et al. (2020); dos @xcite ; bagherian-marandi et al. (2021) who also adopt this approach mention using client information, as well as extracting details from judgments. however, since their data is not published or available for examination the precise nature of the extracted features remains unclear. @xcite , for instance, has the value of the claim as one of the features. this feature could relate either to the sum claimed in the initial submission (appropriate data) or the sum determined during the course of the proceedings, with the latter not being available prior to the judgment (inappropriate data). therefore, the performance of these systems should be tested on data provided by the users rather than relying on extracted data. this approach ensures that the system's performance mirrors a real-world scenario. given the scarcity of papers conducting research on appropriate data and the relatively modest performance of many of those systems, it is evident that there is substantial room for investigation to establish the most effective approaches for ljp. we have offered examples of data that might be appropriate for ljp ('good' data). however further research is needed to establish what 'good' data is available to researchers in different jurisdictions, and, of this data, which may produce optimal results. crucially however, and contrary to the approach adopted in the vast majority of papers, doing ljp 'right' depends on selecting and using data that predates the judgment. ## benchmark datasets as we have discussed, the facts of the judgments that one is predicting are the wrong data for the ljp task. it follows that benchmark datasets, such as cail2018 @xcite , lexglue echr a & b @xcite or ildc cjpe @xcite whose input data consists of facts extracted from judgments, are of little or no assistance for the task of prediction of future judgments. models which predict a future judgment can only be benchmarked on a dataset containing textual representations of facts that are available before the judgment. endorsement by the nlp community of datasets such as cail2018, lexglue echr and ildc cjpe as benchmarks for prediction of judgment has ramifications. @xcite note that "when they institutionalize benchmark datasets, task communities implicitly endorse these data as meaningful abstractions of a task or problem domain. the institutionalization of benchmarks influences the behavior of both researchers and end-users." cail2018, lexglue echr, ildc cjpe and similar datasets implicitly signal to researchers, lawyers and policy makers that it is meaningful to carry out the task of prediction of judgment by using data extracted from cases in which the judgment is already available as the model input. they also implicitly signal that the performance scores of models tested on the datasets are meaningful indicators of performance in the real-world task of prediction of (future) judgment. do the 'predictions' output by models tested against such benchmarks have any real-world utility? we suggest they do not. such 'predictions' involve labelling or categorising input data as correlated with a particular (already known) outcome @xcite . these 'predictions' have no informational value for lawyers, their clients or others who wish to obtain a prediction of a future judgment @xcite . moreover, to our knowledge there is no research indicating that any improvements of the results on such a dataset would translate to improved performance in a real-world context. in fact, we are only aware of research that has demonstrated the contrary (see @xcite . to mitigate these issues, we propose that the ideal test set should -at a minimum -consist of data in which no information has been derived from the judgments that one is predicting. instead, this data should be gathered from the parties involved in the case or from documents accessible before the decision-making process. conducting an experiment in which data is gathered for cases that are still pending decisions, followed by a waiting period until the court issues its judgments for comparison, would be the most dependable method to safeguard against flawed test results. although this approach may present increased difficulties due to its longitudinal nature, it would guarantee the reliability of the test results. ## addressing explainability the real-world utility of ljp systems may also depend on their explainability. "[i]n law the explanation is what matters" @xcite . when lawyers give advice about outcomes, they generally explain why an outcome or range of outcomes might be likely. they explain how the law may apply to the client's problem. similarly, judges usually offer explanations which serve to justify their decisions with reference to the facts found to be established and the relevant law. this aspect of judicial practice relates to the public character of law and the ability of citizens to engage with the law @xcite . both lawyers and judges provide explanations which involve the exercise of legal reasoning @xcite . these explanations have a particular form. they link the facts of the case and the law to the conclusion; they are grounded in law. we also expect these explanations to meet a qualitative standard; we want them to accurately represent the law and to demonstrate sound reasoning. is it necessary that ljp systems issue explanations that are grounded in law? this may depend on how, by whom and for whose benefit these systems will be used. despite our reservations, these systems might be used in place of judges, to make decisions rather than merely offering predictions. in that scenario, it would seem essential for such a system to be capable of providing an explanation of its output that is grounded in law @xcite . otherwise, how could one assess whether the output was justified according to legal standards, or how, and on what legal basis, litigants might challenge the output of the system? similarly, if persons without legal representation use these systems to estimate their prospects of success in a court case, it would seem essential for the system to provide an explanation grounded in law. how else could such persons make sense of those outputs? if on the other hand lawyers use these systems as additional tools, it might be desirable but not essential for the system to provide explanations grounded in law. lawyers should provide their own explanations of the outcome or outcomes they consider likely. a range of approaches to explainability can be found in the ljp literature. for example, machine learning systems which frame ljp as a classification task may output an 'explanation' about which features the system treats as important. the system may output descriptions or visualisations of which words, sentences or paragraphs of the input data (usually the 'facts' of a case) contributed to the output predictions @xcite . alternatively, machine learning systems that treat ljp as a text generation task may generate text in the form of steps of reasoning (a chain of thought) or a legal syllogism together with a conclusion (a prediction) @xcite . the 'reasoning' steps serve as the explanation. hybrid systems which combine machine learning with symbolic reasoning approaches may output justifications of the predictions output by the system @xcite . these justifications would not represent the 'reasoning steps' of the predictive model though they may provide an ex post explanation of the prediction. the focus on explainability is welcome, and certain of these approaches may hold some promise. however, for most users and use cases, the explanations (if they are to be useful) must be grounded in law, relate to predictions of future judgments and evaluated for soundness by suitable methods. we are not aware of any ljp system that outputs 'explanations' that meet all three criteria. this explainability deficit is a significant limitation which is relevant for the evaluation of ljp models, for their real-world utility and for the potential benefits, harms and impacts associated with their use. ## an ethical approach to model evaluation and reporting we have suggested that current practice in relation to the evaluation of ljp models is frequently flawed, involving a race-to-the-top against benchmark datasets unsuited for the task of prediction of future judgments and a failure to consider intended users and use cases. if these issues are relevant for training and testing a system, they are also relevant for further evaluation and reporting. we favour an approach to evaluation which is "applicationcentric" @xcite , that is, an approach which considers the fitness of the model for its real-world application context. an applicationcentric approach would necessarily entail a focus on the intended user and use case and highlight the unsuitability of ljp benchmark datasets comprising formulations of 'facts' only available at the point of judgment. an application-centric approach takes account of direct benefits and harms to users and those affected by the outputs of the system @xcite . adoption of this approach to evaluation therefore requires a good understanding of the context in which a model is likely to be used, who will benefit or suffer adverse consequences as a result of such use and how those consequences are likely to arise. it requires a sufficient understanding of the way in which ljp systems might shape lawyers' advice and judges' determinations, ultimately affecting legal outcomes for citizens, and a recognition that these systems are "intended to inform decisions about matters central to human life or flourishing" @xcite . more concretely, it requires an appreciation that a citi-zen might suffer harm if, for example, a system suggests a claim will succeed when the claim is doomed to failure on legal grounds, or provides a judge with an overly harsh recommendation in relation to sentencing. crucially an application-centric approach takes into account the potential impact of "changes to the ecosystem itself" @xcite . in the case of ljp systems this entails reflection on how the use of these systems may affect the ecosystems of law and legal practice. for example, if judges were to align their decisions with the outputs of ljp systems, this might lend credibility to the notion that machines can engage in legal reasoning, reduce the human element in decision-making, devalue legal reasoning and de-skill judges. lawyers who use these systems might be "encourage @xcite ... to base their litigation strategy on factors other than the legal merits of the case" @xcite . law might stagnate; as bench-capon (2021) notes, the model producing the prediction can only be "trained on past decisions". there is a risk of confusing or conflating what ljp systems can do with the exercise of legal reasoning within a legal institutional framework which allows for "interpretation, contestation and argumentation" @xcite . the impacts are likely to be amplified if the systems are used for decision-making rather than decision-support. in that context, the inability of ljp systems to provide explanations grounded in law could have serious ramifications. of course, the use of these systems might also produce benefits -the point is to reflect on the impact of the research and the technology on the application ecosystem. we suggest that an application-centric approach is both necessary and appropriate where researchers maintain that the systems they develop have real-world utility. it encourages trust; instead of merely claiming real-world utility, researchers would demonstrate their attention to the context of application. it is "better aligned with the needs of decision-makers who consider whether to use a model in an application." @xcite . model cards, @xcite which allow for reporting on the intended user, intended use case, the benefits, harms and social impacts, might be of use in this context. we encourage the use of such a reporting framework as a means of clearly and transparently communicating evaluations of models which look beyond accuracy metrics. ## discussion despite the perception of prolific research activity, the reality is that the ljp field has produced only a meagre 12 papers to date, covering 9 courts, and reporting performance scores between 56% and 79% accuracy. by diverting resources and attention towards systems that do not deliver on their promises, researchers and developers may be discouraged from pursuing more appropriate and impactful avenues. the atmosphere of competition, with numerous models claiming accuracy rates exceeding 90%, may hinder the publication of rigorous initialstage research and incremental advancements in the field that produce much lower performance. consequently, it impedes the overall progress of ljp research. such faulty research not only hampers scientific progress but also poses potential risks to individuals. for instance, a system such as adele, while conceived as an academic project, can be used to make predictions based on claims provided by users, even though it was not trained or evaluated on such data. given the inflated scores of systems claiming to be predicting court decisions, there is a distinct possibility that these systems may eventually find their way onto the market and into courts. we propose a number of recommendations to address the concerns outlined in this paper. we recommend that researchers building and reporting on systems that claim to do ljp identify the enduser, the application of the system and the stage in the legal process when the system would be used. researchers should then evaluate whether the data they plan to use is available to the user at that stage of the proceedings or collect a new dataset. depending on the purpose of the application one may choose to create a system that offers some form of explainability, whether in the sense of explaining how the model makes the prediction (for instance, for error analysis and correction) or, explanations that take the form of legal reasoning. when building such a system, even within academic research, one should keep in mind and report not only potential benefits, but also potential harms that might result from use of the system. we encourage developers to also consider the broader impact that the system may have on individuals, law, legal procedures and society as a whole. we summarise these recommendations in a schematic checklist available in appendix a. ## conclusion our paper delves into the current landscape of legal judgment prediction (ljp) research and its issues. through examination of 171 papers, we find that only a small fraction of ljp papers which claim to predict court decisions achieve this objective. the majority falter by using the wrong data for the task at hand. despite many papers reporting exceptionally high performance, it becomes evident that the task of ljp proves to be considerably more challenging when researchers use an appropriate experimental setup. in addition, we scrutinise when and for whom ljp has real-world utility. we emphasise the importance of considering the end-user and the use case when developing the models. the prevalent reliance on readily available data and benchmark datasets is not in line with this focus. we advocate for a shift in approach to redirect the field away from a potentially futile trajectory, ultimately preventing the misallocation of resources on endeavors that lack practical utility within the legal domain.