Towards Supporting Legal Argumentation with NLP : Is More Data Really All You Need? (2024)

Abstract: Modeling legal reasoning and argumentation justifying decisions in cases has always been central to AI & Law, yet contemporary developments in legal NLP have increasingly focused on statistically classifying legal conclusions from text. While conceptually “simpler’, these approaches often fall short in providing usable justifications connecting to appropriate legal concepts. This paper reviews both traditional symbolic works in AI & Law and recent advances in legal NLP, and distills possibilities of integrating expert-informed knowledge to strike a balance between scalability and explanation in symbolic vs. data-driven approaches. We identify open challenges and discuss the potential of modern NLP models and methods that integrate conceptual legal knowledge.

## introduction law has been an attractive domain for ai in both symbolic knowledge representation and statistical nlp. both strands share the common goal of supporting legal practice through enhancing legal research, document analysis, drafting, and decision making. a focal question distinguishing them remains whether, and how, the process of legal reasoning foot_0 underlying all textual data shall be explicitly represented or left to opaque components, such as generative language models or neural classifiers. in principle, legal reasoning resembles if-then-like inference. legal rules are established from sources @xcite and mandate that certain consequences follow if factual requirements are met in a specific situation. in reality, however, such logic-like inferences are interwoven with areas of ambiguity, vagueness, and human discretion @xcite . at the same time, legal orders evolve over time, continuously refining and adjusting to a dynamic world. in knowledge engineering communities, legal reasoning is characterized as 'defeasible' @xcite ) rather than monotonic. rules that are applicable on their face can be trumped by special exceptions, conflicting superior rules, or by distinguishing the precedent from which the rule derives. thus legal decisions are subject to change, as they can be overturned on appeal. the evolving nature of law to align with shifting social values leads to different legal conclusions. when two parties are in conflict and desire two different resolutions, their argument will combine law and facts in a way that is beneficial to their respective goals -through adversarial discourse @xcite . legal argumentation can be seen as an exercise in competitive theory formation in front of an arbitrator, with each side constructing arguments supported by evidence, written law, cases and other authority to favor their desired conclusions while addressing pitfalls of opposing theories @xcite . ai & law as a field started started in the 1970s, when @xcite suggested that computer modeling of legal reasoning would be a promising area for research to better understand legal reasoning and argumentation. many approaches have been proposed over the past three decades capturing several types of reasoning by means of symbolic representations. some 50 years after the field's beginnings, the legal profession is experiencing considerable disruption by nlp technology, most prominently large language models (llms). in this paper, we provide a review of ai & law work offering faithful modeling of legal reasoning but also requiring expensive legal expertise. we contrast this to modern, largely nonexplainable, data-driven methods, which predict legal conclusions directly without engaging in any explicit legal reasoning. our main contributions are as follows: our discussion makes the following arguments: while prior surveys of @xcite and @xcite focus on cataloging various use cases, tasks, and nlp techniques in legal ai, our paper critically examines the historical integration of expert knowledge into legal systems and advocates for its revival and synthesis with datadriven methods. we emphasize the unique value of expert-informed knowledge in ensuring legal reasoning aligns with established principles, which is not the primary focus of the aforementioned reviews. in contrast to @xcite , which highlights the disconnect between the tasks that are pursued in legal nlp research and the actual needs of legal practitioners, our work emphasizes the critical importance of integrating expert-informed knowledge to avoid this gap. we also present directions for synthesizing expert knowledge with current technological advancements, thereby overcoming traditional bottlenecks in knowledge acquisition and enhancing the efficacy of structured argumentation models. most importantly, we contribute a comprehensive distillation of the conceptual ideas developed and researched by the ai & law community prior to the recently surging interest in law as an application domain for mainstream nlp. in part, our motivation is to connect these communities. much legal nlp work does not build on formal models of legal knowledge and reasoning but characterizes it mostly as precursor work to modern statistical methods. our position is that this view does not do justice to the insights gained and legal authenticity captured in this body of research. symbolic ai & law has thought about how to incorporate legal expertise in models more deeply than most current nlp works, and hence the fields should merge and learn from one another. we strive to drive home the necessity of a paradigm shift in legal nlp, one that values and integrates the profound expertise of domain specialists with the capabilities of data-driven technologies. ## legal systems in a nutshell legal systems revolve around legal subjects, institutions and actors, and sources of law. while there is variation across settings, the most relevant sources typically comprise a national constitution, primary legislation (often referred to as 'statutes', etc.), secondary 'executive' regulation, precedents decided by courts, and other auxiliary sources. a major division exists with regard to the role of precedents relative to written law, as well as the methodology of arguing with them. legal systems primarily influenced by continental europe follow the 'civil law' approach, where important decisions are mostly condensed into context-free interpretive rules to codified law that are compiled in secondary literature (e.g., so-called 'commentaries'). in parts of the world with primarily english legal influence, so-called 'common law' systems, precedents are regularly applied by means of analogizing and distinguishing arguments that take into account the facts of the case in much greater extent than civillaw-type reasoning will. international courts (e.g., the european court of justice, the european court of human rights) usually follow hybrid methodologies that are specific to the legal regime they govern. despite some recent diversification, virtually all ai & law research comes from either civil or common law backgrounds and makes corresponding assumptions, which is why we include this introduction. it is important to note that this coarse systematization is a great simplification of the world's diverse legal systems and cultures, and only intended to supplement our survey. written law generally consists of primary (i.e., parliamentary) legislation and secondary (i.e., executive) regulation. while enacted by two different branches of government, they are structurally similar in that they encode rules that can be formalized in if-then relations. they also contain ambiguous and vague formulations in need of interpretation, for which special methods exist that are beyond the scope of this work. it is typically up to the judiciary (i.e., the courts) to settle open questions through landmark cases, often after arguments being developed in academic literature. these decisions then become part of the discourse accordingly in the applicable methodology. this transition from rule-based to case-based reasoning (in the common law) has intuitively been termed "when the rules run out" by @xcite . when arguing a case relative to a precedent, it is a fundamental principle of justice that similar cases should be treated alike. in common law jurisdictions, this principle is formalised in the doctrine of stare decisis, which obliges decisions of the appropriate status to be followed when deciding a new case. civil law legal orders also recognize a binding effect of high court precedent, but argue with them differently. while higher court decisions bind lower courts, cases move in the opposite direction. they are first filed in, for example, district or trial courts, where evidence is heard and first decisions are made. decisions can then be appealed to the appeals courts, and eventually to supreme courts. at some point in this progression, arguments on evidence will be considered settled and only purely legal errors will be permissible grounds for further escalation. in a legal system, such 'appeals tracks' exist for various jurisdictions (civil, criminal, administrative, etc.) and can be spread across geographic entities (e.g., federal vs state courts). it is worthwhile to acknowledge that legal systems are inherently human-centric, involving complex decision-making processes where persuasion, interpretation, and subjective judgment play critical roles. legal decisions are not solely about determining which side should win as a matter of justice, but about who can present the most convincing argument within the framework of established laws, principles, and precedents. the main vision of ai & law is that state and private actors in all aspects of the legal system can benefit from supporting software that seamlessly connects to the concepts and concerns they have been trained for and work with. notably, legal reasoning not only happens in courts, but also in public administration and law enforcement (i.e., the executive branch), where law needs to be applied to specific situations (e.g., permits, taxes, public safety, etc.). human accountability is paramount for the trust in the overall workings of a democratically governed society. hence, this vision is one of ai supporting human decision makers and not replacing or unduly influencing it. ## knowledge-based approaches ai & law research started with modeling of legal reasoning by means of knowledge representation. rule-based approaches early landmark work demonstrated how british immigration law could be represented in prolog @xcite and outlined challenges faced in this process, including the law's rule-exception pattern, negation-asfailure (i.e., failure to prove true) vs. classical negation (logical, certain falseness), and counterfactual reasoning. @xcite developed a specialized language for rule-based legal inference. rules establish conclusions from antecedents in a forward/backward chaining manner, thereby spanning open a derivation tree of a case outcome. they justify a position and explain how a conclusion can be reached, but they do not capture the dialectical aspects associated with argumentation, since no conflicting arguments are generated and no indeterminacy is accounted for. @xcite extended by using augmented transition networks to model contract formation over time given agent actions with a basic form of uncertainty -if a condition was a 'hard question' and could not be decided, the network would fork into two alternative ways to legally treat the facts. overall, early rule-based systems were still predominantly derivations rather than argumentation models, although they correspond well to how lawyers analyze cases. ## data-driven approaches relationship to political science research: data originating in the legal system has been the subject of extensive analytical study in the field of empirical legal studies, including court and judge decision/voting behavior (e.g., @xcite @xcite . as most of them neither model legal reasoning nor apply nlp techniques, we do not include them in our survey. early ai & law: knowledge-centered approaches can achieve high degrees of faithfulness in their representation and explainability in their inferences, but face the 'knowledge acquisition bottleneck', as they require large amounts of expertise and modeling effort. this is in contrast to data-driven models with less hand-crafted expertise. early works by @xcite used nearest-neighbor methods for outcome classification. in the 1990's, pannu 1995; @xcite @xcite trained neural networks to predict outcomes and derive input feature weights. unsurprisingly, such early applications of ml attracted criticism @xcite . obtaining substantial amounts of processable data was challenging and extensive feature engineering was necessary. these works focused on the application of neural networks to identify how influential certain information is for the decision and did not engage in comparative benchmarking. towards modern legal nlp: recent years saw a resurging interest in case prediction through the use of data-driven methods learning from the large datasets now available from different jurisdictions, such as the ecthr @xcite @xcite @xcite @xcite @xcite ; says, 2020) chinese criminal courts @xcite @xcite @xcite , , us supreme court @xcite , indian courts @xcite french court of cassation @xcite ) supreme court of switzerland @xcite , turkish constitutional court @xcite , uk courts (strickson and de la iglesia, 2020), german courts @xcite , brazilian courts @xcite and philippine courts @xcite . earlier works employed bag-of-words features @xcite @xcite @xcite . more recent approaches use deep learning techniques @xcite @xcite involving convolutional or recurrent networks followed by adoption of pre-trained transformer models @xcite , including legal-domain specific pretrained variants @xcite @xcite @xcite @xcite . classification tasks on legal text interrelate, and so other words have leveraged dependencies between tasks for improving models @xcite @xcite @xcite @xcite @xcite and added additional loss constraints (such as contrastive learning exploiting label information), @xcite @xcite and injected legal knowledge @xcite @xcite @xcite overall, one can observe a trend towards applying nlp models to legal text with little to no architectural bias or explicit domain representation. these are then compared along quantitative metrics, typically with regard to high level classification/prediction goals (e.g., case outcome variables and document-level keywords) at the cost of interpretability. as berman & hafner have observed in the 1990s, however, case outcomes are highly contextual in time, procedure, and socio-legal purpose. classification benchmarks risk decoupling a sense of technical progress towards a notion of model 'understanding' from supporting a realistic task (e.g., legal argumentation) by focusing on a highly reductive representation of its outcome. for instance, case outcome predictions are often treated as binary targets based on the majority opinion, even though judges on the same bench frequently have conflicting reasoning, leading to dissenting or concurrent opinions @xcite . this reductive approach overlooks the nuanced legal argumentation underpinning each decision, focusing on a single outcome instead of capturing the depth of legal reasoning and debate. ## challenges & future directions combining knowledge and data: the pressing question is how best to integrate legal knowledge and ml so that a system can learn from data and still seamlessly interface to a lawyer's understanding of the domain by means of a conceptual representation. a number of such hybrid systems can be found outside of nlp: split up @xcite combined expert-crafted rules and neural networks trained from data in a factor-based model of australian family law to predict divorce asset division. in the cato line of work, both agatha @xcite ) and vjap @xcite leveraged structured legal argumentation for prediction with signals derived from a case base. moving to nlp, one intuitive combination is to ascribe factors from cases using text processing and proceed with formalized legal inference. this was employed in scale @xcite to enable a logic model to predict wipo domain name disputes, and in the ecthr domain by inference using an adf representation @xcite . @xcite automatically identified factors in fourth amendment auto stop cases, demonstrated their predictive value, and used ml techniques to explain case outcomes in terms legal professionals can understand. @xcite apply neural models to identify argument slots in legal provisions and find suitable filling elements from fact descriptions, thereby enabling rule-based inference. similarly, holzenberger and van durme (2023) automates the translation of cases into a knowledge base by posing it as an information extraction task. data utilized: ideally, outcome prediction systems in the legal domain should rely on the information available before proceedings start and legal conclusion are determined (e.g., argumentative memoranda from the parties). most case outcome classification research is conducted based on fact descriptions that are taken from judgments. these are often highly selective summaries tailored to align with the decision @xcite . although they may not explicitly contain outcomes, this can introduce confounding effects as demonstrated in @xcite . to illustrate the effect of proxy data on performance, @xcite utilized data from ecthr 'communicated cases', court-prepared summary data derived from applicant submissions, published before trial and observed a decline compared to facts statements from judgments, highlighting the need to select appropriate data for this task to draw reliable conclusions @xcite . such work may also be subject to data selection bias related to which cases reach which court, and with regard to how they are published. for example, a higher court will receive a different distribution of cases (i.e., such with grounds for appeal) than a district court, and only a subset of them may be published. finally, many cases are settled before or during trial, further skewing the dataset @xcite . temporal dynamics current legal nlp methods often operate under the implicit assumption that past training data is homogeneous and neglect its sequential nature. in reality, attitudes and case law change over time, with later cases altering and superseding the roles of older ones. all shifts in jurisprudence confront the model with a cold start problem of little training data for a new legal rule and copious training data for outdated ones. these dynamics can in principle be modeled. for instance, overruling detection can identify where previous legal precedents have been overturned, and trigger techniques such as model unlearning (see @xcite or selective forgetting. one can also strive to detect updates in beliefs/knowledge expressed in decisions over time, and modify such beliefs within the model @xcite . @xcite accounts for the temporally evolving nature of classification tasks on legal data using continual learning approaches.overall, however, the temporal dynamics of legal corpora remain largely unaddressed in recent works. ## conclusions we believe that knowledge-based approaches to building legal argument support systems deserve the attention of the modern nlp community, as they embody a culture and method of capturing intricacies of legal systems and argumentation that are often simplified away in the increasingly easier application of large mainstream models to legal data. the prominent role of benchmarks compounds this by drawing attention towards quantitative progress instead of real, empirical investigations of downstream benefit to practitioners. at the same time, llms widen the knowledge acquisition bottleneck for structured models considerably, opening up new opportunities. we believe there is great value in combining knowledge-and datadriven systems rather than continuing the assumption that deep expertise will reliably emerge given large enough amounts of data and computation.