Chain of Logic: Rule-Based Reasoning with Large Language Models (2024)

Abstract: Rule-based reasoning, a fundamental type of legal reasoning, enables us to draw conclusions by accurately applying a rule to a set of facts. We explore causal language models as rule-based reasoners, specifically with respect to compositional rules - rules consisting of multiple elements which form a complex logical expression. Reasoning about compositional rules is challenging because it requires multiple reasoning steps, and attending to the logical relationships between elements. We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to resolve the underlying logical expression). This method was inspired by the IRAC (Issue, Rule, Application, Conclusion) framework, a sequential reasoning approach used by lawyers. We evaluate chain of logic across eight rule-based reasoning tasks involving three distinct compositional rules from the LegalBench benchmark and demonstrate it consistently outperforms other prompting methods, including chain of thought and self-ask, using open-source and commercial language models.

## introduction the surge in reasoning capabilities of language models (lms) has the potential to transform the legal industry, a sector inherently reliant on sophisticated reasoning. the development of models that can assist and validate the reasoning processes of legal practitioners promises to usher in a new legal era. such advancements would not only enhance the efficiency of legal services but also expand the capacity of professionals to service more clients, thereby broadening access to justice. at present, language models are prone to hallucinations in a legal setting @xcite , and even powerful lms like gpt-4 struggle to perform basic legal tasks @xcite . color coding is used to identify rule elements and illustrate how these elements form a complex logical expression. reasoning about compositional rules requires not only correctly applying each element to a fact pattern, but also resolving the logical expression. if the logical expression evaluates to true, it triggers a consequence (personal jurisdiction exists). legal tasks typically require sophisticated rulebased reasoning. these rules are written in natural language and expressed in many forms, including statutes, judicial holdings and even contract provisions. similar to an if/then statement, a rule has an antecedent (a condition that can be evaluated to true or false) and a consequent (the outcome triggered if the antecedent is satisfied). even the earliest recorded laws, written in sumerian on clay tablets, have used this conditional structure @xcite . rule-based reasoning allows us to draw conclusions by applying a rule to a set of facts to determine if these preconditions are satisfied. for example, if parking is prohibited (consequent) between 2pm and 4pm (antecedent), and we know it is currently 3pm, then we can conclude that parking is currently prohibited. often rules, especially complex rules, are compositional in nature meaning the antecedent consists of multiple conditions joined by and and or operators forming a complex logical expression (see figure 1 ). in law, these constituent conditions are called rule elements. recent work demonstrates that models can struggle to reason about even basic compositional rules @xcite . we explore large language models as rule-based reasoners, evaluating several common in-context learning methods across 8 tasks and 3 distinct rules from legalbench, a collaboratively constructed legal reasoning benchmark @xcite . legalbench tasks were designed by subject matter experts to evaluate useful legal reasoning skills. prior work on legalbench has focused on evaluating the capacity of models to perform rule-based reasoning given in-context demonstrations of the same rule being applied to varying fact patterns. these few-shot prompts are valuable in law where annotated data is scarce, limiting the ability to finetune models for specific legal tasks. yet requiring several reasoning examples for each rule in a real world setting poses a challenge in terms of both cost and scalability. additionally, a same-rule setting allows the model to largely mimic the reasoning path of the in-context examples. in contrast, a different-rule setting enables us to examine a model's ability to dynamically generate this reasoning path -the same way a lawyer would reason. our work shifts focus to evaluating and improving a model's ability to perform rule-based reasoning given only a single in-context demonstration of a different rule application, thus removing the need to store examples for each rule. in this work, we introduce chain of logic, a new prompting approach to guide lms in reasoning about compositional rules by decomposing rule elements into a series of logical statements (evaluates to true or false), before explicitly recomposing these element answers to resolve the underlying logical expression. understanding natural language questions requires understanding how to break down a question into a set of steps yielding an answer @xcite . the inspiration for our method is rooted in the irac framework, an approach to legal reasoning where lawyers break down the reasoning process into four sequential steps: issue spotting, rule identification, rule application and conclusion. during the rule application step, rules are typically divided into their elements and addressed separately before recombining these threads to arrive at a conclusion. in devising our chain of logic approach, our interdisciplinary team combined this domain knowledge with a variety of insights from prior work on in-context learning. our experiments show chain of logic outperforms other prompting methods, including chain of thought @xcite and self-ask @xcite , across a variety of rule-based reasoning tasks using both open-source and commercial models. we demonstrate that given a single example of chain of logic, a model can learn to generalize this approach to a different rule and fact pattern, and thereby improve its reasoning capabilities. ## background large language models have demonstrated strong capabilities as zero-shot @xcite and few-shot (brown et al., 2020) reasoners. chain of thought prompting enhanced problem solving performance even further by eliciting models to produce intermediate reasoning steps @xcite . this approach proved effective at solving complex tasks, including arithmetic and commonsense reasoning tasks. many other prompting methods followed with a similar aim of decomposing a complex task into a sequence of simpler subtasks. self-ask improved multi-hop question answering performance beyond chain of thought prompting by guiding the model to explicitly pose and answer intermediate questions @xcite . similar to decomposing a multi-hop question into a sequence of intermediate questions, compositional rules can be decomposed into rule elements. however, reasoning over these rules requires not only answering each element correctly, but also understanding the logical relationships that exists between these elements. for example, if a rule with two elements resolves to (true, false), then the final answer depends on whether the boolean relationship between those elements is and or or. in section x, we show existing prompting methods can resolve rule elements correctly while still getting the final answer incorrect. this motivates the need for a more robust prompting method which attends to both the element level answers and the underlying logical structure. further motivating this work, zero-shot methods have been observed outperforming oneshot and few-shot approaches on legal reasoning tasks @xcite , suggesting models can struggle with in-context learning in a legal setting. figure 2 : comparing one-shot examples demonstrating chain of thought, self-ask and our chain of logic method on the personal jurisdiction task. through a sequence of reasoning steps, chain of logic decomposes a rule into elements which are solved independently, before recomposing sub-answers to arrive at a final conclusion. see section 3 for a detailed discussion on the chain of logic approach. ## chain of logic we introduce chain of logic, a new prompting approach to elicit rule-based reasoning in lms through a series of instructive reasoning steps. each step in this series helps inform the next, and enables the model to unravel the many reasoning tasks needed to arrive at the right conclusion. our method builds on chain of thought and self-ask by not only considering problem decomposition, but also attending to relationships between subtasks that can dictate the final answer, which we later show improves performance. our prompt begins with a single example demonstrating the chain of logic approach, appended with the inference-time problem. figure 2 contains a one-shot example of our approach depicting the sequence of reasoning steps which we describe below: during inference, the model performs each step in this process automatically given only a one-shot example and the inference task inputs (formatted as described in step 1). since the one-shot task inputs (rule, facts, issue) are different from the task inputs being tested, the model is reasoning on its own at each step, including deciding how many rule elements exist, the text span of each element and the logical relationships between them. this more closely resembles a real world setting where lawyers apply the irac framework to a rule and fact pattern being observed for the first time. the intermediate steps in our approach provide a detailed accounting of each factor contributing to the final conclusion (see figure 3 for an example of model output). this allows for transparent and explainable decision-making, which is particularly vital in the legal domain where the justification for an answer can be as important as the answer itself. chain of logic also enables us to debug inaccurate conclusions by retracing the reasoning path to locate errors in rule application and logic. ## experiments we conduct a series of experiments to compare our proposed chain of logic approach with existing prompting methods, and to evaluate the rule-based reasoning capabilities of lms in a different-rule setting. while we sometimes observe an inverse relationship between prompt complexity and performance (simpler prompts perform better), we find that chain of logic consistently outperforms other prompting methods across a range of rule-based reasoning tasks. ## results and discussion for each model, chain of logic outperforms all baseline methods, including zero-shot methods. the performance gap between chain of logic and the best performing baseline is 3.9% on average. this gap is smaller for open source models (0.3% and 0.4%) and larger for commercial models, with gpt-3.5 having the largest performance improvement of 10.7%. for this work we observe a large performance difference between the commercial and open source models, and hypothesize the performance gains may be larger for the stronger commercial models which can more easily follow the longer prompts. this can also be seen in the performance differences between the baseline zero and one-shot methods. for gpt-4 we believe the absolute performance gain may be lower due to ceiling effects of the model. the specialized zero-shot legal baselines demonstrate comparable performance to zeroshot for some models, but are also inconsistent. lr prompting slightly outperforms zero-shot using mistral-7b and gpt-4, but also underperforms by 19.1% for gpt-3.5. the one-shot baselines demonstrate the challenges to in-context learning in this setting by consistently underperforming zero-shot methods. the model output depicted in figures 3 and 5 illustrate common errors we observe in rule-based reasoning using chain of thought and self-ask. first, chain of thought often generates fluent but illogical reasoning paths. the output in figure 5 shows the model incorrectly conclude that personal jurisdiction does not exist because dustin plans to move to new york, even though future residency is not an element of the rule. second, the question/answer pairs generated by the self-ask approach commonly decompose the rule incorrectly (figure 3 ) or incompletely (figure 5 ). the output in the latter figure shows the model only asks intermediate questions for two out of three rule elements, omitting whether the defendant had sufficient contact with the forum state. chain of logic prevents these omissions by first restating the rule elements (step 2 in the approach), similar to how system 2 attention (weston and sukhbaatar, 2023) first restates only the relevant portions of a context before attending to a task. in the ablation study, we show that removing this step significantly decreases performance. last, figure 5 also demonstrates how not recognizing logical relationships can lead to an incorrect solution, even when sub-questions are answered correctly. the model correctly determines the defendant is domiciled in the forum state, which is sufficient to conclude personal jurisdiction exists, but still reaches the wrong final answer. ## related work there has been extensive work on improving the problem solving capabilities of lms by enabling a model to use more computation for more complex tasks. graves introduces an adaptive algorithm for dynamically selecting the number of computation steps a recurrent neural network should take based on the task complexity @xcite . ling et al. first demonstrated the use of answer rationales, natural language explanations generated before the solution, to solve algebraic word problems @xcite . several approaches followed this work in exploring the use of intermediate reasoning steps @xcite @xcite . more similar to our work, other methods have explored decomposing compositional questions into simpler sub-questions using supervised models @xcite @xcite and prompting techniques @xcite . talmor and berant also recognized the importance of explicitly identifying relationships between sub-questions @xcite . the authors train a supervised model to translate questions into a computation tree and answer sub-questions by querying a search engine. we extend these works with our decomposition-recomposition approach, exploring simple questions involving compositional rules where the solution depends not only on rule element answers, but also the logical relationships between elements. recently, agent-based systems driven by lms have demonstrated strong compositional reasoning capabilities through task planning and program execution @xcite @xcite . our method is not agentbased -instead of performing many forward passes across an assortment of specialized models and programs, our method demonstrates similar decomposition/recomposition with only a single generation pass on one model. additionally, our approach does not require fine-tuning which is particularly important in the legal domain where the scarcity of annotated data can be prohibitive. prior work in natural language processing has explored a broad range of legal tasks, including clause classification @xcite and recommendation @xcite , contract summarization @xcite , legal judgment prediction @xcite and even answering bar exam questions (au2 and katz, 2022; @xcite . rule-based reasoning can involve rules from many sources. saeidi et al. explored conversational question answering where the solution required rule-based reasoning about a collection of regulations @xcite . servantez et al. captured rules in contract text through graph-based extraction and converted them into code @xcite . similar to our work, holzenberger and van durme introduced an approach to reasoning about tax statutes by decomposing the reasoning process @xcite . this approach first extracts key arguments (entities, dates, dollar amounts) from the statute text and fact pattern using fine-tuned bert models, before arriving at a final answer. we do not explore this method since it requires a fine-tuned model and does not easily translate to the tasks in this work. ## limitations and future work while legalbench tasks were crafted by subject matter experts, the rules have been simplified to ensure answers are objectively correct. this greatly improves evaluation, but also means the scope of complexity is narrower than many real world rules. additionally, our current approach only addresses rules where the solution is based on whether the antecedent has been triggered. real world rules can contain complex consequences which themselves require some form of reasoning. for example, calculating tax liability after determining the applicable tax bracket. in future work, we hope to build on our current approach in several directions. first, for simplicity chain of logic performs all reasoning step in a single forward pass. however, a multiple pass approach would allow us to incorporate other reasoning tasks like rule identification which is not addressed here. this moves us toward more real world scenarios where the applicable rule is not known a priori. second, investigating whether rule-based reasoning could be improved further by dynamically sampling multiple reasoning paths. and third, incorporating retrieval augmented generation @xcite by allowing the model to access external sources of knowledge like term definitions. this is particularly useful in the legal domain where terms or concepts have a distinct meaning based on the jurisdiction or contract. ## conclusion in this work, we explore causal language models as rule-based reasoners, and show these models can have difficulty learning from in-context demonstrations using common prompting approaches. we present a new prompting method, chain of logic, to elicit rule-based reasoning in language models through decomposition and recomposition. our experiments show chain of logic consistently outperforms other prompting methods, including chain of thought and self-ask, across a variety of rule-based reasoning tasks using both open-source and commercial language models. we show how chain of logic creates a coherent and interpretable reasoning path by unraveling the many reasoning steps re-quired by compositional rules. these findings may also be a useful signal for future instruction tuning of language models to imbue them with reasoning. by enhancing the rule-based reasoning capabilities of lms through in-context learning, chain of logic also reduces the need for annotated legal data, which has historically been a bottleneck for the legal domain.