Can C hat GPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer? (2023)

Abstract: Large Language Models (LLMs), such as ChatGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. ChatGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of ChatGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.

## introduction since chatgpt was released by openai in november 2022, there are fast-growing interests in applying llms to analyzing legal documents @xcite and reasoning tasks @xcite . although the recently released llms demonstrate strong abilities to solve challenging tasks requiring reasoning, people find that they often follow different, or even wrong reasoning paths to obtain correct answers @xcite . this issue is also referred to as a misalignment problem between llms and humans. as this problem has not been investigated in the legal domain, this work focuses on understanding to what degree chatgpt is able to perform reasoning for legal scenario analysis in the same way as legal professionals. herein, we chose irac @xcite , standing for issue, rule, application, and conclusion, as the framework for legal analysis because it is the most popular analysis methodology used by legal professionals and law schools. a typical example of irac analysis is depicted in fig. 1 . given a scenario regarding the contract act malaysia (cam), the task is to answer the legal question recognized as an issue in i. in this example, the rules (r) are the relevant statutes in cam. in a common law system, rules may also include precedents. the analysis or application (a) consists of the reasoning paths leading to the answer in the conclusion (c). herein, a reasoning path is a sequence of reasoning steps, where each step involves taking a statement from the available facts in a given scenario and then applying a relevant rule to it. section 2 describes irac in details and its importance in legal reasoning. interpretability of model outputs is crucial for legal professionals in real-world legal applications @xcite . for scenario analysis using irac, it is helpful for models to produce individual reasoning paths and their associated rules or precedents so that legal professionals are able to understand why models draw certain conclusions. moreover, it is noteworthy that legal reasoning is defeasible such that conclusions can be overturned by considering new evidence or making different assumptions due to missing information @xcite . however, prior legal datasets do not include intermediate reasoning paths understandable by legal professionals and neglect the aspect of defeasible reasoning. to address the above issues, we constructed the first semi-structured irac corpus, coined sirac foot_0 , which includes a set of legal scenarios pertinent to cam and australian social act (asa) respectively. to make irac analysis understandable by both llms and legal professionals, we proposed a semi-structured language and used it to codify an irac analysis for each scenario. as all scenarios are analyzed by senior law students with irac, we conducted detailed comparative studies between their results and the ones produced by chatgpt, by applying different prompting and in-context learning techniques @xcite . as our scenarios are complex and involve 7.05 reasoning paths on average, we also decomposed the legal question of a scenario into multiple simpler questions and instructed chatgpt to address them separately. as a result, we obtained the following novel findings via extensive analysis conducted by the law students: ## background a legal problem could be as simple as ascertaining the amount of money due from a tenant to the landlord under a contract affected by a certain unseen event like covid-19, or a more complex one involving contracts between large corporate entities negotiating cross-border sales and related matters. legal problems require unique reasoning skills to be solved. these unique skills are applied for solving legal problems in a rather systematic manner using the irac methodology @xcite . before one begins to solve a legal problem using the irac's legal reasoning process, it is essential to acquire factual details from the client. these facts will lead to the correct identification of the legal issue. given the identified issue, the next step is to determine the law that governs that aspect of the legal problem. the application of the law or the analysis in the third stage is perhaps the most detailed stage of irac. in this stage, the law is practically applied to the facts and the issues in question. as no two legal problems would be identical, one needs to be aware of possible variations. this stage is particularly important because it is here that legal reasoning skills are truly tested. the final stage of irac is the conclusion, which is a natural outcome of applying the law to the facts and issues. however, applying the same law to the same facts and issues could result in different conclusions, and the correct answer depends on the application of the appropriate law to the facts, which could be interpreted differently. ## dataset construction we constructed sirac to evaluate to what degree llms and the other ai models are able to conduct irac analysis in the same manner as legal professionals. our goals are to promote research on devising novel legal ai models that are i) equipped with strong defeasible reasoning capability, and ii) interpretable by both legal and computer science professionals. we start with selecting scenarios pertaining to cam and asa, followed by conducting full-fledged irac analysis on all scenarios using a proposed semi-structured language. the data quality is ensured by our strict procedures. ## assessment of chatgpt our preliminary results show that chatgpt is one of the best performing llms in legal reasoning. therefore, it was chosen to assess its ability in performing irac analysis. instead of evaluating only the final results of statutory reasoning by gpt3 @xcite , we employed chatgpt to perform full irac analysis, and the quality of its outputs was assessed by the law students in the respective sections: rule, application, and conclusion. issues were not evaluated because they are part of the inputs. moreover, we also carried out in-context learning and question decomposition, which have been found useful in other reasoning tasks @xcite , to improve the performance of chatgpt. ## related work in this section, we present related work, highlighting the similarities as well as the differences between theirs and ours. previous work related to our paper can be categorized into the following subsections. ## conclusion we constructed a novel dataset, coined sirac, to evaluate the ability of llms for irac analysis. sirac contains 50 legal scenarios pertinent to asa and cam. every scenario was annotated with reasoning paths with an average length of 7.05, described by our proposed semi-structured language. sirac is not only useful for the studies and analysis by legal professionals, but offers a fertile pool of resources for legal nlp. our evaluation results on chatgpt show that the powerful llms can produce reasonable answers but mostly fail to yield correct reasoning paths aligned with legal experts. its performance can be further improved by providing parts of the annotated reasoning paths, including similar annotated scenarios for in-context learning and decomposing complex issues into simpler questions. as those techniques are not specific to the investigated areas of law, it is desirable to understand the extend to which such empirical findings still hold in the other legal areas in the future work.