Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks (2023)

Abstract: The use of large language models (LLMs) for zero- or few-shot prompting in natural language processing has given rise to a new research area known as prompt engineering. Recent studies have demonstrated that Chain-of-Thought (CoT) prompts can lead to significant improvements in tasks such as arithmetic and common-sense reasoning. This paper explores the use of such approaches in legal reasoning tasks by conducting experiments on the COLIEE entailment task, which is based on the Japanese Bar exam. We evaluate zero-shot/few-shot and fine-tuning approaches with and without explanations, as well as various prompting strategies. Our results indicate that while CoT prompting and fine-tuning with explanations can improve performance, the best results are achieved with prompts derived from specific legal reasoning techniques, such as IRAC (Issue, Rule, Application, Conclusion). In addition, we observe that few-shot learning where the demonstrations are derived from clustering past training data consistently yields high performance on the COLIEE entailment task for both the years of the data that we tested. Through our experiments, we improve the previous best result on the 2021 COLIEE task from 0.7037 to 0.8025 and surpass the best system from 2022 with an accuracy of 0.789.

## introduction application of reason-based prompting mechanisms with large language models is becoming an increasingly prevalent focus area in natural language processing research. models like openai's gpt-3 achieve satisfactory results -81% accuracy with a zero-shot approach and 82.8% accuracy with a few-shot approach -on commonsense reasoning tasks drawn from the physicalqa dataset @xcite , but, as we observe, struggle significantly with more specialized domain data. further research builds on these basic queries by implementing a variety of so-called prompt engineering approaches, which range from soliciting a model * work done while interning at thomson reuters labs. to "think step by step" in producing incremental reasoning to underscore a given response @xcite , to leveraging an iterative cycle of model generated rationales to bootstrap its own ability to produce more elaborate reasoning approaches @xcite ; these approaches have demonstrated a nominal improvement in a language model's ability to rationalize a correct response to a particular baseline query. our research aims to explore the effects of such approaches on highly specialized domain data, namely that in the legal field drawn from the japanese bar exam. 1 we frame our approach around the university of alberta's annual competition on legal information extraction/entailment, coliee @xcite , in which certain subtasks are devoted to reasoning through legal hypotheses given contextual articles. we first explore zero through few-shot approaches using pretrained llms, coupled with prompts either drawn from existing work (e.g., "let's think step by step" @xcite ) or generated by ourselves (e.g., "please determine if the following hypothesis is true or false based on the given premise"). additionally, we assess the impacts of fine tuning an llm to infer binary responses both with and without explanations (either machine generated or extracted from the supporting premise). the best results on the coliee 2021 test set, though, result from a zero-shot, legal-prompt approach, which surpasses state-of-the-art coliee performance by 14.04%. these legal-prompt approaches are motivated by legal reasoning techniques such as issue, rule, application, conclusion @xcite taught at law school. similarly, an 8-shot approach with shots obtained by clustering coliee training data yields the best performance on 2022 coliee test data, with an overall accuracy improvement of 9.5%. our experiments indicate that few-shot and finetuning with explanation approaches show good and consistent results for the two test sets of the col-iee competition we used for evaluation. zero-shot and a fine-tuning approach using the labels show more inconsistent results across the two years. the zero-shot with legal reasoning approach shows the best result for one year only and may be more prone to overfitting to a specific test set indicating that further research on those prompting approaches is needed. the coliee competition @xcite has been carried out since 2014 driving research in the area of legal retrieval and entailment. two of the competition's tasks are using data from the japanese bar exam. the exam requires lawyers to determine whether a given legal statement is true or false. in order to answer the questions, the lawyer has to first determine which articles of the japanese statutes are most relevant to the given question. task 3 of the competition covers this retrieval task. given one or more articles relevant to the question, the lawyer has then to determine whether the question (i.e., the hypothesis) is true or false given the selected articles (i.e., the premise). this task is captured as task 4 in the coliee competition and we are focusing on this task with our work on legal reasoning: hypothesis: if the grounds of commencement of assistance cease to exist, the family court may rescind the decision for commencement of assistance without any party's request. premise: article 18 (1) if the grounds prescribed in the main clause of article 15, paragraph (1) cease to exist, the family court must rescind the decision for commencement of assistance at the request of the person in question, that person's spouse, that person's relative within the fourth degree of kinship, the guardian of a minor, the supervisor of a minor's guardian, the assistant, the assistant's supervisor, or a public prosecutor. entailment: no more formally the task is defined by the organizers as a legal entailment task: given a question q, after retrieving relevant articles s 1 , s 2 , ..., s n determine if the relevant articles entail "q" or "not q". the answer of this task is binary: "yes" ("q") or "no" ("¬q"). the evaluation metric is accuracy and a random baseline (or simply giving always "yes" or always "no") would lead to an accuracy about 0.5 since most test sets have an approximate equal distribution of positive and negative answers. one of the main challenges of this competition is the relatively small size of the training and test set. past competitions provide the answered questions of previous competitions resulting in a total of 806 questions. the 2021 test set contain 81 questions whereas the 2022 test set contained 109 questions. the coliee datasets are accessible upon requesting the organizers. ## prior work an overview of all coliee tasks and their respective approaches can be found in @xcite . most systems addressing task 4 were bertbased @xcite despite the small training set. interestingly enough, the best performing system in 2021 utilized a method for increasing the training pool and deployed an ensemble of different bert-based systems @xcite . other systems relied on different types on language models including distilroberta @xcite , legal-bert @xcite , t5 @xcite and electra @xcite (i.e., @xcite ) other notable past systems used the japanese original text and developed a rule-based system to identify entailment @xcite while another system was based on a graph neural network using distilroberta and legal-bert embeddings as nodes @xcite . only one system @xcite ) looked into a few-shot learning approach utilizing @xcite . the system was based on a few-shot approach addressing task 3 and task 4 together foot_2 . their experiments showed that gpt-3 without any further fine-tuning and without prompting of the relevant articles from the japanese statues does not perform very well on this task reaching results even below the random baseline. these results indicate that even large lms have not stored enough knowledge about japanese law and more advanced prompting and/or fine-tuning techniques are required. other prior work we draw from explores the utility of prompting and explanations in order to have llm solve more complex tasks such as commonsense reasoning or mathematical word problems. we draw from recent work that proposed so-called chain-of-thought (cot) prompts that have shown improved results of zero-shot (zs) and few-shot (fs) approaches @xcite @xcite . previous approaches to incorporating explanations into the reasoning process also show improvements over standard prompting or fine-tuning approaches. by relying on the generation capability of llms to produce reasoning text, those approaches produce text that then is used to finetune the language model. in contrast to earlier approaches that relied on manually created explanations @xcite , @xcite propose a method that automatically creates rationales using gpt-j for arithmetic word problems, commonsense qa, and grad school math problems. their star system keeps the rationales if the llm produces the correct answer and creates a new set of training data they train the lm with. they show significant improvements over previous approaches that are fine-tuned on the answers only and comparable performance to a much larger lm (i.e., gpt-3) on commonsenseqa. ## experiments and results this section outlines the various approaches we explored using an llm such as gpt-3.5 to address the coliee task. our investigation included basic fine-tuning, instruction fine-tuning, n-shot learning, few-shot learning with clustering, prompting with legal reasoning strategies, chain-of-thought prompting, and an attempt to solve this binary task using explanations generated by the llm itself. all experiments were performed using openai's latest model, gpt-3.5 (text-davinci-003), which is the most proficient gpt-3.5 engine among the four available at the time of experimentation (davinci, curie, babbage, and ada). for our experiments, we utilized the collie datasets from 2021 and 2022. we chose accuracy as the evaluation metric due to the approximately equal distribution of positive and negative labels in the coliee test sets.