A Comparative Study of Prompting Strategies for Legal Text Classification (2023)

Abstract: In this study, we explore the performance oflarge language models (LLMs) using differ-ent prompt engineering approaches in the con-text of legal text classification. Prior researchhas demonstrated that various prompting tech-niques can improve the performance of a di-verse array of tasks done by LLMs. However,in this research, we observe that professionaldocuments, and in particular legal documents,pose unique challenges for LLMs. We experi-ment with several LLMs and various promptingtechniques, including zero/few-shot prompting,prompt ensembling, chain-of-thought, and ac-tivation fine-tuning and compare the perfor-mance on legal datasets. Although the newgeneration of LLMs and prompt optimizationtechniques have been shown to improve gener-ation and understanding of generic tasks, ourfindings suggest that such improvements maynot readily transfer to other domains. Specifi-cally, experiments indicate that not all prompt-ing approaches and models are well-suited forthe legal domain which involves complexitiessuch as long documents and domain-specificlanguage.

## introduction large language models (llms) have gained significant popularity recently due to their remarkable versatility across a wide range of natural language processing tasks. these models, such as openai's gpt-3 @xcite ) and facebook's opt @xcite , have exhibited notable ability to generate coherent and fluent text, resulting in their adaptation in various domains @xcite @xcite . in addition, through prompting, llms have demonstrated significant generalizability @xcite . prompt engineering refers to techniques used to steer llm generations by providing specific * these authors contributed equally to this work. previous research has investigated the potential of various prompting techniques, such as zero-and few-shot prompting @xcite , prompt ensembling @xcite , chain-of-thought @xcite , and activation fine-tuning @xcite , in enhancing the performance of llms across different tasks, including textclassification. in addition, text-classification is a central focus of parameter efficient finetuning (peft) research @xcite . however, these studies consider general nlp settings rather than the legal domain. in this paper, we explore various prompting approaches to utilize pre-trained llms for legal document classification. the datasets considered here represent a strong test of llms' out-of-domain generalization. we begin with zero-shot learning and compare with more advanced approaches such as prompt ensembling and chain-of-thought. our findings indicate that legal documents present unique challenges for llms, such as complex terminologies and long documents, that hinder the transferability of prompting techniques to this domain. recently, @xcite observed similar findings in leveraging llms for medical text. ## results and discussion in all cases, we observe notable improvements going from zero-shot to few-shot prompting, which implies that more context and task demonstration are quite helpful. however, performance remains well below the supervised approaches. this suggests that the opt models, despite being large, general purpose language models with similar capabilities to gpt-3, struggle to perform the specialized legal tasks considered here without fine-tuning or domain-adaptation techniques. results comparing question-answering to completion-style prompts for scotus are shown in appendix c, furthermore, we note that performance for cot is lower than zero-shot prompting. we find that the opt output/answer generations are not of high quality, and the generations actually add noise to the prompt, likely causing the degradation. this is an interesting finding as it illustrates that cot improves performance only in cases where the llm is capable of generating relevant reasoning. there is a growing body of evidence that llms whose pre-training corpus meaningfully includes code are ## conclusions in this research, we performed a comparative study of different prompting techniques to assess their utility in the legal domain. the considered prompting techniques are generally helpful, but their applicability to specialized domains requires further exploration, in line with findings for medical texts @xcite . among the approaches tested, activation finetuning improves performance the most, especially when accompanied by a few-show prompt, even with limited labeled data. for opt175b, this approach improves the macro-f1 score zero-shot baselines by 35 and 47 points for echr and scotus, respectively. of the pure prompting approaches, few-shot prompts with and without ensembling perform best, while cot prompting does not work well in this context. in future work, we plan to explore instructiontuned llms, such as opt-iml @xcite , and experiment with peft methods @xcite . we are also interested in assessing whether current llms exhibit notable biases in their legal decision processes with respect to protected groups @xcite . finally, we aim to explore how recent advances in reasoning-based prompting can be used to produce good task performance and more interpretable responses @xcite .