Team UTSA - NLP at S em E val 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT 4 (2024)

Abstract: In this paper, we present our system for the SemEval Task 5, The Legal Argument Reasoning Task in Civil Procedure Challenge. Legal argument reasoning is an essential skill that all law students must master. Moreover, it is important to develop natural language processing solutions that can reason about a question given terse domain-specific contextual information. Our system explores a prompt-based solution using GPT4 to reason over legal arguments. We also evaluate an ensemble of prompting strategies, including chain-of-thought reasoning and in-context learning. Overall, our system results in a Macro F1 of .8095 on the validation dataset and .7315 (5th out of 21 teams) on the final test set. Code for this project is available at https://github.com/danschumac1/CivilPromptReasoningGPT4.

## introduction mastering the reasoning behind legal arguments is a fundamental skill required of all law students. in this study, we develop a novel approach for se-meval task 5, the legal argument reasoning task in civil procedure challenge @xcite . the semeval task released a dataset that was scraped from the glannon guide to civil procedure, a textbook designed for law students. specifically, given case law, a question, and a potential answer to that question, students must be able to reason over the contextual information (case law) to determine if the question is correct or not. there has been substantial research in developing nlp-based reasoning systems @xcite @xcite @xcite . the methods can be categorized into two major frameworks: fine-tuning and large language model-based (llm) solutions. for fine-tuning approaches, @xcite introduced an approach that fine-tunes legalbert @xcite and developed several methods for handling long text that does not fit within the token limitations of legalbert. for llm solutions, @xcite explored the use of chatgpt for solving legal exams. @xcite introduced a more comprehensive evaluation benchmark geared to large language models consisting of 162 tasks. many of their experiments show that gpt4 is one of the top performing approaches for legal reasoning across all language models, while flan-t5-xxl is the best open-source option. although showing substantial generalization is important, developing specific prompting strategies for different reasoning tasks can substantially improve performance. hence, this paper adds to existing literature on the exploration of prompting approaches in the legal domain. for this work, we adapt several prompting-based strategies to develop an llm-based solution for the shared task. specifically, we combine a retrieval system with in-context learning and chainof-thought reasoning. there are several studies showcasing the utility of in-context learning @xcite and chain-of-thought reasoning @xcite @xcite . moreover, there is work using both human-curated and machine-generated reasons. in this work, we focus on human-generated reasons for the training data, and machine-generated examples are only used at test times when human expert annotations are not provided. furthermore, rather than providing a step-by-step reasoning approach, which may not make sense in this context, our approach is more similar to single-step rationales @xcite , which simply provides a single step of reasoning for why an answer is correct or incorrect. in summary, this paper makes the following contributions to our solution for the semeval 2024 task 5 shared task: ## related work overall, there are three major areas of legal nlp: legal question-answering @xcite @xcite @xcite , judgment prediction @xcite @xcite , and corpus mining (e.g., summarization, text classification, information extraction, and retrievalrelated research) @xcite @xcite @xcite . there has also been some broad methodology work that is aimed at working on various legal tasks in general (e.g., legalbert @xcite ). the semeval task is most similar to questionanswering related research. in the domain of legal question-answering, recent research efforts are focused on creating new systems, developing evaluation criteria, and compiling datasets, considering the significant variation across different legal fields. @xcite introduced a commercial question-answering system for legal inquiries, leveraging information retrieval techniques, sparse vector search, embeddings, and a bert-based reranking system, trained on both general and legal domain data. @xcite developed a novel evaluation method for llm-generated texts that assess their validity using retrieval-augmented generation, showing improved alignment with legal experts' assessments and effectiveness in identifying factual errors. wang et al. (2023) created the merger agreement understanding dataset (maud), a unique, expert-annotated dataset for legal text reading comprehension, highlighting promising model performance and the need for further improvement in understanding complex legal documents. from a methodological point-of-view instead of a task-oriented view, recent research efforts have concentrated on the advancement of nlp-based reasoning systems, with a particular focus on applications within the legal domain @xcite @xcite @xcite . these efforts can be broadly classified into two distinct methodologies: fine-tuning approaches and those leveraging large language models (llms). within the fine-tuning paradigm, @xcite have proposed modifications to legalbert @xcite aimed at enhancing its ability to process texts that exceed the model's inherent token limitations. this approach is representative of a broader trend towards tailoring pre-existing models to better suit specific textual analysis tasks in the legal sector. kien et al. (2020) developed a retrieval-based model employing neural attentive text representation with convolutional neural networks and attention mechanisms for accurately matching legal questions to relevant articles, demonstrating superior performance on a vietnamese legal question dataset. conversely, the exploration of llms has also been wide covering new general approaches for legal question answering and reasoning to new datasets and benchmarks. @xcite investigated the impact of chain-of-thought prompts and fine-tuning methods on legal reasoning tasks, specifically the coliee entailment task, and found that prompts based on legal reasoning techniques and few-shot learning with clustered training data significantly enhance performance. chalkidis (2023) demonstrates the potential of utilizing models like chatgpt for complex reasoning tasks, such as solving legal examination questions. building on this, @xcite have introduced a comprehensive evaluation framework designed specifically for assessing the capabilities of llms across a suite of 162 tasks. this benchmark aims to provide a more nuanced understanding of the strengths and limitations of llms in the context of legal reasoning. additionally, while employed within a distinct domain from legal reasoning, a comparable methodology in prompt engineering-encompassing chain of thought prompting and in-context prompting-is illustrated in liu et al.'s recent work @xcite . overall, compared to the prior work that developed methods and datasets for generating answers to questions, the semeval task focuses on understanding whether a provided answer candidate is valid given a specific context. ## method we provide a high-level overview of our approach in figure 1 . overall, we explore three major different prompting approaches: zero-shot prompting, few-shot prompting, and few-shot prompting with chain-of-thought-like reasoning. moreover, we explore an ensemble of multiple approaches. the methods are described in the following subsections. ## results in this section, we present our own results on the validation dataset as well as the final results in the competition. baselines in our experiments, using the validation data, we compare our approach (few-shot & cot & rag) with the other approaches used in the ensemble. moreover, we compare our system to both roberta @xcite and legal-bert. however, because both roberta and legal-bert are limited to 512 tokens, we split the introduction into b pieces. b is calculated by subtracting the number of words in the question and answer from 512. we halve that result to accommodate the fact that the number of tokens typically exceeds the number of words. next, we divide the number of words in the introduction by the previously calculated number to obtain the total number of windows. finally, all of the tokens in the introduction are evenly split into the windows. each piece of the introduction is appended to the question and answer pair independently to generate multiple predictions for each instance. we then use voting to make a final prediction for the entire sequence. this method is similar to what was explored in @xcite . validation results. the validation performances are shown in next, between gpt4 methods (not including the ensemble), we find that performance varies substantially between .6681 and .7162 for macro f1. all methods outperform zero-shot. however, zero-shot & cot achieved the best performance across all baseline methods for macro-f1. when we compare the baseline approaches to our method (few-shot & cot & rag), we find that the method outperforms all variations. from an ablation standpoint, removing rag has the biggest performance drop (.7306 vs. .6762). interestingly, we find that removing cot has the second largest drop in performance (.7306 vs. .6935), and removing the incontext examples has the smallest drop in performance (.7306 vs. .7162). before the study, we expected removing the in-context examples would result in the largest performance drop. ## conclusion in this paper, we described our approach for 2024 semeval task 4, the legal argument reasoning task in civil procedures. specifically, we introduced a gpt4 prompting-based strategy that achieved 5th place in the competition out of 21 participants. overall, we find that combining incontext learning, where we use a retrieval-based approach to find relevant examples, as well as in-context learning improves model performance. based on our experiments, there are three natural areas for future research. first, the actual analysis section used for chain-of-thought reasoning does not match traditional methods which use step-bystep reasoning. hence, a logical next extension is to reword (potentially with gpt4) the analysis section to provide a step-by-step explanation for an answer. second, this work was limited to 2 incontext examples to limit api costs and allow us to test other models in our initial experiments. however, extending that to 10 or more examples can potentially improve performance. third, the current approach relies on a closed-source model (gpt4). exploring open-source models, particularly smaller open-source models such as t5 @xcite and llama2 @xcite , is important to better understand the impact of pretraining data on performance.