HW - TSC at S em E val-2024 Task 5: Self-Eval? A Confident LLM System for Auto Prediction and Evaluation for the Legal Argument Reasoning Task (2024)

Abstract: In this article, we present an effective system for semeval-2024 task 5. The task involves assessing the feasibility of a given solution in civil litigation cases based on relevant legal provisions, issues, solutions, and analysis. This task demands a high level of proficiency in U.S. law and natural language reasoning. In this task, we designed a self-eval LLM system that simultaneously performs reasoning and self-assessment tasks. We created a confidence interval and a prompt instructing the LLM to output the answer to a question along with its confidence level. We designed a series of experiments to prove the effectiveness of the self-eval mechanism. In order to avoid the randomness of the results, the final result is obtained by voting on three results generated by the GPT-4. Our submission was conducted under zero-resource setting, and we achieved first place in the task with an F1-score of 0.8231 and an accuracy of 0.8673.

## introduction in 2023, a significant event in the field of artificial intelligence (ai) was the widespread adoption of chatgpt, particularly the introduction of @xcite , which revolutionized perceptions of ai. gpt-4 exhibited a notable advancement of 11.2 points on the mmlu benchmark @xcite and demonstrated superior performance on various question answering (qa) and natural language inference (nli) datasets. largescale language models (llm) represented by gpt-4 have sprung up, including llama-2 @xcite , falcon @xcite , gemini @xcite , baichuan-2 @xcite , chatglm @xcite , etc. there are many researchers have explored nlp task leveraging gpt-4 in zero-resource and low-resource scenarios. gpt-4 is pretrained on a large amount of internet data initially, and refined through supervised fine-tuning and reinforcement learning from human feedback (rlhf) @xcite . despite these advancements, limited research exists on the direct application of gpt-4 to nlp tasks within the legal domain. this paper aims to comprehensively address this research gap. ## task description this task aims to assess system's ability to reason about legal arguments. the task organizers have introduced a dataset @xcite of civil litigation cases from the u.s. legal system. each instance comprises of an overview of the case, a question, a proposed solution (an answer candidate), and analysis justifying the solution. systems are required to determine if the solutions and analysis are correct (true) or incorrect (false). while similar to a typical classification task, this task demands strong causal reasoning and practical application of knowledge. as shown in figure 1 , evaluating the correctness of an answer candidate demands not only a logical assessment of the question and response but also the application of legal knowledge provided in the introduction. further, this task requires expertise with legal terminology and concepts. an experienced law professor, armed with deep understanding of relevant legal statutes and extensive knowledge in the field, would likely be able to swiftly assess the accuracy of answer candidates and the soundness of their analysis, even with minimal background information. conversely, for those less familiar with the field, even being provided with comprehensive information, identifying key details and reaching the correct conclusion remains a challenging task. notably, the training dataset and development dataset provide analysis of the labels, whereas the test dataset does not. this dataset is extracted from real law teaching books and includes a total of 666 training sets, 84 development sets, and 98 test sets. the training set and development set provide analysis of labels, but introduction my students always get confused about the relationship between removal to federal court and personal jurisdiction. suppose that a defendant is sued in arizona and believes that she is not subject to personal jurisdiction there. naturally, she should object to personal jurisdiction. [...] but generally the scope of personal jurisdiction in the federal court will be the same as that of the state court, because the federal rules require the federal court in most cases to conform to state limits on personal jurisdiction. fed. r. civ. p. 4(k)(1)(a). i've stumped a multitude of students on this point. consider the following two cases to clarify the point. 7. a switch in time. yasuda, from oregon, sues boyle, from idaho, on a state law unfair competition claim, seeking \$250,000 in damages. he sues in state court in oregon. ten days later (before an answer is due in state court), boyle files a notice of removal in federal court. five days after removing, boyle answers the complaint, including in her answer an objection to personal jurisdiction. boyle's objection to personal jurisdiction is question not waived by removal. the court should dismiss if there is no personal jurisdiction over boyle in oregon, even though the case was properly removed. ## conclusion in this article, we propose an llm system with self-eval mechanism for semeval-2024 task 5. we explore the potential for using gpt-4 and prompt learning to obtain causal reasoning capabilities in the field of civil litigation. we have proven that the self-eval mechanism can alleviate the problem of unclear output and can also significantly improve performance. additionally, we found that gpt4 demonstrates greater aptitude for choice tasks than for judgement tasks. with the prompts we provide, the experiment is fully reproducible and the experimental results can be extracted through regular expressions. due to time and space limitations, we leave some questions unresolved. for example, we only used gpt-4 for experiments. the broader applicability of the self-eval mechanism to other llms and its effectiveness in diverse tasks present room for further investigation. we intend to dive deeper into these questions in future work.