Archimedes- AUEB at S em E val-2024 Task 5: LLM explains Civil Procedure (2024)

Abstract: The SemEval task on Argument Reasoning in Civil Procedure is challenging in that it requires understanding legal concepts and inferring complex arguments. Currently, most Large Language Models (LLM) excelling in the legal realm are principally purposed for classification tasks, hence their reasoning rationale is subject to contention. The approach we advocate involves using a powerful teacher-LLM (ChatGPT) to extend the training dataset with explanations and generate synthetic data. The resulting data are then leveraged to fine-tune a small student-LLM. Contrary to previous work, our explanations are not directly derived from the teacher’s internal knowledge. Instead they are grounded in authentic human analyses, therefore delivering a superior reasoning signal. Additionally, a new ‘mutation’ method generates artificial data instances inspired from existing ones. We are publicly releasing the explanations as an extension to the original dataset, along with the synthetic dataset and the prompts that were used to generate both. Our system ranked 15th in the SemEval competition. It outperforms its own teacher and can produce explanations aligned with the original human analyses, as verified by legal experts.

## conclusion this paper described the archimedes-aueb submission to semeval-2024 task 5. the submitted method is based on a llm fine-tuned on cot data in order to offer justifications for the predictions it generates. the training explanations are generated by gpt-3.5 in conjunction with expert analyses. concurrently, we implement an innovative prompting procedure, dubbed multiple choice mutation, designed to produce artificial instances. both approaches contribute to a significant increase in performance. our solution ranks 15 th out of 20 contestants. unlike other competitors, it generates explanations for its predictions. it also surpasses a strong gpt-3.5 baseline that can also produce explanations. we also assessed the output of our system qualitatively, with assistance from legal experts. in many cases the system generated highquality explanations. the errors made by the model can be attributed to either its reasoning limitations or its deficiencies in legal knowledge. we recommend including similar qualitative evaluations in future work, as they reveal pivotal insights about the model that performance metrics alone cannot capture. 12 4 clear unclear figure 3: incorrect predictions annotated for clarity. correct prediction but inaccurate explanation question: a computer programmer living in texas did some work for a corporation which is located in the northern district of illinois, but they refused to pay him.