Syllogistic Reasoning for Legal Judgment Analysis (2023)

Abstract: Legal judgment assistants are developing fast due to impressive progress of large language models (LLMs). However, people can hardly trust the results generated by a model without reliable analysis of legal judgement. For legal practitioners, it is common practice to utilize syllogistic reasoning to select and evaluate the arguments of the parties as part of the legal decision-making process. But the development of syllogistic reasoning for legal judgment analysis is hindered by the lack of resources: (1) there is no large-scale syllogistic reasoning dataset for legal judgment analysis, and (2) there is no set of established benchmarks for legal judgment analysis. In this paper, we construct and manually correct a syllogistic reasoning dataset for legal judgment analysis. The dataset contains 11,239 criminal cases which cover 4 criminal elements, 80 charges and 124 articles. We also select a set of large language models as benchmarks, and conduct a in-depth analysis of the capacity of their legal judgment analysis.

## introduction legal judgment assistants are growing rapidly due to large demands from legal practitioners and normal citizens @xcite and the impressive progress of large language models @xcite @xcite . the core technique employed in those assistants is legal judgment analysis, which aims to analyze the description of a fact and produce reasonable results of a legal judgment. there are two genres of approaches toward legal judgment analysis. one group of researchers is actively working on legal judgment prediction (ljp) @xcite . the main goal is to predict the results of a legal judgment (e.g., charges, legal articles, and terms of penalty) given the description of a fact @xcite . this prediction figure 1 : an example of criminal case of the credit card fraud. the output of task 1 is the main premise. the output of task 2 is the minor premise. the outputs of task 3 and 4 are the conclusion. the corresponding segments to different parts are marked in the same color. we mark the corresponding segments in different parts with the same color. another group of researchers explores court view generation (cvg) @xcite @xcite to enable the interpretability of the results of legal judgment. the core idea is to generate phrases or sentences in the description of a fact as rationales for supporting legal judgment results. however, the generated rationales might be specious @xcite and can not represent the whole analysis process that leads to the final result. in practice, syllogistic reasoning is routinely employed by judges to assess the parties' arguments for legal judgments, thereby maintaining consistency and fairness in their decision-making process. @xcite . it allows judges to break down legal arguments into logical components, including major premise, minor premise and conclusion @xcite . in this work, we seek to answer the following research questions: • rq1: how do llms perform on the syllogistic reasoning for legal judgment analysis? • rq2: how does syllogistic reasoning impact the performance of llms in legal judgment analysis? we make three main contributions to facilitate research into syllogistic legal judgment analysis (slja): a formalization, a dataset, and benchmarks. let us expand on each of these. we formalize slja as four interrelated tasks (see figure 1 ): (1) article retrieval (ar) selects relevant articles from article database as a major premise; (2) criminal element generation (ceg) generates the criminal elements of a fact as a minor premise; we construct a dataset for slja. first, we select criminal cases from the cail-long dataset @xcite with the most common charges whose frequencies are larger than 40. then, we invoke the chatgpt api to generate the results of four tasks for the selected cases. finally, we hire 22 workers with a legal background to revise the results generated by chatgpt. we select four state-of-the-art llms as benchmarks, including chatglm @xcite , davinci002, davinci003, and chatgpt 1 . we compare the performance of each model in each task with and without syllogistic templates (see appendix a) as prompts. through extensive experiments and analyses, we conclude that (1) slja is still challenging for state-of-the-art llms; (2) and syllogistic reasoning can provide effective information for llms to improve the performance of legal judgment analysis. our contributions can be summarized as follows: ## task definition we define the following four tasks for syllogistic legal judgment analysis (slja) based on the concept of syllogistic reasoning: task 1: article retrieval (ar) outputs applicable law articles as major premises given a fact and article library as inputs. task 2: criminal element generation (ceg) outputs four criminal elements as minor premises given a fact as an input. ## dataset construction we construct a dataset for legal judgment analysis with syllogistic reasoning. ## conclusion and future work in this paper, we introduce the resources of syllogistic reasoning for legal judgment analysis: the task, dataset and benchmarks. we define the slja according to the concept of syllogistic reasoning, including four tasks: article retrieval (ar), criminal element generation (ceg), article interpretation generation (aig) and legal judgment prediction (ljp). we release the first chinese slja dataset. the dataset is constructed by correcting the outputs of each task, which are generated by chat-gpt. in order to ensure the quality of the dataset, we hire workers with a legal background for data correction. each case in slja dataset includes a detailed analysis process of syllogistic reasoning and we expect the dataset will facilitate the development of trustworthy legal judgment assistants. besides, we also collect four state-of-the-art llms for extensive experiments. we design specific prompts for each task as inputs of llms. we evaluate each model with and without syllogism and conduct an in-depth analysis. the experimental results indicate that: (1) slja is still challenging for llms. as to future work, on the one hand, we will extend slja dataset with more legal types and theories, such as civil law and two-stage theory. on the other hand, this work is currently releasing a test set for llm evaluation. in the future, we will construct more data to train and evaluate traditional models and conduct a human study to enhance the practicality and interpretability of our approach. legal llms (e.g., lawyer llama) that have emerged during the same period as this work will be evaluated in future research. last but not least, we call for studies to improve the benchmark performance, as well as conduct underexplored research, such as legal judgment analysis for rare charges under low-resource settings.