Evaluation of Commonsense Reasoning and Moral Understanding in Children’s Stories (2024)

Abstract: “This paper provides a comprehensive review of the the CCL24-Eval Task 8: Commonsense Reasoning and Moral Understanding in Children’s Stories(CRMUS). This task has designed two sub-tasks, which aim to assess the commonsense reasoning and implicit meaning comprehension capabilities of Large Language Models(LLMs). We heve received registration forms from 33 teams, 15 of which submitted final results that exceeded the baseline score. We present the results of the top 5 teams and our analysis of these results.”

## introduction stories are essential reading material in education, often containing rich knowledge, vivid plots, memorable characters, and profound implicit meanings. they serve as important vehicles for the dissemination of knowledge, cultural inheritance, and value shaping. story comprehension requires models not only to understand plots based on social, physical, and other common knowledge, but also to analyze character relationships, intentions, and behaviors, and to infer the profound meanings conveyed by the story @xcite @xcite . it is suitable for evaluating the cognitive abilities of llms. therefore, we have constructed a new challenging story comprehension dataset crmus (commonsense reasoning and moral understanding in children's stories), and designed two sub-tasks based on the cognitive process of human comprehension of stories. moreover, we organized ccl24-eval task 8, crmus. this evaluation is divided into two tracks: (1) track 1 allows the use of commercial llms through prompt learning; (2) track 2 allows the use of open-source llms through fine-tuning, but the model parameters must not exceed 7 billion. in the end, we received registration forms from 33 teams, of which 15 submitted final results that exceeded the baseline we provided. we found that although llms already possess certain text comprehension and reasoning abilities, they still perform poorly in deep semantic comprehension and reasoning tasks that extend beyond the surface meaning of the text, such as commonsense reasoning and implicit meanings comprehension. the task description is presented in section 2. the dataset we constructed for this task in section 3. in section 4, we provide baselines for two sub-tasks. we discuss the metrics used to rank participant submissions in section 5. in section 6, we list participants' information and results from their submissions and provide a more in-depth discussion. we introduce the methods of excellent teams in section 7. finally, we conclude the paper in section 8. questions using commonsense knowledge (usually implicit) derived from the story. the questions are in multiple-choice format, each including a question and four options. ## evaluation metrics the final evaluation score of the participating model is the weighted average of the accuracy of the answers in each sub-task. the specific calculation method is as follows: specifically, acc 1 = the accuracy of answers for the commonsense reasoning sub-task acc 2 = the accuracy for the moral understanding sub-task. ## results and analysis based on the model proposals submitted by participating teams, it was observed that most teams employ the prompt design strategy to prompt llms to identify commonsense knowledge within the story, 1 https://github.com/sxu-yaxinguo/crmu c c l 2 0 2 4 the experimental results demonstrate that their method achieved the final score of 79.27, placing first in the closed dataset of track 1 among 10 submitted results, thereby confirming its effectiveness. the results further validate the efficacy of the prompt-based approach in addressing the crmus task. track 2 zzu nlp secured first place in the closed data of track 2. their approach primarily involved designing effective prompt templates, fine-tuning lora parameters, and utilizing data augmentation techniques. in the instruction fine-tuning stage, they chose two llms mainly in chinese, namely qwen1.5-chat-7b @xcite and internlm2-chat-7b @xcite . among them, internlm2-chat-7b is the main fine-tuning model, and qwen1.5-chat-7b is the auxiliary model to verify the optimal lora parameters. by testing the combination of different lora parameters and fine-tuning modules, it was ultimately determined that two sets of parameters can provide the optimal acc indicators for cr and mu, respectively. in the process of conducting commonsense reasoning on the development set, they found that the model performed poorly in terms of temporal, spatial, and physical knowledge, and speculated that this may be due to the small amount of data for several commonsense types. therefore, they used data augmentation methods to address the issue of uneven distribution of different commonsense types in the crmus dataset. they created over 200 commonsense reasoning data using chatgpt, and then manually reviewed and screened 137 high-quality data. the data was then expanded to the development set for fine-tuning, resulting in an improvement in the accuracy. ## conclusion and future work this paper presents an overview of the ccl24-eval task 8, i.e., commonsense reasoning and moral understanding in children's stories(crmus). this evaluation is conducted using our meticulously annotated crmus dataset. these tasks are designed to assess llms' ability to understand and reason about commonsense knowledge in stories, as well as their capacity to capture the deep semantics and implicit meanings within the stories. we received a total of 33 completed registration forms, of which 15 teams submitted the final results that exceeded the baseline we provided. additionally, we offer a comprehensive analysis and summary of the methodologies employed by the participants, which will inform and guide future research in the field of natural language processing. finally, we believe that this evaluation remains challenging for llms, primarily due to the models' insufficient semantic understanding and reasoning abilities. in the future, we will continue to explore and enhance the crmus dataset, aiming to further improve its scale and quality. we also aim to explore additional forms of commonsense and reasoning q&a, as well as moral examination methods, to better evaluate the commonsense reasoning and deep semantic understanding abilities of llms. c c l 2 0 2 4