A Two-stage Prompt-Based Strategy for CRMUS Track 1 (2024)

Abstract: “Large Language Model (LLM) has sparked a new trend in Natural Language Processing, and an increasing number of researchers have recognized the potential of using LLM to unify diverse NLP tasks into a text-generative manner. To explore the potential of LLM for the children’s stories domain, CCL2024 has released the Commonsense Reasoning and Moral Understanding in Children’s Stories (CRMUS) task. This paper presents a straightforward yet effective two-stage prompt-based strategy for the CRMUS Track 1. In the initial stage, we use the same prompt to obtain responses from GPT-4, ERNIE-4, and Qwen-Max. In the subsequent stage, we implement a voting mechanism based on the results from the first stage. For records with inconsistent outcomes, we query GPT-4 for secondary confirmation to determine the final result. Experimental results indicate that our method achieved an average score of 79.27, securing first place in the closed domain among ten participating teams, thereby demonstrating the effectiveness of our approach.”

## introduction with the popularization of chatgpt, large language model (llm) has motivated an increasing trend in both industry and academia; an increasing number of researchers are exploring the potential of llms @xcite @xcite @xcite across various domains, leading to the proposal of new paradigms for nlp tasks. following the trend, the commonsense reasoning and moral understanding in children's stories (crmus) evaluation task 0 is introduced in ccl2024, which aims to evaluate chinese pre-trained language models and large language models from multiple perspectives in terms of commonsense reasoning and moral understanding on the children's education domain. in order to investigate different techniques in the field of llms, the crmus task provides two tracks: prompt-based and fine-tuning of llm parameters. the purpose of the prompt-based track in the crmus task is to assess llm's potential in story commonsense reasoning and moral understanding. the types of commonsense involved in the commonsense reasoning task cover a wide range of aspects: temporal commonsense, spatial commonsense, biological commonsense, physical commonsense, and social commonsense. given the comprehensive reasoning capabilities of llms, we hypothesized that a prompt-based approach would effectively fit the crmus task. to this end, we propose a straightforward yet effective two-stage prompt engineering pipeline: for the inconsistent results, we query gpt-4 for a secondary confirmation, with a slightly different prompt from the first step, which narrows down the range of options using only the options returned from the first step. this secondary confirmed choice is chosen as the final submission result. our experimental results demonstrate that our method achieved an average score of 79.27, ranking first in the closed domain among ten participating teams. this confirms the effectiveness of our approach. furthermore, our method leverages the strengths of multiple llms and a novel two-stage voting mechanism, offering a robust solution for commonsense reasoning and moral understanding tasks without requiring extensive fine-tuning. this innovation lies in the efficient combination of multiple models' strengths and the strategic use of prompt engineering to enhance performance in the crmus task. ## our methods due to the comprehensive reasoning capabilities of llm, we believe the prompt-based method could fit the crmus task well. we propose a simple yet effective two-stage prompt-based pipeline for the evaluation tasks. the overall processing pipeline, illustrated in figure 1 , applies to the commonsense reasoning (cr) subtask. the moral understanding (mr) subtask follows the same pipeline except for the prompt. ## conclusion this work proposes a simple yet effective two-stage prompt-based strategy for the crmus track 1. in the first stage, we employ the same prompt to obtain responses from gpt-4, ernie-4, and qwen-max. we adopt a voting and secondary prompt-based confirmation strategy in the second stage. the experimental results demonstrate that our method achieves an average score of 79.27, ranking first in the closed domain among ten participating teams, thus confirming the effectiveness of our approach. these results further validate the advantages of llms in story reasoning and moral understanding, showcasing their potential in addressing complex tasks in the children's education domain. the successful application of our method underscores the robustness and reliability of combining multiple llms and utilizing a strategic voting mechanism. our approach demonstrates that even with a simple prompt-based strategy, significant improvements can be achieved by leveraging the diverse strengths of different llms. the integration of a secondary confirmation step ensures higher accuracy and consistency in the results, which is particularly beneficial for tasks requiring nuanced understanding and reasoning. we believe that llms will play an increasingly important role in children's education, offering sophisticated tools for story reasoning and moral comprehension that can adapt to the needs of young learners.