Benchmarking C hinese Commonsense Reasoning of LLM s: From C hinese-Specifics to Reasoning-Memorization Correlations (2024)

Abstract: We introduce CHARM, the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative prompt strategies for improving LLMs’ reasoning ability, such as Chain-of-Thought. Our findings indicated that the LLM’s language orientation and the task’s domain influence the effectiveness of the prompt strategy, which enriches previous research findings. We built closely-interconnected reasoning and memorization tasks, and found that some LLMs struggle with memorizing Chinese commonsense, affecting their reasoning ability, while others show differences in reasoning despite similar memorization performance. We also evaluated the LLMs’ memorization-independent reasoning abilities and analyzed the typical errors. Our study precisely identified the LLMs’ strengths and weaknesses, providing the clear direction for optimization. It can also serve as a reference for studies in other fields. We will release CHARM at https://github.com/opendatalab/CHARM.

## introduction commonsense reasoning is important for the enhancement of the large language models (llms) @xcite towards artificial general intelligence (agi) @xcite , therefore requires thorough evaluations. numerous benchmarks evaluate the commonsense reasoning of llms, but most are english-based, limiting non-english evaluations @xcite . this paper focuses on assessing llms' commonsense reasoning in a chinese context. currently, some commonsense reasoning benchmarks in chinese are simply english translations @xcite @xcite , which overlooks unique chinese cultural, linguistic, regional, and historical aspects. these factors matter when chinese users use the llm, hence should be included in benchmarks. to effectively tackle this, we introduce charm, the benchmark designed to thoroughly and in-depth assess the abilities of llms in chinese commonsense reasoning. it covers two domains: globally accepted commonsense (global domain) and chinese-specific commonsense (chinese domain). the latter includes 7 aspects: history (h), traditional culture and arts (ca), daily life and customs (lc), entertainment (e), public @xcite , geography (g), and chinese language (l). therefore charm allows a thorough evaluation of llms' reasoning in a chinese context. prompt strategies like chain of thought (cot) @xcite can significantly improve llms' reasoning performance @xcite . particularly, as the training corpus of llms is primarily in english @xcite , studies @xcite @xcite have shown that for non-english reasoning tasks, some llms perform better when reasoning in english than the native language. we evaluated 7 english and 12 chinese-oriented llms on charm, employing 5 representative prompt strategies. the result showed that prompt strategies' effectiveness depends on the llms' orientation and the benchmark task's domain, which enriches prior research and guides performance assessment and strategy choice for non-english llms. llms' commonsense reasoning relies on memorization. exploring the correlation between memorization and reasoning offers insights into llms, aiding deeper understanding and suggesting ways to enhance these abilities @xcite charm consists closely-interconnected reasoning and memorization tasks. ## related work commonsense reasoning benchmarks there are lots of commonsense reasoning benchmarks, most of them are in english @xcite . some chinese commonsense reasoning benchmarks are directly translated from english benchmarks @xcite @xcite , which lack the chinese specifics. there are some native chinese benchmarks that include some chinese-specific factors and involve commonsense reasoning to a certain extent, such as logiqa @xcite , clue @xcite and cmmlu @xcite . however, they are not designed for commonsense reasoning, therefore containing the large portion of irrelevant tasks and questions. corecode @xcite is the benchmark for chinese commonsense reasoning and commonsense conflict detection, but it is not strictly designed to distinguish the chinese-specific and global domains when compared with charm. in addition, charm has the closely-interconnected reasoning and memorization tasks, which are not included in previous commonsense reasoning benchmarks. the comparison of charm with previous commonsense reasoning benchmarks is shown in prompt strategy prompt strategies such as cot @xcite can effectively boost the reasoning capabilities of llms @xcite . notably, as the llm training corpus is primarily in english @xcite , research revealed that for reasoning tasks in non-english languages, some llms exhibit superior performance when reasoning in english as opposed to the native language @xcite @xcite . @xcite proposed a novel cross-language transfer prompt method, which uses both the source and target languages to construct examples. ## charm charm is built for comprehensive and in-depth evaluation of llms in chinese commonsense reasoning and revealing the intrinsic correlation between memorization and reasoning. therefore, charm covers two domains, global and chinese, using carefully selected tasks for comprehensive coverage. in addition, we chose reasoning tasks and constructed the closely-tied memorization tasks. the construction and main features of charm are in figure 1 . the detailed composition of charm is in ## conclusion this paper introduces charm, the first benchmark designed to comprehensively and thoroughly evaluate llms' commonsense reasoning in chinese. charm encompasses two counterpart commonsense domains, global and chinese-specific, with the carefully selected tasks. we evaluated the representative prompt strategies for improving llms' reasoning ability, and the empirical findings significantly enhances and supplements the conclusions of previous studies. charm comprises closely-interconnected reasoning and memorization tasks, helping to reveal the intrinsic correlation between memorization and reasoning of llms. we evaluated the strengths and weaknesses of different llms and conducted the detailed analysis of memorization-independent reasoning abilities. we hope that charm's approach to studying the correlations between memoriztion and reasoning can serve as a reference for similar research in other fields.