H ella S wag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLM s in Commonsense Reasoning (2025)

Abstract: Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning; however, some variations in questions can trigger incorrect responses. Do these models truly understand commonsense knowledge, or just memorize expression patterns? To investigate this question, we present the first extensive robustness evaluation of LLMs in commonsense reasoning. We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases, by designing and compiling seven types of question variants. To construct this benchmark, we propose a two-stage method to develop Chinese HellaSwag, a finely annotated dataset comprising 12,000 instances across 56 categories. We conduct extensive experiments on 41 representative LLMs, revealing that these LLMs are far from robust in commonsense reasoning. Furthermore, this robustness varies depending on the language in which the LLM is tested. This work establishes a high-quality evaluation benchmark, with extensive experiments offering valuable insights to the community in commonsense reasoning for LLMs.

## introduction commonsense reasoning is a crucial part of intelligence, involving contextual understanding, implicit knowledge, and logical deduction @xcite @xcite . recent studies have focused on enhancing these capabilities in llms, achieving impressive performance @xcite @xcite . however, even slight changes to questions can lead to incorrect responses from the same models. for instance, in binary commonsense questions, human naturally recognizes both correct and incorrect options through a single inference process, while llms, though able to identify the correct answer, struggle to reason about why the alternative is wrong @xcite . therefore, we ask the question: does this high-level performance stem from a genuine understanding of commonsense knowledge, or is it simply a result of memorizing specific expression patterns in pre-training data? to answer this question, an effective approach is to systematically evaluate the robustness of llms in answering commonsense reasoning questions. as illustrated in figure 1 , we find that gpt-4o correctly answers an original question but fails on its variants, i.e., questions about the same commonsense knowledge but in different reasoning forms, such as reverse conversion. this indicates that gpt-4o has not fully grasped the commonsense knowledge behind the question; a genuine understanding of commonsense knowledge should be able to generalize to these question variants. however, existing benchmarks do not yet sup- to address this gap, we present the first extensive evaluation on the robustness of commonsense reasoning for llms, starting with dataset construction. firstly, recognizing that existing benchmarks are predominantly in english, which limits the assessment of non-english llms (davis, 2023), we develop a chinese commonsense reasoning dataset based on the widely-used hellaswag benchmark @xcite , containing 12,000 questions. specifically, we design 56 fine-grained categories, and propose a two-stage data annotation method including initial dataset generation and difficult sample replacement. secondly, we design and compile seven variants from existing studies (cf. using hellaswag-pro, we conduct a comprehensive evaluation on the robustness of 41 closedsource and open-source llms with nine different prompt strategies. we derive several key findings: (1) all llms are far from robust in commonsense reasoning tasks, as evidenced by their poor performance on question variants and the significant gap compared to human performance. neverthe-less, gpt-4o achieves the best robustness among all the evaluated llms. (2) among all types of variants, negative transformation is the most challenging, with an average accuracy of only 9.01%, while problem restatement poses minimal difficulty. our contributions are three-fold. (1) we present the first extensive evaluation on the robustness of commonsense reasoning for llms by designing and compiling seven types of variants. @xcite we have developed a bilingual, large-scale, humanannotated benchmark for evaluating llm robustness in commonsense reasoning, which will be publicly released upon acceptance. (3) we conduct in-depth experiments on 41 representative llms with diverse prompts, yielding critical insights. ## chinese hellaswag given the limitation that most existing benchmarks for commonsense reasoning are in english, we begin by building a chinese benchmark for commonsense reasoning that captures unique aspects of chinese cultural context. firstly, we structure the dataset following the format of hellaswag @xcite , a widely recognized english commonsense reasoning benchmark, which consists of multiple-choice questions with four answer options. secondly, to minimize manual effort, we incorporate qwen-max (yang et al., 2024), a stateof-the-art chinese llm, into the dataset construction process. finally, to enhance the diversity of the dataset, we develop a hierarchical taxonomy of commonsense knowledge, as shown in figure 3 . our taxonomy consists of seven broad categories summarized from existing literature @xcite @xcite , each containing eight subcategories. we aim to construct our dataset based on the taxonomy, where we inject the categorical information into the instruction for llm generation. we propose a two-stage data construction pipeline, initial dataset generation and difficult sample replacement, as shown in figure 2 . ## hellaswag-pro based on the english-chinese hellaswag datasets, we construct hellaswag-pro, the benchmark for extensive robustness evaluation of commonsense reasoning. we begin by designing the seven-type question variants for robustness evaluation, then detail our data generation process. ## experiment in this section, we conduct extensive experiments to evaluate the performance of various llms on our hellaswag-pro benchmark. our study is guided by three key research questions: rq1: how do different llms perform across all variants? rq2: what is the relative difficulty of different variants? rq3: which prompting strategies yield the best robustness in llms? ## related work commonsense reasoning evaluation there are numerous benchmarks and datasets for commonsense reasoning, most of which are in english. some studies focus on evaluating general commonsense knowledge @xcite @xcite . others target specific aspects of commonsense reasoning @xcite @xcite @xcite @xcite . there are some chinese datasets for commonsense reasoning @xcite . for instance, charm @xcite distinguishes between global commonsense and chinese-specific commonsense but includes only a limited number of everyday commonsense cases. however, evaluations aimed at assessing the robustness of commonsense reasoning are still understudied. ## conclusion we conduct a systematic evaluation of the robustness of llms in commonsense reasoning in both chinese and english. to facilitate this evaluation process, we introduce two large-scale, finelyannotated datasets: hellaswag-pro and chinese hellaswag. in addition, we design various prompts to evaluate 41 llms, offering several key findings that may advance the field of commonsense reasoning. we believe this work will serve as a valuable resource to support further research into the commonsense reasoning of llms.