C ² RB ench: A C hinese Complex Reasoning Benchmark for Large Language Models (2025)

Abstract: Large language models (LLMs) have achieved remarkable progress in autonomous reasoning, evolving from basic text processing to sophisticated multimodal reasoning, a critical capability for general-purpose AI assistants. However, existing benchmarks usually fail to adequately capture the intricate multi-step reasoning demands inherent in real-world scenarios. To bridge this gap, we propose **C²RBench**: a **C**hinese **C**omplex **R**easoning **Bench**mark for evaluating multi-step, multimodal advanced reasoning capability of LLMs. C²RBench comprises 1,115 carefully curated Chinese tasks, which are organized into eight domain-specific subsets, each meticulously designed to mirror real-world challenges. This hierarchical benchmark features three difficulty tiers based on the number of reasoning steps required (average 8.44 steps per task), significantly exceeding existing benchmarks in cognitive complexity. Extensive evaluations of 20 LLMs (including DeepSeek-R1) and 24 multimodal large language models (MLLMs) on C²RBench reveal critical performance gaps: GPT-4.1 achieves only 52.11% accuracy, indicating substantial room for improvement. The dataset and evaluation code are publicly available.

## introduction recent advances in reasoning capabilities of large language models (llms), such as o1-preview 3 and deepseek-r1 @xcite have significantly enhanced their ability to engage in more extended and deliberate reasoning processes, allowing them to solve increasingly complex problems compared to earlier llms @xcite . these reasoning capabilities allow models to perform multi-step reasoning, make better decisions, and adapt more effectively to diverse tasks @xcite . reasoning has become a critical factor in evaluating the intelligence and generalization of llms. however, existing reasoning benchmarks often fall short of capturing the requirements of modern reasoning language models @xcite . first, many current benchmarks often focus on a limited set of reasoning tasks with a small number of reasoning steps. for instance, gsm8k @xcite , a purely textual benchmark, tests basic mathematical operations requiring only 2 to 8 reasoning steps. second, benchmarks like drcd @xcite and c-eval @xcite are not well aligned with real-world scenarios and fail to capture the complexity and contextual ambiguity of authentic reasoning problems. third, most reasoning benchmarks are limited to english and unimodal tasks, lacking support for other languages like chinese or multimodal formats that combine text, images, tables, or structured files @xcite . to mitigate these challenges, we propose the chinese complex reasoning benchmark (c 2 rbench), a novel benchmark designed to evaluate llm reasoning across multiple dimensions, including multi-task, multi-step, and multimodal reasoning in chinese. as illustrated in figure 1 , it features broader and more reasoning steps than most existing benchmarks. specifically, c 2 rbench emphasizes multi-step reasoning, with an average of 8.44 reasoning steps per task. the tasks are organized into three levels of reasoning complexity, with level-3 tasks requiring an average of 13.03 reasoning steps. additionally, c 2 rbench supports multimodal reasoning, encompassing text, images, tables, and files. the tasks in c 2 rbench are drawn from real-world scenarios, such as file processing, graph analysis, and scientific reasoning, in order to simulate the evaluation of llm reasoning in practical applications. although similar to the gaia benchmark @xcite , c 2 rbench is larger (1,115 vs. 466 tasks) and more aligned with real-world scenarios, with a focus on chinese rather than english. the main contributions of c 2 rbench are summarized as follows: ## related work as llms continue to advance, their ability to tackle complex reasoning tasks has become a key focus in both research and application @xcite @xcite . existing benchmarks for evaluating reasoning capabilities can be broadly categorized into task-specific and comprehensive benchmarks. task specific reasoning benchmarks. these benchmarks are tailored to assess a model's reasoning ability in a particular domain. for instance, hel-laswag @xcite and siqa @xcite evaluate commonsense reasoning; mathvista @xcite , @xcite and backmath @xcite focus on mathematical reasoning; arc @xcite targets scientific reasoning; codexglue @xcite and probench @xcite benchmark code reasoning and generation tasks; toolbench @xcite and @xcite evaluate the tool using capability of models; and roleeval @xcite aims to evaluate reasoning over knowledge required for role-playing. while these datasets provide valuable insights into specific reasoning capabilities, they fall short in evaluating general-purpose reasoning across diverse settings. comprehensive reasoning benchmarks. to systematically evaluate broader reasoning abilities, several comprehensive benchmarks have been introduced. mmmu @xcite spans multiple domains-ranging from science to humanities-using primarily multiple-choice questions. similarly, c-eval @xcite , m3ke @xcite and openeval @xcite provide large-scale chinese evaluation sets emphasizing advanced knowledge and reasoning. other multimodal benchmarks such as mmbench @xcite and gaokao-mm @xcite focus on visual reasoning in multiple-choice formats. gaia @xcite offers open-ended tasks covering web search, code generation, multimodal understanding, and document processing. despite their contributions, existing benchmarks exhibit several limitations: (i) a lack of real-world contextual complexity, (ii) limited support for multi-step reasoning annotations, and (iii) insufficient coverage of the chinese language and multimodal reasoning settings @xcite . these constraints hinder their applicability for evaluating the capabilities of modern llms in practical, multilingual, and cognitively demanding scenarios. to address these challenges, we propose c 2 rbench, a chinese benchmark designed for multi-task, multi-modal and multi-step reasoning scenarios in real-world applications. detailed comparison of our dataset with previous datasets is presented in ## dataset curation we follow the pipeline illustrated in figure 2 to curate c 2 rbench, aiming to evaluate the multi-step reasoning abilities of llms in real-world chinese ## experiments we conducted extensive experiments with 20 llms and 24 mllms on c 2 rbench to evaluate its effectiveness and utility for assessing the reasoning capabilities of llms and mllms. all models used for evaluation are summarized in ## conclusion in this paper, we have presented c 2 rbench, a novel benchmark designed to evaluate the multi-step reasoning capabilities of llms in chinese. c 2 rbench consists of 1,115 tasks organized into eight distinct subsets. these tasks are categorized into three difficulty levels based on the number of steps involved. the key highlights of c 2 rbench are its real-world scenarios, extensive task coverage, and its focus on chinese. our core contribution lies in providing a comprehensive and challenging benchmark for assessing llms' performance on these complex, multi-step tasks in chinese. experimental evaluations on 24 mllms and 20 text-only llms indicate that c 2 rbench remains challenging for state-of-the-art models. by systematically assessing complex reasoning across eight subsets, c 2 rbench establishes a rigorous and comprehensive benchmark.