LR ² B ench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems (2025)

Abstract: Recent progress in o1-like models has significantly enhanced the reasoning abilities of Large Language Models (LLMs), empowering them to tackle increasingly complex tasks through reflection capabilities, such as making assumptions, backtracking, and self-refinement. However, effectively evaluating such reflection capabilities remains challenging due to the lack of appropriate benchmarks. To bridge this gap, we introduce LR ^2 Bench, a novel benchmark designed to evaluate the Long-chain Reflective Reasoning capabilities of LLMs. LR ^2 Bench comprises 850 samples across six Constraint Satisfaction Problems (CSPs) where reflective reasoning is crucial for deriving solutions that meet all given constraints. Each type of task focuses on distinct constraint patterns, such as knowledge-based, logical, and spatial constraints, providing a comprehensive evaluation of diverse problem-solving scenarios. We conduct extensive evaluation on both conventional models and o1-like models. Our experimental results reveal that even the most advanced reasoning-specific models, such as DeepSeek-R1 and OpenAI o1-preview, struggle with tasks in LR ^2 Bench, achieving an average Exact Match score of only 20.0% and 23.6%, respectively. These findings underscore the significant room for improvement in the reflective reasoning capabilities of current LLMs.

## introduction recent advancements in large reasoning models (lrms), exemplified by qwq-32b, deepseek-r1, and openai-o1 @xcite @xcite , have demonstrated substantial progress in the reasoning capabilities of large language models (llms). these models exhibit more human-like behaviors, such as making assumptions, please solve the sudoku: @xcite , @xcite @xcite , @xcite @xcite verification, backtracking, and self-correction, enabling them to address increasingly complex challenges across various domains @xcite @xcite . despite this progress, the reflection capabilities of llms remain largely unexplored. reflection can be defined as the process of engaging in attentive, critical, exploratory, and iterative self-interactions with one's thoughts, actions, and underlying conceptual frameworks @xcite . existing research predominantly focuses on evaluating llms' abilities to utilize explicit feedback for selfcriticism or self-refinement @xcite @xcite @xcite . however, these approaches overlook a fundamental question: can llms spontaneously engage in the whole reflection process to solve more complex tasks? reflection is not simply about reacting to feedback. it also encompasses exploring possible solutions, assessing and adjusting strategies, and adapting when confronted with contradiction. to better understand how llms might engage in these capabilities, we consider constraint satisfaction problems (csps). csps are defined as a set of variables whose state must satisfy specific constraints, which inherently require iterative exploration in a large search space without predefined solution paths @xcite . as shown in figure 1 , the response of qwq-32b-preview @xcite to sudoku, a classic csp, exemplifies a typical reflection process. this task presents a scenario with row, column, and grid constraints, requiring iterative trial-and-error to determine valid values for each cell. the model initially makes an assumption, then identifies a conflict, and finally backtracks to correct its initial guess. this demonstrates an effective reflective reasoning process often absent from standard nlp tasks, such as summarization, translation, and question-answering. therefore, to systematically investigate reflective reasoning in llms, we propose lr 2 bench, a novel benchmark for evaluating the longchain reflective reasoning capabilities of llms. lr 2 bench consists of six csps: crossword, acrostic, logic puzzle, cryptogram, sudoku, and drop quote. each task necessitates reflection processes and emphasizes specific constraint patterns, such as knowledge-based constraints, logical constraints, and spatial constraints, ensuring a comprehensive assessment of llms' reflective reasoning capabilities across diverse problem-solving scenarios. we manually collect and annotate a total of 850 examples spanning multiple difficulty levels across all task types. through extensive experiments, we observe that even current top-performing lrms (e.g., deepseek-r1 and o1-preview) struggle to complete tasks in lr 2 bench, underscoring sub-stantial room for improvement in this critical area. furthermore, most conventional llms exhibit inferior performance to lrms and achieve near-zero sample-level accuracy in lr 2 bench, highlighting the essential role of reflective reasoning skills in tackling the challenges posed by our benchmark. our contributions are summarized as follows: 1) we introduce lr 2 bench, a novel benchmark with six tasks and diverse difficulty levels, designed to evaluate the long-chain reflective reasoning capabilities of llms. 2) we conduct a comprehensive evaluation of reflective reasoning capabilities in both conventional llms and lrms. 3) we present a detailed analysis of the limitations exhibited by current llms on lr 2 bench, establishing a foundation for future research on the development of advanced reasoning models. ## lr bench in this section, we introduce the construction process of lr 2 bench. section 2.1 provides an overview of each task. section 2.2 discusses the diverse scenarios targeted by lr 2 bench. section 2.3 shows the annotation process and data statistics, and section 2.4 outlines the evaluation metrics. figure 2 illustrates the data collection, annotation, and evaluation pipeline of our lr 2 bench. ## conclusion this paper introduces lr 2 bench, a novel benchmark to comprehensively evaluate the reflection capabilities of llms in long-chain reasoning. lr 2 bench comprises six tasks with varying difficulty levels, providing a thorough analysis across diverse scenarios. the experimental results show that lrms outperform conventional llms, demonstrating their superior performance on reflective reasoning. our findings also highlight the limitation of current reasoning llms and reveal that even the most advanced reasoning models fall short of achieving satisfactory performance, suggesting significant room for enhancement in reflective reasoning capabilities.