Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLM s Falter? (2024)

Abstract: Solving grid puzzles involves a significant amount of logical reasoning. Hence, it is a good domain to evaluate reasoning capability of a model which can then guide us to improve the reasoning ability of models. However, most existing works evaluate only the final predicted answer of a puzzle, without delving into an in-depth analysis of the LLMs’ reasoning chains (such as where they falter) or providing any finer metrics to evaluate them. Since LLMs may rely on simple heuristics or artifacts to predict the final answer, it is crucial to evaluate the generated reasoning chain beyond overall correctness measures, for accurately evaluating the reasoning abilities of LLMs. To this end, we first develop GridPuzzle, an evaluation dataset comprising of 274 grid-based puzzles with different complexities. Second, we propose a new error taxonomy derived from manual analysis of reasoning chains from LLMs including GPT-4, Claude-3, Gemini, Mistral, and Llama-2. Then, we develop a LLM-based framework for large-scale subjective evaluation (i.e., identifying errors) and an objective metric, PuzzleEval, to evaluate the correctness of reasoning chains. Evaluating reasoning chains from LLMs leads to several interesting findings. We further show that existing prompting methods used for enhancing models’ reasoning abilities do not improve performance on GridPuzzle. This highlights the importance of understanding fine-grained errors and presents a challenge for future research to enhance LLMs’ puzzle-solving abilities by developing methods that address these errors.

## introduction recent advancements in llms such as gpt-4, gemini, claude-3 (anthropic, 2024), llama-2 @xcite have achieved remarkable performance on a wide range of natural language understanding (nlu) tasks previously thought to be exclusive to humans. beyond nlu, exploring llms' logical reasoning abilities @xcite @xcite on complex tasks such as puzzle-solving is underexplored. past attempts have been made to evaluate models on logic-intensive grid-based puzzlesolving. however, they either do not focus on evaluating llms @xcite or do not evaluate llms independently, but rather use neuro-symbolic approaches @xcite that use external specialized solvers on llm outputs. here, we aim to evaluate the puzzle-solving abilities of llms by themselves, without the use of any external logic solvers. to understand the reasoning capabilities of llms, it is important to evaluate reasoning chains, rather than the final predicted answer. there have been works that evaluate reasoning chains using objective metrics such as roscoe @xcite , ctc @xcite , and bartscore @xcite , however, they do not focus specifically on evaluating reasoning. some prior works propose metrics for specific reasoning tasks, such as folio @xcite and prontoqa @xcite . however, these methods rely on reference-based evaluation, do not focus on puzzle-solving, and do not aim to identify finegrained errors in reasoning chains. to address these limitations, we propose a reference-free manual and automated subjective evaluation of reasoning chains to understand various fine-grained errors in reasoning chains for grid-based puzzle-solving. motivated by @xcite , we first develop gridpuzzle (figure 1 ), a comprehensive evaluation dataset consisting of grid-based puzzles with grid-size of 3 × 4, 3 × 5, 4 × 4, 4 × 5, and 4 × 6 with three levels of difficulty (easy, medium, and hard). then, we evaluate llms including gpt-2. clue 4 specifies that the dieter who lost 7 lbs is either the person who used the caveman diet or raymond. 3. since stuart lost 7 lbs (from clue 3), he either used the caveman diet or is named raymond. 4. clue 5 states wesley lost 5 lbs. 5. therefore, wesley cannot have used the caveman diet or lost 7 lbs. 6. we don't know his diet yet. . . . 20. that leaves 9 lbs to be the only unassigned number, and since it can't be wesley ( 5 inst. auto-evaluated reasoning chains { ""sentence"": ""clue 3 says stuart lost 7 lbs."", ""premise"": ""clue 3"", ""conclusion"": ""stuart lost 7 lbs."", ""explanation"": ""reiteration of the clue."", ""error category"": ""nc"", ""sub category"": ""-"" }, { ""sentence"": ""clue 4 specifies that the dieter who lost 7 lbs is either the person who used the caveman diet or raymond."", ""premise"": ""clue 4"", ""conclusion"": ""stuart must be on the caveman diet or be raymond."", ""explanation"": ""that is the right reasoning from the given clue."", ""error category"": ""rr"", ""sub category"": ""-"" },...... 4, gemini-pro, claude-3, llama-2, and mistral on gridpuzzle in zero-shot-cot setting (figure 1 ). experimental results show that llms do not fare well and achieve a maximum of 5.1% accuracy. to investigate the reasoning chains, we manually analyze them (figure 1 ) to find fine-grained errors (further details in section 3.3). based on this, we propose a new error taxonomy comprising five broad categories, and nine fine-grained subcategories (tables 1 and 2 ), providing deeper insights into the primary causes of the llms' reasoning failures. however, scaling manual analysis to a larger set is time-consuming and laborious. hence, we propose to leverage llms as auto-evaluators by creating prompts that utilize error taxonomy to automate the analysis of reasoning chains and help in identifying errors (figure 1 ). while evaluating w.r.t. manual annotation, our auto-evaluator model achieves ∼ 86% agreement, hence providing quality error categorization. beyond identifying errors and the accuracy of the final answer, we propose puzzleeval, an llmbased framework to evaluate reasoning chains for grid-based puzzles. puzzleeval involves a multi-stage evaluation using gpt-4o. first, we identify key logical conclusions from the reasoning chain; second, we extract key logical concepts from these conclusions; and finally, we measure the presence of these logical concepts in the final gold answer to assess the correctness of the reasoning chain. evaluating reasoning chains based on error categorization and puzzleeval reveals interesting findings such as llms show lower accuracy despite having more error-free reasoning steps, open-source models lack reasoning skills compared to closed-source models, and the most dominant error categories are wrong reasoning and elimination. additionally, we employ existing prompting methods such as plan-and-solve and self-discover, demonstrating that these methods do not improve performance on gridpuzzle. we believe that our findings will inspire future work in the automated in-depth evaluation of reasoning chains for broader reasoning tasks and enhance the reasoning abilities of models. ## conclusion in this work, we evaluated the logical reasoning abilities of llms through the lens of a grid-based puzzle-solving task. we introduced gridpuzzle, an evaluation dataset of 274 puzzles with various grid sizes. from a manual evaluation of reasoning chains generated by five different llms on gridpuzzle, we developed a fine-grained error taxonomy with five broad categories and nine subcategories. we then created an auto-evaluator to automate the identification of error categories, providing broader insights into error distributions across the dataset. additionally, we proposed puz-zleeval, a reference-free metric to objectively evaluate the correctness of reasoning chains for gridbased puzzles. our analysis of error distributions in gridpuzzle revealed several interesting findings and insights into the logical reasoning abilities of different llms. we further evaluated existing reasoning-specific prompting methods, such as self-discover and self-correct, finding that they do not improve results on gridpuzzle. we believe our work offers a challenging dataset, highlights where these llms make mistakes, and provides insights to develop better logical reasoning systems for complex tasks such as grid puzzle-solving.