Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study (2024)

Abstract: Large Language Models (LLMs) have shown remarkable proficiency in language understanding and have been successfully applied to a variety of real-world tasks through task-specific fine-tuning or prompt engineering. Despite these advancements, it remains an open question whether LLMs are fundamentally capable of reasoning and planning, or if they primarily rely on recalling and synthesizing information from their training data. In our research, we introduce a novel task—Minesweeper—specifically designed in a format unfamiliar to LLMs and absent from their training datasets. This task challenges LLMs to identify the locations of mines based on numerical clues provided by adjacent opened cells. Successfully completing this task requires an understanding of each cell’s state, discerning spatial relationships between the clues and mines, and strategizing actions based on logical deductions drawn from the arrangement of the cells. Our experiments, including trials with the advanced GPT-4 model, indicate that while LLMs possess the foundational abilities required for this task, they struggle to integrate these into a coherent, multi-step logical reasoning process needed to solve Minesweeper. These findings highlight the need for further research to understand the nature of reasoning capabilities in LLMs under similar circumstances, and to explore pathways towards more sophisticated AI reasoning and planning models.

## introduction large language models (llms) have made remarkable strides in the natural language processing (nlp) arena, capturing the spotlight with their multifaceted capabilities. these models have been effectively utilized across a spectrum of nlp tasks, including information extraction @xcite , answering arithmetic and common-sense questions @xcite @xcite , as well as aiding in strategic planning @xcite or acting as game agents @xcite @xcite . with growing scale, llms begin to exhibit "emergent abilities" @xcite , including marked improvements in their ability to follow instructions, perform multi-step reasoning and planning, and even extend to comprehending humor @xcite . such flourishing capability has triggered excitement within the research community, cultivating the belief that llms could be instrumental in achieving artificial general intelligence @xcite @xcite . nevertheless, a crucial question remains underexplored: to what extent can the reasoning capabilities of llms go beyond the scope of their training distribution? this question is critical as it uncovers if llms are reliable for sophisticated hands-off planning in various scenarios such as biomedical experimental design @xcite and autonomous driving @xcite , where the agents have to deal with off-distribution inputs occasionally. unfortunately, the answer is not straightforward, and the debate is going on @xcite @xcite @xcite . the reason is that llms' substantial parameter counts enable them to store and recall vast amounts of information from their training material @xcite @xcite , which may contain conventional reasoning datasets such as gsm8k @xcite , mul-tiarith @xcite , or strategyqa @xcite . this could exaggerate llms' reasoning abilities on such benchmarks @xcite , leading to false a promise when applying llms to practical scenarios. although attempts to develop new benchmarks that challenge llms suggest potential zero-shot learning successes @xcite , there is an underlying concern that these benchmarks may not be distinct enough from the llms' training data, thereby giving llms an undue advantage and masking their true abilities @xcite . for instance, @xcite suggest that smaller llms only learn specific tasks when fine-tuned with data from larger models and that these improvements do not generalize well. drawing an analogy between smaller and larger llms to humans implies that llms may not exhibit genuine intelligence or intrinsic reasoning beyond their training distribution @xcite . to robustly claim that llms possess reasoning abilities, they must be tested under a wider array of conditions. a fundamental approach to overcoming these challenges is to decouple the test inputs from the llms' pre-training data, compelling them to rely on task descriptions for reasoning and planning rather than prior knowledge. this entails an environment with clear instructions, a rich but well-defined action space, and a distinct objective. such a setup facilitates the verification of the reasoning process and qualitative performance assessment, while it also reduces the chances of successful random guessing. minesweeper, a well-known logic puzzle game illustrated in figure 1 , is proposed as an ideal testing ground under these considerations. the game presents a range of complexities: while its fundamental principles are straightforward and accessible to beginners, achieving proficiency requires advanced logical reasoning skills that go beyond mere pattern recognition on the game board. for llms, the challenge in minesweeper lies in interpreting the states of individual cells, making use of numerical hints, and understanding the spatial interconnections between cells. this is essential for accurately determining the locations of mines and strategizing subsequent moves based on incomplete information--a process that is instinctive for human players. despite llms' prior knowledge of the universally known rules of minesweeper, the transformation of game boards from visual representations to machine-readable text has been limited, with no existing format aligning with our representation. consequently, llms must rely on their intrinsic logic for problem-solving by understanding and following the rules, as pattern matching from training data is impractical. building upon our initial motivation, we conduct a comprehensive set of experiments with llms in-cluding gpt-3.5-16k, gpt-3. @xcite , probing their inherent reasoning skills. we introduce the minesweeper game in various formats to determine how different types of input influence llm performance. objective measures like the count and proportion of accurate moves, correctly flagged mines, and completely resolved boards were employed to gauge effectiveness. however, a critical aspect of our evaluation is the manual examination of the llms' reasoning processes, inspecting the validity and logical soundness of their intermediate deductions. our findings indicate that these models generally struggle to maintain consistent logical reasoning chains. the gpt-3.5 versions, in particular, tended to repeat information from provided examples or previous conversations, often failing to adjust to the updated board layouts. while gpt-4 showed improvements in response diversity, relevance, and coherence, it still faced issues with logical inconsistencies and incomplete reasoning. our research categorizes errors in a detailed simulated environment, which sheds light on the specific skills required for solving logical puzzles, underscoring the importance of these competencies in future llm-focused studies. moreover, we notice that llms tend to generate ad-hoc sequences of actions rather than following logical reasoning chains based on action history, a finding not commonly reported, providing novel insights into llm operational behaviors. to support forthcoming research and comprehensive evaluation of llms, we have published our code and data at https://github.com/yinghao-li/minesweeper-forllm . we hope it contributes to llms' broader understanding and future development. ## related works recent studies have explored the reasoning capabilities of llms, with several investigations underway to understand @xcite @xcite @xcite , enhance @xcite , and leverage @xcite these abilities for various applications. some critics argue that the datasets used for evaluation are overly simplistic and potentially compromised by inclusion in the training data of llms @xcite , casting doubt on the authenticity of these purported capabilities @xcite . in response, new tasks and evaluation frameworks (a) "0","1","2","3","4","5","6","7","8","9" "1",`?',`?',`1',`.',`.',`.',`.',`1',`?' "2",`?',`1',`1',`.',`.',`.',`.',`1',`?' "3",`?',`1',`1',`1',`.',`.',`1',`2',`?' "4",`?',`2',`f',`1',`.',`.',`1',`f',`?' "5",`?',`2',`1',`1',`.',`1',`2',`2',`1' "6",`1',`1',`1',`1',`1',`1',`f',`1',`.' "7",`.',`.',`1',`f',`1',`1',`1',`1',`.' "8",`.',`.',`2',`?',`3',`2',`1',`1',`.' "9",`.',`.',`1',`?',`f',`?',`f',`1',`.' ## minesweeper minesweeper, a classic logic game, has been a part of the windows operating system since around 1990s. though the rule is simple, it features a rich and discrete action space, a clear goal, and direct feedbacks and evaluation metrics. the objective is straightforward: players must unveil all cells on a board peppered with hidden mines without detonating any. each cell on the board can exist in one of four possible states (figure 1b ): • unopened ("?"): the state of these cells is unknown; they could transition into any of the other three states. during gameplay, players may execute one of three actions per turn: left-click ("l") to reveal a cell's contents, right-click ("r") to flag a suspected mine, and middle-click ("m") to verify the correctness of flagged cells based on adjacent numbers. on a beginner's 9 × 9 board (figure 1a ), this results in 243 potential actions per round (figure 1d ), creating an extensive action landscape despite numerous actions being invalid (as listed in we employ two plain-text formats to present the game board when interacting with llms, aiming to understand llms' capabilities of comprehending each of them and to reduce the potential influence of input format on the reasoning process. the first format is table representation, depicted in figure 1b . here, the game board is portrayed as a table with the states of each cell enclosed in l a t e x-style quotation marks (grave accent "`" and apostrophe "'") to differentiate them from the cell indices and separators. rows and columns are delineated by line break "\n" and comma "," respectively, with indices for the first row and column serving as the coordinates for the cells. this format is intuitive for human interpretation; llms equipped with capabilities to understand tables, as suggested by the literature @xcite , should process it similarly. for comparison, we also employ coordinate representation (figure 1c ). it explicitly associates each cell's coordinates with its state in the form of a look-up table, potentially offering a more direct description for llms to comprehend the board's configuration. the following sections first explore two critical skills required for minesweeper agents ( § 4). then, ## conclusion in this study, we assessed the capabilities of llms in solving logical puzzles by examining their performance in the game minesweeper, a task for which they have received limited exposure during the pre-training phase. this approach aims to assess the intrinsic reasoning capabilities of llms. our experiments reveal that gpt-3.5 models exhibit basic abilities in spatial navigation, symbol recognition, and counting. however, they struggle with multi-step planning and generating coherent reasoning chains. gpt-4 shows improvement in simpler scenarios with shorter logic chains but still encounters issues with hallucinations and inconsistent context awareness. we find that gpt-4 can learn from examples and apply this knowledge to similar situations but is less adept at deducing underlying rules and applying them to novel scenarios, unlike gpt-3.5 models which tend to replicate examples. our conclusion seems to be in correlation with @xcite : llms do not produce new knowledge (a situation where logical reasoning is necessary); they are just (good) handlers of existing experience. we do not intend to diminish the significance of llms or their transformative impact on natural language processing. rather, we believe it is premature to consider llms as an intelligent object and a threat to human society. they resemble a comprehensive dictionary, useful for reference but lacking in comprehension. we hope that our work can inspire future research on this topic, and we will continue exploring the capabilities of llms.