Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models (2024)

Abstract: Large Language Models (LLMs) demonstrate ever-increasing abilities in mathematical and algorithmic tasks, yet their geometric reasoning skills are underexplored. We investigate LLMs’ abilities in constructive geometric problem-solving, – one of the most fundamental steps in developing human mathematical reasoning, revealing notable challenges in this domain. LLMs exhibit biases in variable names, struggle with 2D spatial relationships and planning, and hallucinate object placements. To this end, we introduce a framework that enhances LLMs’ reasoning potential through a multi-agent system conducting internal dialogue. This work underscores LLMs’ limitations in geometric reasoning and improves their capabilities through self-correction, collaboration, and diverse role specializations.

## introduction recent advancements in large language models (llms) are groundbreaking, demonstrating increasing proficiency in complex mathematical and algorithmic tasks. despite this, llms still face significant challenges in constructive geometry, a field fundamental to human mathematical reasoning involving tool usage, planning, and spatial reasoning. our investigation in this domain reveals several intriguing aspects. in instruction following, llms often exhibit a bias towards the style of the examples rather than focusing on the reasoning necessary for solving these problems. furthermore, llms capable in maths do not necessarily show proficiency in geometrical problems, suggesting that algebraic reasoning does not directly translate to spatial and tool-based problem-solving. another observation is that the choice of variable names in geometric constructions can affect the length and quality of the solutions, pointing to a potential bias where variable names carry unintended semantic weight. moreover, despite being provided with visual aids, multimodal llms such as gpt4-v demonstrate difficulties interpreting 2d spatial relationships. they can identify objects in a scene but struggle to integrate them into a coherent plan involving tools or steps. we propose a framework to overcome these challenges. our solution includes appropriate renaming to mitigate naming biases and an adaptive prompt selection mechanism to focus the llm on relevant information, avoiding overload. the model builds on past geometric tasks, enhancing its reasoning and context awareness. a critical factor in our approach is using simulacra-based conversational agents @xcite with specialized roles, some acting as reasoners while others as solvers or tool users. this cross-domain dialogue leverages the strengths of each agent type and fosters a more effective problem-solving approach than traditional role-playing methods. contributions. the main contributions of our work can be summarized in three points: first, we are the first to provide an extensive analysis of the state-of-the-art leading llms' surprising difficulties in solving fundamental constructive geometric problems, highlighting a critical gap in their reasoning capabilities. second, we introduce three methods that assist llms in overcoming current limitations in the geometry domain. our dynamic prompting mechanism builds on previous interactions instead of uninformative static prompts, our variable renaming technique neutralizes biases from variable name conventions, and our scene description prompt enhances llms' abilities to understand and manipulate spatial relationships. third, we present a novel simulacra-based system that leverages role and domain differentiation to integrate tool usage, instruction following, and collaborative problem-solving. this system surpasses non-collaborative methods and standard simulacra variants, demonstrating adaptability and promising results across various mathematical domains beyond geometry. ## related work our approach is inspired by various research directions, briefly described here. prompt engineering the advanced reasoning abilities of multi-billion-parameter llms @xcite @xcite have transformed prompt engineering into sophisticated interactions for eliciting detailed responses. works like @xcite @xcite use intermediate reasoning steps in prompts, improving performance in arithmetic and symbolic reasoning tasks. diverging from hand-crafted prompts, @xcite @xcite propose automated prompt generation methods, exhibiting better results in reasoning tasks. in multi-agent scenarios, @xcite introduce inception prompting, enabling collaborative environments under role assignment. recently, @xcite @xcite use tree-search and self-evaluation for exploration and strategic lookahead, while @xcite unifies planning and acting, prompting models to generate reasoning traces and actions. ## preliminaries in this section, we present the datasets, models, metrics, and definitions central to our framework. euclidea our primary benchmark is the geometry game euclidea (euclidea), an online construction challenge with eight geometric tools and progressively complex problems. we use the python version (pyeuclidea), which includes ninety-eight challenges across ten levels and a custom api for solution verification. we also compile a natural language version of the euclidea dataset, including solutions from https://euclidea.fandom.com/ wiki/euclidea_wiki , which we will make accessible for future research. ## method this section introduces the components of our proposed framework, each addressing the limitations of llms in solving geometrical problems. ## experiments we present a comprehensive performance analysis on the euclidea dataset in in few-shot, models are prompted with the task, tool descriptions, and five solved examples using geometric tools. in finetuned, all open-source models are finetuned using euclid's "elements" to acquire foundational knowledge. aside from tutorial levels, euclidea challenges do not directly overlap with "elements" problems. we use the same setup for testing as in few-shot. simulacra refers to our proposed multi-agent framework, equipped with an adaptive few-shot mechanism, visual relations prompting, and variable renaming. initially, all models performed modestly in our few-shot experimental setup, with noticeable improvements after fine-tuning. this outcome aligns with our expectation that familiarity with standard mathematical and reasoning scenarios does not ensure proficiency in constructive geometry tasks. we posit that fine-tuning with euclid's "elements" represents the upper limit of improvement achievable by open-source models, constrained by the dataset's size. metamath-mistral 7b is the most promising among the open-source options, which we further examine with our multi-agent setup. our results reveal a significant performance boost in larger models (chatgpt / gpt4) and, notably, in metamath-mistral 7b, under our proposed framework, surpassing the few-shot chatgpt in performance. this finding underscores the adaptability and effectiveness of our approach across a spectrum of model sizes. we also compare against two prior studies on the euclidea dataset. the model by @xcite combines a masked-rcnn detector with an iterative deep search algorithm, using the euclidea api to verify each step until a solution is found within a pre-defined depth limit. euclid-net @xcite uses "deep visual reasoning with backtracking," where a neural network assists in step selection. these models rely on precise visual component extractors and real-time feedback during their trial-and-error process. to assess the possible benefits of real-time feedback, we introduce feedback mode (+fb), where validators have access to ground truth answers instead of relying solely on their internal knowledge but are restricted to simply informing solvers about the correctness of their steps, avoiding any solution leak. our findings show only a slight performance improvement, indicating that solvers struggle to develop complex solutions even with ground truth hints, highlighting the nuanced complexities of applying llms to constructive geometry. ## ablation studies this section presents ablation studies that underpin our model's development, as detailed in the methods section. these studies highlight the iterative refinement and integration of model elements, addressing limitations observed in llms during geometric problem-solving. ## conclusions our study highlights the challenges llms face in constructive geometry, including limited skill transfer from other mathematical domains, inadequacies of typical prompting techniques, and a lack of 2d spatial reasoning. we identify that existing llms struggle with geometric tasks without inductive solid biases like theorem provers @xcite . furthermore, our research demonstrates that a multi-agent system with role and domain spe-cializations can effectively address geometric problems and perform equally well in other domains. we hope our work serves as a foundation for developing training systems that operate in multi-agent settings, which is crucial for domains requiring deep, specific, and accurate cognitive processing.