Forgotten Polygons: Multimodal Large Language Models are Shape-Blind (2025)

Abstract: Despite strong performance on vision-language tasks, Multimodal Large Language Models (MLLMs) struggle with mathematical problem-solving, with both open-source and state-of-the-art models falling short of human performance on visual-math benchmarks. To systematically examine visual-mathematical reasoning in MLLMs, we (1) evaluate their understanding of geometric primitives, (2) test multi-step reasoning, and (3) explore a potential solution to improve visual reasoning capabilities. Our findings reveal fundamental shortcomings in shape recognition, with top models achieving under 50% accuracy in identifying regular polygons. We analyze these failures through the lens of dual-process theory and show that MLLMs rely on System 1 (intuitive, memorized associations) rather than System 2 (deliberate reasoning). Consequently, MLLMs fail to count the sides of both familiar and novel shapes, suggesting they have neither learned the concept of “sides” nor effectively process visual inputs. Finally, we propose Visually Cued Chain-of-Thought (VC-CoT) prompting, which enhances multi-step mathematical reasoning by explicitly referencing visual annotations in diagrams, boosting GPT-4o’s accuracy on an irregular polygon side-counting task from 7% to 93%. Our findings suggest that System 2 reasoning in MLLMs remains an open problem, and visually-guided prompting is essential for successfully engaging visual reasoning.

## introduction as multimodal large language models (mllms) demonstrate success in vision-language tasks @xcite , researchers seek to analyze their underlying mechanisms (golovanevsky et al., 2025; jiang et al., 2024; luo et al., * equal contribution. order determined by coin flip. 2024) and evaluate their ability to perform generalized problem-solving compared to human reasoning @xcite . a key aspect of human reasoning is the interplay between intuitive, reflexive thinking (known as system 1) and deliberate, logical reasoning (known as system 2) @xcite @xcite . while humans engage in both types of reasoning, a key question arises: can mllms move beyond memorized responses of system 1 to engage in the analytical reasoning of system 2? mathematics provides a powerful framework for this investigation, as fundamental reasoning skills are crucial for complex problem-solving. the combination of logic, abstract concepts, and specialized symbolic language makes mathematical reasoning problems a strong test of the extent to which models engage in system 2 reasoning rather than relying solely on memorized concepts @xcite . studying system 2 reasoning abilities is especially relevant given recent progress in natural language processing, where large language models (llms) like gpt-4 @xcite , qwen @xcite , and llama @xcite have demonstrated notable success on mathematical benchmarks like gsm8k @xcite and math @xcite . despite these advancements, mllms remain limited in visual-mathematical reasoning. while closedsource models like gpt-4o demonstrate potential, their performance on benchmarks such as mathverse @xcite remains far below human levels, with open-source models typically performing even worse @xcite . even mllms specially trained for visual, mathematical reasoning (g-llava @xcite , math-puma @xcite , math-llava @xcite cannot close the gap between open-source and closed-source models. many benchmarks feature complex geometric diagrams that require multi-step reasoning, with human performance reaching no more than 70% accuracy @xcite . @xcite show that mllms tend to perform better on more complex mathematical tasks but struggle with simple tasks designed for young children. this counterintuitive result raises concerns about whether mllm performance on visualmathematical benchmarks reflects genuine mathematical reasoning (system 2) or merely the retrieval of familiar concepts from training data (system 1). we design a two-part task to evaluate mllms' reasoning abilities: (1) shape recognition, which relies on visual recall (system 1), and (2) side counting, which engages visual reasoning (system 2). starting with common geometric shapes and properties allows us to pinpoint specific failures in how mllms process mathematical diagrams. our results show that while the underlying llms demonstrate perfect accuracy on non-visual questions about polygon names and side counts, their multimodal counterparts fail when models have to rely on images to answer questions correctly. unlike humans, who can count a shape's sides and infer its identity (e.g., "1, 2, . . . , 7" → "heptagon"), our results suggest that mllms do not attempt such reasoning. instead, they rely on system 1 heuristics like shape memorization and spurious correlations between visual features and labels, ignoring vital information in the image. to understand why mllms struggle with polygon identification from visual inputs, we analyze vision encoder embeddings. our results show that vision encoders are "shape-blind":common shapes form distinct clusters, while less frequent ones like pentagons, heptagons, and octagons overlap (section 3.1.3). even models specifically trained for geometric understanding, such as g-llava, math-llava, and math-puma, exhibit shape-blindness where distinct polygons are embedded in the same region in vector space. we further demonstrate vision encoders' shape-blindness with a two-shape multi-step task: identifying shapes, counting the sides, and computing the total sum of sides, adding complexity by incorporating arithmetic into the reasoning process. on average, models correctly identify both shapes in an image only 27.58% of the time, as shape recognition depends entirely on visual input. however, mllms compute the sum of the identified shapes' sides correctly in 70.75% of cases, indicating that despite errors in recognition, the summation operation remains highly accurate. while open-source models struggle with less common polygons, gpt-4o identifies shapes with at least 49% accuracy. to test whether models use system 2 reasoning, we introduce a dataset of abstract shapes and irregular polygons unlikely to have appeared in training (see figure 1 ). no model accurately counts the sides of these novel shapes, reinforcing their reliance on memorization rather than genuine geometric understanding. as mllms continue to advance, our findings urge the community to re-examine the complexity of visual-mathematical benchmarks. if state-ofthe-art open-source models fail to recognize simple shapes, should we be evaluating them on complex geometric tasks? our results suggest that current models rely heavily on factual recall rather than true geometric reasoning, limiting their ability to generalize beyond familiar shapes. to explore ways of bridging this gap, we take a step forward with visually-cued chain-of-thought (vc-cot) prompting, leveraging the fact that many geometric datasets already contain annotated shapes (e.g., labeled triangle vertices a, b, and c). by explicitly guiding the model to reference image annotations (letters or numbers) and reason about their relationships, vc-cot significantly improves gpt-4o's ability to count the sides of novel shapes, boosting accuracy from 7% to 93%. across molmo, janus-pro, gpt-4-turbo, and gpt-4o, our vc-cot prompts achieve an average accuracy improvement of approximately 5% compared to standard cot prompts on mathverse. these findings highlight the importance of structured prompting in strengthening the connection between visual perception and mathematical reasoning in mllms. ## related works despite the success of mllms, emerging research highlights their struggles with reasoning. @xcite find that vision-language models struggle to negate in retrieval and multiplechoice tasks. beyond general reasoning, open-source, closed-source, and math-specific mllms perform poorly on visual math benchmarks @xcite . a key limitation in mllms is the vision encoder. @xcite show clip-based vision encoders fail to capture fine-grained details -a detrimental trait for tasks requiring precise visual reasoning. similarly, in the visual-mathematics domain, @xcite al. (2023); shi et al. (2024); zhang et al. (2024b); zhuang et al. (2024) demonstrate that vision encoders produce inadequate representations of mathematical diagrams and vision-text misalignments hinder multimodal reasoning. one way to address these challenges is finetuning. recent works adopt a multi-step approach: in the language domain, chain-of-thought (cot) prompting has proven highly effective in encouraging system 2 reasoning in llms @xcite . @xcite further reinforces this, demonstrating that structuring reasoning through cot can significantly improve logical inference and problem-solving. however, applying cot to mllms has been far less successful. @xcite show that open-source mllms struggle with cot, largely due to limitations in existing visual-instruction tuning datasets, which prioritize short, simplistic responses over structured reasoning. thus, recent works explicitly fine-tune mllms for cot reasoning, particularly in object counting and mathematical reasoning tasks @xcite @xcite . however, even these fine-tuned models continue to struggle on complex visual-mathematical benchmarks such as mathvista @xcite , and mathverse @xcite . these challenges highlight that despite recent efforts, system 2 reasoning in mllms remains an open problem, with current approaches failing to achieve generalizable reasoning abilities. ## conclusion our study highlights fundamental limitations in mllms' ability to integrate visual information for reasoning. while llm backbones possess strong geometric knowledge, vision encoders remain the primary bottleneck, forcing mllms to rely on system 1 thinking instead of system 2 reasoning. even in simple tasks like shape identification and side counting, mllms default to memorized patterns rather than systematically analyzing visual inputs. as an initial step to improving mllm reasoning, we introduce visually-cued chain-of-thought (vc-cot) prompting to engage system 2 reasoning. our results show that explicitly guiding models to extract and reason about visual cues improves performance, boosting gpt-4o's accuracy by 86% on our shape dataset as well as enhancing results on mathverse. our findings serve as a broader call to action for the mllm research community. to effectively pinpoint and address limitations in visionlanguage integration, we emphasize the need to explore simple, controlled scenarios before engaging with complex multimodal benchmarks. by improving fine-grained visual perception and utilizing visual cues for prompting, researchers can move toward mllms that engage in true visionbased system 2 reasoning rather than defaulting to llm-driven deduction.