Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding (2025)

Abstract: Vision Language Models (VLMs) have achieved remarkable progress in multimodal tasks, yet they often struggle with visual arithmetic, seemingly simple capabilities like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning. In this work, we first investigate the root causes of this deficiency through a suite of probing tasks focusing on basic visual arithmetic. Our analysis reveals that while pre-trained vision encoders typically capture sufficient information, the text decoder often fails to decode it correctly for arithmetic reasoning. To address this, we propose CogAlign, a novel post-training strategy inspired by Piaget’s theory of cognitive development. CogAlign trains VLMs to recognize invariant properties under visual transformations. We demonstrate that this approach significantly improves the performance of three diverse VLMs on our proposed probing tasks. Furthermore, CogAlign enhances performance by an average of 4.6% on CHOCOLATE and 2.9% on MATH-VISION, outperforming or matching supervised fine-tuning methods while requiring only 60% less training data. These results highlight the effectiveness and generalizability of CogAlign in improving fundamental visual arithmetic capabilities and their transfer to downstream tasks.

## introduction in recent years, vision language models (vlms) have rapidly advanced, demonstrating remarkable capabilities in integrating and processing multimodal information @xcite @xcite . these models have found extensive applications across various domains, ranging from visual commonsense reasoning to sophisticated tasks like web agents @xcite @xcite . by leveraging both visual and textual data, vlms promise a nuanced understanding that surpasses what can be achieved by analyzing them individually. despite these advancements, current vlms exhibit noticeable deficiencies in performing fundamental visual arithmetic: these models struggle with seemingly simple tasks like accurately counting objects, comparing lengths, assessing angles, and evaluating relative sizes or areas @xcite @xcite @xcite . these shortcomings are particularly evident in complex tasks such as chart understanding @xcite and geometric problem-solving @xcite . in this study, we first delve into the root causes of vlms' difficulties with visual arithmetic, exploring several hypotheses to elucidate why vlms often fail when faced with such challenges ( §2). we propose a suite of probing tasks, focusing on basic visual arithmetic such as length comparison, to answer this question. our analysis reveals that pre-trained vision encoders coupled with a simple linear classifier perform poorly on these probing tasks, indicating that a single linear layer is insufficient to decode the complex visual representations for arithmetic reasoning. however, when we finetune the text decoder of a vlm on these tasks, performance significantly improves. this suggests the bottleneck lies in the decoder's ability to effectively process and utilize the visual information, rather than in the visual representation itself. to tackle these challenges, we propose a novel post-training strategy, cogalign, designed to improve the performance of vlms in visual arithmetic tasks ( §3). drawing inspiration from piaget's theory of cognitive development @xcite , our method focuses on enhancing vlms' understanding of conservation (recognizing that certain properties remain unchanged despite transformations) and decentration (considering multiple aspects simultaneously). we train vlms using synthetically generated image pairs that demonstrate transformations, enabling them to compare and evaluate based on specific properties like length, angle, and quantity. by employing direct preference optimization (dpo) @xcite , the model learns from both positive and negative examples, offering a richer learning signal than traditional supervised fine-tuning (sft). our experiments show that cogalign significantly enhances performance across three vlms of different scales and architectures on the proposed probing tasks. furthermore, we evaluate cogalign on two downstream benchmarks: chocolate @xcite for chart understanding, and math-vision @xcite for geometric problem-solving ( §4). our results demonstrate the effectiveness of cogalign in enhancing performance on these complex tasks. on average, cogalign boosts performance by 4.6% and 2.9% on chocolate and math-vision respectively, demonstrating that improving fundamental visual arithmetic capabilities translates to improved performance on downstream tasks. notably, cogalign outperforms or achieves comparable performance to sft methods while requiring 60% less training data, even though cogalign does not involve direct optimization for specific tasks. this showcases its strong generalizability and highlights its potential of focusing on foundational skills to unlock broader capabilities in vlms. our main contributions are as follows: ## why vision language models struggle with visual arithmetic? as suggested in previous studies, vlms struggle with visual arithmetic @xcite , leading to poor performance in tasks involving such capabilities such as chart understanding @xcite and geometric problem-solving @xcite . in this section, we aim to understand the root causes behind such phenomenon. we first propose a suite of probing tasks we design to facilitate our analysis ( §2.1) and then illustrate the various analyses we conduct to validate our hypotheses ( §2.2). ## cogalign to address the challenges vlms face in performing visual arithmetic, we propose a novel post-training method inspired by piaget's theory of cognitive development @xcite , which outlines four stages: sensorimotor, preoperational, concrete operational, and formal operational. each stage represents a different ability to process information and solve problems, culminating in abstract reasoning. the concrete operational stage is particularly relevant. at this stage, children develop (1) conservation, understanding that certain properties like length remains constant despite changes in appearance, and (2) decentration, the ability to consider multiple aspects of a situation at once. these skills are essential for vlms to perform visual arithmetic accurately, recognizing invariant properties such as length or angle across transformations. current vlm training paradigms often neglect these cognitive processes, resulting in models that struggle to maintain key properties during visual transformations and to integrate multiple visual features effectively. while pre-trained visual encoders use losses that encourage some invariance to transformations, the integration of vision and language representa-tions in decoders often lacks explicit enforcement of conservation and decentration principles, leading to models that capture visual features but fail to reason about them effectively. to address these issues, we present a post-training method, cognitive alignment (cogalign), aimed at enhancing vlms' understanding of conservation and decentration. our approach explicitly trains vlms to recognize invariant properties like length, angle, and count across different visual transformations. we achieve this by presenting the model with pairs of figures and associated queries designed to highlight these properties. the queries prompt the model to compare and contrast the figures, focusing on whether a specific property is different or same despite variations in appearance. this approach encourages the model to develop a stronger understanding of geometric concepts and move beyond superficial visual comparisons. furthermore, we leverage dpo @xcite for training, rather than sft. dpo allows the model to learn from both positive and negative examples within the preference framework, providing a richer learning signal compared to sft. by strengthening these cognitive capacities within vlms, our goal query: which circle pairs are further apart? angle query: between these two angles, which one is greater? ## generalizability of cogalign now that we have demonstrated the advantage of cogalign on our probing tasks, we ask: does the improvement on simple visual arithmetic tasks transfer to more complex tasks? to answer this question, we explore whether cogalign en-2 gpt-4o is used for paraphrasing. ## conclusion this study investigates the challenges faced by vlms in performing visual arithmetic, revealing that while visual encoders often capture necessary information, text decoders struggle to effectively utilize it. we introduce cogalign, a novel posttraining strategy inspired by piaget's theory of cognitive development, focusing on enhancing vlms' understanding of conservation and decentration through dpo training. our evaluations show that cogalign not only enhances vlms' understanding of visual arithmetic, but also improves their performance in chart understanding and geometric problem-solving through experiments on the chocolate and math-vision datasets, showcasing its effectiveness and generalizability across various models and tasks. notably, cogalign often outperforms or achieves comparable results to task-specific supervised fine-tuning methods without direct training on the target domain, highlighting the potential of bolstering foundational cognitive skills for broader vlm capabilities. future work could explore how cogalign impacts other multimodal tasks beyond charts and geometry, potentially leading to a more unified approach in vlm training where generalizability is prioritized.