On the Perception Bottleneck of VLM s for Chart Understanding (2025)

Abstract: Chart understanding requires models to effectively analyze and reason about numerical data, textual elements, and complex visual components. Our observations reveal that the perception capabilities of existing large vision-language models (LVLMs) constitute a critical bottleneck in this process. In this study, we delve into this perception bottleneck by decomposing it into two components: the vision encoder bottleneck, where the visual representation may fail to encapsulate the correct information, and the extraction bottleneck, where the language model struggles to extract the necessary information from the provided visual representations. Through comprehensive experiments, we find that (1) the information embedded within visual representations is substantially richer than what is typically captured by linear extractors, such as the widely used retrieval accuracy metric; (2) While instruction tuning effectively enhances the extraction capability of LVLMs, the vision encoder remains a critical bottleneck, demanding focused attention and improvement. Therefore, we further enhance the visual encoder to mitigate the vision encoder bottleneck under a contrastive learning framework. Empirical results demonstrate that our approach significantly mitigates the perception bottleneck and improves the ability of LVLMs to comprehend charts.

## introduction charts are essential for representing and analyzing data, commonly appearing in scientific papers, financial reports, and news articles. unlike natural images, where semantic content is often apparent through object recognition, charts encode dense quantitative and relational information via visual elements such as bars, lines, and points, along with their spatial relationships. this information-dense property creates higher perception challenges for large vision-language models (lvlms), which, despite success in general visual understanding tasks @xcite @xcite , often struggle with chart understanding @xcite @xcite @xcite , as demonstrated in figure 1 . in this work, we systematically examine the perception bottleneck of lvlms by analyzing it through two key components: the vision encoder and the language model. specifically, we define perception as the model's ability to accurately extract visual information from an image. for most lvlms equipped with a dedicated vision encoder, the process of perceiving visual signals can be broken down into two stages: first, the vision encoder encodes the image into compact vector representations; second, the language model extracts the relevant information from these encoded vectors. accordingly, we decompose the perception bottleneck into two categories: the vision encoder bottleneck and the extraction bottleneck. our objective is to investigate how these two distinct bottlenecks impact the overall perception capability of lvlms and how to mitigate them. we begin our study of the vision encoder bottleneck by evaluating the chart understanding ability of clip, a widely used vision encoder in many lvlms @xcite . specifically, we construct an image-text retrieval test set using existing chart-specific datasets and assess clip with the standard retrieval accuracy. we find the clip performs nearly random retrieval accuracy on these chart datasets, which suggests it may experience significant information loss, as several prior studies use clip's retrieval accuracy as an indicator of the information contained in its visual embeddings @xcite . while some researchers attribute this failure to clip's inductive bias or intrinsic limitations @xcite , we figure 1 : an example of perception qa from the plotqa dataset @xcite , along with the responses from gpt4-o @xcite , minicpm @xcite , and idefics2 @xcite for this example. (the chart has been redrawn for clarity in presentation.) successfully develop enhanced clip models with substantially improved retrieval accuracy. specifically, we fine-tune clip on chart-specific datasets within a contrastive learning framework and incorporate hard negative captions @xcite . the gains of over 20 absolute points in our enhanced clip strongly suggest that clip can indeed learn subtle or non-semantic features through further contrastive learning. to investigate the extraction bottleneck in the language model part, we shift our focus to lvlms built on top of these clip vision encoders. specifically, we conduct llava-style training @xcite combined with chart-specific instruction tuning. our initial observations reveal that lvlms trained with the llava data perform poorly on chart understanding tasks, while achieving substantial improvement further fine-tuned on chartspecific data, even with the vision encoder kept frozen. this finding not only indicates that domainspecific instruction tuning effectively addresses the extraction bottleneck, but more interestingly, it suggests that poor clip retrieval accuracy does not necessarily indicate a lack of useful encoded information. in contrast, evaluating across seven chart-related benchmarks, spanning both in-distribution and out-of-distribution scenarios, our enhanced clipsbased lvlms further achieve larger gains due to the mitigation of the vision encoder bottleneck. notably, compared to the original clip-based lvlms, the enhanced clip-based models using the llava-v1.5-13b architecture achieve an average improvement of nearly 3 points, while the model employing the llava-v1.5-phi-3.8b architecture demonstrates an even more significant improvement of 5 points. finally, we conduct an in-depth analysis to understand how the superior performance of clip translates to its lvlm counterpart. by scaling instruction tuning on larger chart datasets and analyzing clip-llava correctness statistics, we observe that samples correctly classified by clip are more easily learned by the lvlm, suggesting that the more salient representations obtained from the enhanced clip facilitate better lvlm learning. these findings further raise rethinking about information encoding in clip and its effect on lvlms. ## the challenge of chart understanding to better illustrate the perceptual challenges in chart understanding, we examine one concrete perception qa example from plotqa @xcite . as shown in figure 1 , to answer "what is the revenue generated (in %) of russian federation in 2006?", models need to: (1) correctly match the dotted green line with its legend label, (2) locate the intersection point between this line and the vertical line at 2006, and (3) accurately map this point to the y-axis scale to obtain the value (∼ 24%). while humans can perform this visual reasoning process effortlessly, current models like gpt4-o @xcite , minicpm @xcite and @xcite often struggle with such perception tasks as demonstrated in figure 1 . unlike natural images, chart understanding presents unique perception challenges as it requires accurately encoding and processing dense quantitative information encoded in visual elements. recent studies have quantitatively revealed these perceptual limitations through new chart-specific benchmarks @xcite @xcite . to better understand the sources 10830 clip 3 bars clip 4 bars llm 4 bars there are 3 bars there are 4 bars clip contrastive learning adapt clip to lvlm llm 4 bars llm 3 bars figure 2: left: a clip-blind case where the original clip fails to discriminate the number of bars in the chart. by leveraging contrastive learning with hard negatives, the enhanced clip model learns more discriminative visual features successfully. right: when adapted to lvlms, after instruction tuning, the original clip-lvlms are possible to correctly interpret the chart information even when the original clip fails to discriminate it. however, the enhanced clip-lvlms enable faster learning and achieve higher overall performance. of these limitations, we decompose the perception bottleneck into two key components. (1) the vision encoder bottleneck: this occurs when the vision encoder fails to encode critical information from the image into its embeddings, leading to inevitable failures in downstream lvlm tasks. (2) the extraction bottleneck: even when the image embeddings contain the necessary information, the llm struggles to extract and interpret them correctly, resulting in erroneous outputs. in our study, we investigate the impact of these two bottlenecks and propose strategies to mitigate them on the chart understanding task. next, we start by analyzing the vision encoder bottleneck. ## the vision encoder bottleneck: investigating and improving clip as clip @xcite serves as the vision encoder in most lvlms @xcite @xcite , we focus on clip to investigate the vision encoder bottleneck. in this section, we construct a framework for training and evaluating clip's chart understanding abilities. ## the extraction bottleneck: connecting clip to lvlm upon finishing our study of clip for the vision encoder bottleneck, we shift our focus to the extraction bottleneck to understand how these clip models impact llavas. having observed the poor performance of the original clip and the improved performance of fine-tuned clips, we aim to answer two questions: ## conclusion this study explores the perception bottlenecks of lvlms for chart understanding through the vision encoder bottleneck and the extraction bottleneck. we address the vision encoder bottleneck through chart-tailored contrastive learning. furthermore, lvlms built on these improved clip models demonstrate substantial performance gains. our findings emphasize how the capabilities of clip influence llava's downstream task performance, offering valuable insights into understanding clip information encoding.