Visual Question Answering on Scientific Charts Using Fine-Tuned Vision-Language Models (2025)

Abstract: Scientific charts often encapsulate the core findings of research papers, making the ability to answer questions about these charts highly valuable. This paper explores recent advancements in scientific chart visual question answering (VQA) enabled by large Vision Language Models (VLMs) and newly curated datasets. As part of the SciVQA shared task from the 5th Workshop on Scholarly Document Processing, we develop and evaluate multimodal Systems capable of answering diverse question types - including multiple-choice, yes/no, unanswerable, and infinite answer set questions - based on chart images extracted from scientific literature. We investigate the effects of zero-shot and one-shot prompting, as well as supervised fine-tuning (SFT), on the performance of Qwen2.5-VL models (7B and 32B variants). We also tried to include more training data from domain-specific datasets (SpiQA and ArXivQA). Our fine-tuned Qwen2.5-VL 32B model achieves a substantial improvement over the GPT-4o-mini baseline and reaches the 4th place in the shared task, highlighting the effectiveness of domain-specific fine-tuning. We published the code for the experiments.

## introduction figures are often the first thing that readers of scientific papers look at @xcite . also, they frequently communicate the main results. therefore, the ability to extract information from scientific chart images would be of great value. however, automatically interpreting charts poses challenges due to their detailed visual components and the complex spatial arrangements of elements. the process requires spatial reasoning and numerical understanding @xcite . new sota vlms like qwen2.5-vl @xcite enable better results in the domain of chart vqa @xcite . furthermore, recent datasets, like 1 https://github.com/flo0620/scientific-chart-qa figure 1 : overview of the system with the four question types: infinite answer set, yes/no, multiple-choice, and unanswerable. spiqa @xcite and arxivqa @xcite , provide large amounts of data on scientific chart vqa. this paper intends to explore these new possibilities in the context of the scivqa shared task @xcite . it challenges participants to answer questions about scientific charts. an example of such questions can be seen in figure 1 . the contributions of this paper are: ## related work the scivqa shared task invites participants to develop multimodal systems for vqa on scientific charts @xcite . to support this, we use the scivqa dataset foot_0 , which contains 3,000 real-world chart images from scientific papers, each paired with seven questions. the dataset features four question types: multiple-choice, yes/no, unanswerable, and infinite answer set questions. these are further categorized into visual and non-visual questions, where visual questions refer to attributes such as size, height, color, direction, shape, or position. this task aligns with growing research interest in chart-based vqa, where existing benchmarks such as chartqa have seen performance plateaus among large vlms, largely due to limited data diversity @xcite . in response, new benchmarks such as spiqa @xcite and arxivqa @xcite have been introduced to address this limitation by using more diverse scientific charts from the real world. these new datasets provide additional training data to fine-tune vlms. @xcite found a positive correlation between the size of the training set and the model performance when fine-tuning a vlm for chart vqa. furthermore, @xcite showed that the used prompt has a significant influence on the results of the task of vqa on charts, underlining the importance of prompt engineering. recent progress in the field of vlms includes models such as qwen2.5-vl @xcite , a successor to qwen2-vl that has achieved sota results in chart vqa tasks @xcite . there are also models specifically developed for chart-related tasks, such as chartllama, which performed fine-tuning on a curated dataset and reached good results on the chartqa benchmark @xcite . chartassistant @xcite and chartvlm @xcite finetuned models to perform chart-to-table translation. the output of these models is then used as input to specialized models fine-tuned for vqa. the fine-tuning of such models is possible through low rank adaptation (lora) @xcite , which drastically reduces the memory and computation requirements for the training. ## experiments to explore the influence of the prompting strategy, the effectiveness of domain-specific fine-tuning, and the importance of the training dataset size, we conducted four different experiments to tackle the task. firstly, we tried zero-shot inference in combination with prompt engineering. secondly, we performed one-shot prompting with one example. third, the vlms were fine-tuned on the dataset provided by the task 2 . lastly, we expanded the finetuning by including the spiqa @xcite and arxivqa @xcite datasets in the training data. all four approaches were tested on the 7b and 32b variants of the qwen2.5-vl model @xcite and the first two on gpt-4o mini foot_1 (openai et al., 2024). ## evaluation this section presents the main experimental results and provides a detailed comparison between our models and the gpt-4o mini model, including a manual error analysis in the ablation study. ## conclusion this paper explored the application of vlms for scientific chart vqa in the context of the scivqa shared task. we evaluated zero-shot and one-shot prompting alongside domain-specific fine-tuning using lora on qwen2.5-vl models. our experiments showed that fine-tuning, especially on the scivqa dataset alone, led to the most significant performance gains, outperforming the gpt-4o mini baseline and reducing hallucinations. in contrast, incorporating external datasets offered limited benefits, possibly due to suboptimal training conditions or data mismatch. overall, the results emphasize the value of targeted fine-tuning and careful dataset curation for improving vqa on scientific charts. future work could include a more sophisticated, possibly manual, selection of training data to further improve the fine-tuning. testing different hyperparameters for the fine-tuning with more data might also improve the results.