Instruction-tuned Q wen C hart for Chart Question Answering (2025)

Abstract: Charts, where information is delivered holistically by visual and textual features, represent a challenge when it comes to downstream tasks such as chart question answering, where both kinds of information contribute to the task. The standard approach is to decouple the task in two steps, first extracting information from the charts, or representing it as a table, text or code, and then a second reasoning step to output the answers. Today, the advancements in visual encoding of Visual Large Language Models (VLLM) have shown their capabilities to solve such complex tasks without using in-between representations of the charts or massive in-domain training. Our new instruction fine-tuned and chain-of-thought model QwenChart showed that even in a complex new benchmark such as SciVQA general models can achieve great performances with low-cost training, matching the capabilities that LLMs have showed in unimodal downstream tasks. An out-of-domain evaluation showed satisfactory results, albeit with an expected drop in performance.

## introduction everything in a chart conveys information: besides labels such as numbers or text, they feature shapes, colors and complex visual elements such as bars, lines or points that contribute to the delivery of their meaning. understanding complex texts such as scientific articles also requires chart comprehension, including answering questions about charts in natural language (qa over charts or chart qa). to tackle this task, previous work has focused on two main aspects: information extraction from charts and complex, often logical or arithmetic, reasoning over that information. early approaches would identify and extract information to feed into a classifier @xcite . since the rise of visual large language models (vllms), many approaches convert charts into a format suitable for a language model, such as text descriptions, @xcite , tables, or code @xcite @xcite , due to the limited resolution capability of the visual encoders, and the conversely great capabilities of the llms. while using tables instead of images leads to some information loss, this approach still remains preferable. despite reaching satisfactory performance in general visual understanding tasks, vllms have struggled with downstream chart understanding tasks @xcite @xcite @xcite . vllms usually consist of a visual encoder and a language decoder. the complexity of the visual features of charts represents a bottleneck for visual encoders, whereas the language decoder struggles to extract the necessary information from the visual representations due to the complexity of the relations between visual and linguistic elements @xcite . a common approach today involves augmenting data with task-specific instructions and fine-tuning a pre-trained model accordingly @xcite @xcite @xcite . some researchers have also opted to train the visual encoder using chart-table pairs to enhance its representational capabilities @xcite @xcite @xcite . @xcite introduced the scientific visual question answering (scivqa) shared task 1 , designed to evaluate multi-modal qa systems on real-world scientific figures through a diverse set of both finite and infinite questions. the task emphasizes reasoning over complex visualizations and includes chart types rarely represented in earlier datasets, such as architecture diagrams, confusion matrices, and compound figures. in this context, we propose qwenchart-7b, a vision-language model specifically designed for the scivqa task which achieved second place in the competition. qwenchart-7b has been instruction-tuned with low rank adaptation (lora, @xcite ) and exploits chain-of-thought (cot, @xcite to improve its reasoning capabilities. we show that qwenchart-7b is capable of achieving good performance in chart qa without pretraining on domain data or using in-between representations of charts, such as tables. furthermore, we show that scaling size of the model does not have a great impact on performance and identify what parameters mostly contribute to the performance of the model. our model is one of the first visual models reaching high performance on a challenging benchmarks such as scivqa without using intermediate representations of charts. our contributions are the following: ## analysis and discussions one of the key contributions of this work is the demonstration that high performance on chart understanding can be achieved using a visual model that does not rely on intermediate representations such as tables or code. this is particularly significant in the context of the scivqa benchmark, which features a diverse set of real-world charts. the strong performance of qwenchart-7b, which surpasses even its larger counterpart (qwenchart-72b), suggests that model architecture and prompt engineering may have a more substantial impact on downstream performance than model size. another advantage lies in the efficient training process enabled by lora. by fine-tuning only 13% of the model's parameters, we achieve competitive results while significantly reducing computational cost and preserving the core capabilities of the pre despite promising results on scivqa, our experiments reveal a performance drop on the chartqa benchmark, indicating that the model's generalization capability to out-of-domain data is limited. this suggests potential overfitting to the prompt format or chart types seen during fine-tuning. further efforts are needed to enhance the robustness of instruction-tuned models across datasets. we observed that the rouge-1, rouge-l, and bertscore metrics exhibit certain limitations when applied to this type of task. compared to bertscore, rouge proves to be more sensitive, as it is better able to highlight performance differences. rouge, in fact, imposes a heavier penalty on responses that do not exactly match the gold standard, making it more suitable for this task. however, this can also lead to an underestimation of model performance when responses are correct but differ in form from the reference answers. ## conclusions in this work, we introduced qwenchart-7b, an instruction-tuned vllm built on qwen2.5-vl for the shared task scivqa. our approach leverages dynamic cot prompting and lora-based parameter-efficient fine-tuning. despite its relatively small size, qwenchart-7b demonstrates state-of-the-art performance on the challenging scivqa benchmark, outperforming even larger models like qwenchart-72b. this suggests that architecture-specific optimization and welldesigned prompts can surpass gains from model scaling alone. however, we also observed limitations in out-of-domain generalization, particularly on the chartqa benchmark, indicating room for improvement. future work will explore richer multimodal alignment, broader datasets, and more generalized instruction strategies to address these challenges and further improve performance across diverse chart types and qa formats.