Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling (2025)

Abstract: Scholarly articles convey valuable information not only through unstructured text but also via (semi-)structured figures such as charts and diagrams. Automatically interpreting the semantics of knowledge encoded in these figures can be beneficial for downstream tasks such as question answering (QA). Current approaches to visual question answering often struggle with the precision required for scientific data interpretation, particularly in handling numerical values, multi-step reasoning over visual elements, and maintaining consistency between visual observation and textual reasoning. We present our approach to the SciVQA 2025 shared task, focusing on answering visual and non-visual questions grounded in scientific figures from scholarly articles.Our strongest individual model, InternVL3, achieved ROUGE-1 and ROUGE-L F1 scores of \textbf{0.740} and a BERTScore of \textbf{0.983} on the SciVQA test split. We also developed an ensemble model with multiple multimodal small language models (MSLMs). Through error analysis on the validation split, our ensemble approach achieves significant improvements over individual models and achieved ROUGE-1 and ROUGE-L F1 scores of \textbf{0.735} and \textbf{0.734} , respectively, and a BERTScore of \textbf{0.979} on the SciVQA test split. Our findings underscore the effectiveness of prompt optimization, chain-of-thought reasoning and ensemble modeling in improving the model’s ability in visual question answering.

## introduction scientific literature communicates complex ideas not only through text but also through carefully designed visual elements including charts, graphs, diagrams, and technical illustrations. these visualizations serve as dense information carriers, encoding quantitative relationships, experimental results, architectural designs, and conceptual frameworks that are essential for scientific understanding. the ability to automatically interpret and reason about these visual elements represents a critical challenge in advancing scientific ai systems. the task of visual question answering (vqa) over scientific figures presents unique challenges that distinguish it from general-domain vqa. scientific visualizations demand mathematical precision, often requiring exact numerical extraction and calculation. they involve complex compositional reasoning across multiple visual elements, and frequently contain domain-specific conventions, symbols, and representations that require specialized understanding @xcite . furthermore, scientific figures often embed multiple layers of information, including raw data points, derived trends, statistical relationships, and comparative analyses. current vqa models, while showing impressive performance on general datasets, often struggle with the precision and reasoning depth required for scientific applications @xcite . common failure modes include visual grounding errors, where models misinterpret chart elements or scales; compositional reasoning failures, where multi-step logical processes break down; and consistency issues between visual observations and textual explanations @xcite . this paper presents our approach to the scivqa shared task foot_0 foot_1 @xcite , focusing on qa over scientific visualizations. the task involves answering closed-ended visual (i.e., addressing visual attributes such as colour, shape, size, height, etc.) and non-visual (not addressing figure visual attributes) questions. we leverage the reasoning and visual understanding capabilities of vlms, and employ task-specific chain-of-thought (cot) @xcite prompting techniques to retrieve and summarize relevant information from the visual-izations. our approach involved testing multiple prompt variants and selecting optimal configurations based on validation performance. to support reproducibility and future research, we make the code publicly available on github foot_2 . our main contributions are: ## related work recent advancements in chart-based qa have focused on various approaches to understanding and generating responses about visualizations. chartllama @xcite and unichart @xcite demonstrate the benefits of chart-specialized language models, showing improved performance in both chart captioning and qa tasks. these works often rely on explicit chart structure parsing as a preprocessing step, achieving strong results on synthetic chart datasets. chart-based reasoning (carbune et al., 2024) and llamav-o1 @xcite propose decomposed reasoning traces and transfer of llm capabilities to visual settings. our work builds on these insights by emphasizing structured reasoning and adopt step-level supervision to encourage coherent and faithful intermediate reasoning in visual contexts. other relevant works include spiqa @xcite and mathvista @xcite , which evaluate visual reasoning in scientific domains. mathvista particularly focuses on precise numerical and symbolic interpretation in mathematical visualizations, similar to our emphasis on scientific accuracy. in the realm of prompt engineering and model alignment, @xcite proposed spri (situated-principles), a framework that automatically generates context-specific guiding principles for each input query to improve model alignment. their approach demonstrates that instance-specific principles can outperform generic ones, which informs our ensemble methodology that combines prompt engineering with multiple vlms. motivated by recent advances in prompt rewriting @xcite , we explore instruction tuning and prompt optimization to enhance model adherence to scientific qa formats. unlike previous work that focuses on architectural innovations requiring additional training, our approach focuses on ensemble strategies and prompt optimization for maximum performance on scientific vqa tasks. ## dataset the scivqa dataset comprises scientific figures from acl anthology and arxiv papers. each figure is annotated with seven question-answer pairs and associated metadata including captions, figure ids, figure types (e.g., compound, line graph, bar chart, scatter plot), and qa pair types, with dataset splits and distributions detailed in section a and tables 3 , 4 , and 5. ## methodology our system integrates three key components: we utilized the vllm @xcite engine for maximum compute utilization during inference. a40 instances were sufficient for 7b models, while 8b models required a100 gpus. cot inference required approximately twice the computation time due to the two-level reasoning process, but provided significant quality improvements. ## conclusion and future work this work demonstrates that advanced vision encoding architecture, combined with systematic prompt engineering, provides a highly effective approach to scientific visual question answering. high-quality visual understanding is critical for strong performance. our approach establishes a strong baseline for the scivqa dataset with a rouge-1 and rouge-l f1 score of 0.740.