E xpert N eurons at S ci VQA -2025: Retrieval Augmented VQA with Vision Language Model ( RAVQA - VLM ) (2025)

Abstract: We introduce RAVQA-VLM, a novel Retrieval-Augmented Generation (RAG) architecture with Vision Language Model for the SciVQA challenge, which targets closed-ended visual and nonvisual questions over scientific figures drawn from ACL Anthology and arXiv papers (Borisova and Rehm, 2025). Our system first encodes each input figure and its accompanying metadata (caption, figure ID, type) into dense embed- dings, then retrieves context passages from the full PDF of the source paper via a Dense Passage Retriever (Karpukhin et al., 2020). The extracted contexts are concatenated with the question and passed to a vision-capable generative backbone (e.g., Phi-3.5, Pixtral-12B, Mixtral-24B-small, InterVL-3-14B) fine-tuned on the 15.1K SciVQA training examples (Yang et al., 2023; Pramanick et al., 2024). We jointly optimize retrieval and generation end-to-end to minimize answer loss and mitigate hallucinations (Lewis et al., 2020; Rujun Han and Castelli, 2024). On the SciVQA test set, RAVQA-VLM achieves significant improvements over parametric only baselines, with relative gains of +5% ROUGE1 and +5% ROUGE-L, demonstrating the efficacy of RAG for multimodal scientific QA.

## introduction scientific literature often conveys core findings through figures such as bar charts, line graphs, scatter plots, and compound diagrams. understanding these figures requires interpreting both visual cues (e.g., color, shape, and size) and associated textual elements (e.g., captions, methodology descriptions, result interpretations) @xcite . this multimodal nature presents challenges for automated systems aiming to answer questions about scientific figures. traditional vision-only architectures such as standard convolutional neural networks (cnns) and object detection models like faster r-cnn @xcite are limited to spatial and visual patterns and typically fail to reason over abstract visual encodings used in scientific plots. on the other hand, language-only models cannot perceive visual structure or layout, making them unsuitable for figure-centric reasoning tasks @xcite . recent advances in large vision-language models (lvlms), such as intervl-3-14b @xcite , qwen-2.5-vl @xcite , phi-3.5 @xcite , and mistral-small-24b (mistral ai, 2025), have enabled more robust multimodal understanding. however, these models often produce hallucinated answers when key context is missing or ambiguous @xcite . retrieval-augmented generation (rag) offers a potential remedy by enriching model inputs with contextually relevant external passages at inference time @xcite . to accelerate research in this area, the scivqa shared task @xcite provides a benchmark dataset of 3,000 figures from scientific documents, each accompanied by seven questionanswer pairs and includes metadata such as caption, figure id, figure type (e.g., compound, line graph, bar chart, scatter plot), qa pair type. the task emphasizes both visual and non-visual question types, facilitating comprehensive evaluation across multimodal reasoning skills @xcite . we build on these insights to propose retrieval-augmented generation architecture with vision language model (ravqa-vlm), a unified framework that: ## dataset the scivqa dataset foot_1 comprises scientific figures extracted from papers in the acl anthology and arxiv, each annotated with question-answer (qa) pairs and associated metadata. the dataset is organized into three splits: a training set with approximately 15k instances, a validation set with 1.7k instances, and a test set containing 4.2k instances. an instance is one datapoint consisting of figure and its respective question answer pair. each qa pair in scivqa is categorized along two key dimensions: answerability and visual grounding. based on answerability, qa pairs are labeled as either closed-ended (answerable solely from the image or image+caption), unanswerable (not inferable from the given source), finite answer set (with binary or multiple-choice answers), or infinite answer set (requiring open-form answers, such as numerical sums). based on visual grounding, qa pairs are classified as either visual-requiring interpretation of figure elements like shape, size, position, height, direction, or colour-or non-visual, which do not involve these aspects. the dataset also provides annotations for figure types, distinguishing between compound figures-those composed of multiple subfigures-and non-compound figures, which depict a single visual element. figure types span common scientific visualizations such as line charts, bar charts, box plots, confusion matrices, and pie charts. we perform all the evaluation on test set only. figure 1 presents a sample qa pair along with its corresponding figure from the test set. in the scivqa dataset, each figure is accompanied by a caption and is paired with seven distinct qa pairs, each corresponding to a different qa pair type. the example shown illustrates one such qa pair, demonstrating the format of the image, caption, and associated multiple-choice question and answer. we also do preprocessing of images in the later step as mentioned in setting c. captions and questions are tokenized using the bert tokenizer, with a maximum sequence length of 512 subword tokens. this preprocessing ensures a consistent input structure for our rag-based architecture. the term subword tokens refers to the output units produced by the bert tokenizer after applying wordpiece tokenization to the input text. ## methodology the overall flow of our proposed approach is illustrated in figure 2 . we conducted experiments across multiple distinct configurations, each incrementally improving upon the last to evaluate model capabilities comprehensively. below are details of input meta information across settings. setting a: for inference, the prompt includes image_file, caption, and question. setting b: during fine-tuning, only image_file is used. for inference, the prompt includes image_file, caption, and question. setting c: fine-tuning uses image_file and the corresponding pdf. inference is performed using image_file, caption, and question. setting d (final approach): identical to setting c, fine-tuning utilizes image_file and pdf, and inference uses image_file, caption, and question. ## limitations although our approach demonstrates promising results, it still has several limitations stemming from two primary factors. firstly, certain challenges arise from the data itself. these include poor image quality, lack of contextual information, or missing visual elements. additionally, in some instances, the correct answer is visually ambiguous or difficult to distinguish from the figure for example, differentiating between values such as 0.54 and 0.56 in a bar graph. secondly, while our method incorporates additional contextual information to support answer prediction, this context is not always sufficient or fully relevant. although the inclusion of retrieved context generally improves performance, there are edge cases where questions originally labeled as unanswerable could become answerable with the added context potentially leading to inconsistencies in evaluation and lower performance. a systematic analysis of these cases is currently lacking and would require additional strategies to robustly identify and handle such cases. furthermore, although the auxiliary dataset used to enhance model performance contributes positively, it does not comprehensively capture the full complexity and diversity of question types presented in the shared task. ## conclusion we present a retrieval-augmented vqa pipeline that combines vision-language modeling with document-aware context retrieval to improve scientific chart understanding. through progressive experimentation and enhancement, our method achieved significant gains in accuracy, reasoning depth, and answer quality. by integrating image sharpening, textual retrieval, and dataset augmentation, the system successfully bridges the gap between purely visual inputs and the rich semantic context needed for effective scientific qa. our approach demonstrates the potential of llm enhanced vision-language systems in handling complex academic visual data. future work will explore multi-modal attention mechanisms across figure-caption-text triplets and generalize the framework to broader scientific domains, enabling more diverse and open-ended question answering capabilities.