S ci VQA 2025: Overview of the First Scientific Visual Question Answering Shared Task (2025)

Abstract: This paper provides an overview of the First Scientific Visual Question Answering (SciVQA) shared task conducted as part of the Fifth Scholarly Document Processing workshop (SDP 2025). SciVQA aims to explore the capabilities of current multimodal large language models (MLLMs) in reasoning over figures from scholarly publications for question answering (QA). The main focus of the challenge is on closed-ended visual and non-visual QA pairs. We developed the novel SciVQA benchmark comprising 3,000 images of figures and a total of 21,000 QA pairs. The shared task received seven submissions, with the best performing system achieving an average F1 score of approx. 0.86 across ROUGE-1, ROUGE-L, and BertScore metrics. Participating teams explored various fine-tuning and prompting strategies, as well as augmenting the SciVQA dataset with out-of-domain data and incorporating relevant context from source publications. The findings indicate that while MLLMs demonstrate strong performance on SciVQA, they face challenges in visual reasoning and still fall behind human judgments.

## introduction graphical representations such as figures (e. g., charts and diagrams), combined with natural language, serve as essential tools for identifying patterns, analysing trends, and extracting insights from data. in academic research, this dual-modality is particularly prominent, with scientific publications conveying large amounts of valuable information through both unstructured text and (semi-)structured figures. automatically decoding and processing data from figures available in scholarly papers (i. e., scientific figures) can be beneficial for downstream tasks such as visual question answering (vqa). however, vqa over figures is challenging due to their diverse types (e. g., line charts, box plots, pie charts), multimodal nature (combining visuals, numerical data, text), and complex relationships between various components (e. g., axes and labels) @xcite . for scientific figures, the task is further complicated by the presence of domain-specific terminology and principles @xcite . hence, efficient vqa requires accurate information extraction, strong reasoning skills, and expertise in the target research field @xcite @xcite . although vqa has been extensively studied @xcite , its application to scientific figures is still an emerging area of research @xcite . existing real-world datasets are limited, containing figures sourced exclusively from arxiv foot_0 @xcite , ignoring other scientific contexts, such as peer-reviewed conference and journal publications. furthermore, while several works examine the robustness of current multimodal large language models (mllms) for figure @xcite @xcite , none specifically focus on extensive evaluation of models' abilities to accurately recognise, process, and link visual attributes (e. g., colour, shape, size) of scientific figures with textual content (e. g., captions, legends, axis labels). to bridge the mentioned gaps and promote further research, we organised the first scientific visual question answering (scivqa) shared task as part of the fifth scholarly document processing workshop (sdp 2025) 2 at acl 2025. this challenge aims to shed light on the capabilities and limitations of current mllms in handling both questions addressing visual elements of scientific figures and those without visual information. participants were invited to build vqa systems using a novel dataset of 3,000 images of scientific figures from two distinct sources, acl anthology foot_2 and arxiv, associated with a total of 21,000 visual and non-visual qa pairs. the competition attracted 20 registered teams, seven of which submitted their results. this paper presents an overview of the scivqa shared task, including the dataset, baseline, and submitted systems description, summary of the results, comparison of automatic solutions to human performance, and an analysis of common challenges and errors faced by mllms. ## shared task overview in the scivqa challenge, the task is to develop multimodal qa systems using images of scientific figures, their captions, associated natural language qa pairs, and optionally additional metadata (e. g., figure type). the shared task was hosted on the codabench platform @xcite from april 1, 2025, to may 16, 2025. 4 in what follows, qa pair types schema ( §3.1), dataset ( §3.2), and metrics used for evaluation ( §3.3) are described in detail. ## system descriptions for the scivqa challenge, we provide both a baseline model and human judgments to evaluate the task's difficulty and establish an upper-bound benchmark. in this section, we first outline our methodology for evaluating human performance on scivqa. then we describe our baseline model and the systems from five teams that submitted results to the leaderboard and corresponding reports. ## conclusion in this paper, we presented an overview of the first scivqa shared task. the challenge attracted seven submissions, five of which outperformed our baseline. the results reveal that, while automated systems can achieve strong performance on the newly proposed scivqa benchmark, they remain behind human judgments. furthermore, the findings indicate that fine-tuning on cross-domain data, combined with relevant contextual information from source papers, leads to the best results. however, domain adaptation and data augmentation is not always required, and carefully designed prompting strategies can achieve very close results (about 2% gap). additionally, we observe that current mllms struggle most with visual reasoning, as their accuracy drops on qa pairs addressing visual attributes of figures.