A Simple Data Augmentation Strategy for Text-in-Image Scientific VQA (2025)

Abstract: Scientific visual question answering poses significant challenges for vision-language models due to the complexity of scientific figures and their multimodal context. Traditional approaches treat the figure and accompanying text (e.g., questions and answer options) as separate inputs. EXAMS-V introduced a new paradigm by embedding both visual and textual content into a single image. However, even state-of-the-art proprietary models perform poorly on this setup in zero-shot settings, underscoring the need for task-specific fine-tuning. To address the scarcity of training data in this “text-in-image” format, we synthesize a new dataset by converting existing separate image-text pairs into unified images. Fine-tuning a small multilingual multimodal model on a mix of our synthetic data and EXAMS-V yields notable gains across 13 languages, demonstrating strong average improvements and cross-lingual transfer.

## introduction vision-language models (vlms) have advanced ai by enabling multimodal reasoning, facilitating more natural user interaction in tasks such as visual question answering (vqa) and captioning. @xcite proposed vqa as a task that spans language and image to generate an accurate response. the vqa task has evolved rapidly with applications and benchmarks in domains such as science @xcite , chart understanding @xcite , document analysis @xcite , medical imaging @xcite , and other real-world applications. vqa tasks typically follow either a multiple-choice or open-ended format. in multiple-choice scientific vqa, the input typically consists of an image (figure, table, chart) accompanied by a question and answer choices in text form. the task requires reasoning over both image 1 dataset: https://huggingface.co/datasets/shoir/ scientific_vqa the circuit diagram below represents four resistors connected to a 12-volt source. what is the total current in the circuit? and text components to select the correct answer. these modalities are often processed separately in multimodal models. however, in practice, the text is often embedded within the visual modality, for example, screenshots of digital exams or textbook photos. to address this, das et al. ( 2024 ) introduced a new scientific vqa benchmark that consists of images with embedded questions, providing a robust benchmark for evaluating model performance under realistic conditions. the exams-v dataset includes two splits: train (16.5k instances) and test (4.8k instances), spanning 15 languages. this text-in-image formulation of scientific vqa either requires a separate optical character recognition (ocr) step, which may introduce noise, or, preferably, vlms with strong inherent ocr capabilities that can jointly reason over visual content and embedded text. yet, current vlms typically benefit from text as opposed to text-inimage format, even if the text is just describing the contents of the image itself @xcite . exams-v approached the task in a zero-shot man-ner, without leveraging the training data. finetuning for reasoning over visually-embedded text is likely to improve performance. while we already have the exams-v training split in the text-in-image format, we find that it provides limited coverage, with an average of 1,415 data points per language. to address this, we augment the training set by synthesizing text-in-image data points derived from disjoint scientific vqa datasets, in four languages: chinese, english, italian, and german. this results in approximately 1,742 additional examples on average per language. ## background in this section, we provide the necessary background for our work. we begin by reviewing traditional datasets and benchmarks commonly used in scientific vqa, highlighting their strengths and limitations. we then review exams-v, the primary benchmark our work builds upon, and describe our chosen vlm, paligemma @xcite , explaining the rationale behind its selection. ## data augmentation this section outlines our method for generating synthetic text-in-image instances for vqa and introduces the pre-trained vlm used in our experiments. we focus on the text-with-visual format rather than the text-only due to its limited presence in exams-v, with only 5,162 such images. we use data from five datasets that provide separate text and image pairs: m3exam, cmmu, m4u, mmmu-pro, and pinocchio. these datasets span multiple languages and subjects. we focus on chinese, english, italian, and german. we filter the data to retain only science-related instances, primarily from chemistry, physics, biology, biochemistry, and engineering. each instance is formatted consistently, with the question at the top, followed by the figure and answer options. for an example of our method, refer to figure 1 . to simulate realistic exam formats, hanzi and latin scripts are rendered using randomly selected fonts and dark text colors. we use common fonts such as simsun and simhei for hanzi, and arial and times new roman for latin script. to reflect typical document formatting, text colors are sampled from a set of dark grayscale tones, with a strong bias toward black as detailed in appendix b. we fix the random seed to 42 for reproducibility. to encourage generalization during fine-tuning, the option format (letters or numbers) is chosen uniformly at random for each synthetic instance. ## conclusion in this study, we augment the exams-v dataset with synthetic text-in-image instances for 4 languages. our experiments demonstrate improved performance across the four languages on average, albeit on text-only questions and not on questions containing visuals. we find that multilingual finetuning outperforms monolingual fine-tuning on average, indicating a positive cross-lingual transfer.