Multimodal A r X iv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models (2024)

Abstract: Large vision-language models (LVLMs) excel across diverse tasks involving concrete images from natural scenes. However, their ability to interpret abstract figures, such as geometry shapes and scientific plots, remains limited due to a scarcity of training datasets in scientific domains.To fill this gap, we introduce Multimodal ArXiv, consisting of ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension.ArXivCap is a figure-caption dataset comprising 6.4M images and 3.9M captions, sourced from 572K ArXiv papers spanning various scientific domains.Drawing from ArXivCap, we introduce ArXivQA, a question-answering dataset generated by prompting GPT-4V based on scientific figures. ArXivQA greatly enhances open-sourced LVLMs’ mathematical reasoning capabilities, achieving a 10.4% absolute accuracy gain on a multimodal mathematical reasoning benchmark.Furthermore, employing ArXivCap, we devise four vision-to-text tasks for benchmarking LVLMs.Evaluation results with state-of-the-art LVLMs underscore their struggle with the nuanced semantics of academic figures, while domain-specific training yields substantial performance gains.Our error analysis uncovers misinterpretations of visual context, recognition errors, and the production of overly simplified captions by current LVLMs, shedding light on future improvements.

## introduction large vision-language models (lvlms), which integrate large language models (llms) @xcite with pre-trained vision encoders through cross-modal alignment training @xcite @xcite , have demonstrated remarkable perceptual and cognitive capabilities in processing concrete images from everyday scenes @xcite @xcite . however, recent studies have shown that open-source lvlms struggle to understand abstract figures, such as geometric shapes in multimodal mathematical reasoning @xcite and scientific plots @xcite . the inadequacy of training datasets in scientific domains that involve complex reasoning with abstract figures is the main underlying cause. to address this, we construct multimodal arxiv by utilizing the rich resources in preprints hosted on arxiv to improve the ability to understand scientific literature in lvlms. we first curate arxivcap, a diverse scientific figure-caption dataset. in contrast to previous scientific figure datasets, which consist of synthesized figures @xcite or are restricted to simple captioning scenarios in the computer science domain @xcite , our dataset is composed of figures extracted from academic papers across a range of domains. arxivcap has 6.4m images and 3.9m captions from 572k papers. we also keep the subfigure structure, and titles of original papers, thereby supporting diverse evaluation tasks. we further instruct gpt-4v to generate 100k multiple-choice question-answering (qa) pairs for the figures in arxivcap. the resulting arxivqa dataset could naturally serve as a pivotal resource for improving the scientific reasoning abilities of lvlms. we validate the effectiveness of our multimodal arxiv dataset from two dimensions: reasoning ability measured by qa accuracy and generation performance through novel vision-to-text tasks. our experiments demonstrate that arxivqa brings a significant 10.4% absolute accuracy boost for qwen-vl-chat @xcite , on the math-vista @xcite , a challenging benchmark for multimodal mathematical reasoning. additionally, detailed analysis uncovers the relationship between paper domains and fine-grained task performance. moreover, using arxivcap, we define four generation tasks of varying complexity to benchmark the ability of lvlms to comprehend scientific plots: ## related work recent advancements in lvlms have seen notable progress in model architecture, training paradigms, and dataset creation @xcite . model architecture lvlms typically comprise three core modules: (i) a vision encoder for image feature extraction, (ii) a modality alignment module to integrate visual features into the language model embedding space, and (iii) an llm backbone for decoding multimodal context. clip @xcite is widely used for image encoding, while llama @xcite and vicuna @xcite serve as popular choices for llms. the alignment module varies from simple linear projections @xcite to more complex architectures like gated cross-attention layers substantiated by flamingo and idefics @xcite . innovations such as the q-former module in blip2 @xcite and instruction integration in instructblip @xcite further enhance alignment capabilities. additionally, fuyu-8b @xcite introduces a novel framework mapping raw image pixels directly to the llm embedding space. training paradigms regarding the training recipes, pali-x (chen et al., 2023b) investigates the scaling effects of both vision encoders and language models, highlighting the advantages of scal-ing both components. qwen-vl @xcite increases input image resolution and explores different module unfreezing strategies. alignment methodologies such as rlhf training @xcite , e.g., llava-rlhf @xcite , and preference optimization through ai feedback @xcite demonstrate effectiveness in aligning lvlms with human preferences. dataset curation dataset quality significantly impacts lvlm performance. modality alignment training often utilizes web-scale image-caption pairs such as laion-400m @xcite , with recent studies favoring cleaned captions @xcite . instruction fine-tuning (ift) helps lvlms respond according to user queries, triggering the exploration of high-quality ift datasets. efforts include multimodal instruction collections such as mul-tiinstruct @xcite and m 3 it @xcite , dialog-style datasets such as llava @xcite and domain-specific datasets for medical @xcite ) and text-rich images @xcite . in the scientific domain, @xcite and figureqa @xcite are created based on synthetic figures. dvqa @xcite ) creates heuristicbased questions for bar charts only. scicap @xcite , scicap+ @xcite , and m-paper @xcite collect figure-caption pairs from specific domains such as computer science. compared with these datasets, our arxivcap is sourced from diverse scientific domains with a much larger scale, enabling more comprehensive improvements and evaluations. besides, we employ gpt-4v for creating arxivqa with challenging questions, showcasing its effectiveness in boosting the mathematical reasoning ability of lvlms. ## multimodal arxiv this section presents a detailed construction process of our multimodal arxiv dataset, consisting of two sets: arxivcap ( §3.1) and arxivqa ( §3.2). ## experiments we conduct experiments to (i) validate the effectiveness of arxivqa for boosting multimodal scientific reasoning for open-source lvlms ( §4.1) and (ii) benchmark lvlms capability to comprehend scientific figures with arxivcap ( §4.2). ## conclusion our work introduces multimodal arxiv, comprising arxivcap and arxivqa, aims at advancing the scientific comprehension of lvlms. experiments show that fine-tuning on arxivqa notably enhances lvlms' mathematical reasoning capabilities. moreover, our comprehensive evaluations across four vision-to-text tasks on arxivcap underscore the challenges in understanding scientific figures for lvlms, while highlighting the substantial improvements achieved by in-domain training. our error analysis offers valuable insights for the ongoing development of lvlms.