Math- LL a VA : Bootstrapping Mathematical Reasoning for Multimodal Large Language Models (2024)

Abstract: Large language models (LLMs) have demonstrated impressive reasoning capabilities, particularly in textual mathematical problem-solving. However, existing open-source image instruction fine-tuning datasets, containing limited question-answer pairs per image, do not fully exploit visual information to enhance the multimodal mathematical reasoning capabilities of Multimodal LLMs (MLLMs). To bridge this gap, we address the lack of high-quality, diverse multimodal mathematical datasets by collecting 40K high-quality images with question-answer pairs from 24 existing datasets and synthesizing 320K new pairs, creating the MathV360K dataset, which enhances both the breadth and depth of multimodal mathematical questions. We introduce Math-LLaVA, a LLaVA-1.5-based model fine-tuned with MathV360K. This novel approach significantly improves the multimodal mathematical reasoning capabilities of LLaVA-1.5, achieving a 19-point increase and comparable performance to GPT-4V on MathVista’s minitest split, and yielding leading performance on Math-V and MathVerse. Furthermore, Math-LLaVA demonstrates enhanced generalizability, showing substantial improvements on the MMMU benchmark. Our research highlights the importance of dataset diversity and synthesis in advancing MLLMs’ mathematical reasoning abilities. The code and data are available at: https://github.com/HZQ950419/Math-LLaVA .

## introduction motivation. large language models (llms) exhibit impressive reasoning capabilities, drawing significant research interest in mathematical problemsolving in textual form @xcite @xcite @xcite @xcite . however, the task of multimodal mathematical reasoning @xcite requires models to interpret diverse images and apply advanced reasoning skills. while open-source multimodal large language models (mllms) like llava @xcite and mini-gpt4 @xcite perform well on vqa tasks @xcite , they fall short of proprietary mllms (openai; google) in solving complex mathematical problems involving visual content. two common approaches to enhance mllms' mathematical reasoning skills are prompt methods and fine-tuning methods. prompt methods @xcite ) leverage mllms' latent abilities through carefully designed prompts, while fine-tuning methods @xcite @xcite adjust model parameters using reasoning data collected from realworld or synthetic data from advanced llms (e.g., gpt-4). however, existing open-source image instruction fine-tuning datasets @xcite @xcite , which contain limited question-answer pairs per image, do not fully exploit visual information to enhance mllms' multimodal mathematical reasoning capabilities. research objectives. to bridge this gap, we select 40k high-quality images with corresponding question-answer pairs from 24 pre-existing datasets. these images and queries span various subjects, including algebra, arithmetic, geometry, logic, numeric commonsense, science, and visual question answering. the selection criteria were based on image clarity and comprehension complexity. additionally, we propose a pipeline to synthesize 320k new pairs based on the 40k images and seed inquiries. constructing such a dataset presents significant challenges, including selecting diverse and highquality multimodal question-answer data and enhancing question diversity. selecting suitable data involves filtering for image clarity and comprehen-sion complexity, ensuring the dataset covers a wide range of mathematical concepts and question types. enhancing question diversity requires synthesizing new questions that probe different aspects of the images and involve multiple reasoning steps. to further improve model robustness and comprehension, we focus on enhancing logical consistency @xcite and the ability to understand underspecified language @xcite . contributions. using the selected 40k data, the fine-tuned llava-1.5 model, named math-llava-ds, achieved a significant improvement of 10.6% on mathvista @xcite . to further enhance multimodal mathematical reasoning capabilities, we synthesized an additional 320k question-answer pairs based on the 40k images and seed questions, resulting in the mathv360k dataset. this comprehensive dataset, containing around 40k images and 360k question-answer pairs, significantly expands the coverage of multimodal mathematical reasoning. fine-tuning llava-1.5 with mathv360k, we developed math-llava, which outperforms the original llava-1.5 by 19% on mathvista's minitest split and achieves leading performance on math-v @xcite and mathverse @xcite . we also evaluated math-llava on mmmu @xcite , demonstrating its improved generalizability. ## data synthesis existing open-source image instruction fine-tuning datasets @xcite @xcite , containing limited question-answer pairs per image, do not fully exploit visual information to enhance the multimodal mathematical reasoning capabilities of mllms. to address this, we propose mathv360k, a robust dataset synthesized based on the 40k selected images and seed question-answer pairs from multiple sub-domains. as shown in the left side of figure 1 , we first select 40k highquality data points based on the image clarity and plotqa … fqa tabmwp iconqa … mwp geometry3k unigeo … gps ai2d scienceqa … tqa super-clevr vqa-as … vqa image clarity classifier not clear (clarity: 0) clear (clarity: 1) image comprehension complexity classifier easy complex 1 0 2 3 data selection proportion from 0-3 is 2:3:4:1 data augmentation and synthesis finetune llava-1.5 original question ： a dog show enthusiast recorded the weight of the winning dog at recent dog shows. according to the table, what was the rate of change between 2017 and 2018? the answer is 6. new asked question: what was the average weight of the winning dog over the five years shown in the table? the answer is 16.2 kg. new asked question: what was the greatest weight change between consecutive years? the answer is 6 kg. new asked question: how much did the weight of the winning dog decrease from the heaviest year to the lightest year? the answer is 7. more complex question: what is the average rate of change in the weight of the winning dog from 2016 to 2020? the answer is -1. rephrased question: based on the recorded weights of the winning dogs at recent dog show, how did it change between 2017 and 2018 according to the following table? the answer is 6. simplified question: based on the table, what was the rate of change in weight between 2017 and 2018? the answer is 6. comprehension complexity from 24 open-source multimodal question-answering datasets. in the second step, illustrated in the top right of figure 1 , we attempt to fully mine the visual information of the images to generate additional questions. the data generation pipeline includes creating diverse new questions to fully exploit the visual information, more complex questions to further improve the reasoning capabilities, rephrased questions and underspecified questions to improve the robustness of the model. with the data generation pipeline, we collected 360k high-quality and diverse instruction-tuning data for the selected 40k data points to enhance the image understanding and mathematical reasoning capabilities of the llava-1.5 open-source model. ## conclusions we addressed the shortage of high-quality and diverse multimodal mathematical training datasets by creating mathv360k, which consists of 40k high-quality multimodal questions and answers from 24 existing datasets, along with 320k newly synthesized question-answer pairs. this comprehensive dataset enhances both the breadth and depth of multimodal mathematical questions. using mathv360k, we fine-tuned math-llava, significantly improving its capability in multimodal mathematical reasoning, outperforming llava-1.5 by 19 points on the minitest split of mathvista, and yielding leading performance on math-v and mathverse. additionally, math-llava was validated on the mmmu benchmark, demonstrating its generalizability. our research underscores the importance of dataset diversity and synthesis in enhancing the mathematical reasoning abilities of mllms.