I nfi MM - W eb M ath-40 B : Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning (2025)

Abstract: Pre-training on large, high-quality datasets is essential for improving the reasoning abilities of Large Language Models (LLMs), particularly in specialized fields like mathematics. However, the field of Multimodal LLMs (MLLMs) lacks a comprehensive, open-source dataset for mathematical reasoning. To fill this gap, we present InfiMM-WebMath-40B, a high-quality dataset of interleaved image-text documents. It consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens, all carefully extracted and filtered from CommonCrawl. We outline our data collection and processing pipeline in detail. Models trained on InfiMM-WebMath-40B demonstrate strong performance in both text-only and multimodal settings, setting a new state-of-the-art on multimodal math benchmarks such as MathVerse and We-Math.

## introduction recent advancements in large language models (llms) @xcite @xcite have improved their ability to handle complex reasoning and multi-step mathematical problems through techniques like chain-of-thought (cot) prompting @xcite . these models excel from basic gsm8k word problems @xcite to high school-level math tasks @xcite . specialized smaller llms like deepseekmath-7b @xcite and internlm-math @xcite have also made notable progress in mathematics, demonstrating strong performance in focused domains. although most mathematical knowledge is textbased, visual elements such as figures and diagrams are essential for understanding abstract concepts. to integrate these visual components, multimodal llms (mllms) like g-llava (gao 1 we have released our data at huggingface. et al., 2023b), math-llava @xcite , and mavis @xcite have been developed. these models enhance reasoning by incorporating visual inputs through embeddings from pre-trained models like clip @xcite and siglip @xcite , and use multimodal instruction datasets such as geo170k @xcite , mathv360k @xcite , and mavis-instruct @xcite . however, introducing new knowledge during instruction fine-tuning is challenging @xcite , often leading to hallucinations @xcite , particularly due to limitations in dataset scale and quality. while large corporations benefit from proprietary datasets, the open-source community lacks comprehensive pretraining datasets for mathematical reasoning that integrate text and visual data. to address this gap, we introduce infimm-webmath-40b, the first large-scale, publicly available multimodal mathematics pre-training dataset. comprising 24 million web documents, 85 million image urls, and 40 billion text tokens, it provides a valuable resource for training multimodal llms (mllms). we validate the effectiveness of infimm-webmath-40b through experiments on benchmarks like mathverse @xcite and wemath @xcite , showing improved performance in multimodal math reasoning. our contributions include: (1) we introduce infimm-webmath-40b, the first large-scale, multimodal math dataset for pre-training, filling a critical gap in open-source research. (2) we provide a detailed preprocessing pipeline for filtering relevant content from commoncrawl to ensure high-quality, relevant data. (3) we demonstrate the impact of infimm-webmath-40b through experiments, where our models excel on multimodal mathematical benchmarks, showcasing the dataset's potential for advancing mllm research. ## related work llms have demonstrated potential in mathematical reasoning across various studies. to evaluate and enhance their capabilities, several math-specific benchmarks @xcite @xcite @xcite and training datasets, both proprietary @xcite @xcite and open-source @xcite @xcite @xcite , have been introduced. the rise of multimodal llms (mllms) has sparked interest in enhancing their multimodal reasoning capabilities. to support this, various evaluation benchmarks @xcite @xcite @xcite @xcite @xcite and training datasets @xcite @xcite @xcite @xcite have been developed to assess and enhance mllms' mathematical reasoning skills. ## dataset construction in this section, we detail the methodology used to construct infimm-webmath-40b from the com-moncrawl archives. infimm-webmath-40b is a large-scale multimodal math dataset integrating interleaved text and image data, following approaches used in prior works @xcite @xcite . we enhance the methodology used in the obelics dataset @xcite by incorporating both text and corresponding image urls. ## cpt and ift dataset ablations on mathverse in this section, we compare models trained with and without our own mathematical multi-modal pre-training dataset, infimm-webmath-40b. additionally, we evaluate two ift dataset configurations: (a) a combination of mavis-captionto-qa, mavis-existing-dataset-augment, mavis-caption, mavis-dataengine-geometry, and mavis-meta-question (referred to as the mavis dataset); and (b) a broader set consisting of the mavis datasets along with vflan, visu-alwebinstruct, ai2d, chartqa, docvqa, dvqa, geoqa, dart-math, and numina-math (referred to as the extended dataset). as shown in ## conclusions in this work, we introduced infimm-webmath-40b, the first large-scale multimodal pretraining dataset for mathematical reasoning, filling a crucial gap in open-source research. our dataset significantly enhances models' performances on key benchmarks.