Train a Unified Multimodal Data Quality Classifier with Synthetic Data (2025)

Abstract: The Multimodal Large Language Models (MLLMs) are continually pre-trained on a mixture of image-text caption data and interleaved document data, while the high-quality data filtering towards image-text interleaved document data is under-explored. We propose to train an efficient MLLM as a Unified Mulitmodal Data Quality Classifier to Filter both high-quality image-text caption and interleaved data (UniFilter). To address the challenge of collecting diverse labeled multimodal data, we introduce a semi-synthetic approach that leverages readily available raw images and generates corresponding text across four quality levels. This method enables efficient creation of sample-score pairs for both caption and interleaved document data to train UniFilter. We apply UniFilter to curate high-quality caption data from DataComp caption dataset and interleaved data from the OBELICS image-text interleaved dataset. MLLMs pre-trained on the filtered data demonstrate significantly enhanced capabilities compared to those trained on baseline-filtered data, achieving stronger zero-shot reasoning and in-context learning capabilities. After visual supervised fine-tuning, these UniFilter-induced MLLMs achieve stronger performance on various benchmarks, highlighting the downstream benefits of high-quality multimodal pre-training.

## introduction large-scale multimodal datasets significantly motivates the recent advances in vision language models (vlms) @xcite @xcite and multimodal large language models (mllms) @xcite @xcite . the scaled up data allows the mllms to harvest the knowledge in the training corpora to the greatest extent and promotes the state-of-the-art mllms. the mllms are trained on a mixture of imagetext caption data and interleaved document data to enhance both zero-shot and few-shot capability. moreover, with the limited computing resources but the overwhelming number of data mining from commoncrawl snapshots, the recent large-scale mllms are only trained on a data subset for less than one epoch. therefore, the data quality became the major bottleneck in training stronger models. in selecting high-quality image-text caption dataset, the representative model-based filter, clipscore filter @xcite has become the predominant data filtering method. however, clipscore can only deal with imagetext caption data based on the similarity between a single image and a short text caption. it is completely un-explored on how to select high-quality image-text interleaved data, which contains multiple images and long text paragraphs interleaving in one document. to address this problem, we propose to train an efficient mllm as a unified mulitmodal data quality classifier to filter both high-quality image-text caption and interleaved data (unifilter). adopting an mllm architecture for the proposed data quality classifier effectively overcomes the limitation of clipscore, which can only process single image-text pairs. the proposed unifilter can process both image-text paired and interleaved data and output a float quality score to indicate the quality of this multimodal data sample. meanwhile, it outperforms clipscore on curating highquality image-text caption data for enhancing both vlm and mllm pre-training. simultaneously, it achieves a high inference throughput of 130 samples/s by leveraging qwen-2.5-0.5b as llm backbone, sligtly outperforming the clipscore method's 128 samples/s on the same hardware. the key to train an effective data quality classifier lies in constructing accurate sample-score pairs @xcite . human annotations for these pairs are costly and challenging to maintain consistency across different annotators. to address this, we propose a novel semi-synthetic multimodal data generation method by leveraging the proprietary mllms. given that the proprietary mllms excel in text generation given multimodal inputs and the raw images are readily available, we sample a diverse set of original images from captioned or interleaved data. we then use proprietary mllms to generate the full multimodal data following quality requirements across 4 quality levels (section 2.1), in which a similar 4 level quality score is also used in fineweb-edu-quality-classifier @xcite . then the synthetic data can be easily constructed as sample-score pairs, with score labels 0, 1, 2, and 3 corresponding to the defined quality levels in the prompts. in addition to well-designed synthetic training data construction, we conduct comprehensive ablation studies on the effective and efficient multimodal model architecture of unifilter on the held-out validation synthetic sample-score data. we experiment with 6 combinations of choices of the vision encoder, visual projector, and the llm backbone for constructing the unifilter architecture. the architecture designs of siglip-so-400m vision encoder @xcite , adaptive average pooling projector and the qwen-2.5-0.5b llm @xcite achieves the best tradeoff between data quality classification performance and efficiency. we conduct comprehensive experiments on image-text caption data and interleaved document data filtering to demonstrate the effectiveness of our method over strong baselines. we firstly validate the priority of unifilter on curating highquality image-text caption data over strong baselines through the experiments on mllm pretraining with curated caption data only. our method unifilter outperforms the state-of-the-art (sota) clip-based filtering method, data filtering network (dfn) @xcite and the sotaq mllm-based data filtering model, mlm-filter @xcite on all 5 zero-shot vqa datasets. secondly, pre-training mllms on the high-quality image-text interleaved document data curated by unifilter promotes the few-shot learning capabilities of mllms by +0.7 and +2.8 average scores on 4-shot and 8-shot vqa performance over the baseline data filters. finally, instructiontuned mllm based on unifilter pre-training outperforms baselines and achieves +3.1 average improvement on vqa tasks and +1.5 improvement on mmmu benchmark, demonstrating the broad benefits of our approach. the summarization of the contributions of unifilter are as follows: ## synthetic data construction compared with collecting training data of data quality classification tasks from web-resources or human annotators, the synthetic data can be easily generated on a large scale to provide sufficient training data for models, which is a more feasible solution to empowering the effective training of data quality classifier. furthermore, by incorporating controlled 4-level quality requirements ( ## unifilter architecture to achieve a unified architecture to process both the image-text caption and interleaved data, we construct the unifilter based on a mllm architecture. figure 2 presents how a mllm-based multimodal data quality classifier can process the two major types of image-text data. for the image-text interleaved data, the images and texts are encoded separately with vision encoder and word embedding layer and then reconstructed in original interleaving order. for the caption data, the image encoding and caption embeddings are concatenated and forwarded into the llm backbone. a trainable one-dimensional classification head is appended on the top of llm backbone to output a logit indicating the quality score of the input caption or interleaved data sample. adopting a mllm-based data quality classifier can substantially improve the quality classification performance compared with clip-based architectures, while the introduced billion-level model parameters will bring huge inference cost to the data quality label inferences on the pre-training data scale. in order to to train the unifilter to be both efficient and capable, we perform comprehensive ablation study on the mllm architecture of the unifilter. the recent advances on the architecture design of mllms @xcite @xcite all deploy the modality fusion architecture with 3 major modules of vision encoder, vision-language projector, and the llm. we inherit this architecture and perform detailed ablations on design choices on different modules. the configuration for model architecture ablations om each module is as follows: because of the quadratic time complexity of llms with respect to number of concatenated multimodal input tokens, the computation efficiency of mllms are heavily affected by the number of image tokens for representing one image. therefore, conducting compression on image patches to fixed number of image tokens is mandatory to ensure the efficiency, especially for highresolution vision encoders, i.e. siglip-so400m-384px and clip-large-336px. @xcite compares the performance of 4 popular compressive vision projectors, q-former @xcite , c-abstractor @xcite , d-abstractor @xcite and adaptiveaveragepooling (avg-pool), and the avgpool significantly outperforms other competitors. thus, we adopt the twodimensional avgpool as the compressive vision projector and compare it with the non-compressive mlp projector. we train each unifilter variant for 10 epochs on synthetic contrastive data and the best model is selected based on validation accuracy. the unifilter based on qwen2.5-1.5b achieves the best quality classification performance while introducing significant computational overhead due to the additional 1b parameters compared with qwen2.5-0.5b model. among all group of mllm architecture configurations, the architecture with siglip-so400m vision encoder, avgpool visual projector of 144 tokens per image, and the qwen-2.5-0.5b llm achieves the best trade-off between quality classification performance and efficiency in figure 3 , which is used as the final unifilter model. surprisingly, the final unifilter model can achieve comparable inference speed with dfn-clip-large @xcite . ## conclusion we propose an efficient mllm-based unified multimodal data quality classifier to filter both highquality image-text caption and interleaved data. pre-training mllms on the high-quality data curated by the proposed unifilter can significantly enhance the capability of these general-purpose models on downstream tasks. unifilter overcomes the limitation of being only capable of filtering caption data in clip-based data filters and paves a way to steadily improve both zero-shot and fewshot multimodal in-context learning capability of pre-trained and fine-tuned mllms via unified multimodal high-quality data filtering.