A lign ^2 LL a VA : Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation (2025)

Abstract: Recent advances in Multi-modal Large Language Models (MLLMs), such as LLaVA-series models, are driven by massive machine-generated instruction-following data tuning. Such automatic instruction collection pipelines, however, inadvertently introduce significant variability in data quality. This paper introduces a novel instruction curation algorithm, derived from two unique perspectives, human and LLM preference alignment, to compress this vast corpus of machine-generated multimodal instructions to a compact and high-quality form: (i) For human preference alignment, we have collected a machine-generated multimodal instruction dataset and established a comprehensive set of both subjective and objective criteria to guide the data quality assessment critically from human experts. By doing so, a reward model was trained on the annotated dataset to internalize the nuanced human understanding of instruction alignment. (ii) For LLM preference alignment, given the instruction selected by the reward model, we propose leveraging the inner LLM used in MLLM to align the writing style of visual instructions with that of the inner LLM itself, resulting in LLM-aligned instruction improvement. Extensive experiments demonstrate that we can maintain or even improve model performance by compressing synthetic multimodal instructions by up to 90%. Impressively, by aggressively reducing the training instructions from 158k to 14k (9× smaller), our model consistently outperforms its full-size dataset counterpart across various MLLM benchmarks. Our project is available at https://github.com/DCDmllm/Align2LLaVA.

## introduction recent triumphs in artificial intelligence have been driven notably by the development of large language models (llms) @xcite @xcite . inspired by the impressive instruction-following capability of llms, visual instruction tuning @xcite has been proposed to extend llms into multimodal llms (mllms) to perceive and understand visual signals (e.g., images) @xcite @xcite @xcite . due to the scarcity of visuallanguage instruction-following data, contemporary mllms prefer to utilize the data reformation approach @xcite , which leverages textonly llms conditioned on image captions and bounding boxes to create synthetic multi-modal instructions involving visual content. following the evolution, other works such as allava @xcite and vigc @xcite , have extended this automatic paradigm by utilizing visually-capable llms, demonstrating the potential value of utilizing synthetic instruction generation. despite their efforts and achievements, the simply synthetic instruction generation paradigm may result in sub-optimal multimodal instruction following capability, which can be summarized as two key aspects: (i) inherently noisy instructions: the generated questions and corresponding responses of instructions may be incongruent or incomplete with the visual content, as they are produced by text-only large language models. regarding question generation, common issues include irrelevance to the image, independence from visual cues, and repetitive queries in multi-turn dialogues. on the other hand, the corresponding responses may suffer from textual hallucinations or ignored visual details, as they are produced by text-only large language models. figure 1 (a) shows an example of the synthetic instructions. of the two llmgenerated questions, the previous one demonstrates superior quality by incorporating queries relevant to image content, and providing subtle prompts to stimulate reasoning without revealing specific visual information. conversely, the subsequent question lacks pertinence to the image, as it can- ## method our visual instruction curation pipeline comprises three sequential steps, as depicted in figure 2 . initially, two distinct datasets, enriched with human preferences, are constructed focusing on the question and answer components of instructions, respectively (section 3.1). leveraging these humanranked datasets, two separate reward models are trained to align with human values, and subsequently serve as human-like evaluators to filter large-scale synthetic visual instruction data through a two-stage process (section 3.2). finally, an inner llm is employed to refine the selected instructions in a rewrite-review manner, ensuring alignment with the llm's writing style while preserving the original semantic content (section 3.3). ## conclusion this paper contributes to the advancement of highquality visual instruction synthesis by introducing a novel filtration pipeline that significantly reduces dataset size. our pipeline sequentially refines instruction quality through alignment with human preferences and llm writing style. by applying this pipeline to synthetic data, we demonstrate that the resulting substantially smaller dataset yields performance on par with the full instruction dataset when fine-tuning mllms. our results highlight the potential for improving instruction quality via human-in-the-loop refinement and offer a path towards more efficient mllm training.