ASPO : Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning (2025)

Abstract: Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models.

## introduction large language models (llms) have demonstrated remarkable capabilities in natural language processing (nlp) @xcite @xcite , which greatly accelerates the development of multimodal large language models (mllms) @xcite @xcite . the training of mllms typically involves a pre-training stage followed by the supervised fine-tuning (sft) stage @xcite . sft enhances the model's ability to understand and execute multimodal instructions @xcite @xcite . models trained in this man-ner exhibit impressive multimodal conversational abilities @xcite . despite these advancements, sft often leads to hallucinated outputs, causing performance saturation @xcite . as noted in @xcite , increasing the probability of preferred outputs can inadvertently raise the likelihood of dis-preferred outputs, making models more prone to errors in complex reasoning tasks @xcite . additionally, as response length grows, models become increasingly susceptible to generating hallucinations @xcite . this issue is exacerbated by machine-generated data containing noisy sentences, where segments may be partially correct or incorrect @xcite . training on such data risks convergence to suboptimal solutions, emphasizing the need for strategies to suppress undesirable outputs @xcite . in response, researchers have developed various preference fine-tuning methods to better align mllms with human preferences @xcite @xcite . among these, dpo-based methods @xcite have gained prominence due to their simplicity and low computational cost @xcite . however, traditional dpo methods rely on binary preference data and a coarse reward mechanism, which lacks the fine-grained preference granularity needed to identify specific errors in responses @xcite . this limitation hampers the model's ability to refine its reasoning capabilities @xcite . to this end, we introduce a fine-grained preference mechanism that assigns variable reward values, enabling more fine-grained reasoning optimization. as shown in figure 1 , we present an adaptive sentence-level preference optimization (aspo) approach, where each sentence in a response serves as the fundamental unit of preference optimization. unlike traditional dpo, which evaluates preferences at the response level @xcite , aspo systematically evaluates the correctness and importance of each sentence and assigns appropriate rewards. by prioritizing accurate reasoning and rejecting errors based on finegrained criteria, our method efficiently identifies erroneous sentences for targeted optimization. this reduces the impact of noisy data and substantially improves the model's comprehensive capability. our contributions are summarized as follows: ## methodology in this section, we introduce aspo for fine-grained multimodal reasoning. aspo incorporates two adaptive reward weights into dpo, i.e., the imagetext similarity weight, which optimizes multimodal alignment to mitigate hallucination, and the textual perplexity weight, which enhances text confidence to improve the model's reasoning capabilities. ## experiment in this section, we conduct comprehensive experiments to evaluate the effectiveness of the proposed aspo. we detail the experimental setups, results, and analysis in the following sections. preference data. we sample about 20k instructions from llava-instruct-150k @xcite and go through seva pipline @xcite to produce the preference dataset. the data collection pipeline is shown in figure 2 . specifically, diffusion noise is added to the images in the instruction data, and the original and augmented images with the instruction are then fed into the mllm to be train to obtain the chosen and rejected responses, respectively. the identical paired preference data is filtered out and we finally get about 16k preference data pairs. for all experiments, the noise baselines. we compare aspo with several previous state-of-the-art (sota) open-source mllms and preference optimization-based meth-ods. the open-source mllms include blip-2 @xcite , minigpt-4 @xcite , instructblip @xcite , llava @xcite , llava-v1.5 @xcite , shikra @xcite , qwen-vl @xcite , qwen-vl-chat @xcite , mplug-owl2 @xcite and idefics @xcite . for preference optimization- ## conclusion in this paper, we introduce aspo, a method designed to overcome the limitations of traditional dpo approaches, which treat entire responses as indivisible units during optimization. by employing a fine-grained optimization strategy that treats sentences as the fundamental units, aspo enables more precise and effective alignment of mllms with human preferences. by moving beyond response-level optimization to embrace a sentence-level approach, our method opens new avenues for improving the comprehensive capabilities of mllms, paving the way for more accurate and reliable alignment with human preferences.