Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025)

Abstract: Recent research increasingly focuses on training vision-language models (VLMs) with long, detailed image captions. However, small-scale VLMs often struggle to balance the richness of these captions with the risk of hallucinating content during fine-tuning. In this paper, we explore how well VLMs adapt to such captions. To quantify caption quality, we propose Decomposed NLI (DNLI), an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. This fine-grained analysis reveals a critical balance between capturing descriptive details and preventing hallucinations. Our findings show that simply reducing caption complexity or employing standard data curation techniques does not effectively resolve this issue. To tackle this challenge, we introduce Knowledge Adapted (KnowAda) fine-tuning, a data-centric approach that automatically adapts training data with the model’s existing knowledge and visual understanding. KnowAda minimizes hallucinations while preserving high descriptiveness. We validate this approach across several small-scale VLMs (up to 7B parameters) and dense caption datasets, demonstrating that KnowAda effectively balances hallucination reduction and descriptiveness. Our results show that KnowAda outperforms various baselines in both automatic metrics and human evaluations.

## introduction fine-tuning pretrained multimodal models for generating dense image captions is common in both research and practical applications, such as assisting visually impaired individuals. recent work has focused on creating high-quality, descriptive captions through human annotations @xcite @xcite and synthetic generations from models like gpt-4 @xcite and gemini @xcite , including extensions that integrate expert models for enhanced detail @xcite . these datasets enable the fine-tuning of models to create detailed descriptions in specific styles, distinguishing them from the zero-shot capabilities of pretrained models. however, smaller multimodal models (e.g., up to 7 billion parameters), which are essential for real-time applications, frequently face challenges figure 2 : our proposed knowada pipeline. we first probe the knowledge of the vlm, identifying the known and unknown parts of the image description, by generating questions about the visual content of the image mentioned in the caption. then, knowada identifies the knowledge gaps by judging the vlm answers to these questions. finally, knowada adapt the description to match these gaps (e.g., removing the number of limousines mentioned in the caption, which relates to a question the model failed to answer). consider a fine-tuning dataset for dense captioning, like the one in figure 1 from docci @xcite , alongside a pretrained visionlanguage model trained on tasks like captioning, vqa, and ocr. if, for instance, the model has only encountered low-resolution images during pretraining, it may struggle to identify fine details, such as the drawing on the purple van or the hotel name in the background, which require higher resolution. this issue extends beyond resolution to other visual challenges in modern vlms @xcite @xcite . we hypothesize that fine-tuning the model on overly complex captions may increase hallucinations, as the model is compelled to predict details it cannot accurately perceive or understand. recent works on large language models (llms) have shown that fine-tuning primarily adapts preexisting factual knowledge for specific tasks, with the majority of this knowledge being encoded during the pretraining phase @xcite @xcite . rather than acquiring new information, fine-tuning typically activates and refines pretrained knowledge, which remains largely stable throughout the process @xcite . furthermore, @xcite demonstrated that fine-tuning on con-tent not grounded in a model's pre-existing factualknowledge can lead to an increase in hallucinations. similarly, @xcite showed that attempting to distill gpt4v into a smaller, less capable vlm significantly increases hallucinations. motivated by these findings, we hypothesize that one possible cause of increased hallucinations is the excessive visual complexity of captions relative to the model's pretrained capabilities. we propose a method to mitigate this effect. to better adapt pretrained models to dense caption datasets, we introduce knowada, a modelspecific adaptation technique that simplifies complex details in dense captions. the knowada pipeline, shown in figure 2 , automatically identifies visual knowledge gaps between a pretrained vlm and an image-caption pair by generating questions related to the image's visual content. it then modifies the captions to align with the model's visual knowledge and capabilities, producing adapted captions that enhance fine-grained control while balancing a low hallucination rate with high descriptiveness. evaluating dense image captions requires attention to two critical factors: descriptiveness, which captures the image's details, and hallucination rate, which measures factual accuracy. traditional metrics, such as those based on similarity to reference captions @xcite @xcite or clipbased alignment @xcite , often fall short when applied to long captions. these approaches penalize valid variations in phrasing and fail to distinguish between factual accuracy and token overlap or semantic similarity, rendering them inadequate for identifying hallucinations in detailed captions. existing hallucination metrics are similarly limited, focusing primarily on short captions or object-level errors @xcite @xcite , or relying on other vlms @xcite , which themselves can hallucinate during the verification process. to address these limitations, we propose decomposed nli (dnli), a novel evaluation framework that breaks captions into propositions and assesses their entailment with the detailed ground truth description. dnli offers a more reliable measure of both descriptiveness and accuracy, demonstrating strong alignment with human judgments. our results show that training with knowada captions offers a favorable balance between descriptiveness and fidelity when fine-tuning llava-7b, outperforming other data-centric baselines. to demonstrate the consistency of knowada across different models and datasets, we fine-tuned several multimodal models, ranging from 2 billion to 7 billion parameters, on two different dense captioning datasets. across all models, knowada consistently reduced hallucinations compared to training with the original captions, as confirmed by both automatic and human evaluations. stated explicitly, our contributions are: (i) we show that small-to-medium-scale vlms underperform when fine-tuned directly on dense caption datasets, exhibiting increased hallucinations and reduced descriptive accuracy; (ii) to address this we propose knowada, a model-dependent augmentation method that fills knowledge gaps in captions, reducing hallucinations while preserving high descriptive accuracy; (iii) we demonstrate the effectiveness of knowada through extensive experiments, supported by both quantitative metrics and qualitative human evaluations; (iv) we introduce dnli, a novel evaluation framework for dense captions that offers a more fine-grained analysis and shows strong correlation with human annotations. ## knowada we begin by introducing our caption adaptation method for dense captioning datasets. knowada comprises three stages, as shown in figure 2 . first, we use an llm to generate visual questions from each dense caption. next, these questions are employed to probe the vlm's pretrained visual knowledge and identify parts of the image caption the model struggles with. finally, an llm adapts the image descriptions by editing out the unknown parts. below, we elaborate on each step of the pipeline. ## decomposed nli evaluation next, we introduce dnli, a propositiondecomposition evaluation framework for dense-captioning. it evaluates the quality of a generated dense image caption using two criteria: descriptiveness, which measures accuracy and detail, and contradiction, which measures how much the caption contradicts the ground truth. inspired by prior work on paragraph summarization evaluation @xcite @xcite and retrieval @xcite , dnli assesses descriptiveness and consistency through proposition extraction, as outlined below and shown in figure 4 . additional qualitative examples are in the appendix. ## experimental settings our experiments concentrate on training dense captioning models. we evaluate the performance of models fine-tuned on captions curated through various methods against those fine-tuned on captions adapted using knowada. we start by comparing knowada to data curation methods for multimodal datasets according to the datacomp challenge (gadre et al., 2024), specifically icc @xcite and @xcite . we also compare our approach to the following baselines that might address the hallucination-descriptiveness trade-off: for all methods in this experiment, we fine-tuned llava-1.5-7b @xcite using low rank adaptation @xcite . to ensure that knowada is robust across multiple models and datasets, we fix the threshold at 20% for classifying questions as unknown and finetune three models: paligemma @xcite , tinyllava @xcite , and llava-1.5-7b @xcite . we fine-tune on two docci variations: one using the original docci captions, and another using synthetically generated captions created by gemini, which were prompted to be visually descriptive. we refer to appendix a. @xcite et al., 2024) dataset. we evaluate the models using an automatic nli model and human annotators (section 3). in all experiments, we split the docci test set into 1,000 samples for evaluation, while 4,000 samples are used for the reported test set. ## results this section presents our experiments, highlighting the impact of knowada over competing baselines. ## conclusions this work focuses on fine-tuning small-to-mediumscale vision-language models by aligning dense captions with the models' existing knowledge and visual understanding. it aims to address the challenge of balancing rich descriptiveness with factual correctness in multimodal models, especially when models have limited capacity to process complex visual details in dense captions. by probing models with visual questions and adapting captions to exclude unknown information, we reduce hallucinations while maintaining high descriptive accuracy. our results, supported by both human and automatic evaluations, suggest that this strategy can lower hallucinations compared to training on original captions or competing data curation baselines. furthermore, we introduce a proposition-based evaluation framework that provides fine-grained analysis of generated captions, offering deeper insights into the balance between descriptiveness and factuality. we believe that our findings contribute to improving fine-tuning methods in vlms from a data-centric approach, particularly in resourceconstrained environments where balancing descriptiveness and accuracy is important for real-world applications. we hope that this work will encourage further research into addressing the challenges of dense captions in vlms.