Can Hallucination Correction Improve Video-Language Alignment? (2025)

Abstract: Large Vision-Language Models often generate hallucinated content that is not grounded in its visual inputs. While prior work focuses on mitigating hallucinations, we instead explore leveraging hallucination correction as a training objective to improve video-language alignment. We introduce HACA, a self-training framework learning to correct hallucinations in descriptions that do not align with the video content. By identifying and correcting inconsistencies, HACA enhances the model’s ability to align video and textual representations for spatio-temporal reasoning. Our experimental results show consistent gains in video-caption binding and text-to-video retrieval tasks, demonstrating that hallucination correction-inspired tasks serve as an effective strategy for improving vision and language alignment.

## introduction aligning representations across modalities involves creating joint embeddings that map visual and linguistic features to a shared space, enabling the model to assess their similarity. this is crucial for tasks including cross-modal retrieval @xcite , mitigating hallucinations @xcite , compositional reasoning @xcite , visual-to-text generation @xcite , visual question answering @xcite , and visual-language navigation @xcite . while progress has been made in aligning image representations with text @xcite @xcite , advancements in video-language alignment remain limited. videos pose unique challenges due to their rich spatio-temporal information, involving multiple entities and scenes that dynamically interact and change over time. video-language input: does this caption accurately describe the video? → caption: outside, a blonde-haired woman is bending down to grab an object. video-llava output: yes, the caption accurately describes the video. the woman is seen bending down to pick up an object, and she is blonde. [p @xmath0.85 ] haca output: yes. this caption accurately describes the video. [p @xmath1.75 ] contrast input: does this caption accurately describe the video? → caption: outside, a female with blonde hair is standing up, grabbing an object. video-llava output: yes, the caption accurately describes the video. the female with blonde hair is seen standing up and grabbing an object. [p @xmath2.85 ] haca output: no. this caption shall be corrected as: outside, a female with blonde hair is lying down, grabbing an object. [p @xmath3.36 ] figure 1 : models tasked with determining whether a given video entails a caption, where the contrast caption closely resembles the correct one. haca effectively differentiates between the correct caption (top) and the incorrect one (bottom), and corrects hallucination in the latter. in contrast, video-llava fails to distinguish between those captions or correct the hallucination. models (video-llms in §3.1) can compute alignment scores @xcite but struggle to distinguish between similar videos and descriptions @xcite @xcite , as illustrated in figure 1 . one promising approach is to fine-tune video-llms on entailment tasks using similar captions @xcite , where the model is prompted to answer yes or no to whether a video is aligned with a given caption ( §3.1). however, using a single binary label as a learning signal fails to indicate which parts of the description misalign with the video. @xcite generates natural language explanations for mismatches but requires costly dataset construction with additional models and annotations. to this end, we introduce haca, a self-training framework grounded in hallucination correction for video-language alignment ( § 3.2). hallucination (or confabulation) refers to a mismatch between textual descriptions and the corresponding factual content of an image or video @xcite . haca requires the model to pre-dict whether a description entails the video content. if the description does not align, the model corrects the hallucinations to better match the video. instead of relying solely on a binary entailment label, haca uses hallucination correction as a finergrained learning signal to enhance the alignment of video and language representations. given that misalignment between modalities is a key factor in hallucination @xcite , we hypothesize that introducing a hallucination correction task can improve video-language alignment. haca also requires no external models or annotations beyond the ground-truth video description. to further enhance haca, we introduce a masking correction task as data augmentation ( §3.3). we fine-tune two video-llms with haca, and evaluate these fine-tuned models in a zeroshot manner on two spatio-temporally challenging downstream tasks ( § 4): velociti @xcite , a video-caption binding dataset, and ssv2-temporal @xcite and ssv2-events @xcite , which are text-tovideo retrieval datasets emphasizing action recognition. the models fine-tuned with haca outperform baseline models by up to 17.9% accuracy and 5.7 map points, demonstrating that haca effectively improves video-text alignments, and generalizes beyond in-domain data ( § 5). our code and data will be available upon acceptance. ## related work in addition to the video-language alignment approaches discussed in § 1, several methods leverage a contrastive learning objective to learn a shared video-language embedding space @xcite @xcite , and @xcite further introduces a contrastive loss in video-llms to enforce timeorder consistency. however, most of these models lack robustness to semantically plausible manipulations @xcite . @xcite also finds that applying a contrastive objective to video-caption datasets does not promote the model's ability to capture fine-grained details. in contrast, our approach encourages video-llms to capture more nuanced semantic mismatches by learning to correct hallucinations, extending beyond sentence-level hallucination detection. more discussion is provided in appendix b. ## haca: hallucination correction for video-language alignment to investigate whether hallucination correction can improve video-language alignment, we introduce haca, a fine-tuning objective for video-llm as a sequence-to-sequence generation task. ## experimental setup data (detailed in §a.5). we train haca using videos and their ground-truth and contrastive descriptions from videocon @xcite , generating 115,536 (video, description, correction) triplets for training and 8,312 for validation, which is used for model selection. synthetic contrast captions are also used to fine-tune the baseline entailment task with the same dataset sizes. we evaluate our trained models on text-to-video retrieval using the temporally-challenging ssv2-temporal @xcite and actionintensive ssv2-events @xcite datasets. additionally, we evaluate our models on compositional ability over time using the velociti benchmark @xcite . each video in the dataset includes a correct caption and an incorrect one. baselines. (i) pretrained video-llms: we employ two pre-trained models with different architectures, video-llava @xcite and videochat2 @xcite . more details in §a.4. (ii) entailment: we fine-tune the pretrained video-llms using the entailment task described in §3.1. more details about the implementation are in §a.1. ## analysis performance on text-to-video retrieval. haca consistently outperforms both the pretrained model and the entailment fine-tuned model, as illustrated in figure 3 . this demonstrates haca's ability to effectively capture the rich temporal information present in videos. on the ssv2-events dataset, while the entailment objective yields performance comparable to the pretrained video-llava, haca achieves better results on this action-intensive dataset, despite being fine-tuned on the same amount of data. additional comparisons with other models are provided in §a.2. performance on video-language binding. qualitative examples. figure 4 presents an example where haca outperforms the entailment baseline on the velociti dataset, and delivers ac-curate corrections. additional qualitative examples are provided in §a.3. ## conclusion video understanding through language is vital for applications like human-robot interaction and autonomous driving. we propose a novel approach to enhance video-language alignment by connecting it to the hallucination problem in visual-language models, paving the way for future advancements.