OVF act: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models (2025)

Abstract: Large vision-language models (VLMs) often struggle to generate long and factual captions. However, traditional measures for hallucination and factuality are not well suited for evaluating longer, more diverse captions and in settings where ground-truth human-annotated captions are unavailable. We introduce OVFact, a novel method for measuring caption factuality of long captions that leverages open-vocabulary visual grounding and tool-based verification without depending on human annotations. Our method improves agreement with human judgements and captures both caption descriptiveness (recall) and factual precision in the same metric. Furthermore, unlike previous metrics, our reference-free method design enables new applications towards factuality-based data filtering. We observe models trained on an OVFact-filtered (2.5-5x less) subset of a large-scale, noisy (VLM-generated) pretraining set meaningfully improve factuality precision without sacrificing caption descriptiveness across a range of downstream long caption benchmarks.

## introduction large vision-language models (vlms) are fundamental to a range of grounded language understanding applications, including multimodal ai assistants and tools @xcite . these models have grown in capability from generating short sentence descriptions @xcite to long paragraphs @xcite @xcite . however, long caption models struggle to maintain factuality 1 over these long descriptions @xcite , including hallucinations of objects that are not present in the input image. as such, there is a crucial need to both measure and improve this limitation. prior work for measuring factuality for vlms has been promising but limited. questionanswering approaches @xcite @xcite are indirect, and do not assess if a particular output from a model is factual or not @xcite , as is important in safety-critical settings @xcite . on the other hand, metrics for directly assessing caption text @xcite @xcite @xcite have traditionally been tailored for short captions or specific datasets with limited vocabularies (e.g., ms-coco @xcite ). notably, many of these methods shown in fig. 1 also rely on ground-truth human annotations, which means they do not work well when such references are unavailable @xcite , as is often the case when vlms are deployed at scale. finally, these metrics often do not capture "recall", the coverage of detailed objects in the caption @xcite . this leads to a conflicting incentive for long caption models to output short, conservative text with fewer objects, reducing the risk of a mistake but sacrificing important details @xcite . the simultaneous lack of these attributes in prior metric designs, highlighted in fig. 1 , also inhibits integrations with techniques that potentially improve factuality in long caption models. for example, outputs from strong vlm models have been used to generate large-scale pretraining datasets for long captions @xcite , but these captions can be prone to factuality errors. automatically filtering this data to ensure higher quality with previous metrics is not possible since this would require human ground truth. thus, there is a critical need for a unified, reference-free, open-vocabulary method for both measuring and improving factuality in long captions and models. in this work, we make the following contribution: (i) we introduce ovfact, a new method for measuring and improving open-vocabulary factuality in vision-language models (vlms) that output long, descriptive captions. our method leverages openvocabulary visual grounding and tool-based verification to operate robustly in settings without human annotations. it combines aspects of both precision (minimizing object hallucinations) and recall (ensuring coverage of diverse objects) in the same metric. we validate the combination of these tools with careful analysis and observe that our method improves agreement with human judgments for a range of vlm model outputs, particularly in reference-free settings. (ii) by addressing the combination of limitations in previous metrics, our method design enables new applications towards factuality-based data filtering. we observe that models trained on an ovfact-filtered subset (with 2.5-5x size reduction) of a large-scale, noisy (vlm-generated) pretraining set @xcite meaningfully improve factuality precision without sacrificing caption descriptiveness across a range of downstream benchmarks @xcite @xcite . these findings are consistent across different model scales and are further validated with human evaluation. ## related work object-level factuality in vlms. vlms, which are natural extensions of llms @xcite @xcite , exacerbate existing issues with hallucinations by introducing an additional visual modality for potential errors @xcite @xcite . a significant line of work @xcite @xcite @xcite @xcite relies on questionanswering style verification, where an instructiontuned vlm is expected to answer whether a particular fact about an image is valid. however, such an approach suffers from several limitations: first, the lack of interpretability makes understanding where the model fails challenging. in the case of binary (yes/no) questions, there is a high probability (50%) of the model answering correctly, but for the wrong reason. second, the qa abilities of the models do not necessarily translate into captioning capabilities @xcite . the alternative approach is to check if the facts mentioned in a model's output align with what is shown in the input images. so far, this has been limited to the presence of objects defined in classic captioning datasets with groundtruth annotations. for instance, the seminal work, chair @xcite , is strictly bound to the ms coco dataset @xcite as it requires a dataset with paired captions and objectlabel annotations. chair extracts nouns from a predicted caption using traditional nlp techniques and verifies their presence in an image using ground-truth object annotations. throne @xcite , aloha @xcite , and valor-bench @xcite improve on chair by employing an llm to parse generated free-form captions. while those methods can handle concepts outside of ms coco classes, they can only do so to a limited extent. during parsing, throne only considers a closed set of objects specifically defined within a dataset (ms coco or object365 @xcite ). valor-bench @xcite only operates on a carefully selected small subset of images from visual genome @xcite . aloha relaxes chair's string matching with a text similarity between annotated and predicted objects, yet also assumes access to an exhaustive list of human-figure 2 : ovfact overview. our reference-free method assesses two aspects of long captioning -precision and descriptiveness (recall) -in a unified manner, without requiring ground-truth reference annotations. we process a model output caption into a set of candidate entities c, and assess which subset g is groundable in the input image with open-vocabulary detection and segmentation tools. precision is then measured as a ratio of entities remaining (e.g. above"red blanket" and "white curtains"detected as hallucinated). to measure descriptiveness, we assess a large open-vocabulary concept set v and identify which concepts are grounded in the image r, then measure their recall within the candidate entities from the vlm c caption using maximum similarity scoring. unlike prior work @xcite @xcite , our method can be directly applied to assess factuality in settings where only model-generated caption outputs are available, such as data filtering. data filtering. data filtering for contrastive image-text pretraining @xcite on web-scale data has shown it can lead to stronger models and improved training efficiency @xcite @xcite @xcite . meanwhile, data filtering for generative vlms training has seen less attention, discussing data curation for instruction fine-tuning @xcite @xcite . their primary focus, however, is to improve the instruction-following of the model and not on the factual quality of captioning outputs as we do here. @xcite considers data curation for image captioning models by filtering or replacing examples with high training losses on small-scale datasets with human-annotated short captions. in contrast, our proposed metric can operate on large-scale pretraining datasets with noisy (vlm-generated) long captions in terms of their factuality; we show this leads to consistent improvements across a range of measures. crucially, this application is only possible as our metric is reference-free and does not require ground-truth object annotations like prior works @xcite @xcite . ## method we detail ovfact in sec. 3.1 to measure openvocabulary object factuality and then describe how our proposed metric can be used for data filtering to improve a vlm's factuality in sec. 3.2. finally, sec. 3.3 summarizes how ovfact addresses the limitations of existing approaches. ## experiments in this section, we first describe our overall experimental setup and technical details in sec. 4.1. next, in sec. 4.2, we discuss the verification of ovfact as a method for measuring factuality, including ablation studies of its components. finally, in sec. 4.3, we present results for improving factuality, applying ovfact for data selection. ## conclusions we introduced ovfact, a novel method for measuring caption factuality that leverages openvocabulary visual grounding and tool-based verification. unlike previous metrics, ovfact is not dataset-specific and effectively evaluates the factuality of long, descriptive captions. we further demonstrated how ovfact can be used to improve the factuality of vlms by using it as a metric for data filtering. we show that models trained on an ovfact-filtered subset (2.5-5x smaller) of a largescale, noisy (vlm-generated) pretraining set meaningfully improved factuality precision without sacrificing caption descriptiveness when evaluating zero-shot on various downstream datasets.