VALOR - EVAL : Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models (2024)

Abstract: Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs, undermining their reliability. A comprehensive quantitative evaluation is necessary to identify and understand the extent of hallucinations in these models. However, existing benchmarks are often limited in scope, focusing mainly on object hallucinations. Furthermore, current evaluation methods struggle to effectively address the subtle semantic distinctions between model outputs and reference data, as well as the balance between hallucination and informativeness. To address these issues, we introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases. Moreover, we propose a large language model (LLM)-based two-stage evaluation framework that generalizes the popular CHAIR metric and incorporates both faithfulness and coverage into the evaluation. Experiments on 10 established LVLMs demonstrate that our evaluation metric is more comprehensive and better correlated with humans than existing work when evaluating on our challenging human-annotated benchmark dataset. Our work also highlights the critical balance between faithfulness and coverage of model outputs, and encourages future works to address hallucinations in LVLMs while keeping their outputs informative.

## introduction large vision-language models (lvlms) @xcite @xcite have shown remarkable performance across a broad range of vision-language tasks. despite the promising progress, the issue of hallucinations has emerged as a critical concern. hallucination refers to the generation of plausible-sounding but inaccurate or fabricated textual descriptions for a given image, which can compromise the reliability and trustworthiness of the models. recent studies have proposed various methods to evaluate models' generative hallucinations @xcite @xcite and discriminative hallucinations @xcite @xcite . however, they predominantly focus on hallucinations concerning object existence and their faithfulness within generated content, often neglecting other critical types of hallucinations and the assessment of coverage. this oversight can result in a lack of attention to the variety and depth of hallucinations evaluation hallucination type human faithfulness coverage open vocab. ## existing lvlms hallucination evaluation benchmarks and metrics as shown in regarding benchmark annotations, many existing benchmarks employ different ways of annotating the evaluation datasets automatically. for example, @xcite employ object detectors to identify all objects in an image; @xcite employ gpt-4v(ision) to generate ground-truth annotations. there are also approaches to developing models specifically for automatic evaluation, thereby bypassing the need for benchmark collections process @xcite . given the challenges and potential inaccuracies associated with automated models, our study opts to annotate the evaluation dataset manually to ensure the annotation accuracy and encompass the distinct categories of hallucinations. additionally, most existing benchmarks focus exclusively on hallucination evaluation, which can favor precise but uninformative model outputs, overlooking the aspect of coverage. to address the issue, we incorporate coverage scores in our evaluation. we note that two relevant concurrent works @xcite also include the coverage scores. however, compared with our work, they are either limited in scope, focusing only on objects or simple attributes and relations, or are unable to be adopted in openvocabulary generation settings. besides, along with the benchmark, we propose an evaluation metric generalizing their adopted chair metric. ## valor-bench in this section, we detail the methodology employed to create the benchmark, which aims to evaluate the hallucination issues of lvlms. as illustrated in figure 2 , constructing this benchmark (2) human annotations ( §3.2): finally, we manually annotate each image within the distinct feature subsets, adhering to the definition in §3.1.1. here, we provide an example of how we use the co-occurrence statistics to select images for object subsets and add human annotations for later evaluation. ## valor-eval we propose a framework valor-eval that generalizes chair, a metric that is widely adopted in existing studies @xcite , by introducing semantic matching and incorporating both the faithfulness and coverage aspects into the evaluation. as shown in figure 3 , our evaluation process has two steps: feature extraction and matching ( §4.1) and scoring ( §4.2). the image features four women standing ... from left to right, the first woman is wearing a black suit, the second woman is wearing a pink outfit, the third woman is wearing a gray suit, and the fourth woman is wearing a white suit. (woman, (black, outfit)), (woman, (pink, suit)), (woman, (gray, suit)), (woman, (white, suit)) (woman, (black, suit)), (woman, (pink, suit)), (woman, (coral, suit)), (woman, (gray, suit)) 0.5 0.25 attribute in the image, a young girl is blow-drying her hair with a pink hair drier. she is smiling and appears to be enjoying the process. the scene takes place in a bathroom, with a sink and a toilet visible in the background. there are two towels in the room, ... the image features a wooden dining table with a plate, a glass, a mouse, and a knife on it. the plate is placed in the center of the table, while the glass is next to it. the mouse is positioned on the left side of the plate, and the knife is located on the right side of the plate. ... generated attributes ground-truth attributes matched attributes faithfulness score coverage score llms object existence girl, hair, hair drier, bathroom, sink, toilet, towel, towel kid, hair, hair dryer, sink, brush, comb, room, toilet paper, basket, towel, bag, plate, faucet generated objects ground-truth objects hair, hair dryer, sink, towel, room matched objects kid broader conceptual objects relation (plate is on table) (plate is in the center of table), (glass is next to table), (mouse is to the left of plate), ...... (plate is in the center of table), (glass is above the plate), (mouse is to the right of plate), ...... ground-truth relations generated relations (plate is in the center of table), ... matched relations (glass is next to the plate) broader conceptual relations lvlms generated captions benchmark images gpt-4 gpt-4 gpt-4 llms gpt-4 gpt-4 gpt-4 benchmark collection caption generation feature extraction feature matching evaluation metric (woman, (pink, suit)) broader conceptual attributes (woman, (black, outfit)) 0.875 0.43 faithfulness score coverage score 0.75 0.45 faithfulness score coverage score figure 3: overview of valor-eval evaluation framework: (1) firstly, lvlms generate captions from valor-bench benchmark images. (2) following this, llms are employed to extract pivotal features that encapsulate from the generated descriptions. ## conclusion we introduce a comprehensive multi-dimensional benchmark, named valor-bench, dedicated to the evaluation of lvlms, with a particular focus on measuring hallucinations in generative tasks. our benchmark categorizes hallucinations into three distinct types -object, attribute, and relation -offering a detailed understanding of model inaccuracies. furthermore, our novel evaluation framework, referred to as valor-eval, employs a two-stage approach that integrates an llm, effectively addressing the complexities related to open vocabularies, semantic similarities, and the intricate assessment of attributes and relationships. this method significantly enhances the precision and depth of image captioning evaluations compared to previous methods. our experimental findings highlight the persistent challenges in this field, demonstrating that even state-of-the-art models such as gpt-4v, are prone to a considerable degree of hallucination. this study emphasizes the imperative for continuous advancements in lvlm evaluation techniques and establishes a new benchmark for future endeavors aimed at reducing hallucination and bolstering the reliability of content generated by lvlms.