Pelican: Correcting Hallucination in Vision- LLM s via Claim Decomposition and Program of Thought Verification (2024)

Abstract: Large Visual Language Models (LVLMs) struggle with hallucinations in visual instruction following task(s). These issues hinder their trustworthiness and real-world applicability. We propose Pelican – a novel framework designed to detect and mitigate hallucinations through claim verification. Pelican first decomposes the visual claim into a chain of sub-claims based on first-order predicates. These sub-claims consists of (predicate, question) pairs and can be conceptualized as nodes of a computational graph. We then use use Program-of-Thought prompting to generate Python code for answering these questions through flexible composition of external tools. Pelican improves over prior work by introducing (1) intermediate variables for precise grounding of object instances, and (2) shared computation for answering the sub-question to enable adaptive corrections and inconsistency identification. We finally use reasoning abilities of LLM to verify the correctness of the the claim by considering the consistency and confidence of the (question, answer) pairs from each sub-claim. Our experiments demonstrate consistent performance improvements over various baseline LVLMs and existing hallucination mitigation approaches across several benchmarks.

## introduction large vision language models (lvlm) have seen significant advancements in recent years @xcite . they typically integrate visual tokens into the embedding space of a large language model (llm), leveraging the linguistic capabilities of llms while incorporating visual information for multimodal understanding. despite substantial performance gains, lvlms suffer from hallucinations due to limited training data, lack of * equal contribution precise grounding, and over-reliance on language priors @xcite . prior works have focused on scaling training data for reducing hallucinations, as demonstrated by the improved performance of llava-1.5 @xcite ) that used many academic datasets during instruction tuning. other works have created highquality visual instruction tuning datasets. for example, lrv @xcite included diverse examples as well as adversarial questions referencing non-existent objects. another promising direction has been to improve the model by using variants of reinforcement learning with human feedback (rlhf) to align the model with human preferences @xcite or by training the model to correct itself through self-feedback during inference @xcite . woodpecker @xcite , inspired by the fact-checking task in nlp @xcite , recently proposed to correct hallucinations by modeling the problem as verifying and correcting visual claims generated by lvlms. the method involved extracting key concepts from the outputs, formulating questions, answering them using visual tools such as vqa, and collating the outputs into a visual claim, which is then used to refine the original output with an llm. our work builds upon and extends this claim verification paradigm. we propose pelican (figure 1 ), a novel and structured pipeline for verifying visual claims to detect and correct hallucinations. pelican addresses the weaknesses of prior methods for claim verification such as lack of precise grounding, weak contextual integration and visual referencing, and ineffective reasoning over the claim and the visual context. pelican first breaks down a complex claim into more manageable sub-claims by using a set of predefined first-order predicates tailored to the visual question answering (vqa) task. these sub-claims are represented as a chain of (predicate, question) pairs, with each question stemming from its corresponding predicate. the resulting chain can be interpreted as a computational graph (figure 2 ), where each node corresponds to a predicate/question. the verification of the overall claim is then accomplished by answering the questions in a sequential manner. pelican uses program-of-thought prompting to synthesize python code that seamlessly integrates external tools with python operators, offering greater flexibility compared to previous methods like woodpecker. the introduction of intermediate variables to reference specific object instances is another innovation, which is crucial for precise grounding, particularly in claims involving multiple objects. furthermore, pelican shares computations from previous nodes in the chain while answering questions, enabling adaptive corrections and the identification of inconsistencies in the reasoning process, distinguishing it from earlier works. pelican creates a visual table representation, that includes key visual entities, stored as a pandas dataframe to simplify manipulation of key entities in the code generation step. the information from the sub-claims is then combined, and the reasoning capabilities of llm are used to assess the correctness of the claim and generate a refinement in case of hallucinations. robustness in this step is ensured by using in-context examples and promoting cot-style reasoning that considers the correctness, confidence, and relevance of the generated answers for each sub-claim. by integrating the flexibility of reasoning in language with the precision of computational methods, pelican is able to achieve strong improvements over sota methods. we show significant drop in hallucinations on standard benchmarks (mmhal-bench, gavie) and improve visual understanding accuracy on the mme dataset. our ablation study reveals the contribution of the key innovations on the final performance. through qualitative examples, we demonstrate how the model identifies and corrects hallucinated locations. our contributions are as follows, we: