S nap NT ell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM (2024)

Abstract: Vision-extended LLMs have made significant strides in Visual Question Answering (VQA). Despite these advancements, VLLMs still encounter substantial difficulties in handling queries involving long-tail entities, with a tendency to produce erroneous or hallucinated responses. In this work, we introduce a novel evaluative benchmark named SnapNTell , specifically tailored for entity-centric VQA. This task aims to test the models’ capabilities in identifying entities and providing detailed, entity-specific knowledge. We have developed the SnapNTell Dataset , distinct from traditional VQA datasets: (1) It encompasses a wide range of categorized entities, each represented by images and explicitly named in the answers; (2) It features QA pairs that require extensive knowledge for accurate responses. The dataset is organized into 22 major categories, containing 7,568 unique entities in total. For each entity, we curated 10 illustrative images and crafted 10 knowledge-intensive QA pairs. To address this novel task, we devised a scalable, efficient, and transparent retrieval-augmented multimodal LLM. Our approach markedly outperforms existing methods on the SnapNTell dataset, achieving a 66.5% improvement in the BELURT score.

## introduction vision-extended llms have shown significant advancements, excelling at capturing complex semantics and context-aware attributes needed for intricate tasks. however, their abilities in factual vqa tasks, which demand accurate, concrete answers about real-world entities and phenomena, expose certain limitations. particularly, torso-to-tail or long-tail entities, which constitute a large proportion of real-world data but appear infrequently in training datasets, pose a challenge. this scarcity in representation often leads to vllms resorting * work done while at meta figure 1 : comparing snapntell with existing methods reveals a distinctive focus. in the snapntell benchmark, the answers are predominantly entity-centric, characterized by a greater depth of knowledgeable information pertaining to the specific entity depicted in the image as the answer. to generating plausible but incorrect or imaginative content in their outputs, a problem that manifests as "hallucinations" within the context of model responses. to ensure the confident deployment of vllms in practical scenarios, there is an urgent need for dedicated research that not only recognizes but actively strives to tackle and reduce instances of hallucinations, especially in the context of factual queries involving these long-tail entities. the lack of publicly available evaluation datasets specifically tailored to assess models' ability in recognizing real-world long-tailed entities presents a notable gap in vqa. existing datasets fall short in serving this purpose due to a narrow range of entity categories, the prevalence of overly simplistic yes/no qa pairs, and a general lack of entity specificity, often using broad terms like "tiger" instead of more specific ones like "siberian tiger". to address this gap, we introduce a novel evaluation task called snapntell, which focuses on entity-centric knowledge-based vqa. the snap-ntell benchmark has been designed to evaluate models' abilities in accurately identifying entities and generating responses that showcase a deep understanding of these entities. to support this task, we have curated a new evaluation dataset that departs from existing datasets in two crucial ways: (1) it includes a wide range of fine-grained and catego-rized entities, each accompanied by corresponding images and clear mention of the entity name within the answer sets. (2) it features qa pairs designed to prompt knowledge-intensive responses, moving beyond the binary yes/no format to challenge and assess the depth of the model's comprehension. furthermore, the limitations identified in factual query generation underscore the need for new solutions to address the problem of hallucinations. recent advancements suggest that retrieval-based approaches hold significant promise in this regard @xcite @xcite . these methods enhance llms by integrating external knowledge sources or incorporating retrieval mechanisms to access relevant information from extensive knowledge bases. the synergy between the advanced inference capabilities of llms and the wealth of external knowledge has the potential to significantly reduce issues related to long-tail entities and, consequently, decrease the occurrence of hallucinatory responses. in this work, we aim to propose an evaluation task to investigate the model's ability to recognize real-world long-tailed entities and provide knowledge-intensive answers. we also propose a retrieval-augmented method to reduce hallucinations and enhance the precision and trustworthiness of generated responses. our contribution is summarized as follows: ## method in this section, we will introduce the details of our proposed retrieval-augmented multimodal llm model. the architecture of our model is shown in figure 3 (larger figure in appendix d due to space limit). our model can be considered twofold: (1) retrieval augmentation. given the input image-question pair, we retrieve useful entitycentric information within knowledge sources. (2) entity-centric knowledge-based answer generation. the retrieved information will be combined with the image and question together to generate a knowledgeable answer. ## conclusion in this work, we tackle the significant challenge vllms face with long-tail entity queries, which often lead to inaccurate or hallucinated responses. to address these issues, we introduce an entity-centric vqa task named snapntell. this task is designed to test models on entity recognition and their ability to provide detailed, entity-specific knowledge in their responses. we collected a unique evaluation dataset for this task, which distinguishes itself from existing vqa datasets by including a wide array of fine-grained categorized entities, supported by images and explicit entity mentions in the answers. this dataset emphasizes knowledgeintensive responses over simple binary answers. in addition, we propose a retrieval-augmented multimodal llm solution for the snapntell task as an effective baseline. our experimental results show that our model outperforms existing approaches, providing more accurate and coherent answers.