MAR : Matching-Augmented Reasoning for Enhancing Visual-based Entity Question Answering (2024)

Abstract: A multimodal large language model MLLMs may struggle with answering visual-based (personal) entity questions (VEQA), such as ”who is A?” or ”who is A that B is talking to?” for various reasons, e.g., the absence of the name of A in the caption or the inability of MLLMs to recognize A, particularly for less common entities. Furthermore, even if the MLLMs can identify A, it may refrain from answering due to privacy concerns. In this paper, we introduce a novel method called Matching-Augmented Reasoning (MAR) to enhance VEQA. Given a collection of visual objects with captions, MAR preprocesses each object individually, identifying faces, names, and their alignments within the object. It encodes this information and stores their vector representations in vector databases. When handling VEQA, MAR retrieves matching faces and names and organizes these entities into a matching graph. MAR then derives the answer to the query by reasoning over this matching graph. Extensive experiments show that MAR significantly improves VEQA compared with the state-of-the-art methods using MLLMs.

## introduction multimodal language models (mllms) @xcite like gpt-4v @xcite and llava @xcite have significantly improved visual question answering (vqa) by integrating text and images. however, they still face challenges in visual-based entity question answering (veqa), a crucial subset of vqa that focuses on extracting information about specific entities @xcite . ## related work we categorize related work as follows. ## visual-based entity question answering (veqa) captioned visual objects. we consider a captioned visual object o as a pair o : (v, t ) where v is an image, and t is an optional text description relative to the image v . figure 1(a) and figure 1 (b) provide two sample captioned visual objects, (v 1 , t 1 ) and (v 2 , t 2 ), respectively. let @xmath0, o 2 , . . . , o n } be a group of captioned visual objects, sourced from public or enterprise datasets without privacy concerns. note that, such a group is common in practice, e.g., a collection of news articles. ## a new newspersonqa benchmark the problem of veqa needs to address complex interactions between multiple visual and textual data. despite its growing importance, existing benchmarks fall short in adequately representing the diverse challenges posed by veqa tasks. particularly in the domain of news qa, where the accurate identification and understanding of both common and uncommon persons are crucial, current datasets (e.g., @xcite and newsqa @xcite ) do not provide the necessary depth and breadth. to bridge this gap, based on goodnews (biten et al., 2019), we are constructing a new benchmark, namely newspersonqa, that encompasses a wide range of scenarios, including both well-known and obscure individuals. total queries 5,941 ## conclusion in this paper, we explore a novel visual-based (personal) entity questions (veqa) problem that focuses on aggregating clues from multiple captioned visual objects. we introduce matching graphs designed to capture the relationships between identical entities across various visual objects. extensive experiments demonstrate the high accuracy of our method. while our work has primarily focused on matching person entities, future research can aim to extend matching-augmented reasoning to other tasks. a experimental details 1. setup and environment: the experiments were conducted in a zero-shot setting using rtx 4090 gpus, with pytorch version 1.12.0. for gpt-4v, we used the interface of the gpt-4-vision-preview model. it is worth noting that gpt-4v often refrains from answering person identification questions without additional clues due to policy reasons. however, with the incorporation of matching graph techniques, it can leverage weak signals and combine them with its own knowledge base.