Visual Text Matters: Improving Text- KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant (2024)

Abstract: We revisit knowledge-aware text-based visual question answering, also known as Text-KVQA in the light of modern advancements in large multimodal models (LMMs), and make the following contributions: (i) We propose VisTEL – a principled approach to perform visual text entity linking. The proposed VisTEL module harnesses a state-of-the-art visual text recognition engine and the power of a large multimodal model to jointly reason using textual and visual context obtained using surrounding cues in the image to link visual text entity to the correct knowledge base entity. (ii) We present KaLMA – knowledge-aware large multimodal assistant that augments an LMM with knowledge associated with visual text entity in the image to arrive at an accurate answer. Further, we provide a comprehensive experimental analysis and comparison of our approach with traditional visual question answering, pre-large multimodal models, and large multimodal models, as well as prior top-performing approaches. Averaging over three splits of Text-KVQA, our proposed approach surpasses the previous best approach by a substantial 23.3% on an absolute scale and establishes a new state of the art. We make our implementation publicly available.

## introduction in the past few years, the research community has shown significant interest in visual question answering based on text appearing in images, as evidenced by the emergence of ocr-vqa @xcite , st-vqa @xcite and textvqa @xcite . giving another aspect to these problems by leveraging external knowledge for text-based visual question answering, @xcite introduced a task called text-kvqa. the text-kvqa presents a unique challenge: given an image containing textual entities like business brands, book titles, or movie titles, the task is to answer questions that require external knowledge about these entities. addressing text-kvqa involves detecting text in images, recognizing it, linking it to a knowledge base, and employing visual context and knowledge base for reasoning to provide an answer. since the introduction of this problem, several advancements have happened in visual text understanding as well as vision and language models. in this work, we revisit text-kvqa by leveraging these modern advancements and propose a framework that judiciously integrates various components of contemporary architecture. the emergence of large multimodal models (lmms) foot_0 represents a significant trend in the literature on vision and language @xcite @xcite @xcite @xcite . over the past few years, many large-scale language and vision models have been developed, demonstrating exceptional performance across various tasks including, but not limited to, image captioning, visual question answering, multimodal reasoning, and visual grounding. we believe that pretrained lmms hold great potential for addressing text-kvqa. these models are rich in the implicit knowledge learned by large-scale pretraining. however, despite their numerous advantages, they are not without drawbacks, notably hallucinations. this challenge becomes particularly apparent in text-kvqa, where precise reasoning about entities depicted in images and associated knowledge is required. consider the following scenario where a customer, after finishing their meal at a restaurant store, takes a picture of the store signboard and enquires about a possible future online delivery, asking, 'where can i place an online order from this store?' (figure 1(a) ). existing lmms often hallucinate over @xcite : given an image containing a named entity as visual text, e.g., "domino's" in this illustration, the aim is to answer the question by leveraging explicit knowledge about the visual text entity. (b) large multimodal models are one obvious choice for solving such tasks today. however, they alone are insufficient as they hallucinate on visual objects. (c) we propose a novel approach -kalma that augments an lmm with specialized visual text recognition and retrieved relevant knowledge obtained using visual text entity linking by proposed vistel. our approach establishes a new state-of-the-art for this task. the pizza present in the image and points to the website of 'pizza hut' instead of 'domino's' (figure 1(b) ); whereas complementing the lmm with an explicit visual text entity linking followed by knowledge-retrieval helps overcome hallucination (figure 1(c )), thereby generating an accurate answer to the given question. our model is developed on this hypothesis. we address text-kvqa by introducing an architecture, namely kalma -knowledge-aware large multimodal assistant that first invokes our proposed visual text entity linker or vistel -an lmm-architecture that links visual text entities to the associated knowledge base (illustrated in figure 1 (c )). once the entities are linked to the knowledge base, the associated knowledge is retrieved and augmented to a large multimodal model to answer visual questions. to summarize, our contributions are as follows: (i) we revisit text-kvqa -a task originally introduced by @xcite in the light of the latest advancements in large multimodal models. to this end, we benchmark latest lmms on text-kvqa. our study highlights that lmms although powerful, often ignore visual text present in the images, resulting in hallucinations. st-vqa @xcite , textvqa @xcite , docvqa @xcite , and ocr-vqa @xcite . while these benchmarks were successful in their intent of integrating reading and reasoning abilities in vqa, they are often restricted to reasoning around what is visually apparent. to address this gap and encourage models to perform reasoning beyond visually apparent facts, @xcite introduced knowledgeaware text-based vqa task. distinctively different from other knowledge-aware visual question answering tasks such as kb-vqa @xcite , fvqa @xcite , kvqa @xcite , ok-vqa @xcite , and infoseek @xcite ), text-kvqa deals with reasoning over visual text entities and associated knowledge to arrive at answer. ## methodology problem statement: text-kvqa @xcite is a knowledge-intensive visual questionanswering task that requires a system to read and interpret the visual text in an image and leverage it as a gateway to access and reason over external knowledge to answer the question. the external knowledge base k consists of a set of n entities @xmath0, e 2 , ..., e n } and their corresponding knowledge @xmath1, k 2 , ..., k n }, where each k i is a set of facts. for example, domino's pizza is an entity whose associated knowledge facts, obtained in the form of triplets from wikidata, are concatenated to form simple sentences such as "domino's pizza is a restaurant", "its headquarters are in ann arbor charter township", "it belongs to the fast food industry", and so on. in this section, we describe our approach, whose overall architecture is illustrated in figure 4 . our approach first links visual text entities using the proposed vistel module and retrieves relevant knowledge to the entity (section 3.1), it then reasons over the image and the retrieved knowledge to answer the question (section 3.2). ## conclusion we have revisited the text-kvqa and significantly advanced state of the art on this task. our findings suggest that visual text entity linking, combined with seamless reasoning using both visual and textual cues, as well as explicit external knowledge via lmm, is key to our success. we performed extensive ablation studies and analyses to support our claims. the future scope of this work is to expand the dataset with more visual-intensive queries and address text-kvqa for multilingual societies.