Combo of Thinking and Observing for Outside-Knowledge VQA (2023)

Abstract: Outside-knowledge visual question answering is a challenging task that requires both the acquisition and the use of open-ended real-world knowledge. Some existing solutions draw external knowledge into the cross-modality space which overlooks the much vaster textual knowledge in natural-language space, while others transform the image into a text which further fuses with the textual knowledge into the natural-language space and completely abandons the use of visual features. In this paper, we are inspired to constrain the cross-modality space into the same space of natural-language space which makes the visual features preserved directly, and the model still benefits from the vast knowledge in natural-language space. To this end, we propose a novel framework consisting of a multimodal encoder, a textual encoder and an answer decoder. Such structure allows us to introduce more types of knowledge including explicit and implicit multimodal and textual knowledge. Extensive experiments validate the superiority of the proposed method which outperforms the state-of-the-art by 6.17% accuracy. We also conduct comprehensive ablations of each component, and systematically study the roles of varying types of knowledge. Codes and knowledge data are to be released.

## introduction conventional visual question answering (vqa) @xcite tasks require models to answer questions based on image content. such tasks have been thoroughly studied @xcite @xcite on conventional vqa datasets vqav2 @xcite . however, real-world questions often rely on a certain amount of knowledge beyond images. therefore, knowledge base question answering (kb-vqa) tasks (a) the conventional vqa paradigm fuses image and question text into the cross-modality space, and then predicts answers in a close-set classification manner. (b) language-centric paradigm applies captioning and tagging tools to describe the visual context, and abandons the visual features to convert the vqa task into an open-ended generative qa task. (c) the proposed paradigm intends to constrain the cross-modality space into the same space as natural-language space so that models can directly decode both text and multimodal embeddings. @xcite @xcite @xcite always require models to answer questions by referring to the corresponding knowledge facts in a specific pre-defined knowledge base. yet any pre-defined knowledge base is far from covering real-world knowledge. recently, the outside-knowledge visual question answering (ok-vqa) task has been proposed @xcite and provides the most open vqa setting. that is, any knowledge resource can be used to answer its challenging and diverse questions. most previous work @xcite @xcite on ok-vqa follows conventional vqa paradigm (as shown in figure 1 (a) ) based on visual-language pre-trained (vlp) models, and injects knowledge into the same cross-modality space afterward. however, knowl-edge in cross-modality space is much less than that in natural-language space gao et al.. this paradigm excels at visual understanding, but refers to little knowledge, like a human who focuses on observing but does not think enough. to take the advantage of the vast knowledge in natural-language space, state-of-the-art methods @xcite @xcite on ok-vqa follow language-centric paradigm (as shown in figure 1 (b) ) based on pre-trained language models (plms). however, although more knowledge can be introduced, the paradigm is counter-intuitive because many visual details are lost when converting an image into text. therefore, it is like a human who starts thinking after brief observing. for a human, a feasible solution to ok-vqa is combo thinking while observing. to this end, we propose two, which is a framework consisting of a multimodal encoder, a textual encoder and an answer decoder. as shown in figure 1 (c), the multimodal encoder directly encodes the visual features and acts as the role of observer, while the textual encoder encodes a range of knowledge resources and acts as the role of thinker. finally, the answer decoder decodes the latent embeddings from both encoders to generate the final answer. in addition, a pre-training stage is added to help constrain the output of both encoders to the same latent space. previous methods @xcite @xcite have thoroughly studied explicit textual knowledge such as wikipedia, as well as implicit textual knowledge in gpt-3 @xcite . however, the discussion of multimodal knowledge, which further utilizes visual features, is still in its infancy in ok-vqa. in this paper, we accumulate explicit multimodal knowledge during pre-training on vqav2 @xcite . besides, inspired by prompting gpt-3 @xcite for implicit textual knowledge, we use prompt to bring in implicit multimodal knowledge stored in the unifying vlp model ofa @xcite . moreover, we refine a taxonomy of existing methods by knowledge (refer to figure 2 ) where our method is the first to bring in all types of knowledge. to summarize, our contributions are as follows: ## qualitative analysis case study on two encoders. to explore the respective roles of the two encoders, the upper part of figure 5 shows the examples that can be answered correctly by one of the two single-encoder models. plot (a) and (b) of figure 5 show that ours-mm excels at answering questions that need comprehension about image scenes and objects. for example, the orientation and the relative position between tv and sofa in plot (b) help generate the answer "watch tv". such scene information is easily omitted by a single textual encoder. this further validates that the multimodal encoder supplements the missing image information, and makes better use of the image when combining knowledge. plot (c) and (d) shows that ours-txt is an expert in answering questions that require focusing more on external knowledge rather than image understanding, since the textual encoder is the primary channel for receiving knowledge from multiple sources. ## conclusion and future work this paper proposes a simple and effective method that mimics human behavior "thinking while observing", i.e., benefiting from the vast knowledge in natural-language space while making the most of the visual features for better image understanding. our method establishes a new sota accuracy of 56.67% with a 6.17% improvement on ok-vqa. moreover, we consider more comprehensive types of knowledge, and systematically analyze the role of each type of knowledge in detail. we hope our work can stimulate followers to explore ok-vqa further along the direction of how to fuse both nature-language and cross-modality spaces better.