Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? (2023)

Abstract: Pre-trained vision and language models have demonstrated state-of-the-art capabilities over existing tasks involving images and texts, including visual question answering. However, it remains unclear whether these models possess the capability to answer questions that are not only querying visual content but knowledge-intensive and information-seeking. In this study, we introduce InfoSeek, a visual question answering dataset tailored for information-seeking questions that cannot be answered with only common sense knowledge. Using InfoSeek, we analyze various pre-trained visual question answering models and gain insights into their characteristics. Our findings reveal that state-of-the-art pre-trained multi-modal models (e.g., PaLI-X, BLIP2, InstructBLIP) face challenges in answering visual information-seeking questions, but fine-tuning on the InfoSeek dataset elicits models to use fine-grained knowledge that was learned during pre-training. Furthermore, we show that accurate visual entity recognition can be used to improve performance on InfoSeek by retrieving relevant documents, showing a significant space for improvement.

## introduction the acquisition of knowledge occurs in the pretraining of large language models @xcite , demonstrated as their emergent ability to answer informationseeking questions in the open-world, where the questioner does not have easy access to the information. while prior works have analyzed models' capabilities to answer textual information-seeking (or info-seeking) questions, much less is known for visual info-seeking questions. for example, after taking a picture of the specific church in figure 1 , a person might want to know the date of construction, or who decorated the interior of the church. although the entity is presented in the image (the specific church), the relevant knowledge (e.g., the date) is not. given recent advances on pre-trained visual and language models @xcite @xcite , do these models also understand how to answer visual information-seeking questions? to study this research question, a visual question answering (vqa) dataset focusing on info-seeking questions is inevitably required. however, not all vqa datasets meet this criterion. for example, by design, the majority of questions in datasets such as vqa v2 @xcite focus on visual attributes and object detection that does not require information beyond the image to answer. while models capable of answering these types of questions have the potential to aid visually impaired individuals @xcite , there is a broader class of info-seeking questions that cannot be easily answered by sighted adults. handling such questions (e.g., when was this building constructed? 1955) is critical as they come closer to the natural distribution of human questions. in this paper, we present infoseek, a natural vqa dataset that focuses on visual info-seeking questions. different from previous vqa datasets, the testing subset of infoseek is collected in multiple stages from human annotators to evaluate vqa where the question can not be answered from only the visual content (see a comparison of datasets in § 2). in addition to this manually curated test set, which enables realistic evaluation of info-seeking vqa, we also join annotations from a recent visual entity recognition dataset @xcite with the wikidata database @xcite , and employ human annotators to write templates to semi-automatically generate a large corpus of visual info-seeking qa pairs. over 1 million {image, question, answer} triplets are generated to support fine-tuning multimodal models for info-seeking vqa. we split data to ensure memorizing knowledge during fine-tuning is useless -models either have to learn to use knowledge learned during pre-training or learn to retrieve knowledge from an external knowledge base. using infoseek, we analyze the ability of stateof-the-art models to answer visual info-seeking questions. we found pre-trained vision-language models, such as models pre-trained end-to-end (e.g., pali-x by @xcite , and models pre-trained with frozen llm (e.g., blip2 by @xcite , both struggle to answer info-seeking questions in zeroshot, though blip2 outperforms pali-x by a margin. surprisingly, after fine-tuning on our (large, semi-automatically curated) training set, pali-x yields a significant improvement and outperforms the fine-tuned blip2 models on queries that are unseen during fine-tuning. this suggests that while pre-trained pali-x has a significant amount of knowledge, it requires a small amount of finetuning data to fully awaken its capabilities. furthermore, we show that infoseek fine-tuned models can even generalize to questions and entity types completely unseen during fine-tuning (e.g., art & fashion). when incorporating a visual entity recognition component, and conditioning models on the wikipedia articles of the relevant entities, we show that models accessing such a knowledge base (with-kb) perform better overall than those that rely on knowledge learned during pre-training. however, end-to-end (no-kb) models were found better on certain classes of questions that require coarse-grained answers ("which continent is this building located on?"), even on tail entities. our dataset ok-vqa viquae infoseek palm (q-only) 23.8 31.5 5.6 current sota 66.1 22.1 18.2 require knowledge † 29.2% 95.2% 95.6% † :% of questions that require knowledge to answer. palm (q-only): a question-only baseline using palm. while there have been plenty of knowledgeintensive vqa (ki-vqa) benchmarks, we show that none of these meet the criteria to effectively evaluate info-seeking vqa. early efforts in this area, such as kbqa @xcite and fvqa @xcite , were based on domainspecific knowledge graphs, while recent datasets like ok-vqa @xcite and its variants such as s3vqa @xcite and a-okvqa @xcite have improved upon this foundation by incorporating an opendomain approach and highlighting common-sense knowledge. among the existing benchmarks, k-vqa (sanket shah and talukdar, 2019) and vi-quae @xcite are the most relevant, but they have severe limitations in their question generation process, as discussed below. ## infoseek: a vqa benchmark of visual information-seeking questions the infoseek dataset consists of two components, (1) infoseek human : a collection of humanwritten visual info-seeking questions (8.9k) to simulate information seeking intent (see § 3.1); and (2) dataset # {i, q, a} len of q/a # entity # ent. type ok-vqa 14k 8.1/1.3 --⋆ k-vqa 183k 10.1/1.6 18,880 1 † viquae 3.6k 12.4/1.7 2,397 980 infoseek -wikidata 1.35m 8.9/1.5 11,481 2,739 -human 8.9k 7.8/2.3 806 527 ⋆: ok-vqa does not specify visual entities. †: k-vqa only covers entities from the human category. infoseek wikidata : an automated dataset (1.3m) to cover diverse entities for large-scale training and evaluation purposes (see § 3.2). we split the dataset to ensure memorizing the training set is useless, thereby emphasizing the importance of pre-training to acquire knowledge (see § 3.3). due to space limitations, we summarize the key essence in this section and defer details to the appendix. image sources for diverse entity coverage. we sourced images from 9 image classification and retrieval datasets used in hu et al., including landmarks (17%), animals (13%), food (5%), aircraft (3%), etc. we utilize their annotation, which links visual entities to their corresponding wikipedia articles, to construct our infoseek dataset. ## protocols and models for infoseek motivated by previous research on text-based question benchmarks @xcite , we introduce two evaluation protocols, i.e, no kb and with kb, to evaluate models with different information accessible from infoseek. the no-kb protocol. models are tasked to directly predict the answer by examining the image and question, similar to traditional vqa systems. this requires the model to store world knowledge in its parameters for effective question answering. the research question focuses on how much knowledge can an end-to-end model memorize in its parameters during pre-training, and how well can it utilize this knowledge after fine-tuning? we use the standard vqa formatted data, i.e, {image (i), question(q), answer(a)} triplets for training / validation, and {i, q} for testing. the with-kb protocol. the goal is to analyze headroom for improvement when a viable reasoning chain is explicitly provided. therefore, this protocol encourages an extra step of visual entity recognition, grounding the task on a knowledge base. the vqa task is transformed into a two-step pipeline, i.e, (1) visual entity recognition; and (2) language qa with entity information. we provide training signals to first recognize the queried visual entity and then leverage the information to query a large language model for answers, or identify relevant wikipedia articles for extracting the answer. specifically, we provide a 100k wikipedia kb (articles and infobox images) that includes visual entities from infoseek and top frequent entities from wikipedia. during training and validation, with kb protocol provides entity labels for each queried visual entity. during testing, the model is evaluated based on the {i, q} pairs only. 4.1 models without kb information random & prior. random answers sampled from the training set; the majority answer based on the question prior, which is calculated using the train-ing set questions grouped by question 4-gram. palm (q-only) model. to validate the importance of visual content in infoseek, we build a question-only baseline with palm (540b) (chowdhery et al., 2022), using text question as the only input and with 5-shot in-context-learning. ## related work pre-trained vision language models. there has been significant growth in the development of vision-language models pre-trained on large-scale image-text datasets @xcite @xcite @xcite . one line of research aims to augment a pre-trained language model with visual modality by learning a mapping from an external visual encoder to the frozen large language model @xcite @xcite , to fully leverage textual knowledge from the language model @xcite @xcite @xcite . ## conclusion we introduced infoseek, a large-scale vqa dataset that focuses on answering visual information seeking questions. with infoseek, we found that current state-of-the-art pre-trained visuallanguage models struggle to answer visual infoseeking questions requiring fine-grained knowledge, such as questions about time and numerical information of a visual entity. our analysis using pipeline systems, which ground visual entities to an external knowledge base, suggests that incorporating fine-grained knowledge into the pre-training process holds significant potential to improve endto-end pre-training models.