COFAR : Commonsense and Factual Reasoning in Image Search (2022)

Abstract: One characteristic that makes humans superior to modern artificially intelligent models is the ability to interpret images beyond what is visually apparent. Consider the following two natural language search queries – (i) “a queue of customers patiently waiting to buy ice cream” and (ii) “a queue of tourists going to see a famous Mughal architecture in India”. Interpreting these queries requires one to reason with (i) Commonsense such as interpreting people as customers or tourists, actions as waiting to buy or going to see; and (ii) Fact or world knowledge associated with named visual entities, for example, whether the store in the image sells ice cream or whether the landmark in the image is a Mughal architecture located in India. Such reasoning goes beyond just visual recognition. To enable both commonsense and factual reasoning in the image search, we present a unified framework namely Knowledge Retrieval-Augmented Multimodal Transformer (KRAMT) that treats the named visual entities in an image as a gateway to encyclopedic knowledge and leverages them along with natural language query to ground relevant knowledge. Further, KRAMT seamlessly integrates visual content and grounded knowledge to learn alignment between images and search queries. This unified framework is then used to perform image search requiring commonsense and factual reasoning. The retrieval performance of KRAMT is evaluated and compared with related approaches on a new dataset we introduce – namely COFAR.

## introduction retrieving relevant images for a natural language query has been an exciting field of research in the vision-and-language community @xcite @xcite . most of the available literature focuses on querying visually-evident aspects in the images, such as searching for objects or their interactions in natural scenes. however, as illustrated in figure 1 , users often require an image search engine that can perform commonsense reasoning and leverage facts (world knowledge) about the image content. to fill this gap, we propose a novel image search task requiring commonsense and factual reasoning associated with named visual entities. to study this problem, a suitable dataset is required. while many text-to-image search datasets are publicly available @xcite @xcite , they have not been explicitly created to study our proposed task. few of the recently introduced knowledge-enabled vqa datasets such as ok-vqa @xcite , kvqa @xcite ), text-kvqa @xcite , fvqa @xcite require either factual or commonsense or a combination of both. however, they may not be well-suited for studying the "image search" task we are interested in. note that in the conventional vqa task, a query (question) is evaluated against a single image which is often directly relevant to the query; whereas, in image search, a query needs to be evaluated against several thousands of images, including distractors and then needs to rank the relevant image as the top result. moreover, to our knowledge, there is no dataset available that includes natural scene images containing a diverse set of visual named entities (such as business brands, celebrities, and world landmarks), visual details of the natural scene along with annotations that demands commonsense and factual reasoning associated with the images. to meet these requirements, we present cofar, which contains manually annotated english language queries for natural scenes containing named visual entities. a plausible approach to addressing our image search problem on cofar is large-scale visionlanguage pretraining @xcite and learning the associations between commonsense-factual concepts and images. this can be successful in learning popular associations, e.g., starbucks to coffee, eiffel tower to paris if it has seen such samples during training. however, such methods often require large data and generalize poorly to unseen or rare entities. in contrast, we take a distinct path in this work and ground external knowledge associated with entities in the images to perform commonsense and fac-tual reasoning. to this end, we present a unified model, namely knowledge retrieval-augmented multimodal transformer (kramt), that retrieves relevant knowledge from wikipedia by performing query-knowledge similarity-guided visual entity linking. it then encodes the retrieved knowledge, query and visual features, and learns image-query alignment using a multimodal transformer to perform knowledge-aware image search. contributions of this paper: (i) we study the problem of image search requiring both commonsense and factual reasoning associated with named visual named entities such as business brands, celebrities, and world landmarks for the first time and introduce a novel dataset, viz. cofar for this task. we firmly believe that the proposed task, accompanying dataset, and benchmarks presented in this paper will open up future research avenues. (section 3) (ii) we introduce a knowledge retrieval augmented multimodal transformer (kramt) -a unified framework that learns to align queries with the relevant images by performing visual entity linking, retrieving relevant knowledge, and seamlessly integrating it with visual content. the experimental results demonstrate that kramt, besides visual reasoning, can perform commonsense and factual reasoning (section 4 and section 5). ## cofar: dataset for image search requiring commonsense and factual reasoning we introduce cofar, a dataset for studying the novel problem of image search that requires commonsense and factual reasoning. a detailed comparison with related datasets is made in ## knowledge retrieval-augmented multimodal transformer (kramt) given a natural language query and a large gallery of images each containing a visual named entity, our goal is to retrieve relevant images. to this end, we present knowledge retrieval-augmented multimodal transformer (kramt) -an unified framework that contains two major modules: (i) visual entity and query-aware knowledge retrieval and (ii) knowledge-infused multimodal transformer as illustrated in figure 5 . ## experiments and results we group image retrieval baseline approaches into three categories: (i) knowledge-only, (ii) visiononly, and (iii) knowledge-aware vision and language (v-l) models to investigate the following questions respectively: under knowledge-only, we utilize bert @xcite to perform query-knowledge sentence-matching. in vl models, we use modern text-to-image retrieval methods, namely vse++ @xcite , and competitive vision-and-language transformers such as visual-bert (li et al., 2020), vilbert @xcite , and vinvl @xcite . knowledgeaware vl models: as there are no directly comparable knowledge-aware image-retrieval methods in current literature, we implement a few knowledgeaware visual question answering-based models with appropriate modifications to make them compatible for our task: (i) modified memory network: memory networks, and their variations have shown to yield state-of-the-art performance on knowledge-aware vqa benchmarks @xcite . we implement this baseline by using top-k knowledge texts. these texts are scored with a query, and the weighted sum of this representation, cnn features of the image, and query representation are passed to a binary classifier that classifies if the image is relevant to the query. (ii) krisp-inspired model: krisp @xcite addresses open knowledge-based vqa using implicit and symbolic knowledge stored in a graph data structure. in our setting, we use unstructured knowledge text in place of symbolic knowledge. we model implicit knowledge using mm-bert, similar to krisp, and for unstructured text, we use bert embedding of the knowledge text. the output of these representations along with bert-based query representation is fed to an mlp for learning alignment. (iii) kqia: here, knowledge text, along with queries and images, are encoded using gated recurrent units and cnn, respectively, and are then projected into a common space to learn alignment. all baselines are pretrained on the coco dataset unless mentioned otherwise. ## conclusion in information retrieval and nlp community, knowledge bases are instrumental in enabling commonsense and semantic search. however, their utility in semantic image search has not been extensively explored in the literature. we have drawn the attention of the vision and language community towards this issue through our work and presented a novel multimodal transformer namely kramt which seamlessly combines image, query, and knowledge encoding to learn alignment between the image with associated knowledge and query. we firmly believe that image search requiring commonsense and factual reasoning and the new dataset viz. cofar introduced in this work will open up several future research avenues.