KAT : A Knowledge Augmented Transformer for Vision-and-Language (2022)

Abstract: The primary focus of recent work with large-scale transformers has been on optimizing the amount of information packed into the model’s parameters. In this work, we ask a complementary question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6% absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. Additionally, explicit knowledge integration improves interpretability of model predictions in our analysis.

## introduction there has been a revival of interest in knowledgeintensive tasks which require an external knowledge source for humans to perform. many applications in real-world scenarios, such as autonomous ai agents, need to seamlessly integrate implicit (i.e., commonsense) and explicit knowledge (e.g., wikidata) to answer questions. in this work, we investigate how to effectively integrate implicit and explicit knowledge for reasoning. tasks like outside knowledge visual question answering (ok-vqa) @xcite require that models use knowledge not present in the input to answer ques-work done when liangke and borui interned at microsoft research. tions, making it an ideal test bed for investigating this implicit-explicit knowledge trade-off. consider the examples from ok-vqa shown in figure 1 . to answer the question in the left example, the system needs to both ground organism to bird through explicit knowledge and then apply the implicit knowledge birds evolved from reptiles to answer the question. similarly for the question in the right example, the system needs to recognize boats and harbor and requires the implicit knowledge anchors are used to stop boats from moving. a key challenge here is to accurately link image content to abstract external knowledge. there have been a number of recent developments demonstrating the feasibility of incorporating external knowledge into question answering models @xcite @xcite @xcite . existing methods first retrieve external knowledge from external knowledge resources, such as dbpedia @xcite and conceptnet @xcite before jointly reasoning over the retrieved knowledge and image content to predict an answer. however, most existing approaches have several drawbacks. first, explicit knowledge retrieved using keywords from questions or image tags may be too generic, which leads noise or irrelevant knowledge during knowledge reasoning. second, existing work mainly focuses on explicit knowledge which is often in the form of encyclopedia articles or knowledge graphs. while this type of knowledge can be useful, it is insufficient to answer many knowledge-based questions. as shown in figure 1 , questions require the system to jointly reason over explicit and implicit knowledge, which is analogous to the way humans do. to address these challenges, we propose an approach, kat, to effectively integrate implicit and explicit knowledge during reasoning. the main contributions of our work are as follows: i) knowledge extraction. we adopt two novel methods for knowledge extraction that significantly improve the quality and relevance of extracted knowledge: for implicit knowledge, we design new prompts to extract both tentative answers and supporting evidence from a frozen gpt-3 model; for explicit knowledge, we design a contrastivelearning-based explicit knowledge retriever using the clip model, where all the retrieved knowledge are centered around visually-aligned entities. ii) reasoning in an encoder-decoder transformer. we design a novel reasoning module in kat to perform jointly reasoning over explicit and implicit knowledge during answer generation, which is trained by using an end-to-end encoderdecoder transformer architecture. iii) ok-vqa performance. kat sets a new state of the art on the challenging ok-vqa @xcite benchmark, and significantly outperforms existing approaches. ## related work vision-language transformer. multimodal transformers have made significant progress over the past few years, by pre-trained on largescale image and text pairs, then finetuned on downstream tasks. visualbert @xcite , unicoder-vl @xcite , nice @xcite , and vl-bert @xcite propose the single-stream architecture to work on both images and text. vilbert @xcite and lxmert @xcite propose a two-stream architecture to process images and text independently and fused by a third transformer in ta later stage. while these models have shown to store in-depth cross-modal knowledge and achieved impressive performance on knowledge-based vqa @xcite @xcite , this type of implicitly learned knowledge is not sufficient to answer many knowledge-based questions @xcite . another line of work for multimodal transformers, such as clip @xcite or align @xcite , aligns visual and language representations by contrastive learning. these models achieve state-of-the-art performance on image-text retrieval tasks. different from existing work that uses multimodal transformers as implicit knowledge bases, we focus primarily on how to associate images with external knowledge. importantly, our model only relies on multimodal transformers learned by contrastive learning which do not require any labeled images. this makes our model more flexible in real-world scenarios. knowledge-based vqa. some knowledgebased visual language tasks requires external knowledge beyond the image to answer a question. early exploration, such as fvqa @xcite , creates a fact-based vqa dataset by selecting a fact (e.g., <cat, capableof, climb-@xmath0. a recent outside knowledge vqa (ok-vqa) dataset is a more challenging dataset, covering a wide range of knowledge categories. in our work, we focus on ok-vqa due to its large-scale knowledge-based questions as well as its open-ended nature. recent approaches have shown a great potential to incorporate external knowledge for knowledgebased vqa. several methods explore aggregating the external knowledge either in the form of structured knowledge graphs @xcite @xcite , unstructured knowledge bases @xcite @xcite , and neural-symbolic inference based knowledge @xcite . in these methods, object detectors @xcite and scene classifiers @xcite are used to associate images with external knowledge. further, external apis, such as google @xcite , microsoft @xcite and ocr @xcite are used to enrich the associated knowledge. finally, pre-trained transformerbased language models @xcite , or multimodal models @xcite @xcite @xcite are leveraged as implicit knowledge bases for answer predictions. different from previous approaches, our work aims to develop a single, unified architecture, by jointly reasoning over explicit and implicit knowledge to augment generative language models. while part of our approach is similar to pica @xcite which considers gpt-3 as implicit knowledge base, our model takes one step further by showing that how explicit and implicit knowledge can be integrated during knowledge reasoning. another similar work vis-dpr @xcite collects a knowledge corpus from training set by google search which is specific to a certain dataset. our proposed model is more generic by collecting entities from wikidata and not limited to the training set. ## ablation study to unpack the performance gain and understand the impact of different components, we ablate and compare different model architectures, types of knowledge and the number of explicit knowledge. specifically, as shown in ## conclusion this paper takes a step towards understanding the complementary role of implicit knowledge gained from continuing to scale models and explicit knowledge from structured knowledge bases. importantly, it appears that there is headroom in both directions (i.g. improving retrieval and reasoning). our conceptually simple yet effective approach for knowledge-based vqa makes these relationships explicit while still achieving a significant improvement against state-of-the-art results. additional challenges remain, for example how best to align image regions with meaningful external semantics deserves and how to efficiently and accurately integrate multiple knowledge bases. and implicit knowledge is designed to enhance semantic alignment and generate representations with stronger knowledge-awareness.