Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge (2021)

Abstract: The limits of applicability of vision-and language models are defined by the coverage of their training data. Tasks like vision question answering (VQA) often require commonsense and factual information beyond what can be learned from task-specific datasets. This paper investigates the injection of knowledge from general-purpose knowledge bases (KBs) into vision-and-language transformers. We use an auxiliary training objective that encourages the learned representations to align with graph embeddings of matching entities in a KB. We empirically study the relevance of various KBs to multiple tasks and benchmarks. The technique brings clear benefits to knowledge-demanding question answering tasks (OK-VQA, FVQA) by capturing semantic and relational knowledge absent from existing models. More surprisingly, the technique also benefits visual reasoning tasks (NLVR2, SNLI-VE). We perform probing experiments and show that the injection of additional knowledge regularizes the space of embeddings, which improves the representation of lexical and semantic similarities. The technique is model-agnostic and can expand the applicability of any vision-and-language transformer with minimal computational overhead.

## introduction the last few years have seen a surge of interest for vision and language (v&l) tasks. these include visual question answering for example (vqa), in which the machine must answer a question about an image (fig. 1 ). v&l tasks require processing two modalities and reasoning over textual, visual, and abstract concepts. the current state of the art in v&l are models based on transformers such as bert @xcite that have been extended to handle visual inputs (e.g. @xcite ). these we first preprocess the kb into knowledge embeddings. when training the transformer, we use an auxiliary objective that aligns its learned representations with knowledge embeddings. in the context of visual question answering (pictured), the model is able to answer questions that require knowledge that is typically not captured in task-specific datasets. models are usually pretrained on a collection of datasets of paired textual and visual data. suitable datasets include image captioning data @xcite and vqa data @xcite . despite their large scale, these datasets only cover limited domains. many are based on images from coco @xcite and visual genome @xcite and the linguistic diversity of textual annotation is limited. the v&l tasks that we are ultimately interested in require knowledge beyond current datasets, e.g. about specific events, named entities, common sense, and abstract concepts (see fig. 3 ). this paper focuses on the expansion of the applicability of v&l models with additional knowledge. during training, we infuse the model with knowledge from an external source, distinct from datasets of paired v&l data.the challenge is that standard v&l data is not annotated or paired with such additional knowledge. even though techniques have been proposed to exploit additional data in nlp, including text-based question answering @xcite @xcite , little work has been done on the extension to v&l. works in nlp with benchmarks of knowledge-demanding questions @xcite @xcite @xcite have shown that knowledge bases (kbs) contain information that can benefit large-scale pretrained models. motivated by this line of evidence, we aim to evaluate similar mechanisms for v&l tasks. in the context of vqa, datasets of knowledgedemanding questions @xcite @xcite have proved challenging for existing methods. for example, a question like is the man eating healthy food ? requires recognizing a type of food and relating it to its nutritional quality. learning this type of knowledge from vqa training examples would be clearly inefficient. other examples of challenging questions involve references to named entities such as brands, locations, or movie titles. for example, the question levi's is a popular brand of what item shown here ? requires knowledge of the brand's specialization before identifying the correct element in the image. we believe that embedding this type of knowledge in a model is a necessary step to enable progress on complex multimodal question answering. this paper describes a technique to inject information from kbs into a transformer-based model during its training (see fig. 1 ). we take inspiration from @xcite and adapt their regularizer from text-based models to v&l transformers. we provide an implementation of our method on top of the popular lxmert model @xcite . we use it to investigate the suitability of several kbs to different tasks and benchmarks. our contributions are summarized as follows. additional knowledge in nlp can help with tasks requiring commonsense or factual information @xcite . various techniques have been proposed to improve transformers such as bert @xcite . zhang et al. (2019) proposed ernie, which feeds graph embeddings of text entities to the model. peters et al. (2019) proposed knowbert, a similar technique suitable to multiple kbs. levine et al. (2020) used wordnet to aid in the masked-word prediction objective, and improve lexical understanding in downstream tasks. ye et al. (2019) proposed a multiple-choice question answering pretraining task and improved performance on multiple datasets requiring commonsense reasoning. liu et al. (2020) addressed noise issues by controlling the amount of domain-specific knowledge infused into the model. goodwin and demner-fushman (2019) proposed oscar, a regularization method to inject ontological knowledge in a pretrained language model. wang et al. (2019) proposed to simultaneously learn knowledge rep- knowledge-based vqa refers to benchmarks designed to require additional information @xcite @xcite . models have been proposed that retrieve such information from kbs based on question and image contents @xcite @xcite . some works use other sources of external information at training ( teney and van den hengel, 2016) or test time (teney and van den hengel, 2018, 2019) but they showed limited improvements on vqa benchmarks. recently proposed concept-bert (gardères et al., 2020) model jointly learns visual, textual and knowledge embeddings to fuse commonsense information into vqa models. this paper describes a method applicable to a variety of sources of information and to tasks beyond vqa. we also show that different tasks benefit from different types of information. ## methodology we describe a general, simple yet effective technique to embed additional knowledge into a transformer-based model. it is compatible with existing multimodal transformers and thus suitable to a variety of tasks. the method proceeds in three stages (see fig. 2 ). 1. we preprocess the additional knowledge into a set of vector representations that we call knowledge embeddings. for example, with a relational knowledge base, we apply a graph embedding method to obtain a vector representation of every of its entities. each is associated with a textual expression. 2. we match sentences in the v&l training data with knowledge embeddings. matches in the training data are referred to as knowledge-rich expressions. 3. during the training of the transformer (pretraining and/or fine-tuning), we optimize an additional objective that aligns its learned representations of knowledge-rich expressions with the matching knowledge embeddings. we now describe each stage in details. ## experiments we performed a suite of experiments with multiple tasks and benchmarks. our objective is to evaluate the suitability of two popular knowledge bases (conceptnet and wikidata) to these tasks. ## conclusions in this paper we described a general-purpose technique to inject external information from knowledge bases into multimodal transformers for vision-and-language tasks. the current prevailing paradigm is to pretrain large models on collections of datasets. our experiments demonstrate that some types of commonsense and factual knowledge are not captured within these models. knowledge bases like conceptnet and wikidata can fill in these deficiencies. we showed clear improvements in performance on a variety of tasks and benchmarks requiring visual and multimodal reasoning, demonstrating the versatility of the procedure. the value of these results for future research are twofold. on the one hand, they indicate that the combination of heterogeneous sources of information is a promising way to expand the applicability of current machine learning models. on the other hand, by improving the availability of supporting knowledge, the approach opens the door to future advances on reasoning procedures that process this information. advances on this front would lead to improved capabilities on tasks that require highlevel or multi-hop reasoning, for example.