Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it? (2021)

Abstract: Large language models are known to suffer from the hallucination problem in that they are prone to output statements that are false or inconsistent, indicating a lack of knowledge. A proposed solution to this is to provide the model with additional data modalities that complements the knowledge obtained through text. We investigate the use of visual data to complement the knowledge of large language models by proposing a method for evaluating visual knowledge transfer to text for uni- or multimodal language models. The method is based on two steps, 1) a novel task querying for knowledge of memory colors, i.e. typical colors of well-known objects, and 2) filtering of model training data to clearly separate knowledge contributions. Additionally, we introduce a model architecture that involves a visual imagination step and evaluate it with our proposed method. We find that our method can successfully be used to measure visual knowledge transfer capabilities in models and that our novel model architecture shows promising results for leveraging multimodal knowledge in a unimodal setting.

## introduction large language models have proved performant across a diverse set of tasks in nlp, and most recently even as unsupervised multitask learners @xcite ). an important contributing factor to this is the capability of the models to hold large amounts of linguistic as well as factual knowledge in their parameters. while impressive, without strong task-specific fine-tuning these models are prone to outputting false or inconsistent statements, often also referred to as hallucination @xcite . this has been particularly studied for generative tasks such as abstractive text summarization @xcite and dialog systems @xcite , but the problem is also apparent for models applied to cloze-style fill-in-the-blank tasks @xcite . having truthful nlp systems is a core requirement for most applications, which is why this is an important problem to address. grounding has been proposed as a potential way to mitigate this problem, e.g by providing broader world information from for example multimodal perception @xcite . information from multimodal perception may actually provide a significant amount of additional world information to an nlp model, since text data suffers from the problem of reporting bias. that is, humans generally communicate novel information rather than trivial, leading to a discrepancy between reality and what gets described in text @xcite . consequently, perceptual information may contain complementing world knowledge that cannot be found in text data, and has the potential to mitigate the aforementioned problem of hallucinating nlp models. previous works have evaluated how grounded language representations impact performance on common nlp benchmarks @xcite @xcite , but little has been done on investigating grounding specifically as an additional source of knowledge. in this work, we take a focused look at how data from a visual modality can augment the knowledge a language model expresses. we design an experimental setup to enable the development of strategies for maximizing visual-to-textual knowledge transfer. in the setup, we create a small knowledgecentric cloze-style task in english named memory colors that is tailored to test for visual knowledge by querying for the typical colors of well-known items. we also build a large vision-and-language dataset in the english language, where we carefully control for the modality from which the necessary visual knowledge can be learnt. finally, we use this data to train self-supervised multimodal models, and compare strategies to query for the visual knowledge. based on intuitions of how humans are able to store and retrieve such knowledge, we also propose a querying strategy that involves "imagining" a visual representation from which the answer then can be decoded. to summarize, our contribution is twofold: ## an experimental setup evaluating visual knowledge transfer humans have the ability to learn knowledge from non-linguistic modalities (such as visual perception) and express this in language, making them able to e.g. textually reason about what an elephant looks like because they have previously seen said animal in an image. many models that integrate the textual and visual modalities exist, but the majority of them have been created with the purpose of reasoning about properties of individual images provided to the system: for instance, to ask about an elephant, you need to simultaneously provide the model with an image of an elephant. we hypothesize that the capability to incorporate knowledge from different modalities and expressing it textually could improve on the common sense as well as in-domain knowledge that language models possess. to this end, we wish to create an experimental setup in which we can measure how well a model can acquire visual knowledge and then express it in text. a simple way to evaluate a model for its capability to transfer visual knowledge into text is to query it about typical colors of certain objects -memory colors -while making sure that the model cannot acquire this knowledge through a text signal, i.e. it has not previously been told what the colors should be. consequently, we create a zero-shot cloze-style task of predicting memory colors of common objects, described in section 2.1. we also collect a large vision-and-language dataset for model training in which we carefully control for whether the knowledge necessary for solving the memory color task is available strictly in the visual, textual, or both modalities, described in section 2.2. ## models when a human is asked a question like what is the color of your house? it typically requires retrieving a mental picture from memory of what the house looks like. based on this mental picture, the answer can then easily be inferred. the mental picture provides an efficient means to store knowledge about the appearance of the house, as other questions like how many floors does it have? or does it have a garden? can as easily be inferred. we hypothesize that this idea of visual imagination could also provide an efficient means of visual knowledge transfer, and propose a model for performing the "imagination" explicitly. we take inspiration from recent works in visionand-language modeling, where the transformer architecture @xcite has become the de facto standard @xcite @xcite . in a typical setup, an image representation is fed to the transformer encoder jointly with the text tokens, and the encoder is then pre-trained using various denoising and contrastive objectives. in this work, we perform experiments on a simple yet novel variant to accomodate for the visual imagination. while common practice is to use visual features from an object detector (ren et al., 2015), we extract visual representations using the image encoder of a pretrained clip model (radford et al., 2021) instead. clip consists of two networks for encoding an image and a text sentence respectively, and is trained to align these representations in a joint space using a contrastive training objective. the resulting visual encoder is shown to clip text encoder e[ cls] et he ecol or eof ebl ood ei s [cls] the color of blood is v h[ cls] ht he hcol or hof hbl ood hi s the color of blood is [mask] tokenize e[ mask] [mask] h[ mask] hv red e[ cls] et he ecol or eof ebl ood ei s [cls] the color of blood is the color of blood is [mask] tokenize e[ mask] [mask] t r a n s f o r m e r h[ cls] ht he hcol or hof hbl ood hi s h[ mask] have great discriminatory performance, for example when applied to zero-shot image classification. however, the main benefit of using clip to extract visual features is the joint feature space between its visual and textual encoder, enabling us to generate "visual" features from text. in our experiments, we start from the popular pretrained bert base model 4 , and continue training on our visual-language dataset from section 2.2, using only the masked language modelling (mlm) objective with 15% random dynamic masking ratio. specifically, we train two models on the filtered and unfiltered versions respectively: bert-base we continue training of bert base using mlm only on the captions part of the visualand-language dataset. this provides a baseline of the amount of color knowledge that can be picked up from text alone. clip-bert we continue training of bert base using mlm but on both the captions and the images of the visual-and-language dataset. the image representation is transformed through a projection layer and appended to the transformer input without adding any positional or segment embeddings. the mlm objective is only applied on the textual positions. an illustration of the training of clip-bert is shown in figure 2 . ## results and analysis we evaluate the transfer capabilities of our aforementioned models both to assess the functionality of our measurement method and to investigate the effect of implicit and explicit visual knowledge transfer. the results on memory colors for the different experiments are displayed in are humans the top performers on the memory colors dataset? we can conclude that the human baseline results are better than those of any model, training procedure and querying strategy evaluated. this baseline is expected to be high because the task is inherently based on the notions of color according to the majority of the humans that were evaluated. furthermore, as language models lack much knowledge compared to humans, we expect them to perform worse than humans on this task. not even the clip-bert model provided with gold standard images and unfiltered textual information on colors matches the performance of the human annotators. there may be several reasons for this, for instance that the capacity of the multimodal models is not sufficient, or that humans are privy to additional information that helps them solve the memory colors task better. is the filtering of the training data necessary for our experimental setup of evaluating visual knowledge transfer? we see that the bertbase model without any further training has a bad performance on the memory colors dataset, only slightly better than the majority baseline. on the other hand, the model shows significant performance improvement if it is trained on our unfiltered visual-and-language data. this suggests that the unfiltered training dataset contains much information about the objects' color textually. this is perhaps not surprising, as it is common that captions describe what colors the objects in the image have. however, for our purposes it is problematic as we wish to constrain this information to be learnt from the visual modality solely. based on this, we conclude that the filtering of the training data is necessary in our experimental setup for evaluating visual knowledge transfer. does the filtering of the training data work as intended according to the memory colors dataset? if we filter the training data of the bert-base model, the performance drops from 0.724 to 0.460, indicating that a large portion of the necessary information has been removed. however, the model performance does not drop to that of the original bert-base model, so seemingly some color information still reaches the model through the text despite the data filtering. this leakage is undesirable from the perspective of evaluating visual knowledge transfer, since the model should not be able to perform well on memory colors without visual knowledge transfer capabilities. ## related work there are multiple perspectives on how our contributions relate to previous work, and we elaborate on this in the subsequent sections. ## conclusions and future work we have introduced a methodology for measuring visual knowledge transfer in multimodal language models. the centerpiece is a new benchmark memory colors designed to test how well such models incorporate knowledge about colors of common objects. we find that careful filtering of the underlying training data can provide an effective means to attribute the acquired knowledge to the individual source modalities. our results based on this methodology also showcase that vision-and-language pre-trained language models are able to textually express knowledge obtained from a separate (e.g. visual) modality. we also found that there is some information leakage in our filtering method, as the performance of filtered bert-base improves over the bertbase baseline. this improvement in model performance cannot be explained based on the method and results of this work. thus, it remains to be investigated what kind of information leakage takes place in spite of the filtering. potential explanations are that the model learns the color of an item through second-order effects, e.g. by learning the color of a synonymous item that we have not filtered for, or that the original bert-base model already contains textual knowledge relevant to memory colors, while it needs further training on a visual-language dataset to access that knowledge. future work should ensure that the experimental setup for evaluating visual knowledge works as intended. additionally, it is worth investigating what other evaluation alternatives we have for measuring the cross-modal capabilities of nlp models. can we create an evaluation methodology that is not taskbased, can we find some other task to evaluate on, or can we improve on the statistical robustness of our evaluation methodology? we observed that a model with implicit transfer performs better on our evaluation task than a unimodal language model, while a model with explicit transfer through prediction performs even better on the task. this implies that both implicit and explicit knowledge transfer are promising directions for efficient visual knowledge transfer to text, although the explicit transfer may be more promising. while we here only investigate knowledge transfer from a visual modality, it is likely that this model design also can be successfully implemented for other modalities. the experimental setup proposed in this work helped us discover and validate the potential of explicit transfer. we can conclude that more work on understanding how multimodal training of language models affect their predictions is an interesting direction towards more robust and trustworthy nlp systems.