Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022)

Abstract: Pre-trained language models are still far from human performance in tasks that need understanding of properties (e.g. appearance, measurable quantity) and affordances of everyday objects in the real world since the text lacks such information due to reporting bias. In this work, we study whether integrating visual knowledge into a language model can fill the gap. We investigate two types of knowledge transfer: (1) text knowledge transfer using image captions that may contain enriched visual knowledge and (2)

## introduction pre-trained language models (ptlms) such as bert @xcite , roberta @xcite , and t5 @xcite have shown impressive results in various conventional natural language understanding (nlu) tasks by capturing syntactic and semantic knowledge from the pretraining tasks of masked language modeling and masked span infilling tasks on massive text corpora. though yielding good performance on various nlu downstream tasks, these pre-training objectives suffer from a lack of out-of-domain knowledge that is not explicitly present in the pre-training corpus @xcite @xcite . specifically, one type of knowledge that models often struggle with is the visual knowledge of common objects such as attributes (e.g. appearance, measurable quantity) and affordances. this is because this kind of knowledge is rarely explicitly described in the training text due to reporting bias. for example, as shown in figure 1 , people tend to report what interests them rather than general facts such as a shape or color of oranges they already know. towards better knowledge-enhanced ptlms, recent works incorporate external knowledge bases (e.g., knowledge graph, dictionary) to inject entity knowledge into ptlms @xcite @xcite or retrieve knowledge from external knowledge bases to solve the problem @xcite . however, these approaches still suffer from a lack of visual knowledge that is important to understand the real world. in this paper, we conduct systematic experiments to understand whether such visual knowledge can be transferred into lms, and if so, how to perform effective knowledge transfer. specifically, we look into a series of analysis question as follows: for the text knowledge transfer, we utilize text corpus from visual domain, e.g., image captions. we leverage two training objectives for the language model: (1) masked language modeling follows the domain adaptive pre-training scheme @xcite , assuming the corpus contains enriched visual knowledge or physical commonsense knowledge; (2) text contrastive learning augments the sentence representation with dropout to create positive samples while considering all others in the batch as negative samples for the contrastive learning @xcite , assuming training better sentence representations leads to better understanding of the corpus. for the cross-modal knowledge transfer, we explore multiple methods to transfer visual-related knowledge to lms: (1) masked language modeling with visual clues incorporates visual clues to capture dependencies between visual and linguistic contents @xcite ; (2) voken classification contextually aligns language tokens to their related images (called "vokens") to transfer visual knowledge into lms @xcite ; (3) cross-modal contrastive learning aims to improve text representations by maximizing the agreement between correct image-text pairs versus random (inbatch) and adversarial negative pairs by contrastive learning between image and text modalities; and (4) cross-modal knowledge distillation transfers the knowledge from the teacher model, which is trained by cross-modal contrastive learning on image and text modalities, to the student language model using knowledge distillation. we perform comprehensive comparisons on five downstream tasks that may require visual or physical commonsense knowledge, including piqa @xcite , visual paraphrasing (vp) @xcite , csqa @xcite , obqa @xcite , and rid-dlesense @xcite . results suggest that: (1) simple intermediate pre-training on captions can help improving performance on commonsense reasoning that needs physical or visual knowledge. ## analysis setup in this work, we study how to transfer the visual knowledge into language models. for this study, we introduce our analysis setup: problem formulation, analysis questions, and knowledge corpora. ## method in this section, we introduce the following two approaches to integrate visual knowledge into lms: ## experimental settings for all the approaches, we use bert-base-uncased @xcite as text encoder f l and resnext101 @xcite as an image encoder f v . we continue to pre-train the encoders in our experiments. for text knowledge transfer, (1) mlm follows the exact setting of codebase in huggingface 2 which uses dynamic masking strategy to conduct language modeling task. (2) tcl conducts contrastive learning with f l . we choose the best checkpoint by the best spearman correlation on @xcite . for cross-modal knowledge transfer, (1) cmkd explores vl-bert, vokenization, and vidlankd approaches. here, we use vl-bertlarge model to do cmkd. we use the vl-bert and vokenization checkpoints from their official 2 https://github.com/huggingface/transformers/ tree/master/examples/pytorch/language-modeling codebases foot_0 . vidlankd trains a teacher model by two versions of contrastive learning (equations ( 6 ) and ( 8 )) on ms coco dataset. we set α = 1 in vidlankd (equation ( 8 )). (2) cmcl conducts contrastive learning with f l and f v . here, we set τ = 0.05 (equations ( 2 ) and ( 4 )). ( 3 ) cmcl with ans chooses three noun words or verb words to do masked prediction and use top-5 predictions from f l as replacement. we filter out synonyms and hypernyms of original words using wordnet @xcite . (4) cmcl with psa includes the perturbed sentences with synonyms and hypernyms as additional positive samples. in cmcl, we adopt resnext101 @xcite as an image encoder f v and bert as a text encoder f l . tcl and cmcl train with batch size 64, maximum sequence length 20, learning rate 1e-4 for 3 epochs. for fine-tuning on downstream tasks, we do grid search on learning rates {5e-5, 1e-4, 3e-4, 4e-4, 5e-4, 6e-4} and choose the best learning rate. we set maximum epochs to 30 in low-resource and 15 in fully supervised settings. ## results and analysis we analyze the main results of intermediate pretraining. tables 2 and 3 show the main results of low-resource learning and fully supervised learning with the ms coco captioning dataset, respectively. we train the models with a few training examples, 64 and 128, to understand the better initialization. we argue that if a model obtains better performance in the low-resource setup, then it is a faster learner and has better generalization on downstream tasks. can text intermediate pre-training help improve text encoders? text intermediate pre-training using mlm and tcl on a caption corpus improves the performance on downstream tasks in both lowresource and fully supervised settings. in particular, tcl shows significant improvement on obqa and riddlesense over bert (p-@xmath0.01). these results suggest that text intermediate pre-training on visual-related datasets helps performance on commonsense reasoning tasks. can cross-modal intermediate pre-training help transfer visual knowledge to augment text encoders? we observe that cross-modal intermediate pre-training is helpful in both fully supervised and low-resource settings (see what intermediate pre-training objectives are effective for cross-modal knowledge transfer? among various cross-modal knowledge transfer methods, we study which method is the most effective for cross-modal knowledge transfer. overall, cmcl with psa and ans shows the best performance among all cross-modal methods. interestingly, vl-bert also shows better performance than bert-base on all datasets in the low-resource setting. this suggests that exploiting images in masked language modeling task help transfer the knowledge to language models. what types of knowledge sources are most helpful? here, we investigate whether using an image source in addition to a text source can further improve the model. to answer this question, we analyze methods from different types of sources: text-only and text-image pair sources. we focus on the methods that use the contrastive learning objective: tcl and cmcl. note that these two methods share the same objective but cmcl trains on cross modalities which are images and captions while tcl only trains on captions. overall, tcl performs slightly better than cmcl in low-resource and fully supervised settings. interestingly, additional negative samples (ans) and positive samples in tcl decreases the performance while they help cmcl to improve the performance. we conjecture that perturbed sentences in ans might not be semantically negative to the original sentence so models learn from wrong labels. ## conclusion we study whether intermediate pre-training on visual knowledge can help transfer visual knowledge into lms. we investigate text knowledge transfer and cross-modal knowledge transfer using images and captions. in our empirical analysis, we observe that intermediate pre-training on captions can help improving performance and cross-modal knowledge transfer approaches consistently improve performance. when the transfer methods are equipped with additional positive and negative samples, they show better performance. future works include improving both commonsense reasoning and general language understanding.