XDBERT : D istilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding (2022)

Abstract: Transformer-based models are widely used in natural language understanding (NLU) tasks, and multimodal transformers have been effective in visual-language tasks. This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders’ success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU. After training with a small number of extra adapting steps and finetuned, the proposed XDBERT (cross-modal distilled BERT) outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG) benchmarks, and readability benchmarks. We analyze the performance of XDBERT on GLUE to show that the improvement is likely visually grounded.

## introduction transformer-based models are extensively used in natural language understanding (nlu) tasks, and some prominent pretraining strategies include bert @xcite , roberta @xcite , albert @xcite , and elec-tra @xcite . despite their differences in curating the learning objectives, they all utilize text-based datasets only. in the real world, however, humans can benefit from the visual modality when acquiring knowledge from language; an obvious example is learning visually grounded words, such as colors and shapes. some studies have succeeded with visually grounded information used in nlu. vico @xcite learned visual co-occurrences in text and reported superior performance to glove in word analogy problems. @xcite and @xcite used images to boost translation performance in supervised and unsupervised bird : "a winged @xcite soars in the sky." ## proposed method the training process consists of three phases: pretraining, adaptation, and finetuning (figure 2 ). our proposed method focuses on the adaptation phase with pretrained models, so pretraining is not a part of our experiment, but we explain all three phases for completeness. the adaptation phase incorporates the cross-modal transformer structure to jointly learn from clip-t and bert outputs. ## experimental results we evaluated our model on three nlu benchmarks, namely glue, swag and read. we tested our adaptation strategy on three different language encoders coupled with clip-t, including bert-base, electra-base, and electra-large. we fix the finetuning parameters between models where comparison is intended, and select the median result of multiple runs. details of finetuning are provided in appendix c. we show that the results of finetuning clip-t alone on glue does not perform well. since the language capability of the clip-t model is weak, the distilled information obtained by xd-bert/xdelectra is predominantly visual. it is also possible to finetune the entire crossmodal transformer after adaptation. the performance further increases but the model has more parameters. the results are in appendix c.3. ## analysis to justify the use of a cross-modal encoder, we first conducted a pairwise projection weighted canonical correlation analysis (pwcca) on word embeddings. the pwcca is a good measure to determine how close the distributions of two vector groups are to each other. the pwcca results in we inspect rte, mrpc, and cola results of 5 runs in detail to show that the improvements are likely from visual information of clip-t. over the 5 runs, @xcite the green plus symbol denotes the mean value. the visually grounded ratio estimation follows @xcite . and cola show +0.3% and +0.9% gains in accuracy respectively, and translates to a larger gain in performance with their original metric (mrpc f1: +0.83%, cola corr: +2.2%). we then separate each of the glue datasets entries into two categories: entries that xdbert-b improves classification over bert-b, and entries of the opposite. entries where both models obtain the same performance are set aside. analyzing the separated entries as a whole, we discovered that the betterperforming entries have a larger visually grounded ratio (figure 4 ), as the quartile, median and mean values are generally higher for improved samples. the enhancement of visually grounded token representations is a rough indicator that xdbert has obtained distilled visual information from clip-t. we show examples of each category in appendix d. ## ablation study we tried various combinations of adaptation tasks and found out that using all three yielded the best results. we also tried to reduce the number of cross-modal encoder layers to one; however, no further improvements were made upon the visually grounded language encoder. other experiments include changing the number of layers in the crossmodal encoder, training for longer, and swapping to a much larger wiki (14g). swapping to wiki reduces potential overfitting from the 20 epochs setting trained on wiki103, as training for the same amount of steps on wiki is less than 1 epoch. we tested these changes on rte, mprc, stsb, and cola on 5 random seeds, and the results are shown in besides experimental evidence, we also justify the cliptc loss via further analysis, as the cliptc objective can theoretically be trivially solved by identity mapping. despite this possibility, we find that the loss is crucial to cross attention learning. since we do not impose negative hard samples from sampled sentences, the match objective can be solved sufficiently simply by guiding the cross attention to focus on common trivial words. with the cliptc objective, the diversity of the input embeddings corresponding to different tokens must be retained in the cross-modal encoder, leading to more robust cross-modal attention. we show comparisons of the attention maps generated from the cross-modal encoders with a random se-quence from rte in ## conclusion in this study, we explored using cross-modal encoders to distill visual information to bert. we adapted the model with multiple objectives, and we were able to achieve improved performance on nlu tasks. our adaptation techniques are computationally inexpensive and straightforward. furthermore, our method is language encoder agnostic, as we show similar performance gains on xdelec-tra.