MSD : Saliency-aware Knowledge Distillation for Multimodal Understanding (2021)

Abstract: To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large “teacher” model to a smaller “student” model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework, modality-specific distillation (MSD), to transfer knowledge from a teacher on multimodal tasks by learning the teacher’s behavior within each modality. The idea aims at mimicking a teacher’s modality-specific predictions by introducing auxiliary loss terms for each modality. Furthermore, because each modality has different saliency for predictions, we define saliency scores for each modality and investigate saliency-based weighting schemes for the auxiliary losses. We further study a weight learning approach to learn the optimal weights on these loss terms. In our empirical analysis, we examine the saliency of each modality in KD, demonstrate the effectiveness of the weighting scheme in MSD, and show that it achieves better performance than KD on four multimodal datasets.

## introduction recent advances in computer vision and natural language processing are attributed to deep neural networks with a large number of layers. current state-of-the-art architectures are getting wider and deeper with billions of parameters, e.g., bert @xcite and @xcite . such wide and deep models suffer from high computational costs and latencies at inference. to mitigiven only image modality as input (image), and given only text modality as input (text). kd denotes a student model with knowledge distillation and the small model is a student model without distillation. we observe that there is still a prediction gap between the teacher and the student trained by kd. in this paper, we study saliency explanations for each modality and propose modality-specific distillation (msd) to minimize the gap. there has been a lot of interest in multimodal distillation setup such as cross-modal distillation @xcite . multi-modal problems involve relating information from multiple sources. for example, visual question answering (vqa) requires answering questions about an image @xcite @xcite and models should incorporate information from the text and image sources to answer the questions. multimodal problems are important because many real-world problems require understanding signals from different modalities to make accurate predictions; information on the web and social media is often represented as textual and visual descriptions. digesting such multimodal information in an effective manner is challenging due to their different types of information on each modality. in this paper, we offer a large-scale, systematic study on the effects of each modality through saliency explanations in kd. while kd approaches can be applied to multimodal applications, the student and teacher models may significantly differ in their outputs using each modality as input. we illustrate the point in fig. 1 . to minimize the gaps, we introduce a multimodal kd framework, modalityspecific distillation (msd), that aims to mimic the teacher's modality-specific predictions. we show that the samples' modalities have a different amount of information. based on this observation, we improve the knowledge transfer by splitting the multimodality into separate modalities, using them as additional inputs, and thus distilling the modality-specific behavior of the teacher. msd introduces auxiliary losses per modality to encourage each modality to be distilled effectively. to maximize the effect of modality-specific distillation, we investigate multiple weighting schemes to balance out the auxiliary losses. one of the weighting schemes is based on modality saliency scores that are proxy scores to modality importance. furthermore, we leverage a meta-learning method to introduce weight-learning to automatically learn optimal weights per sample per modality. ## preliminaries in this section, we define notations and revisit conventional knowledge distillation (kd). ## analysis setup in this section, we introduce a multimodal kd approach, modality-specific distillation, to understand the importance of each modality ( §3.1), experimental setup ( §3.2), and datasets for the experiments ( §3.3). ## modality weighting methods for the analysis, we introduce three categories of weighting schemes for msd, presented in the order of complexity: a) population-based ( §4.1), b) saliency-based ( §4.2) weighting approaches for the losses, and c) weight-learning approach ( §4.3) to find the optimal weights. ## empirical analysis in this section, we revisit and discuss the questions we raised in §3.2. q1. how salient is each modality for predictions? to answer the question, we visualize saliency scores in the hateful-memes, mm-imdb, and snli-ve datasets in figs. 2 and 3 . we use kl divergence in eq. ( 5 ). we observe that the mm-imdb dataset shows higher saliency scores of text modality than those of image modality, which implies that text information has important information in general. on the other hand, hateful-memes dataset does not show such a global pattern but one here, we use msd on top of conventional kd @xcite . also, our weight learning for weights shows the best performance. q2. can the saliency scores aid students? q3. can we learn a sample weighting strategy to better aid students? we observe that among weighting strategies, msd with weight learning shows the best performance in hateful-memes and mm-imdb, indicating it finds better weights for each dataset in in addition, we measure kullback-leibler (kl) divergence between the teacher's outputs and other models' outputs on the mm-imdb test set in fig 5 . this is to measure the difference between teacher's multi image text 0.08 0.10 0.12 0.14 kl divergence 0.114 0.122 0.142 0.113 0.117 0.134 0.098 0.119 0.093 0.104 0.113 0.109 0.093 0.112 0.097 small kd msd (population) msd (saliency) msd (learning) figure 5: kullback-leibler divergence on the mm-imdb test set between the teacher's outputs and other models' outputs. this is a measure of how the teacher's probability distribution is different from other models'. the lower divergence is, the closer a model is to the teacher. q4. is the student with the weighting strategies consistent with the teacher? to showcase that our approach helps the student model to be more sensitive to important changes in modalities, we take examples from the hateful-memes test set and randomly replace one of the modalities with a modality from another sample. hateful-memes is a multimodal dataset and changing the modalities might or might not change the final label. in this case, we do not have the ground truth, but we use the teacher's predicted label on the newly generated sample as a proxy for ground truth and count the times that the student/small model is consistent with the teacher on these generated samples. we define the ratio of such consistent predictions over the total generated samples as "teacher-student consistency ratio". note that none of the models have seen these samples during the training. as it can be seen from fig. 6 , the msd approach with weight learning has a larger "teacher-student consistency ratio" than the small model with and without kd. this indicates that msd not only improves the accuracy but also improves the sensitivity of the student model to better match the teacher on the changes in modalities on unseen data. please refer to case study in §c of appendix. q5. can this be applicable to other distillation methods? we present improvements over kd approaches with/without msd in ## related work knowledge distillation. there have been several studies of transferring knowledge from one model to another @xcite @xcite @xcite @xcite . @xcite improve the accuracy of a shallow neural network by training it to mimic a deep neural network by penalizing the difference of logits between the two networks. @xcite introduced knowledge distillation (kd) that trains a student model with the objective of matching the softmax distribution of a teacher model at the output layer. park et al. @xcite focused on mutual relations of data examples instead and they proposed relational knowledge distillation. tian et al. @xcite proposed to distill from the penultimate layer using a contrastive loss for cross-modal transfer. a few recent papers @xcite have shown that distilling a teacher model into a student model of identical architecture, i.e., self-distillation, can improve the student over the teacher. ## conclusion we studied knowledge distillation on multimodal datasets; we observed that conventional kd may lead to inefficient distillation since a student model does not fully mimic a teacher's modality-specific predictions. to better understand knowledge from a teacher on the multimodal datasets, we introduced saliency scores for a modality and modality-specific distillation; the student mimics the teacher's outputs on each modality based on saliency scores. furthermore, we investigated weighting approaches, population-based and saliency-based weighting schemes, and a weightlearning approach for weighting the auxiliary losses to take the importance of each modality into consideration. we empirically showed that we can improve the student's performance with modalityspecific distillation compared to conventional distillation. more importantly, we observe choosing the right weighting approach boosted the student's performance. we believe that future work can expand on our methods, and search the space of weighting approaches beyond multimodal setups.