Modality-specific Distillation (2021)

Abstract: Large neural networks are impractical to deploy on mobile devices due to their heavy computational cost and slow inference. Knowledge distillation (KD) is a technique to reduce the model size while retaining performance by transferring knowledge from a large “teacher” model to a smaller “student” model. However, KD on multimodal datasets such as vision-language datasets is relatively unexplored and digesting such multimodal information is challenging since different modalities present different types of information. In this paper, we propose modality-specific distillation (MSD) to effectively transfer knowledge from a teacher on multimodal datasets. Existing KD approaches can be applied to multimodal setup, but a student doesn’t have access to modality-specific predictions. Our idea aims at mimicking a teacher’s modality-specific predictions by introducing an auxiliary loss term for each modality. Because each modality has different importance for predictions, we also propose weighting approaches for the auxiliary losses; a meta-learning approach to learn the optimal weights on these loss terms. In our experiments, we demonstrate the effectiveness of our MSD and the weighting scheme and show that it achieves better performance than KD.

## introduction recent advances in computer vision and natural language processing are attributed to deep neural networks with large number of layers. current state-of-the-art architectures are getting wider and deeper with billions of parameters, e.g., bert @xcite and @xcite . in addition to their huge sizes, such wide and deep models suffer from high computational costs and latencies at inference. these shortcomings greatly given only image modality as input (image), and given only text modality as input (text). kd denotes conventional knowledge distillation and the small model is a model with distillation. we observe that there is still a prediction gap between the teacher and the student trained by kd. to minimize the gap, we propose modality-specific distillation (msd). to mitigate the heavy computational cost and the memory requirement, there have been several attempts to compress a larger model (a teacher) into a smaller model (a student) @xcite @xcite @xcite . among them, knowledge distillation (kd) @xcite assumes the knowledge in the teacher as a learned mapping from inputs to outputs, and transfers the knowledge by training the student model with the teacher's outputs (of the last or a hidden layer) as targets. recently, kd has been explored in various studies such as improving a student model @xcite @xcite @xcite and improving a teacher model itself by self-distillation @xcite @xcite . there has been a surge of interest in distillation in a multimodal setup such as cross-modal distillation @xcite . multimodal problems involve relating information from multiple sources. for example, visual question answering (vqa) requires answering questions about an image @xcite @xcite and models should incorporate information from the text and image sources to answer the questions. multimodal problems are important because many real-world problems requires understanding signals from different modalities to make accurate predictions; information on the web and social media is often represented as textual and visual description. digesting such multimodal information in an effective manner is challenging due to their different natures, e.g., visual and textual sources present different types of information. also, they don't have comparable amounts of information in each modality; usually the textual modality tends to dominate and have more information. while kd approaches can be applied to multimodal applications, student models in these approaches are directly trained to mimic a teacher's outputs without access to teacher's modalityspecific behaviors. as a result, the student and teacher models may significantly differ in their outputs using each modality as input. we illustrate the point in fig 1 . the gap verifies that the student's and the teacher's modality-specific behaviors are not well matched. we hypothesize that it may lead to inefficient distillation, because the student does not carefully mimic the teacher's modality-specific predictions. thus, we propose modality-specific distillation (msd) which is to mimic the teacher's modalityspecific behavior to minimize the gaps. we improve the transfer by splitting the multimodality into separate modalities, using them as additional inputs, and thus distilling modality-specific behavior of the teacher. our msd introduces auxiliary losses per modality to encourage each modality to be distilled effectively; we transfer the modalityspecific knowledge from the teacher. furthermore, we propose weighting approaches for weighting the auxiliary losses to take importance of each modality into account; one modality might have more important information. there are two main strategies to weight these auxiliary losses in the objective; population-based and instance-wise weighting schemes. in the population based, the weight of each loss term is fixed for the whole popula-tion. but in many cases the samples' modalities might carry different amount of information; one of modalities has more important information for predictions. thus, we explore an intuitive instancewise weighting scheme. in the end, we propose a meta-learning approach to find optimal weights. as we will see in our empirical study on multimodal datasets, msd significantly improves the performance of student models over kd. also, our extensive experiments verify that msd with weighting functions learned by our method shows the best performance among other weighting schemes. in our analysis, we show that datasets are different in the need of population-based or samplespecific weighting; the mm-imdb dataset, for example, shows less improvement on instance-wise weighting compared to population-based weighting. ## background in this section, we first define notations and revisit conventional knowledge distillation (kd). ## proposed method in this section, we introduce our proposed approach, modality-specific distillation (msd) for multimodal datasets. ## experiments in this section, we empirically show the effectiveness of our proposed approaches. ## related work knowledge distillation. there have been several studies of transferring knowledge from one model to another @xcite @xcite @xcite @xcite . @xcite improve the accuracy of a shallow neural network by training it to mimic a deep neural network with penalizing the difference of logits between the two networks. @xcite introduced knowledge distillation (kd) that trains a student model with the objective of matching the softmax distribution of a teacher model at the output layer. romero et al. @xcite distill a teacher using additional linear projection layers and minimize l 2 loss at the earlier layers to train a students. park et al. @xcite focused on mutual relations of data examples instead and they proposed relational knowledge distillation. the transfer works best when there are many possible classes because more information can be transferred, but in cases where there are only a few possible classes the transfer is less effective. to deal with the problem, müller et al. @xcite improved the transfer by forcing the teacher to divide each class into many subclasses. tian et al. @xcite proposed to distill from the penultimate layer using a contrastive loss for cross-modal transfer. a few recent papers @xcite have shown that distilling a teacher model into a student model of identical architecture, i.e., self-distillation, can improve the student over the teacher. ## conclusion we studied knowledge distillation on multimodal datasets; we observed that conventional kd may lead to inefficient distillation since a student model does not fully mimic a teacher's modalityspecific predictions. to better transfer knowledge from a teacher on the multimodal datasets, we proposed modality-specific distillation; the student mimics the teacher's outputs on each modality. furthermore, we proposed weighting approaches, population-based and instance-wise weighting schemes, and a meta-learning approach for weighting the auxiliary losses to take importance of each modality into consideration. we empirically showed that we can improve the student's performance with our modality-specific distillation compared to conventional distillation. our msd approach results on modality specific outputs that better resemble the teacher's outputs. we showed that the results hold across different student sizes. moreover, our meta-learning approach is flexible enough to find different effective weighting schemes, depending on the dataset. ). we further replaced its image modality with a background picture that makes it benign and examined models on both examples (right).