The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains (2022)

Abstract: Recent model pruning methods have demonstrated the ability to remove redundant parameters without sacrificing model performance. Common methods remove redundant parameters according to the parameter sensitivity, a gradient-based measure reflecting the contribution of the parameters. In this paper, however, we argue that redundant parameters can be trained to make beneficial contributions. We first highlight the large sensitivity (contribution) gap among high-sensitivity and low-sensitivity parameters and show that the model generalization performance can be significantly improved after balancing the contribution of all parameters. Our goal is to balance the sensitivity of all parameters and encourage all of them to contribute equally. We propose a general task-agnostic method, namely intra-distillation, appended to the regular training loss to balance parameter sensitivity. Moreover, we also design a novel adaptive learning method to control the strength of intra-distillation loss for faster convergence. Our experiments show the strong effectiveness of our methods on machine translation, natural language understanding, and zero-shot cross-lingual transfer across up to 48 languages, e.g., a gain of 3.54 BLEU on average across 8 language pairs from the IWSLT’14 dataset.

## introduction exploring efficient parameter use in neural networks is critical for improving computational quality and decreasing storage requirements @xcite . the lottery ticket hypothesis @xcite suggests that a small subset of parameters, namely 'winning tickets', can reach similar performance compared to a dense model through iterative retraining. recent techniques for pruning models @xcite @xcite figure 1 : illustration of the intra-distillation process. we pass the model k (@xmath0, p 2 , p 3 . each time we disable different subsets of parameters (illustrated by different colors). minimizing the difference of the k outputs approximates minimizing the contribution difference of these disabled subsets of parameters, which can significantly improve model generalization performance. @xcite have shown to be successful in reducing redundant parameters of trained networks by over 80% without obvious loss of the model quality. despite the success of winning tickets, it usually does not offer better performance and is actually computationally expensive to obtain (needing iterative pruning and retraining). moreover, unstructured pruning barely accelerates inference computation because the device still computes the dense tensors, with the only difference being that they are filled with zeros. the main motivation behind pruning is the existence of redundant parameters which basically have no contribution to the model. taking an opposite approach from pruning redundant parameters, we encourage all parameters to contribute. @xcite ; @xcite recently showed that redundant parameters are insufficiently trained, and can actually contribute more when they are trained properly. following this line, we also argue that there is a large room for improving model general-ization by making redundant parameters contribute instead of discarding them. however, our approach differs in that we change the training objective, as opposed to learning rates. in this paper, we show significant improvement after balancing the contribution of all parameters on various tasks. our goal is to balance the sensitivity of parameters to encourage the equal contribution of each parameter, where sensitivity is a gradient-based measure reflecting the degree of parameter contribution. usually, lower-sensitivity parameters are considered redundant. however, in an extreme case of our goal, no parameter is redundant. thus, we propose intra-distillation, a task-agnostic method aiming to minimize the sensitivity difference among each subset of parameters. specifically, we obtain k outputs by forward passing the model k times, where we randomly disable a different subset of parameters for each pass (figure 1 ). we deduce that minimizing the difference of these k outputs approximates minimizing the sensitivity of the disabled parameters. therefore, in each step of training, we can minimize the sensitivity of k random groups of parameters. we list our main contributions are summarized as follows: ## why balance the contribution? we investigate the contribution difference among parameters based on an important metric, parameter sensitivity, which has been widely used in pruning under the name "importance scores" @xcite @xcite . then, we highlight that model performance benefits from balanced parameter contribution in a case study of knowledge distillation. ## proposed method in the previous section, we showed two important findings; the large contribution gap between highand low-sensitivity parameters, and that there is little understanding of the correlation between the better performance and more balanced contribution in self-distillation. in this section, we propose a general method to balance the parameter sensitivity (contribution) to improve the model performance. ## conclusions taking an opposite view from pruning redundant parameters, we questioned whether we overlook the potential of these redundant parameters and encouraged all parameters to contribute equally to improve the model generalization performance. we first introduced the concept of degree of contribution balance to describe how balanced of all parameters is. then, we used balanced parameter contribution to explain the 'dark knowledge' that successfully transfers in knowledge distillation, by analyzing the contribution gap among parameters within a model. with the goal of adding a constraint term to balance the parameter contribution, we proposed intra-distillation with a strength-adaptive learning method. with wide-ranging experiments and analysis on machine translation, natural language understanding and zero-shot cross-lingual transfer tasks, we demonstrated that intra-distillation are capable of improving the model performance significantly and balance the parameter contribution effectively.