On the Sparsity of Neural Machine Translation Models (2020)

Abstract: Modern neural machine translation (NMT) models employ a large number of parameters, which leads to serious over-parameterization and typically causes the underutilization of computational resources. In response to this problem, we empirically investigate whether the redundant parameters can be reused to achieve better performance. Experiments and analyses are systematically conducted on different datasets and NMT architectures. We show that: 1) the pruned parameters can be rejuvenated to improve the baseline model by up to +0.8 BLEU points; 2) the rejuvenated parameters are reallocated to enhance the ability of modeling low-level lexical information.

## introduction modern neural machine translation (nmt) @xcite @xcite models employ sufficient capacity to fit the massive data well by utilizing a large number of parameters, and suffer from the widely recognized issue, namely, over-parameterization. for example, @xcite showed that over 40% of the parameters in an rnn-based nmt model can be pruned with negligible performance loss. however, the low utilization efficiency of parameters results in a waste of computational resources @xcite , as well as renders the model stuck in a local optimum @xcite . in response to the over-parameterization issue, network pruning has been widely investigated for both computer vision (cv) @xcite and natural language processing (nlp) tasks @xcite . recent work has proven that such spare parameters can be reused to maximize the utilization of models in cv tasks such as image classification (han et al., 2017; * work was done when interning at tencent ai lab. @xcite . the leverage of parameter rejuvenation in sequence-to-sequence learning, however, has received relatively little attention from the research community. in this paper, we empirically study the efficiency issue for nmt models. specifically, we first investigate the effects of weight pruning on advanced transformer models, showing that 20% parameters can be directly pruned, and by continuously training the sparse networks, we can prune 50% with no performance loss. starting from this observation, we then exploit whether these redundant parameters are able to be re-utilized for improving the performance of nmt models. experiments are systematically conducted on different datasets (i.e. zh⇒en, de⇒en and en⇒fr) and nmt architectures (i.e. transformer, rnnsearch and lightconv). results demonstrate that the rejuvenation approach can significantly and consistently improve the translation quality by up to +0.8 bleu points. further analyses reveal that the rejuvenated parameters are reallocated to enhance the ability to model the source-side low-level information, lacking of which leads to a number of problems in nmt models @xcite @xcite . ## approach a standard nmt model directly optimizes the conditional probability of a target sentence @xmath0, . . . , y j given its corresponding source sentence @xmath1, . . . , x i , namely p (y|x; θ) = the parameters of the nmt model are trained to maximize the likelihood of a set of training examples. given a well-trained nmt model, we first prune its inactive parameters, and then rejuvenate them. our implementation details are as follows. pruning the redundant parameters in neural networks can be pruned according to a certain criterion while the left ones are significant to preserve the accuracy of the model. specifically, we mask weight connections with low magnitudes in the forward pass and these weights are not updated during optimization. given the weight matrix w with n parameters, we rank the parameters according to their absolute values. supposed that the pruning ratio is γ (i.e. γ% of parameters should be pruned), we keep the top n parameters (@xmath2, and remove the others with a binary mask matrix, which is the same size of w . we denote the pruned parameters as θ p , subject to θ p ⊂ θ. there are two pruning strategies @xcite : 1) local pruning, which prunes γ% of parameters in each layer; and 2) global pruning, which compares the importance of parameters across layers. following @xcite , we retrain the pruned networks after the pruning phase. specifically, we continue to train the remaining parameters, but maintain the sparse structure, that is we optimize p (y|x; θ) with the constraint: @xmath3, ∀@xmath4. rejuvenation after the pruning and retraining phases, we aim to restore the model capacity by rejuvenating the pruned parameters. this is a common method in optimization to avoid useless computations and further improve performances (han pruning rejuvenation bleu local global zero external × × 28.12 × × 28.08 × × 28.12 × × 28.14 ## conclusion in this paper, we prove that existing nmt systems are over-parameterized and propose to improve the utilization efficiency of parameters in nmt models by introducing a rejuvenation approach. empirical results on a variety of language pairs and architectures demonstrate the effectiveness and universality of the presented method. we also analyze the gains from perspectives of learning dynamics and linguistic probing, which give insightful research directions for future work. future directions include continuing the exploration of this research topic for large sequenceto-sequence pre-training models @xcite and multi-domain translation models @xcite . we will employ recent analysis methods to better understand the behaviors of rejuvenated models @xcite .