Data Pruning for Efficient Model Pruning in Neural Machine Translation (2023)

Abstract: Model pruning methods reduce memory requirements and inference time of large-scale pre-trained language models after deployment. However, the actual pruning procedure is computationally intensive, involving repeated training and pruning until the required sparsity is achieved. This paper combines data pruning with movement pruning for Neural Machine Translation (NMT) to enable efficient fine-pruning. We design a dataset pruning strategy by leveraging cross-entropy scores of individual training instances. We conduct pruning experiments on the task of machine translation from Romanian-to-English and Turkish-to-English, and demonstrate that selecting hard-to-learn examples (top-k) based on training cross-entropy scores outperforms other dataset pruning methods. We empirically demonstrate that data pruning reduces the overall steps required for convergence and the training time of movement pruning. Finally, we perform a series of experiments to tease apart the role of training data during movement pruning and uncover new insights to understand the interplay between data and model pruning in the context of NMT.

## introduction large-scale pre-trained language models have demonstrated encouraging performance in various nlp tasks at the cost of over-parametrized networks, and high memory requirements @xcite . this has led to the development of several pruning approaches for reducing model size, such as magnitude pruning @xcite , movement pruning (fine-pruning) @xcite , block movement pruning @xcite and lottery ticket hypothesis for bert @xcite . although model pruning is effective at reducing the inference time after deployment, the actual pruning procedure is computationally intensive and unsuitable to be performed in resource-constrained settings. for example, bert requires six iterations to reach 40% sparsity with iterative magnitude pruning (imp) -requiring training to convergence, pruning, and retraining to recover the lost accuracy @xcite . recent work @xcite attempts to decrease the training time by identifying structured winning tickets early in training but the implementation does not allow general application to other model pruning algorithms. in contrast, we examine the problem of increasing the efficiency of model pruning techniques through the lens of reducing data requirements and thus ask the following questions -how much data is superfluous in fine-pruning language models for machine translation? can we develop a metric for identifying the informative training examples and significantly prune training data to decrease the training time and memory requirements during language model pruning? in this work, we develop a dataset pruning algorithm for efficient movement pruning of t5 language model @xcite on the task of neural machine translation (nmt) across two datasets, wmt16 en-ro and en-tr. we begin by leveraging the training dynamics and use cross-entropy score for ranking each example according to its difficulty. we utilize this ranking to prune the datasets by selecting hard-to-learn training examples and pruning the remaining examples. we compare this approach with multiple data pruning baselines used in standard vision and speech tasks, including stratified (representative) selection @xcite , random selection, and easiest example selection @xcite . the pruned subsets are then used for movement pruning at varying levels of model sparsity. finally, we perform a series of experiments in the context of nmt to tease apart the role of training data during movement pruning and make the following contributions. ## preliminaries in this section, we first introduce neural machine translation (nmt) and then present model pruning and data pruning methods in the context of nmt. ## method we consider a language model l(x; θ) (θ ∈ r d ) pretrained on a generic dataset through objective l p . this model is fine-pruned for the downstream task of machine translation through movement pruning on the dataset @xmath0. d t consists of sequence pairs (x i , y i ) where x i is the source sentence in one language and y i is the translation in another language. our goal is to prune d t through different heuristics to obtain a smaller dataset d s and analyze the impact of fine-pruning the nmt model l(x; θ) using this limited data. specifically, we consider the changes in test bleu performance of pruned model and the impact on the training time during movement pruning using limited data. ## results and discussion figure 2 shows the complete results for bleu on the development datasets after fine-pruning t5 on the subsets selected through top-k, bottom-k, random and stratified strategies. the sweep across pruning percentage demonstrates consistently higher bleu for fine-pruning on subsets consisting of hard-to-learn examples (top-k strategy). bottom-k performs the worst, indicating that the selection of the easy-to-learn examples is not a good choice in a limited data regime, especially for challenging datasets like en-tr. to identify the subpopulations being selected by each pruning strategy, we analyze the distribution of training cross-entropy scores (figure 3 ). for en-ro dataset, we observe a long-tail of hard-tolearn examples and thus top-k strategy is selecting these examples to an extent determined by pruning percentage. in contrast, en-tr being a challenging dataset, has a significantly smaller number of easyto-learn examples. despite having different training distribution, the better performance of top-k compared to other strategies ( the original pruning algorithm considers the crossentropy loss of individual examples as the pruning metric. we now consider using the bleu score of individual training examples for data pruning. this is an extrinsic metric that compares the candidate translation with the reference translation. the distribution of the training bleu score (shown in figure 4 ) is different from the cross-entropy distribution (figure 3 ). particularly, the long-tail we observe in the distribution of cross-entropy scores of en-ro dataset corresponding to the rare, hardto-learn examples is not present for training bleu scores distribution of en-ro. we next evaluate our pruning algorithm with bleu training scores instead of the cross-entropy pruning scores ( ## limitations we list the potential limitations of our work below: other datasets. we evaluated our data pruning algorithm on two nmt datasets -en-ro and en-tr. further empirical evaluation will help verify the generalization of our approach to other types of nmt datasets -for example, high-resource languages like wmt14 en-de @xcite , noiser datasets like mtnt @xcite , datasets with high oov rate like gnome and the ones from a different domain like the ubuntu technical dataset. cheaper pruning metrics. the proposed pruning method requires a fine-tuning run to compute the cross-entropy scores and construct the ranking for all the training examples. although this procedure is done before fine-pruning, it still contributes to the end-to-end cost. cheaper example scoring metrics, e.g., self-supervised metrics that do not require a complete training run @xcite might reduce the initial cost of data pruning and yield more efficient results. ## conclusion in this work, we leverage training dynamics to devise a dataset pruning algorithm for efficient movement pruning in nmt. experiments on two nmt datasets of varying difficulty show the advantage of selecting hard-to-learn examples for fine-pruning t5 language model. finally, we demonstrate the desirable properties of the proposed pruning method, including minimal vocabulary changes and transferability to other models. future work includes experimentation with the proposed pruning strategy on other downstream tasks and an in-depth analysis of the pruned subsets.