The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021)

Abstract: A “bigger is better” explosion in the number of parameters in deep neural networks has made it increasingly challenging to make state-of-the-art networks accessible in compute-restricted environments. Compression techniques have taken on renewed importance as a way to bridge the gap. However, evaluation of the trade-offs incurred by popular compression techniques has been centered on high-resource datasets. In this work, we instead consider the impact of compression in a data-limited regime. We introduce the term low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. This is a common setting for NLP for low-resource languages, yet the trade-offs in performance are poorly studied. Our work offers surprising insights into the relationship between capacity and generalization in data-limited regimes for the task of machine translation. Our experiments on magnitude pruning for translations from English into Yoruba, Hausa, Igbo and German show that in low-resource regimes, sparsity preserves performance on frequent sentences but has a disparate impact on infrequent ones. However, it improves robustness to out-of-distribution shifts, especially for datasets that are very distinct from the training distribution. Our findings suggest that sparsity can play a beneficial role at curbing memorization of low frequency attributes, and therefore offers a promising solution to the low-resource double bind.

## introduction over the years, the size of language models have grown exponentially @xcite @xcite . additional parameters have improved quality on a variety of downstream nlp tasks, but drive up the cost of training @xcite @xcite and increase the latency and memory footprint at inference time @xcite . figure 1 : cost of mobile data by country per language rank according to the taxonomy by @xcite . extending state-of-the-art language models to low-resource languages requires addressing what we term the low-resource double bind. lowresourcedness goes beyond mere data availability and reflects systemic issues in society @xcite . classifications of languages with respect to "resourcedness" have focused on the relative availability of data @xcite , and the concentration of nlp researchers from these regions or the over-fitting of model design around a small set of high resource languages @xcite . less well documented and explored is the overindexing of low-resource languages in ecosystems which simultaneously present severe constraints of computational resource. in fig. 1 we plot 22 languages grouped by the availability of labelled and unlabelled data as proposed by @xcite against the cost of 1 gb of data as a percentage of monthly income. each language is mapped to the country with the most speakers. the cost of data is a valuable proxy for the cost of access to technology in an ecosystem @xcite . here, this visibly co-occurs with the limitations in available data for different languages. in computationally constrained environments, access to machine learning technology depends upon optimizing jointly for both model performance and compactness. pruning and quantization are widely applied techniques for compressing deep neural networks prior to deployment, as compressed models require less memory, energy consumption and have lower inference latency @xcite @xcite . to-date, evaluating the merits and tradeoffs incurred by compression have overwhelmingly centered on settings where the data is relatively abundant @xcite @xcite @xcite . in this work, we instead ask how these design choices trade-off with performance in data-limited regimes typical of low resource languages. we conduct large scale experiments on neural machine translation (nmt) models trained to translate between english and three low resource african languages (yoruba, igbo and hausa) and one high resourced language (german). we compare performance across models independently trained to very different levels of sparsity -ranging from 50 % to 98 % -and evaluate performance on the original distribution, in addition to establishing sensitivity to distribution shift across multiple corpora. recent work restricted to the computer vision domain has found that sparse models with comparable top-line performance metrics diverge considerably in behavior on the long-tail of the distribution and are sensitive to distribution shifts @xcite . here, we rigorously characterize the impact of sparsity on learned decision boundaries in nmt. in addition to heldout set bleu, we measure sub-group performance on sentences grouped by prototypicality and study generalization properties over test corpora with different out-of-vocabulary ratios. we also evaluate whether humans prefer translations from sparse or dense models. our contributions can be enumerated as follows: this holds until extremely high levels of sparsity (95%-99% of all weights removed) where a severe decline in bleu is notable. implications of our work understanding the impact of compression on low-resource languages is key to making technology accessible and inclusive. our work suggests that compression in these settings alters generalization in ways that can be beneficial and go beyond merely fulfilling deployment constraints. a challenge in low-resource nlp is that the existing publicly available corpora often come from very specific domains, such as missionary websites or translations of religious texts. these sources do not adequately reflect the reality of the potential applications of nlp technologies, and are rarely sufficient for deployment @xcite @xcite . thus, a task of great interest is establishing what model design choices can lead to generalization properties that extend beyond the immediate task at hand. our work suggests that sparsity can play an important role in aiding generalization by curbing the memorization of rare long-tail instances. ## methodology addressing the low-resource double bind requires a careful setup of experiments to reflect the realities of low-resource translation. in particular, we want to control the effects of (1) network sparsity, (2) training data size, (3) target language, and (4) domain shifts. in this work we focus on pruning, a widely favored compression technique due to remarkably high levels of compression that can be achieved while retaining top-line performance @xcite . pruning typically involves three separate stages: 1) training a dense model, 2) progressively removing a subset of weights estimated to be unimportant, and 3) continuing to train the smaller sparse network for a certain number of steps to recoup performance @xcite . pruning is the subject of considerable research and numerous techniques have been proposed, which differ in how weights are identified for removal and the schedule for introducing sparsity/allowing recovery @xcite @xcite @xcite @xcite . the development of specialized software kernels has enabled the acceleration of sparse networks on traditional hardware @xcite @xcite with new generations of hardware directly facilitating sparse training @xcite . state of art pruning techniques can achieve a far higher level of compression and performance than simply using a smaller dense network @xcite . in our setting, a 90% sparse base transformer greatly outperforms a tiny dense one across all the languages despite having a fraction of the parameters (14m vs 4.6m) (appendix ## results sparsity bleu trade-off in figure 2 , we can see that models are tolerant of moderate to high levels of sparsity (50% -80%) while retaining bleu performance relative to dense. between 50% and 80% sparsity, any degradation is minimal as sparse performance remains at 95% or more of dense performance for all languages. hausa even has a slight exception where pruning 70% or 80% of the model parameters performs on par with the baseline or even better. however, for both global and random test sets, there is a noticeably sharp degradation in bleu when progressing to extremely high sparsity levels of 90% and beyond. ## conclusion we demonstrate the effectiveness of introducing sparsity when training nmt models for lowresourced languages. we show that small performance drops in extremely sparse regimes according to automatic metrics are not reflected in humanjudged translation quality. our extensive study of the impact of pruning on out-of-distribution generalization reveals that sparse models improve over dense models in a limited data regime. overall, these insights are promising for overcoming the low-resource double bind: pruned models reduce resource requirements for deployment, and increase the robustness towards out-of-domain samples due to reduced memorization during training.