Data and Parameter Scaling Laws for Neural Machine Translation (2021)

Abstract: We observe that the development cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.

## introduction as training neural networks becomes an organizational and multi-million dollar venture (brown et al., 2020), it is imperative to quantifiably predict the benefits of scaling up neural networks. in this paradigm, machine learning is an engineering effort, in which money can buy resources (data, compute) and the main concern is to predict returnon-investment (roi) while avoiding bottlenecks. recent work has observed that the cross entropy loss of neural language models and other autoregressive generative models scales like a power law in the amount of training data, compute, and number of model parameters over several orders of magnitude (hestness et al., 2019; kaplan et al., 2020; henighan et al., 2020). similar intuitions exist in the realm of supervised mt: doubling the amount of parallel training data leads to roughly a fixed improvement in bleu in both phrase-based statistical mt (irvine and callison-burch, 2013; turchi et al., 2008) and neural mt (koehn and knowles, 2017; sennrich and zhang, 2019). in section 2, we show that these mt intuitions can be quantified and explained via cross-entropy power law scaling; using a handful of experiments on small subsets of mt datasets, we precisely predict the performance of large systems trained on orders of magnitude more data. in section 3, we demonstrate how these trends might be utilized to make roi predictions when annotating more data for low-resource language pairs. ## machine translation scaling laws to investigate the predictability of mt system performance as parameters/data increase, we train many transformers of various sizes ( we use three language pairs in our experiments: german-english (de-en), russian-english (ru-en), and chinese-english (zh-en). the data for each language pair is a concatenation of wmt 2017 data (which includes news commentary, parliamentary proceedings, and web-crawled data) and open-subtitles2018 (lison and tiedemann, 2016; tiedemann, 2016). datasets are tokenized using the moses 1 tokenizer, after which a 30k bpe vocabulary is constructed using the full dataset. for evaluation, we use newstest2016 concatenated with the last 2500 lines of opensubtitles2018. transformers are trained with early stopping and a learning rate of 0.0002 2 with a plateau-reduce schedule for a maximum of 350k updates. other training details can be found in the code supplement. @xcite the resulting losses are plotted in figure 1 , with model sizes ranging from 393k-56m parameters and data sizes from 40k-50m lines of text. as either n or d approaches infinity, l(n, d) simplifies to a "pure power law" in the other variable, which looks like a straight line on a log-log graph. for example, if we assume all models are large enough that data becomes the main performance bottleneck, then: we will use this assumption later when dealing with very low-resource language pairs. ## predicting roi of annotating low-resource language pairs if we assume that data is the main performance bottleneck (as it is in many low-resource language pairs), we can plug equation 2 into equation 3 to directly model the relationship between bleu and data size: ## conclusion we have shown that supervised neural machine translation performance with transformers scales like a power law in non-embedding parameters and training data, aligning with similar observations in unsupervised auto-regressive modeling. we've also seen that as development cross-entropy decreases, bleu exponentially increases. these two relationships can be combined to predict an effective usd-to-bleu trade-off when annotating more data, even in low-resource regimes.