Original: Data and Parameter Scaling Laws for Neural Machine Translation
Revised: Data and Parameter Scaling Laws for Neural Machine Translation

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: We observe that the development crossentropy loss of supervised neural machinetranslation models scales like a power law withthe amount of training data and the number ofnon-embedding parameters in the model.
Revised: We observe that the development crossentropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model.

Original: Wediscuss some practical implications of these results, such as predicting BLEU achieved bylarge scale models and predicting the ROI oflabeling data in low-resource language pairs.
Revised: We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.

Original:  Introduction
Revised:  Introduction

Original: As training neural networks becomes an organizational and multi-million dollar venture (Brownet al., 2020), it is imperative to quantiﬁably predict the beneﬁts of scaling up neural networks.
Revised: As training neural networks becomes an organizational and multi-million dollar venture (Brown et al., 2020), it is imperative to quantiﬁably predict the beneﬁts of scaling up neural networks.

Original: Inthis paradigm, machine learning is an engineeringeffort, in which money can buy resources (data,compute) and the main concern is to predict returnon-investment (ROI) while avoiding bottlenecks.
Revised: In this paradigm, machine learning is an engineering effort, in which money can buy resources (data, compute) and the main concern is to predict returnon-investment (ROI) while avoiding bottlenecks.

Original: Recent work has observed that the cross entropyloss of neural language models and other autoregressive generative models scales like a power lawin the amount of training data, compute, and number of model parameters over several orders of magnitude (Hestness et al., 2019; Kaplan et al., 2020;
Revised: Recent work has observed that the cross entropy loss of neural language models and other autoregressive generative models scales like a power law in the amount of training data, compute, and number of model parameters over several orders of magnitude (Hestness et al., 2019; Kaplan et al., 2020;

Original: Henighan et al., 2020).
Revised: Henighan et al., 2020).

Original: Similar intuitions exist in the realm of supervised MT: doubling the amount of parallel training data leads to roughly a ﬁxed improvement in BLEU in both phrase-based statistical MT (Irvine and Callison-Burch, 2013; Turchi et al., 2008) and neural MT (Koehn and Knowles, 2017; Sennrich and Zhang, 2019).
Revised: Similar intuitions exist in the realm of supervised MT: doubling the amount of parallel training data leads to roughly a ﬁxed improvement in BLEU in both phrase-based statistical MT (Irvine and Callison-Burch, 2013; Turchi et al., 2008) and neural MT (Koehn and Knowles, 2017; Sennrich and Zhang, 2019).

Original: In Section 2, we show that these MT intuitions cansimilarly be quantiﬁed as power law scaling; using a handful of experiments on small subsets of MT datasets, we precisely predict the performance(both cross-entropy loss and BLEU) of large systems trained on orders of magnitude more data.
Revised: In Section 2, we show that these MT intuitions can be quantiﬁed and explained via cross-entropy power law scaling; using a handful of experiments on small subsets of MT datasets, we precisely predict the performance of large systems trained on orders of magnitude more data.

Original: In Section 3, we demonstrate how these trends might be utilized to make ROI predictions when annotating more data for low-resource language pairs.
Revised: In Section 3, we demonstrate how these trends might be utilized to make ROI predictions when annotating more data for low-resource language pairs.

Original:  Machine Translation Scaling Laws
Revised:  Machine Translation Scaling Laws

Original: To investigate the predictability of MT system performance as parameters/data increase, we train many Transformers of various sizes (Table 2) on randomly selected subsets of data ( 12 , 14 , 18 , ... ) for several standard MT datasets.
Revised: To investigate the predictability of MT system performance as parameters/data increase, we train many Transformers of various sizes (Table 2) on randomly selected subsets of data ( 12 , 14 , 18 , ... ) for several standard MT datasets.

Original: The smallest data subsets contain ∼ 0.1% of the total data available.
Revised: The smallest data subsets contain ∼ 0.1% of the total data available.

Original: We use three language pairs in our experiments:
Revised: We use three language pairs in our experiments:

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: German-English (de-en), Russian-English (ru-en),and Chinese-English (zh-en).
Revised: German-English (de-en), Russian-English (ru-en), and Chinese-English (zh-en).

Original: The data for eachlanguage pair is a concatenation of WMT 2017data (which includes news commentary, parliamentary proceedings, and web-crawled data) and Open- [Table] Subtitles2018 (Lison and Tiedemann, 2016; Tiedemann, 2016).
Revised: The data for each language pair is a concatenation of WMTdata (which includes news commentary, parliamentary proceedings, and web-crawled data) and OpenSubtitles2018 (Lison and Tiedemann, 2016; Tiedemann, 2016).

Original: Datasets are tokenized using the
Revised: Datasets are tokenized using the

Original: [Table]

Original: Moses 1 tokenizer, after which a 30k BPE vocabu- [Table] lary is constructed using the full dataset.
Revised: Moses 1 tokenizer, after which a 30k BPE vocabulary is constructed using the full dataset.

Original: For evalu- [Table] ation, we use newstest2016 concatenated with thelast 2500 lines of OpenSubtitles2018.
Revised: For evaluation, we use newstest2016 concatenated with the last 2500 lines of OpenSubtitles2018.

Original: Transformers are trained with early stopping anda learning rate of 0.0002 2 with a plateau-reduceschedule for a maximum of 350k updates.
Revised: Transformers are trained with early stopping and a learning rate of 0.0002 2 with a plateau-reduce schedule for a maximum of 350k updates.

Original: Othertraining details can be found in the code supplement.
Revised: Other training details can be found in the code supplement.

Original: The resulting losses are plotted in Figure 1,with model sizes ranging from 393k-56M parameters and data sizes from 40k-50M lines of text.
Revised: The resulting losses are plotted in Figure 1, with model sizes ranging from 393k-56M parameters and data sizes from 40k-50M lines of text

Original:  2. Cross-entropy vs. Data/Parameters
Revised: . 2. Cross-entropy vs. Data/Parameters

Original: Kaplan et al.
Revised: Kaplan et al.

Original: (2020) provide an ansatz that predictscross-entropy loss given the amount of training dataand the size of the neural model:
Revised: (2020) provide an ansatz that predicts cross-entropy loss given the amount of training data and the size of the neural model:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where L is the per-token development crossentropy loss (in nats), N is the number of nonembedding parameters, D is the amount of training[Table] data (in bytes), and α N , α D , N C , and D C are con- [Table]
Revised: where L is the per-token development crossentropy loss (in nats), N is the number of nonembedding parameters, D is the amount of training data (in bytes), and α N , α D , N C , and D C are constants determined by the particulars of the data distribution and training setup.

Original: stants determined by the particulars of the data

Original: [Table]

Original: distribution and training setup.

Original: Figure 1 shows that this equation is highly predictive of our results.
Revised: Figure 1 shows that this equation is highly predictive of our results.

Original: The predictions are also fairly stable; Table 3 shows that the best-ﬁt parameters of this equation stay similar even when restricting ourselves to using only 3-6% of the data.
Revised: The predictions are also fairly stable; Table 3 shows that the best-ﬁt parameters of this equation stay similar even when restricting ourselves to using only 3-6% of the data.

Original: In Appendix A we perform a retrospective analysis of the results from Zhang and Duh (2020) to give some insight into how different hyper-parameter settings may inﬂuence scaling coefﬁcients.
Revised: In Appendix A we perform a retrospective analysis of the results from Zhang and Duh (2020) to give some insight into how different hyper-parameter settings may inﬂuence scaling coefﬁcients.

Original: As either N or D approaches inﬁnity, L ( N, D ) simpliﬁes to a “pure power law" in the other variable, which looks like a straight line on a log-log graph.
Revised: As either N or D approaches inﬁnity, L ( N, D ) simpliﬁes to a “pure power law" in the other variable, which looks like a straight line on a log-log graph.

Original: For example, if we assume all models are large enough that data becomes the main performance bottleneck, then:
Revised: For example, if we assume all models are large enough that data becomes the main performance bottleneck, then:

Original: [Equation]
Revised: [Equation]

Original: We will use this assumption later when dealing with very low-resource language pairs.
Revised: We will use this assumption later when dealing with very low-resource language pairs.

Original: 2. BLEU vs. Cross-Entropy Loss
Revised: 2. BLEU vs. Cross-Entropy Loss

Original: Predicting cross-entropy loss by itself does not tell us much about the quality of the translation system; we would really like to predict the achieved
Revised: Predicting cross-entropy loss by itself does not tell us much about the quality of the translation system; we would really like to predict the achieved

Original: BLEU score, which is more interpretable to humans as a measure of adequacy, ﬁdelity, and ﬂuency (Papineni et al., 2002).
Revised: BLEU score, which is more interpretable to humans as a measure of adequacy, ﬁdelity, and ﬂuency (Papineni et al., 2002).

Original: Figure 2 shows thatthe relationship between BLEU and cross-entropycan vary between different language pairs and BPEsettings.
Revised: Figure 2 shows that the relationship between BLEU and cross-entropy can vary between different language pairs and BPE settings.

Original: However, when these factors are ﬁxed,
Revised: However, when these factors are ﬁxed,

Original: BLEU seems to exponentially increase as crossentropy decreases:
Revised: BLEU seems to exponentially increase as crossentropy decreases:

Original: [Equation]
Revised: [Equation]

Original: This relationship is fairly predictable for high
Revised: This relationship is fairly predictable for high

Original: BLEU values, but becomes noisier as BLEU dropsbelow 15.
Revised: BLEU values, but becomes noisier as BLEU drops below 15.

Original: Notably, changing the BPE encodingdoes not seem to affect k , but does change themultiplying constant C .[-
-]
Revised: Notably, changing the BPE encoding does not seem to affect k , but does change the multiplying constant C .[-
-]

Original: Why should this relationship be exponential?
Revised: Why should this relationship be exponential?

Original: We might gain some insight by re-writing [-Equation
-] in terms of the per-token perplexity (P):
Revised: We might gain some insight by re-writing [-Equation
-]Equation 3 in terms of the per-token perplexity (P):

Original: [Equation]
Revised: [Equation]

Original: where (1 /P ) can intuitively be interpreted as theexpected unigram precision of an autoregressivelysampled translation with the same length as thereference sentence (Manning and Schutze, 1999).
Revised: where (1 /P ) can intuitively be interpreted as the expected unigram precision of an autoregressively sampled translation with the same length as the reference sentence (Manning and Schutze, 1999).

Original: This is only intuition, however: in practice, wedo not sample translations but decode using beamsearch, and BLEU combines multiple modiﬁed ngram precisions besides unigram precision.
Revised: This is only intuition, however: in practice, we do not sample translations but decode using beam search, and BLEU combines multiple modiﬁed ngram precisions besides unigram precision.

Original: 2. Preventing Breakdown At Smaller 
Revised: 2. Preventing Breakdown At Smaller Dataset Sizes

Original: Dataset Sizes

Original: Some extremely low-resource MT datasets (which we examine in Section 3) can have less than 5 MB of data ( ∼ 40k sentence pairs).
Revised: Some extremely low-resource MT datasets (which we examine in Section 3) can have less than 5 MB of data ( ∼ 40k sentence pairs).

Original: Figure 3 shows that when we extend our previous experiments to datasets smaller than this size, using 0.05% - 0.0125% of the data, the data scaling power law seems to break down, casting doubt on our ability to extrapolate extremely low-resource results to medium and high-resource data regimes.
Revised: Figure 3 shows that when we extend our previous experiments to datasets smaller than this size, using 0.05% - 0.0125% of the data, the data scaling power law seems to break down, casting doubt on our ability to extrapolate extremely low-resource results to medium and high-resource data regimes.

Original: However, the results are not simply noisy but predictably plateau to an apparent ceiling of 7. nats.
Revised: However, the results are not simply noisy but predictably plateau to an apparent ceiling of 7. nats.

Original: For reference, a unigram language model trained on only the English part of the training data (with a 30k BPE vocab) achieves a per-token crossentropy of ∼ 7 nats.
Revised: For reference, a unigram language model trained on only the English part of the training data (with a 30k BPE vocab) achieves a per-token crossentropy of ∼ 7 nats.

Original: This leads us to suspect that models in this data regime are learning to rely on simple unigram statistics that do not change much as we decrease the data size.
Revised: This leads us to suspect that models in this data regime are learning to rely on simple unigram statistics that do not change much as we decrease the data size.

Original: Using a much smaller BPE vocabulary of 2k tokens rectiﬁes this plateau and returns to power law scaling, even with datasets < 5 MB.
Revised: Using a much smaller BPE vocabulary of 2k tokens rectiﬁes this plateau and returns to power law scaling, even with datasets < 5 MB.

Original: We believe this is because the smaller vocabulary makes it difﬁcult to exploit unigram statistics for rare words.
Revised: We believe this is because the smaller vocabulary makes it difﬁcult to exploit unigram statistics for rare words.

Original: While this is not conclusive evidence, we recommend that cross-entropies near or above unigram LM performance should not be relied upon to extrapolate performance.
Revised: While this is not conclusive evidence, we recommend that cross-entropies near or above unigram LM performance should not be relied upon to extrapolate performance.

Original: Dataset subsets which contain less than half of the BPE vocabulary should similarly be avoided.
Revised: Dataset subsets which contain less than half of the BPE vocabulary should similarly be avoided.

Revised: during training.

Revised: In contrast, ∼ 10 MB datasets contain almost every vocab token.

Revised: See Appendix

Original:  Predicting ROI of AnnotatingLow-Resource Language Pairs
Revised:  Predicting ROI of Annotating Low-Resource Language Pairs

Original: If we assume that data is the main performancebottleneck (as it is in many low-resource languagepairs), we can plug Equation 2 into Equation 3 todirectly model the relationship between BLEU anddata size:
Revised: If we assume that data is the main performance bottleneck (as it is in many low-resource language pairs), we can plug Equation 2 into Equation 3 to directly model the relationship between BLEU and data size:

Original: [Equation]
Revised: [Equation]

Original: where K = k ( D C ) α D .
Revised: where K = k ( D C ) α D .

Original: This can be further combined with the hourly cost of ﬂuent human translators to give us an approximate USD-to-BLEU tradeoff when annotating more data for low-resourcelanguage pairs.
Revised: This can be further combined with the hourly cost of ﬂuent human translators to give us an approximate USD-to-BLEU tradeoff when annotating more data for low-resource language pairs.

Original: Figure 4 shows some example projections for
Revised: Figure 4 shows some example projections for

Original: Tagalog-English (tl-en) and Swahili-English (swen), with each dataset containing less than 50k sentence pairs (Zavorin et al., 2020).
Revised: Tagalog-English (tl-en) and Swahili-English (swen), with each dataset containing less than 50k sentence pairs (Zavorin et al., 2020).

Original: Under someassumptions about the costs of human translation 9 ,we predict that spending ∼ $60k USD to acquiremore tl-en/sw-en data (which would roughly double the size of the either dataset) would lead to animprovement of around 10-15 BLEU.
Revised: Under some assumptions about the costs of human translation 9 , we predict that spending ∼ $60k USD to acquire more tl-en/sw-en data (which would roughly double the size of the either dataset) would lead to an improvement of around 10-15 BLEU.

Original: 3. Limitations
Revised: 3. Limitations

Original: There is a reasonable amount of noise in the crossentropy/BLEU relationship at this scale (shown in Appendix Figure 8) which limits the precision and reliability of these predictions.
Revised: There is a reasonable amount of noise in the crossentropy/BLEU relationship at this scale (shown in Appendix Figure 8) which limits the precision and reliability of these predictions.

Original: In practice, we expect small amounts of data can be acquired in batches and predictions can be re-evaluated before deciding to continue.
Revised: In practice, we expect small amounts of data can be acquired in batches and predictions can be re-evaluated before deciding to continue.

Original: However, these predictions give a general sense of the cost of progress in lowresource machine translation.
Revised: However, these predictions give a general sense of the cost of progress in lowresource machine translation.

Original: When engineering a real-world system, the simple option of acquiring more data and predictably improving performance should always be carefully weighed against more complicated and less predictable options.
Revised: When engineering a real-world system, the simple option of acquiring more data and predictably improving performance should always be carefully weighed against more complicated and less predictable options.

Original: That being said, predictably achieving a high
Revised: That being said, predictably achieving a high

Original: BLEU score on a test dataset is not equivalent to “solving translation" for that language pair.
Revised: BLEU score on a test dataset is not equivalent to “solving translation" for that language pair.

Original: Underspeciﬁcation (D’Amour et al., 2020) still poses a challenge for effectively evaluating machine translation systems in real-world scenarios, especially in low-resource language pairs where evaluation data is usually from a narrow domain.
Revised: Underspeciﬁcation (D’Amour et al., 2020) still poses a challenge for effectively evaluating machine translation systems in real-world scenarios, especially in low-resource language pairs where evaluation data is usually from a narrow domain.

Original: More robust evaluation methods are needed, and it is not clear whether the output of these methods will be as predictable as cross-entropy loss or BLEU.
Revised: More robust evaluation methods are needed, and it is not clear whether the output of these methods will be as predictable as cross-entropy loss or BLEU.

Original: And ﬁnally, while our work demonstrates empirical power law scaling of NMT systems, it does not attempt to provide any causal explanation for these results.
Revised: And ﬁnally, while our work demonstrates empirical power law scaling of NMT systems, it does not attempt to provide any causal explanation for these results.

Original: We also do not investigate the speciﬁc training factors that lead to a particular scaling exponent, but we expect this to be a fruitful research direction for future exploration.
Revised: We also do not investigate the speciﬁc training factors that lead to a particular scaling exponent, but we expect this to be a fruitful research direction for future exploration.

Original:  Conclusion
Revised:  Conclusion

Original: We have shown that supervised neural machine translation performance with Transformers scales like a power law in non-embedding parameters and training data, aligning with similar observations in unsupervised auto-regressive modeling.
Revised: We have shown that supervised neural machine translation performance with Transformers scales like a power law in non-embedding parameters and training data, aligning with similar observations in unsupervised auto-regressive modeling.

Original: We’ve also seen that as development cross-entropy decreases, BLEU exponentially increases.
Revised: We’ve also seen that as development cross-entropy decreases, BLEU exponentially increases.

Original: These two relationships can be combined to predict an effective USD-to-BLEU trade-off when annotating more data, even in low-resource regimes.
Revised: These two relationships can be combined to predict an effective USD-to-BLEU trade-off when annotating more data, even in low-resource regimes.

Original: A Parameter Scaling in 
Revised: A Parameter Scaling in Japanese-English Translation

Original: Japanese-English Translation

Original: In this section, we provide a brief retrostpectiveanalysis of the results of Zhang and Duh (2020),in which many MT systems were trained to evaluate the efﬁcacy of hyper-parameter optimizationtechniques.
Revised: In this section we provide a brief retrostpective analysis of the results of Zhang and Duh (2020), in which many MT systems were trained to evaluate the efﬁcacy of hyper-parameter optimization techniques.

Original: Speciﬁcally, we examine their resultson the Japanese-English WMT 2019 Robustnesstask (Li et al., 2019).
Revised: Speciﬁcally, we examine their results on the Japanese-English WMT 2019 Robustness task (Li et al., 2019).

Original: Figure 9 shows power-lawscaling of the development cross-entropy loss withthe number of non-embedding parameters.
Revised: Figure 9 shows power-law scaling of the development cross-entropy loss with the number of non-embedding parameters.

Original: 12 Wesee that changing the BPE encoding vocabularysize and the number of layers can affect the constant multiplier N C , but does not seem to affectthe exponent α N .
Revised: 12 We see that changing the BPE encoding vocabulary size and the number of layers can affect the constant multiplier N C , but does not seem to affect the exponent α N .

Original: Furthermore, multiple attention head settings (8, 16) were trained for each modelsize but they do not seem to impact scaling trends.
Revised: Furthermore, multiple attention head settings (8, 16) were trained for each model size but they do not seem to impact scaling trends.

Original: We exclude some outliers with unexpectedlylarge losses at for larger model sizes.
Revised: We exclude some outliers with unexpectedly large losses at for larger model sizes.

Original: This onlyoccurs for speciﬁc learning rates, so we believethose models failed to converge due to improperlearning rate tuning.
Revised: This only occurs for speciﬁc learning rates, so we believe those models failed to converge due to improper learning rate tuning.