E XAMINING S CALING AND T RANSFER OF L ANGUAGE M ODEL A RCHITECTURES FOR M ACHINE T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing.
In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs.
In this work, we thoroughly examine the role of several architectural design choices on the performance of LMs on bilingual, (massively) multilingual and zero-shot translation tasks, under systematic variations of data conditions and model sizes.
Our results show that: (i) Different
LMs have different scaling properties, where architectural differences often have a signiﬁcant impact on model performance at small scales, but the performance gap narrows as the number of parameters increases, (ii) Several design choices, including causal masking and language-modeling objectives for the source sequence, have detrimental effects on translation quality, and (iii) When paired with full-visible masking for source sequences, LMs could perform on par with EncDec on supervised bilingual and multilingual translation tasks, but improve greatly on zero-shot directions by facilitating the reduction of off-target translations.
I NTRODUCTION
The popularity of large, general-purpose text generation models has skyrocketed in recent years due to their outstanding performance across a wide range of natural language processing tasks (Brown et al., 2020; Raffel et al., 2020; Liu et al., 2020; Xue et al., 2021).
These studies, under large-scale pretraining and also model scaling, show the promise of moving to uniﬁed neural architectures and that dropping various task-speciﬁc inductive biases improves model generalization and/or performance (Devlin et al., 2019; Kaplan et al., 2020). In neural machine translation (NMT), one essential task-speciﬁc inductive bias is to separately handle source sentence understanding and target sentence generation with the encoder-decoder paradigm (EncDec).
Although such bias signiﬁcantly beneﬁts translation (Vaswani et al., 2017; Chen et al., 2018; Aharoni et al., 2019; Barrault et al., 2020; Ansari et al., 2020), some recent work shows insights challenging it, for example, aggressively simplifying the decoder yields little to no compromise on translation quality (Kasai et al., 2021). This thereby inspires the question how the removal of this separation, i.e. using a single uniﬁed module for both encoding and decoding (LMs), works for translation, and whether we can get any beneﬁts out of that.
Although some studies reported promising translation quality with LMs (He et al., 2018; Wang et al., 2021), they compare models under merely one conﬁguration (model size), neglecting that neural models follow scaling laws (Kaplan et al., 2020; Ghorbani et al., 2021; Gordon et al., 2021) where the impact of each added parameter on model performance might vary across different models.
How the inductive biases of LMs and EncDec impact the model’s performance as we increase their size and the amounts of training data are intriguing yet missing in the literature.
In this paper, we explore various conﬁgurations of LM architectures for translation as in Figure 1, and compare them with EncDec on model scaling and cross-lingual transfer.
We conduct a systematic study under a variety of data conditions, tasks (bilingual, multilingual and zero-shot) and model
[Figure]
x
[Figure]
xx
[Figure]
< s > yy
[Figure] [Figure]
PrefixLM CasualLM
[Figure]
Encoding x 1 x 2 x 3 < s
[Figure]
y 1 y 2 x
[Figure]
x 2 x
[Figure]
y
[Figure]
y [Figure] <
[Figure]
x 1 x 2 x [Figure] < s > y
[Figure]
y
[Figure]
capacities. We examine architectural design choices associated with LMs, including causal masking (CausalLM) vs. full-visible masking (PreﬁxLM) for source sequences, 1 layer-wise coordination (He et al., 2018) vs. ﬁnal-layer source encodings (TopOnly) for target sequence generation, increasing
LM depth vs. width, and also the effect of adding source language modeling loss for CausalLM.
We evaluate how these architectural priors affect translation quality as we increase the number of parameters available to the model, under a diverse set of bilingual translation settings and also (massively) multilingual settings, with a special focus on transfer to low-resource languages and zero-shot directions.
Our main ﬁndings are summarized below:
• LMs show different scaling properties compared to EncDec.
The architectural differences become less important as models scale, measured by reduced quality gap against EncDec, regardless of the language similarities, training data conditions and evaluation settings.
• PreﬁxLM variants often outperform their CausalLM counterparts; increasing LM depth beneﬁts the translation task more than increasing the width; and adding a source-side language modeling objective to CausalLM doesn’t affect translation quality.
• Cross-lingual transfer also beneﬁts from model scaling, where EncDec almost always dominates the quality Pareto frontier on supervised directions while zero-shot translation favors PreﬁxLM LMs.
PreﬁxLM LMs signiﬁcantly reduce off-target translations.
• Although LMs could reach or even surpass the translation quality of EncDec, they still lag far behind EncDec with respect to computational efﬁciency as measured in FLOPs.
R ELATED W ORK
Using language models in the task of translation has a long history, particularly in the era of statistical machine translation (SMT) where LM was used as a separate yet crucial component ensuring the ﬂuency of generation (Stolcke, 2002; Heaﬁeld, 2011; Koehn, 2010).
With neural networks,
NMT uniﬁed those isolated SMT components including LM under the encoder-decoder formulation (Kalchbrenner & Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015), which makes use of separate modules to process input and output.
Further studies exploring architectural modiﬁcations by using LM alone as a translation model, nevertheless, got much less attention.
He et al.
(2018) proposed layer-wise coordination between encoder and decoder with tied weights, where each decoder layer attends to its corresponding encoder layer at the same depth as opposed to the conventional method of attending the top-most encoder representations.
Later,
[Table]
Fonollosa et al.
(2019) extended it with locality constraint.
Dong et al.
(2019) explored LMs for sequence generation under large-scale pretraining.
Despite reporting promising results, these prior studies either focus only on bilingual tasks or do not consider the scaling properties of the models, leaving the picture incomplete: how the ﬁndings will change as we scale the models and how the languages beneﬁt from/interfere each other as the architectural priors (inductive biases) change.
Neural models follow some scaling laws.
Kaplan et al.
(2020) reported the test cross-entropy loss of LMs can be formulated as a power-law scaling function of either model size (excluding embedding parameters) or dataset size.
Later on, researchers examined and conﬁrmed such ﬁndings across different domains, including vision modeling (Zhai et al., 2021), knowledge transfer from pretraining (Hernandez et al., 2021), autoregressive generative modeling (Henighan et al., 2020)
[Table]
and neural machine translation (Gordon et al., 2021; Ghorbani et al., 2021), to name a few.
We ﬁnd it essential to study the scaling behavior of new architectures and approaches given the recent evidence on the emergent properties of the models at scale (Brown et al., 2020).
Another critical component in machine translation is the number of languages being considered with the models, which is the very focus of multilingual NMT (Firat et al., 2016).
Cross-lingual transfer in multilingual NMT often results from parameter sharing across languages, which beneﬁts low-resource languages and also enables zero-shot translation (Johnson et al., 2017), although the quality on zero-shot directions is largely hindered by the off-target translation problem (Arivazhagan et al., 2019; Zhang et al., 2020).
The structure of LMs further encourages parameter sharing, offering a chance to improve the transfer while magnifying the problem of interference (negative-transfer) (Wang et al., 2020; Zhang et al., 2021).
Very recently, Wang et al.
(2021) analyzed the cross-lingual transfer behavior of CausalLM, and reported encouraging zero-shot performance.
However, we did not observe the same results likely because of data sampling, model architecture and optimization differences which zero-shot transfer is sensitive to.
L ANGUAGE M ODEL A RCHITECTURES FOR T RANSLATION
In this section, we brieﬂy review EncDec and then present LM architectures for translation based on
Transformer (Vaswani et al., 2017).
Table 1 compares different models.
Given a source sequence
X of length | X | and its target translation Y of length | Y | , EncDec performs translation via the following structure:
[Equation]
where l denotes the layer index and ◦ indicates consecutive sublayers.
[Equation]
Y l ∈ R | Y |× d are the layer representations of the source and target sequence respectively, with a model dimension of d .
The ﬁrst input layer ( X 0 , Y 0 ) is the summation of token embeddings and their positional encodings.
We drop all the layer normalization and residual connections in our formulations for brevity.
The encoder is a stack of L layers, each of which includes a multi-head self-attention sublayer (SAtt) followed by a feed-forward sublayer (FFN).
SAtt in the encoder is bidirectional with fullvisible masking that has full visibility to all source tokens, preceding and following.
Its ﬁnal-layer representations X L are fed to the decoder, which shares a similar structure to the encoder but with [Table] an additional (multi-head) cross-attention sublayer (CAtt).
Unlike encoder, SAtt in the decoder is unidirectional with causal masking , where attention to following tokens is disabled (masked).
CAtt can always access all source inputs, though.
Note we set the encoder and decoder depth equally to
L [Table] and use d ff to denote the intermediate dimension of FFN.
EncDec is often optimized with the target translation objective based on Y L :
[Equation]
Rather than separately modeling source and target sequences
[Table]
LM handles both with a single module:
[Equation]
where M ∈ { 0 , 1 } ( | X | + | Y | ) × ( | X | + | Y | ) is the attention mask that controls the information ﬂow within the concatenated sequences ( [ · , · ] ).
2 Two LM variants explored by changing the structure of mask
M , PreﬁxLM and CausalLM .
PreﬁxLM merges different modules of EncDec, trained with L TGT .
Its attention mask
[Equation]
combines the encoder/decoder self-attention mask and the cross-attention mask of EncDec.
CausalLM , by contrast, is a strict LM that applies causal masking to both sequences:
[Equation]
Apart from L TGT , CausalLM also includes the source-side language modeling loss for training:
[Equation]
To improve our understanding of LMs for translation, we further incorporate two extensions:
PreﬁxLM + TopOnly The model deﬁned in Equation 3 performs attention over the source and target sequence within the same layer.
In contrast, EncDec always uses the topmost-layer source encodings for translation.
We mimic this with TopOnly extension by feeding toplayer encodings, i.e. X L instead of X l − 1 , to each attention sublayer.
It operates the same as EncDec but with the parameters of encoder and decoder tied.
CausalLM
+ TgtOnly The inclusion of the source-side objective enriches CausalLM’s learning signal and encourages the model to absorb source language characteristics.
However, it requires and occupies part of modeling capacity, which might negatively affect translation.
To offset this impact, we add the TgtOnly extension that optimizes CausalLM with the target translation objective L TGTC alone, which also aligns better with EncDec and PreﬁxLM.
S ETUP
Model Setting We use Transformer for experiments.
By default, we adopt the base setting, with d = 512 , d ff = 2048 and 8 attention heads.
We also work with the Transformer big setting where each hyper-parameter above is doubled.
Training and inference details are in Appendix A.
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure]
[Table]
[Figure]
Datasets and Evaluation We use WMT14 English-French (En-Fr), WMT14 English-German (En-De), WMT19 English-Chinese (En-Zh) and an in-house web-crawled (Web) En-De dataset for experiments, whose statistics are summarized in Table 2.
We also report results on OPUS100 (Zhang et al., 2020), a massively multilingual corpus containing 100 languages.
All datasets are pre-processed with byte pair encoding (Sennrich et al., 2016, BPE) implemented by SentencePiece (Kudo & Richardson, 2018).
We set the BPE vocabulary size to 32K by default.
We report test log-perplexity score (PPL) for scaling study particularly and also show SacreBLEU (Post, 2018) 3 .
E XPERIMENTS FOR M ODEL S CALING
Kaplan et al.
(2020) reported the model performance can be described with a power-law, with respect to its parameters, as below:
[Equation]
where L ( N ) ﬁts test PPL, and N denotes the number of parameters.
N 0 is a constant used for numerical stability which is obtained from 1-layer EncDec model.
α, p, L ∞ are ﬁtted parameters, and we mainly analyze the estimated scaling exponent p and the irreducible loss L ∞ for different models.
The way of increasing model parameters varies for the same model and also across different models.
We perform scaling ﬁrstly for EncDec by changing the depth L (from 1 to 26 layers, equally for its encoder and decoder) while keeping the other hyper-parameters intact following Ghorbani et al.
We then align the scaling settings of LM with its EncDec counterpart in term of model parameters through increasing either its depth or width:
• LM + Deep adds parameters via stacking more Transformer layers, which was also used in previous studies (He et al., 2018;
Wang et al., 2021). [Figure]
• LM
+ Wide , instead, grows the model width.
We choose to enlarge the feed-forward dimension from d ff to 3 d ff .
Note other strategies for width scaling are possible and many, but exploring them is resource-consuming and beyond the scope of our paper.
We distinguish data-limited regime from model size-limited regime for model scaling (Bahri et al., 2021), where the former has relatively fewer training samples than model parameters thus likely suffers from overﬁtting (e.g. with WMT14 En-Fr and WMT19 En-Zh), while the latter has enough samples for model ﬁtting (e.g. with Web En-De).
5. S CALING IN D ATA -L IMITED R EGIME
Architectural difference matters most when the model is at a small scale.
Figure 2 summarizes the scaling results on WMT14 En-Fr and WMT19 En-Zh.
When there are fewer parameters, the model with inductive biases favoring translation will achieve better quality.
Such inductive bias includes 1) allowing the full visibility to the source input as in PreﬁxLM 4 rather than causal masking; 2) using topmost-layer source encodings for translation ( TopOnly ) rather than layer-wise coordinated encodings; 3) deeper LMs ( Deep ) rather than wider; and 4) training LMs without source-side language modeling loss ( TgtOnly ).
The fact that LM + Deep outperforms LM + Wide shows that not only the number of parameters matters, but also the way parameters are added.
This aligns with the previous ﬁndings: deeper models apply more non-linear operations and induce more abstract representations, which often helps translation (Wang et al., 2019).
This also applies to TopOnly.
Most of these ﬁndings are consistent across different languages and evaluation metrics (Figure 2a-2d).
Different models show different scaling properties, but the gap narrows at scale.
The impact of added parameters on translation performance differs across different models.
The LMs performing poorly at small scale often gain more from the increased capacity.
For instance, the difference between LM + Deep and LM
+ Wide almost disappears at the end, resonating with the optimal depth-vs.-width theory (Levine et al., 2020).
We observe that PreﬁxLM and EncDec converge to a similar region, followed by CausalLM + TgtOnly while CausalLM still retains a clear gap against the others.
This performance gap on WMT19 En-Zh is smaller, mainly because of model overﬁtting.
BLEU scores in Figure 2c and 2d also show such a trend, although the relationship between BLEU and PPL is non-trivial (Ghorbani et al., 2021).
These tell us that the success of architectural modiﬁcations on small-scale models might not transfer to large-scale settings, and that comparing different models under one model conﬁguration might result in incomplete and misleading conclusions.
Note we also observe reduced gap when considering the number of layers (see Figure 9 in the Appendix).
Do sentence length and originality of test samples affect scaling properties?
Not much!
We further test how the scaling changes across different evaluation settings, and show the results on
WMT14 En-Fr in Figure 3.
The scaling exponent changes marginally over different settings (often less than 0.05), suggesting that the scaling curves are quite similar in these settings (see Figure 8, 10, 11 in Appendix), although sentences of different originalities differ largely in style and natural- [Figure] [Figure] ness (Graham et al., 2020; Freitag et al., 2020).
The estimated irreducible loss shows that targetoriginal parallel sentences are harder to model than the source-original ones, and that translating medium-length sentences is much easier.
The loss ranking of different models changes little over these settings, supporting PreﬁxLM and EncDec generally more than CausalLM.
Do LMs deal with computational efﬁciency better than EncDec?
No!
The FLOPs results in
Figure 4 show that EncDec demands generally less computation than LM, but the gap narrows at scale.
Note LM doesn’t save computations.
By contrast, to perform similarly to EncDec, LM often goes wider or deeper, which even deteriorates its running (training and decoding) efﬁciency.
Besides, EncDec allows arbitrary decoders, e.g. shallow decoders for faster inference, which is nonfeasible for LMs.
Figure 4 also shows that adding the source-side loss hurts CausalLM’s efﬁciency.
5. S CALING IN M ODEL S IZE -L IMITED R EGIME
Figure 5 shows the in-domain scaling performance on Web En-De.
Overall, we observe similar scaling patterns as discovered above, and such pattern transfers to out-of-domain evaluation, FLOPs and BLEU scores.
More results are available in the Appendix (Figure 12, 13 and 14).
E XPERIMENTS FOR C ROSS -L INGUAL T RANSFER
Based on the literature (Wang et al., 2020; Zhang et al., 2021), sharing capacity across languages could encourage knowledge transfer but might also gain the risk of negative interference.
In this section, we further compare different models but on multilingual many-to-many translation.
To enable multilingual NMT, we append a target language tag to each source sentence following Johnson et al.
We perform over-sampling to balance the training data with a temperature of T = 5 .
Do LMs facilitate the transfer to low-resource languages?
Not really!
We start with multilingual translation for WMT En-De/Fr/Zh, and regard En-De as a relatively low-resource language pair.
One reason behind the popularity of multilingual NMT is its transfer capability to low-resource languages.
We analyze this transfer behavior for LMs and explore transfer (to De) from similar (Fr) and [Figure] [Figure] l
[Figure] t i o n L a n g u a g [Figure] a c y
[Figure] distant (Zh) languages separately. Figure 6 shows the results. PreﬁxLM produces comparable results to EncDec, while CausalLM lags far behind, and the incorporation of source-side objective actually hurts translation.
Overall, we observe that EncDec almost dominates the transfer performance under different model sizes, regardless of language similarity.
Similar results are also observed for low-resource to high-resource transfer (see Figure 15 in the Appendix).
Do LMs beneﬁt zero-shot transfer?
PreﬁxLM does!
We further test how LMs perform on zeroshot translation.
We use the newstest2019 De-Fr test set as the in-domain zero-shot eval set, and an internal sports-domain N-way test set for De-Fr-Zh (2000 samples) as the out-of-domain eval
[Table]
set.
Figure 7 shows the results.
Scaling improves knowledge transfer for almost all models, while
PreﬁxLM performs surprisingly well on zero-shot directions.
In most settings, PreﬁxLM surpasses
EncDec signiﬁcantly with respect to BLEU, and such superiority is more obvious on out-of-domain evaluation and for distant language pairs.
Nevertheless, we ﬁnd that PreﬁxLM usually underperforms EncDec in terms of PPL.
In other words,
EncDec still possesses the best ﬁtting ability on zero-shot language pairs.
Results on translation language accuracy explains this mismatch: compared to EncDec, PreﬁxLM drastically reduces offtarget translation – a bottleneck of zero-shot translation (Zhang et al., 2020).
This also suggests that
EncDec suffers from more serious searching errors during inference (Stahlberg & Byrne, 2019), which the inductive biases of PreﬁxLM help.
In addition, we observe no beneﬁts from CausalLM on zero-shot translation, with or without the source-side language modeling objective.
This ﬁnding disagrees with that of Wang et al.
(2021), which we ascribe to various differences in model, data and optimization.
Note that Wang et al.
(2021) adopted more aggressive data oversampling, didn’t consider distant languages, proposed dedicated optimization with the source-side loss, used a different way to count model parameters, and designed different language tags for multilingual translation that could greatly affect zero-shot results (Wu et al., 2021).
We leave the study of these differences to the future.
How do LMs perform on massively multilingual translation?
We further examine the scalability of LMs with respect to the number of languages, and experiment on massively multilingual translation using OPUS-100.
We enlarge the BPE size to 64K to handle multilingual lexicons.
Following Zhang et al.
(2020), we divide the test language pairs into high-resource (High, > 0.9M), low-resource (Low, < 0.1M), and medium-resource (Med, others) groups, and report average scores for each group.
Table 3 summarizes the results.
EncDec outperforms LMs on supervised directions, with larger gap on low-resource languages and for XX → En translation.
By contrast, LMs, particularly PreﬁxLM, perform better on zero-shot directions, with improved translation language accuracy.
Overall, PreﬁxLM outperforms CausalLM, and also performs comparably to EncDec on supervised directions (often < − 1 BLEU on average), echoing with our above ﬁndings.
C ONCLUSION AND F UTURE W ORK
In this paper, we revisited language model architectures for machine translation in the perspective of model scaling and cross-lingual transfer.
Extensive experiments show that LMs often have different scaling properties where the impact of architectural differences gradually reduces as model scales up, and that LMs often deliver better zero-shot transfer than its EncDec counterpart with improved off-target translation although its cross-lingual transfer on supervised directions is suboptimal.
PreﬁxLM, the one with full visibility to the source input, shows consistent superiority to its
CausalLM counterpart, and performs similarly well to EncDec across different settings especially when paired with deep modeling.
These ﬁndings show that while current product offerings for major language pairs or small on-device models should continue using EncDec, LMs can be an effective architecture for giant multilingual models with zero-shot transfer as a primary focus.
We notice that the performance gap caused by architectural differences gradually disappears as the model size increases.
This has several implications for future researches: 1) Comparing NMT models in one model setting is not enough, particularly with the widely adopted 6-layer Transformer base setting, because of the scaling property difference.
We believe the best practice should portray the whole scaling picture for model comparison.
Just like NMT models optimized for high-resource translation transfer poorly to low-resource scenarios, many models developed in the past with claims outperforming Transformer might not transfer to large-scale model settings.
These studies ideally should be revisited in the face of model scaling. When models are compared in a single setting due to shortage of resources, best practice should state clearly which model size the conclusion is based on. 4) PreﬁxLM matches the performance of EncDec at large scale, and delivers promising zero-shot transfer, which deserves more efforts to study. This is also an encouraging signal for model uniﬁcation, i.e. supporting disparate tasks with a large enough yet single model (including the translation tasks), which is an exciting future direction. We also notice that LMs underperform EncDec signiﬁcantly on computational efﬁciency.
In the future, we will put more efforts on developing more efﬁcient networks for LM scaling.
We are also interested in exploring the applicability of translation-optimized LMs to downstream monolingual and cross-lingual tasks.
A M ODEL T RAINING AND I NFERENCE
We update model parameters via Adafactor (Shazeer & Stern, 2018) with label smoothing of value 0.1, and scheduled learning rate of warmup steps 40K.
We apply dropout of 0.1 to residuals, feedforward activations and attentions.
We employ the post-norm Transformer by default; for some exceptional cases (often with deep models where training is unstable) we use the pre-norm one instead.
Batch size is set to about 128K tokens.
We train models for up to 1M steps on different tasks, except Web En-De where 500K steps is used.
We average 10 checkpoints for evaluation.
For bilingual experiments, these checkpoints are selected according to the dev set performance; for multilingual experiments, we use the last 10 checkpoints.
Beam search is used for inference, with a beam size of 8 and length penalty of 0.
[Figure]
B M ORE E XPERIMENTAL R ESULTS
[Figure]
our test set according to source sentence length, and then split it into 8 disjoint groups.
This shows the results on the longest group.
[Figure]
C R ELATIVE VS . A BSOLUTE T RANSFER R ESULTS
[Figure]
[Figure] [Figure]
[Figure] [Figure]
En → De) as a function of model parameters.
Models are trained in the Transformer big setting
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]