I MPROVED T RAINING T ECHNIQUES FOR O NLINE N EURAL M ACHINE T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural sequence-to-sequence models are at the basis of state-of-the-art solutions for sequential prediction problems such as machine translation and speech recognition.
The models typically assume that the entire input is available when starting target generation.
In some applications, however, it is desirable to start the decoding process before the entire input is available, e.g . to reduce the latency in automatic speech recognition.
We consider state-of-the-art “wait- k ” decoders, that ﬁrst read k tokens from the source and then alternate between reading tokens from the input and writing to the output.
We investigate the sensitivity of such models to the value of k that is used during training and when deploying the model, and the effect of updating the hidden states in transformer models as new source tokens are read.
We experiment with German-English translation on the IWSLT dataset and the larger WMT15 dataset.
Our results signiﬁcantly improve over earlier state-of-the-art results for German-English translation on the WMT15 dataset across different latency levels.
I NTRODUCTION
Sequence-to-Sequence (S2S) models are state-of-the-art for tasks where source and target sequences have different lengths, including automatic speech recognition, machine translation, speech translation, text-to-speech synthesis, etc .
The most common models are composed of an encoder that reads the entire input sequence, while a decoder (often equipped with an attention mechanism) iteratively produces the next output token given the input and the partial output decoded so far.
While these models perform very well in the typical ofﬂine decoding use case, few studies consider how S2S models are affected by low-latency constraints, and which architectures and strategies are the most efﬁcient.
Low-latency decoding is desirable for applications such as online speech recognition, and as-you-type machine translation.
In such scenarios, the decoding process starts before the entire input sequence is available, and the output sequence is produced in an on-the-ﬂy manner.
However, if we consider for instance machine translation, online prediction generally comes at the cost of reduced translation quality and more research is needed to reach the grail of natural and high-quality online speech-to-speech interpretation.
In this paper we consider deterministic “wait- k ” decoders that are state of the art for low-latency decoding (Ma et al., 2019; Zheng et al., 2019b).
These decoders ﬁrst read k tokens from the source, after which they proceed to alternatingly produce a target symbol and read another source symbol.
We compare two architectures to implement such models: one based on a 2D-convolutional sequence-to-sequence model (Elbayad et al., 2018), and one based on the attention-based transformer architecture (Vaswani et al., 2017).
For these models, we investigate the impact of the choice of k when training the models, and when using them to generate translations.
For the transformer model, we also consider the effect of updating the hidden states of previous target symbols based on the full source context that is available at any moment.
These updates are inspired from the 2D convolutional model, where such “updates” are an inherent consequence of the architecture.
In summary, our contributions are the following:
We compare transformer and 2D convolutional architectures for online machine translation.
We propose improved training techniques for wait-
k by ﬁrst using uni-directional encoders and training across multiple values of k
We boost the translation quality in test-time by updating the preﬁx hidden-states in transformer decoders.
We set a new state of the art for online translation on WMT15 German-English, improving over the recent state-of-the-art results by Ma et al.
and
Zheng et al.
(2019b)
across a full range of latency levels.
The rest of this paper is organized as follows:
Section 2 presents related work on low-latency machine translation; Section 3 details our low-latency models, which we evaluate in Section 4 for
German-English machine translation on the IWSLT14 dataset as well as the larger WMT15 dataset.
Section 5 concludes the paper.
R ELATED WORK
In order to study differences between translated and simultaneously interpreted text, He et al.
(2016a) produced a parallel corpus between both and showed that human interpreters regularly apply several tactics to reduce translation latency, including sentence segmentation and passivization.
In machine translation, after pioneering work from Fügen et al.
, Yarmohammadi et al.
and
He et al.
(2015) proposed methods to increase the alignment monotonicity for statistical machine translation.
To allow for a more ﬂexible segmentation, Grissom et al.
(2014) introduced a trainable segmentation module that decides whether to write or read tokens based on the prediction of the ﬁnal verb and the next token in the source sequence.
Similarly, Oda et al.
(2015) use output of a syntax-based statistical translation system to ﬁnd the optimal segmentation strategy that maximizes the quality of the translations via greedy search and dynamic programming.
One of the ﬁrst works on online translation to use attention-based sequence-to-sequence models is that of Cho & Esipova (2016), which uses manually designed non-trainable waiting criteria that dictate whether the model should commit to a read/write operation.
The neural transducer of Jaitly et al.
(2016) reads equally-sized chunks of the source sequence and generates output sub-sequences of variable lengths, each ending with a special token marking the end of the writing.
For training, a single segmentation is chosen to optimize the likelihood of the full output sequence.
Raffel et al.
(2017) propose an alternative to attention using monotonic alignments.
Whereas attention requires access to the full source sequence to compute the weights, monotonic alignments enable linear time computation of the weights and online decoding.
Another line of research treats general alignments of source and target sequence as a latent variable.
Inspired from the HMM word alignment model used in statistical machine translation (Vogel et al., 1996), the segment-to-segment neural transducer model of Yu et al.
(2016) integrates a latent alignment variable into an LSTM-based sequence-to-sequence model where the transition probabilities are conditioned on the encoder-decoder hidden states.
During training the alignment is marginalized out with a forward-backward algorithm (Rabiner, 1989).
However, their approach is used as an alternative to attention-based models, not for low-latency translation.
Luo et al.
(2017) introduce a recurrent sequence-to-sequence model with binary stochastic decision variables to either emit the next output token or advance in encoding the source sequence with a unidirectional LSTM that jointly encodes the partially generated output sequence.
The stochastic decisions are optimized using a standard policy gradient reinforcement learning approach.
Gu et al.
(2017) propose a trainable agent emitting read/write decisions modeled as a recurrent neural network fed with the encoder and decoder current hidden states.
In their framework, a left-to-right recurrent sequence-to-sequence model is ﬁrst pre-trained on the full bi-texts and then ﬁne-tuned with policy gradient to optimize a reward balancing the quality and the latency of the translations.
Dalvi et al.
(2018) used a static decoding algorithm that starts with k read operations then alternates between blocks of l write operations and l read operations.
Albeit simple, this approach outperforms the information based criteria of Cho & Esipova (2016) and allows for complete control of the translation delay.
Their attempt to integrate incrementality in the training by pre-aligning the sourcetarget sequences and then attending over a constrained span of source positions failed to improve the translation quality.
The work of Press & Smith (2018), although not tackling the simultaneous translation task, allows for emitting target tokens before reading the full input sequence.
Similar to the incremental training of Dalvi et al.
, it requires pre-aligning the bitexts.
More recently, Ma et al.
(2019) trained a sequence-to-sequence model based on the transformer architecture (Vaswani et al., 2017) with an integrated “wait- k ” agent that ﬁrst reads k source tokens then alternate single t z
j ă s ą 135 2 3 4 5 6 7k =
t z j ă s ą 135 2 3 4 5 6 7k =
t
z
j ă s ą 135 2 3 4 5 6 7k =
t z j ă s ą 135 2 3 4 5 6 7k = read-writes, similar to Dalvi et al.
but using l “ 1 .
Wait- k approaches, were found most effective by Zheng et al.
(2019b) when trained for the speciﬁc k that is used to generate translations.
This, however, requires training separate models for each potential value of k used for translation.
As an alternative, Zheng et al.
(2019a;b) both learn an adaptive policy to produce read/write decisions in a pre-trained ofﬂine translation model.
Zheng et al.
(2019a) train the model for decoding along two wait-
k paths and Zheng et al.
(2019b) use supervised training based on an oracle read/write sequence derived from the pre-trained ofﬂine translation model.
Different latency levels are achieved by thresholding the policy’s conﬁdence, however, on most regimes their model performs worse than using transformer models directly trained for
wait-
k online translation.
In our work, we also use wait-
k decoders, but unlike Ma et al.
; Zheng et al.
(2019a;b) we opted for uni-directional encoders and show that they are efﬁcient to train in an online setup and can achieve better results if we update the preﬁx decoder states during inference.
L OW - LATENCY SEQUENCE - TO - SEQUENCE MODELING
Let p x , y q be a pair of source-target sequences of respective lengths | x | and | y | .
Low-latency de- coding consists of executing a sequence of interleaved reading and writing operations, consuming tokens from the source x and producing tokens of the target y .
Low-latency decoding paths can be represented on a | y | ˆ | x | grid, where we advance from the top left to the bottom right in a total of | x | read steps and | y | write steps.
See Figure 1 for an illustration.
We consider two recent sequence-to-sequence models for low-latency decoding: the transformer architecture of Vaswani et al.
(2017) and the 2D convolutional architecture of Elbayad et al.
We describe in Section 3.1 how each of these architectures is adapted for the task of simultaneous translation.
In Section 3.2 we describe how these architectures can be trained for online decoding, based on one or more decoding paths.
3. L OW - LATENCY DECODING ARCHITECTURES
To formalize the low-latency decoding process we model it as a sequence of | y | write steps.
At step t P t 1, . . .
, | y |u we decode the t -th target token y t , conditioning on the target preﬁx y ă t and the source sequence read up that point x ď z t .
The value of z t is dictated by the decoding path.
The decoder computes the distribution over the next target token given the source and target contexts,
[Equation]
(1) where θ denotes the parameters of the writer.
It depends on the architecture of the decoder whether, given z t , the prediction of y t further depends on the full decoding z ă t path that led up to this point.
3.1. C ONVOLUTION - BASED LOW - LATENCY DECODER
In order for the decoder to be trained efﬁciently for different source/target context sizes, we build upon the “pervasive uses masked 2D convolutions across a source-target grid, as in Figure 1.
The masked convolutions ensure that only information from past target tokens is used to predict the next one.
Here we adapt the masking pattern so as to construct the ﬁeld-of-view in a way that it only extends over previous positions in both source and target dimensions.
More concretely, we use a 11 ˆ 11 ﬁlter where only the top-left 6 ˆ 6 weights are non-zero.
In this manner features computed by a sequence of convolutional layers at a position p t , j q in the grid can be used to deﬁne p θ p
y t | y ă t , x ď j q .
As input, at each site p t , j q , the network takes the concatenation of the (sub-)word embedding of the t -th token in target and the j -th token in the source sequence.
The input is then progressively transformed along a number of convolutional network layers.
Let H “ p h tj q for 1 ď t ď | y | andď j ď | x | be the ﬁnal features of the masked convolutional neural network.
Elbayad et al.
(2018) apply max-pooling across the source dimension to map H to a ﬁxed-sized vector for each target position.
To deﬁne p θ p
y t | y ă t , x ď j q in our low-latency model, we use
[Equation]
where Pool p¨q pools H ď t , ď j in the source and/or target direction, or neither in which case f tj “ h tj .
Preliminary experiments with the pervasive attention architecture showed that max-pooling across the source dimension only is the best option.
For the remainder of this paper we have:
[Equation]
From there we generate the emission probabilities by linearly projecting the feature f tj to the dimension of the output vocabulary with the matrix W , followed by a soft-max normalization:
[Equation]
Due to the 2D convolutional structure of the network, the path leading up to position p t , j q in the network does not impact the prediction of y t .
Therefore, we drop the dependence on z ă t in Eq.
3.1. A TTENTION - BASED LOW - LATENCY DECODER
The second online decoder we consider is based on the “transformer” model of Vaswani et al.
Both the encoder and decoder consist of a stack of blocks.
Each block consists of a multi-head selfattention followed by a position-wise fully connected feed-forward network, or equivalently a 1 ˆconvolution.
The decoder uses an additional multi-head attention block that ranges over the source token encodings.
Given a query vector q t , an attention component aggregates a set of value vectors v j in a weighted sum, based on scoring a corresponding set of key vectors k j against the query.
Using the dot-product as the score, we obtain the attention aggregate a t as
[Equation]
The self-attention block is designed such that only previous output tokens can be attended over, since future tokens are not available when generating targets, ensuring the decoder is autoregressive (AR).
In our work, the transformer model for online translation has the following properties:
A uni-directional encoder: we make the self-attention in the encoder autoregressive, so that it can encode all source preﬁxes in parallel during training, and encode them progressively during generation.
This is achieved by subsituting the STACL mask in Eq.
(6) with the AR mask of Eq.
A constrained encoder-decoder interaction: we mask the source attention in the decoder, so that when producing the t -th output token attention is limited to the z t source tokens read so far.
This is guaranteed with the enc-dec mask
of Eq
[Equation]
Cached preﬁx ... hp z 2 q hp z 1 q h t ´p z t ´ 2 q y t ´h t ´p z t ´ 1 q
Self-attention (a)
Updated preﬁx ... hp z t ´ 1 q hp z t ´ 1 q h t ´p z t ´ 1 q y t ´h t ´p z t ´ 1 q
Self-attention
[Figure]
In this transformer-based decoder, the prediction of y t depends on the full decoding path z
ă t .
This is because the decoder self-attention ranges over decoder hidden states of previous time steps t 1 ă t , which themselves used z t 1 to attend over the source encoding.
In addition to the transformer-decoder described above, we consider a second transformer-based variant which removes the dependency of y t on z ă t .
To achieve this, each time a source token is read, we update all hidden states in the decoder based on the source context x z t available at that point.
Essentially, we will re-run the decoder across the entire output sequence, using the new source context.
This makes the decoder self-attention insensitive to z ă t given z t , increasing the decoding cost from O p| x | ˆ | y |q to O p| x | 2 ˆ | y |q .
Note that the encoder still needs to be run only once, since its self-attention has been constrained to be autoregressive.
We refer to this decoder as “transformer-update”, see Figure 2 for an illustration.
3. T RAINING LOW - LATENCY DECODERS
For our decoders we use pre-deﬁned “wait- k ” schedules (Dalvi et al., 2018; Ma et al., 2019).
First k source tokens are read, before alternating reading and writing a single token at a time, until either the full source has been read, or the target generation has been terminated.
To train our models, we consider several strategies.
The ﬁrst, similar to Ma et al.
(2019) is to train the model using the same wait-
k decoding path that will be used for generation.
That is, to train the model, we optimize a sum of loss terms, each of which corresponds to the negative log-likelihood of predicting a target token:
[Equation]
Training with k “8 corresponds to an ofﬂine (wait-until-the-end) decoder, with the exception that here we still use autoregressive dependencies in the encoder or along the source dimension in the 2D convolutional network.
In addition to training according to a single wait-
k decoding path, we can use losses associated with multiple paths.
The additional loss terms may provide a richer training signal, and potentially yield models that could perform well across a range of values for k during decoding.
Due to the dependence of y t on the full decoding path z ă t in the transformer-based model, it is not possible to improve over simply training in parallel across the different values of k .
When training for multiple values of k , we encode the source sentence once, and then forward it multiple times in the decoder, once for each value of k .
The 2D CNN-based architecture, however, does not include this full dependency, and allows parallel computation of all p θ p
y t | y ă t , x ď j q in single evaluation of the CNN.
For this architecture we therefore use a loss of the form
[Equation]
where q tj weight the terms corresponding to different positions in the decoding grid of Figure 1.
The weights q tj can be set to zero/one to reﬂect all positions corresponding to wait- k decoding paths for one or more values of k .
In our experiments we also consider the option of activating all loss terms above a certain wait- k decoding path.
The latter corresponds to the sum of all loss terms on all wait- k 1 decoding paths with k ď k 1 ď | x | .
The transformer-update architecture also avoids the full dependency on z ă t , but in order to train from all loss terms above a certain decoding path, it requires a separate decoder run for each source context size, i.e . for each column of the decoding grid in Figure 1.
E XPERIMENTAL EVALUATION
In this section we present our experimental setup, followed by quantitative results.
4. D ATASETS AND EXPERIMENTAL SETUP
Datasets.
We evaluate our approach on IWSLT14 En-De (Cettolo et al., 2014) and on the WMT
De-En datasets.
2 Replicating the setup of Elbayad et al.
(2018) for IWSLT
De-En, we train on 160K sentence pairs, develop on 7K held-out pairs and test on 6.7K pairs.
The vocabulary consists of 8.8K types on the source side and 6.6K types on the target side that were obtained from a joint source and target byte pair encoding (BPE; Sennrich et al. 2016).
For WMT15 De-En we use a joint vocabulary of 32k BPE types.
We train on 4.5M pairs, develop on newstest2013 (3K pairs) and test on newstest15 (2.2K pairs).
Evaluation metrics.
We use beam-search decoding, with a beam of size 5, for ofﬂine models and only greedy decoding online.
We evaluate the translation quality by measuring case-sensitive tokenized BLEU (Papineni et al., 2002) with multi-bleu.pl .
3 Cho & Esipova (2016) measure the decoding latency using the proportion of the source sentence which has been read when producing target tokens, and average this proportion across the tokens in the generated sentence:
[Equation] [Equation]
More recently, average lagging (AL) and differentiable average lagging (DAL), were proposed to measure translation latency (Ma et al., 2019; Cherry & Foster, 2019; Arivazhagan et al., 2019):
[Equation]
where γ “ | y |{| x | and τ “ min t t | g p t q “ | x |u .
These metrics handle differing source and target lengths more properly, and have an intuitive interpretation as the average by which the system lags behind an ideal instantaneous translator.
We refer the reader to (Cherry & Foster, 2019) for details.
We report the mean of these metric across translations.
In this section we report the average lagging (AL) metric, while the other metrics (AP, DAL, leading to a similar trend) are given in Appendix B.
AL is chosen to make our results comparable with those of Ma et al.
and
Zheng et al.
(2019b) for German-English WMT15 dataset.
Architectures.
For the pervasive attention model, to reduce the memory footprint, we use residual skip connections (He et al., 2016b) rather than the dense connections (Huang et al., 2017) used by Elbayad et al.
See Appendix A for more details about other minor changes made to the original architecture.
We consider an ofﬂine pervasive attention baseline (PA) that reads the source sequence bi-directionally via asymmetric convolution ﬁlters that are only masking the future targets.
Note that this baseline is not directly applicable for low-latency decoding, due to the bidirectional source encoding.
We use a second baseline where convolutions are masked along the source dimension, which we refer to as masked-PA baseline (MPA).
For the Transformer model, we use a small architecture on IWSLT with an embedding dimension d enc “ 512 for the encoder, d dec “ 256 for the decoder, and N “ 6, d ﬀ “ 1024, h “ 4, P drop “ 0.3 .
On WMT, we use a Transformer base (Vaswani et al., 2017) with tied embeddings.
Similar to the pervasive attention baseline, we consider the ofﬂine baseline (T) with bidirectional self-attention in the source side and a masked baseline (MT) with left-to-right self-attention in the encoder.
[Table]
[Figure] [Figure]
O FFLINE DECODING BASELINE RESULTS
In Table 1 we report the ofﬂine performance of the Pervasive Attention (PA), Masked Pervasive
Attention (MPA), Transformer (T) and Masked Transformer (MT) baselines.
Each with greedy decoding (G) and with beam search (BS).
Interestingly, the masked and non-masked versions perform quite similar, across both datasets and architectures.
This validates the use of autoregressive source encoders in the low-latency models we consider below.
The transformer and convolutional architectures perform similarly on IWSLT14, while the former performs signiﬁcantly better on WMT15.
4. O NLINE DECODING RESULTS
In our ﬁrst experiment we evaluate the wait- k online decoding for different MT and MPA models, trained for different wait-
k decoding paths.
We denote with k “8 the wait-until-end training where the full source is read before decoding.
In each ﬁgure the ofﬂine results are added for reference, the ofﬂine model has a latency of AL “ | x | .
Here we show results for IWSLT De-En, similar trends are observed for IWSLT En-De (see Appendix B.3).
Impact of the architecture.
Figure 3 presents the performance of models trained for wait-
k decoding across a range of latencies k eval P t 1, 3, 5, 7, 9 u .
Each trained model is represented by a curve, by evaluating it across different wait-
k decoding paths.
For the convolutional architecture we observe that a single model trained for a relatively large value of k , e.g . 7 or 9, provides good performance when using it to generate using different k values for generation.
The performance of the transformer-based model is more sensitive to the value of k used for training.
This difference in sensitivity could be due to the use of input-independent convolution ﬁlters vs .
self-attention which relies on scoring context to weight its features.
Given an appropriate value for k during training, the transformer-based models yield better translations for a given latency level.
Averagelagging
B L E U
k “ 1 k ě 1 k “ k eval k ě k eval       32 Ofﬂine (a) MPA above path
Averagelagging
B L E U
k “ 1 k “ 3 k “ 5 k P r 1..5 s       32 Ofﬂine (b) MT k P r 1..5 s
Averagelagging
B L E U k “ 9 k P r 1..5 s k
P r 1..9 s 2318     32 Ofﬂine (c
[Figure] [Figure]
s
Training
Update k eval “k eval “k eval “k eval “k eval “k “w/o 22.
k “with 24.
k “w/o 22.
k “with 24.
k P r 1..5 s w/o 22.
[Table]
k P r 1..5 s with
[Table]
k P r 1..9 s w/o
[Table]
k P r 1..
[Table]
with
[Table]
Impact of multiple path training.
For the convolutional model (MPA), we consider activating loss terms in the area above the wait- k path.
Training on this area covers a combinatorially large number of paths with latency higher than k .
The results in Figure 4a show that for the MPA architecture, multiple path training can be beneﬁcial compared to single path ( k “ k eval vs. k ě k eval ).
More importantly, the model trained with all loss terms above the diagonal of the decoding grid ( k ě 1 ), performs equal or better than models trained with only a subset of the terms, across all tested latency levels.
For transformer model (MT), training for all decoding paths above a certain wait- k path is prohibitively costly (since the cost is linear in the number of training paths).
Therefore, we opted instead for joint training for a selected set of wait-
k paths.
Results in Figure 4b show that jointly training on the wait- k paths from k “ 1 to k “ 5 improves over the training on individual paths in lowlatency regimes ( k eval ď 3 ).
For higher latency a relatively small drop in performance is observed compared to training with k “ 5 .
Where the convolutional model could be trained near optimal by training across all values of k , in
Figure 4c we see that training the paths from k “ 1 to k “ 9 performs similarly to just training with k “ 9 .
The latter model is (near) optimal for AL ě 3 , but for very low-latency the model trained on k P r 1..5 s is better.
Impact of hidden state updates.
The standard decoding approach with a Transformer is to use previously evaluated hidden states to encode the target preﬁx written so far.
This is not the case for MPA where new source context is integrated in the convolution for all previous time-steps.
To bridge the gap between the two decoding paradigms, we update previous hidden states to account for new source contexts.
In Table 2 we compare transformer models with and without hidden state updates, trained along one or multiple decoding paths.
Among the 20 comparisons (4 models, and 5 wait-
k decoders)
we
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
observe only a single case where state updates drop the BLEU score by 0.1.
In all other cases, the state updates lead to improvements, by up to 2.4 BLEU points in very low latency settings.
Experiments on the WMT15 De-En corpus.
In Figure 5a we present our results on the WMT
De-En task using transformer model, which gave the best results in the IWSLT experiments.
The results conﬁrm observations made on IWSLT: (i) Hidden state updates consistently improve the performance of the tested models.
(ii) When using hidden state updates, a single low-latency model ( k “ 7 ) performs better or comparable to using separate models trained speciﬁcally for the value of k used for decoding (k eval ).
The main differences w.r.t.
the IWSLT experimental results are observed for the model trained for ofﬂine decoding ( k “ 8 ).
Whereas on IWSLT it was competitive with the best models trained for online decoding, for this larger corpus with longer sentences, this is no longer the case and the ofﬂine model performs signiﬁcantly worse than the ones speciﬁcally trained for online decoding.
Comparison to state-of-the-art.
Figure 5b compares our results with state-of-the-art performance reported by Ma et al.
and
Zheng et al.
(2019b) for German-English translation on the WMT dataset.
Our model ( k “ 7 , with update) establishes a new state of the art for this task, signiﬁcantly improving over previous work across the full range of latency levels.
It is important to note that to obtain our results we use a single trained model regardless of the latency level considered at decoding time, like Zheng et al.
(2019b)
but unlike Ma et al.
Moreover, we use simple wait-
k decoders to attain different latency levels.
Zheng et al.
(2019b), on the other hand, trained a data adaptive controller to decide on the read/write actions, but used an underlying decoder trained for ofﬂine translation.
Given our results, we expect that further gains are possible by training controllers on top of our model, which is trained speciﬁcally for online decoding.
Training time.
We compare in Table 3 the training time of our models to the training time of the baseline and to that of our implementation of the original wait-k paper (Ma et al., 2019).
If training on a wait- k path, we report the average training time of k P t 1, 3, 5, 7, 9 u .
Updating either the encoder states to use bi-directionally or the preﬁx decoder states increases the training time dramatically.
In fact, when training along the wait- k path, the encoder update scales the encoder forward-passing time by min p| x | , | y |q ´ k ` 1 to produce states for each context size.
Similarly, the decoder update increases the decoder forward-passing time by the same factor to re-compute the states with growing context.
Decoding speed.
We compare in Table 4 the decoding speed of our models with and without updating the decoder states.
We also include in the comparison the decoding speed if we update the encoder states as we advance in reading the source sequence, similar to Ma et al.
To factor out the training approach, we only consider the models trained for ofﬂine decoding (MT and T of
[Table]
Table 1) and evaluate them on the wait- k paths.
The reported speeds are the average of evaluating with k P t 1, 3, 5, 7, 9 u .
[Table]
with and without updating the decoder states.
The decoding speeds show that updating the preﬁx decoder states costs less than updating the encoder states with a bi-directional encoder on both GPU (IWSLT: ˆ 1.8 , WMT: ˆ 1.2 speed-up) and
CPU (IWSLT: ˆ 1.8 , WMT: ˆ 1.3 speed-up).
C ONCLUSION
We compared transformer and 2D convolutional architectures for online machine translation with “wait- k ” decoders, and proposed improved training techniques for these models.
We ﬁnd the transformer architecture to perform best, and improved its performance using hidden state updates inspired from the 2D convolutional architecture.
We ﬁnd that training a single model for relatively high values of k , e.g . 7 or 9, yields a performance that is comparable to using separate models trained for each speciﬁc value of k .
Training a single model for multiple values of k improves performance for very low-latency regimes.
Our results establish a new state of the art for online machine translation on WMT15 De-En dataset.
We improve over recent results obtained with “wait- k ” decoders (Ma et al., 2019) and trained dataadaptive controllers (Zheng et al., 2019b) to schedule read/write actions.
Our results, together with those of Zheng et al.
(2019b), suggest that further performance improvements are possible by integrating our models trained for online decoding with data-adaptive controllers.
A D ETAILS ON THE CONVOLUTIONAL ARCHITECTURE
In this section we provide more details on the modiﬁcations we made to the convolutional “pervasive attention” architecture of Elbayad et al.
In Figure 6 we provide a schematic overview of the original convolutional architecture of Elbayad et al.
[Figure]
In our work
[Figure]
we made the following modiﬁcations to reduce the memory footprint of the model and to improve its performance.
We added feed-forward 1 ˆ 1 convolutional layers after the masked convolutions for their important role in existing encoder architectures as they boost the representational power of the model.
We substituted the dense layer connections with residual ones to reduce the size of the features that keep increasing and costing more memory.
We opted for layer-normalization (Ba et al., 2016) instead of batch-normalization as it is more stable and more appropriate for causal sequences.
We use depth-wise separable convolutions for Conv2D(k) instead of ordinary convolutions.
In Figure 7 we provide a schematic overview of the adapted convolutional architecture.
[Figure]
Finally, we found that removing the layer normalizations all-together and summing the block outputs before projecting on the target vocabulary to give best results.
The ﬁnal high-level architecture with block output addition is given in Figure 8.
[Figure]
B A DDITIONAL EVALUATION RESULTS
In this appendix we provide additional evaluation results.
In addition to results in the main paper measuring latency with AL, in Appendix B.1 we provide results measuring latency using the AP and DAL metrics.
In Appendix B.2 we provide the numerical underlying the plots for IWSLT De-En in the main paper and appendix.
Finally, we provide results for the reverse En-De translation direction for both IWSLT in Appendix B.3 and WMT in Appendix B.4.
B. IWSLT D E -E N : AP AND DAL LATENCY
[Table] [Table]
[Table] [Table]
[Table] [Table]
Averageproportion
[Table]
U
[Table] [Table]
[Table] [Table]
(a)
[Table]
Averageproportion
[Table] [Table]
) MPA above path
[Table] [Table]
[Table] [Table]
[Table] [Table]
(c) MPA
[Table]
k
[Table]
DAL)
[Table] [Table]
[Table] [Table]
[Table]
32 Ofﬂine (d) MPA above path (DAL) 0.
[Figure]
k “ 9 k “        32 Ofﬂine (e) MT wait-k (AP) 0.
Averageproportion
B L E U
k “ 1 k “ 3 k “ 5 k P r 1..5 s        32 Ofﬂine (f) MT k P r 1..
5 s (AP) 0.
Averageproportion
[Table]
U k
“ 9 k P
[Table] [Table]
.5 s k
P r
[Table]
9 s        32 Ofﬂine (g) MT k P r 1..9 s (AP)
Differentiableaveragelagging
B L E U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “        32 Ofﬂine (h) MT wait-k (DAL)
Differentiableaveragelagging
B L E U
k “ 1 k “ 3 k “ 5 k P r 1..5 s 23      32 Ofﬂine (i) MT k P r 1..5 s (DAL)
Differentiableaveragelagging
B L E U k “ 9 k P r 1..5 s k
P r 1..9 s
[Figure]
[Table] [Table]
Averageproportion
B L E U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “     26 Ofﬂine (a) MPA wait-k (AP)
Averagelagging
B L E U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “22    26 Ofﬂine (b) MPA wait-k (AL)
Differentiableaveragelagging
B L E U
k “ 1 k “ 3 k “ 5 k “ 7 k “ 9 k “22    26 Ofﬂine (c) MPA wait-k (DAL) 0.
Averageproportion
B L E U k “
[Table]
ě 1 k “
[Table]
eval      26 Ofﬂine (d) MPA above path (AP)    8
Averagelagging
B L E U
k “ 1 k ě 1 k “ k eval k ě k eval 22    26 Ofﬂine (e) MPA above path (AL)
Differentiableaveragelagging
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]