Original: P AY LESS ATTENTION WITH L IGHTWEIGHT AND D YNAMIC C ONVOLUTIONS
Revised: P AY LESS ATTENTION WITH L IGHTWEIGHT AND D YNAMIC C ONVOLUTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-attention is a useful mechanism to build generative models for language and images.
Revised: Self-attention is a useful mechanism to build generative models for language and images.

Original: It determines the importance of context elements by comparing each element to the current time step.
Revised: It determines the importance of context elements by comparing each element to the current time step.

Original: In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results.
Revised: In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results.

Original: Next, we introduce dynamic convolutions which are simpler and more efﬁcient than self-attention.
Revised: Next, we introduce dynamic convolutions which are simpler and more efﬁcient than self-attention.

Original: We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements.
Revised: We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements.

Original: The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic.
Revised: The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic.

Original: Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models.
Revised: Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models.

Original: On the WMT’14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.
Revised: On the WMT’14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There has been much recent progress in sequence modeling through recurrent neural networks (RNN; Sutskever et al. 2014;
Revised: There has been much recent progress in sequence modeling through recurrent neural networks (RNN; Sutskever et al. 2014;

Original: Bahdanau et al.
Revised: Bahdanau et al.

Original: ; Wu et al. 2016), convolutional networks (CNN;
Revised: ; Wu et al. 2016), convolutional networks (CNN;

Original: Kalchbrenner et al. 2016; Gehring et al. 2016; 2017; Kaiser et al. 2017) and self-attention models (Paulus et al., 2017; Vaswani et al., 2017).
Revised: Kalchbrenner et al. 2016; Gehring et al. 2016; 2017; Kaiser et al. 2017) and self-attention models (Paulus et al., 2017; Vaswani et al., 2017).

Original: RNNs integrate context information by updating a hidden state at every time-step, CNNs summarize a ﬁxed size context through multiple layers, while as self-attention directly summarizes all context.
Revised: RNNs integrate context information by updating a hidden state at every time-step, CNNs summarize a ﬁxed size context through multiple layers, while as self-attention directly summarizes all context.

Original: Attention assigns context elements attention weights which deﬁne a weighted sum over context representations (Bahdanau et al., 2015;  Chorowski et al., 2015; Luong et al., 2015).
Revised: Attention assigns context elements attention weights which deﬁne a weighted sum over context representations (Bahdanau et al., 2015; Sukhbaatar et al., 2015; Chorowski et al., 2015; Luong et al., 2015).

Original: Source-target attention summarizes information from another sequence such as in machine translation while as self-attention operates over the current sequence.
Revised: Source-target attention summarizes information from another sequence such as in machine translation while as self-attention operates over the current sequence.

Original: Self-attention has been formulated as contentbased where attention weights are computed by comparing the current time-step to all elements in the context (Figure 1a).
Revised: Self-attention has been formulated as content-based where attention weights are computed by comparing the current time-step to all elements in the context (Figure 1a).

Original: The ability to compute comparisons over such unrestricted context sizes are seen as a key characteristic of self-attention (Vaswani et al., 2017).
Revised: The ability to compute comparisons over such unrestricted context sizes are seen as a key characteristic of self-attention (Vaswani et al., 2017).

Original: [Figure]
Revised: [Figure]

Original: However, the ability of self-attention to model long-range dependencies has recently come into question (Tang et al., 2018) and the unlimited context size is computationally very challenging due to the quadratic complexity in the input length.
Revised: However, the ability of self-attention to model long-range dependencies has recently come into question (Tang et al., 2018) and the unlimited context size is computationally very challenging due to the quadratic complexity in the input length.

Original: Furthermore, in practice long sequences require the introduction of hierarchies (Liu et al., 2018).
Revised: Furthermore, in practice long sequences require the introduction of hierarchies (Liu et al., 2018).

Original: In this paper, we introduce lightweight convolutions which are depth-wise separable (Chollet, 2017;
Revised: In this paper, we introduce lightweight convolutions which are depth-wise separable (Chollet, 2017;

Original: Kaiser et al., 2017), softmax-normalized and share weights over the channel dimension.
Revised: Kaiser et al., 2017), softmax-normalized and share weights over the channel dimension.

Original: The result is a convolution with several orders of magnitude fewer weights than a standard non-separable convolution.
Revised: The result is a convolution with several orders of magnitude fewer weights than a standard non-separable convolution.

Original: Different to self-attention, lightweight convolutions reuse the same weights for context elements, regardless of the current time-step.
Revised: Different to self-attention, lightweight convolutions reuse the same weights for context elements, regardless of the current time-step.

Original: Dynamic convolutions build on lightweight convolutions by predicting a different convolution kernel at every time-step.
Revised: Dynamic convolutions build on lightweight convolutions by predicting a different convolution kernel at every time-step.

Original: The kernel is a function of the current time-step only as opposed to the entire context as in self-attention (Figure 1b).
Revised: The kernel is a function of the current time-step only as opposed to the entire context as in self-attention (Figure 1b).

Original: Dynamic convolutions are similar to locally connected layers in the sense that the weights change at every position, however, the difference is that weights are dynamically generated by the model rather than ﬁxed after training (LeCun et al., 1998; Taigman et al., 2014; Chen et al., 2015).
Revised: Dynamic convolutions are similar to locally connected layers in the sense that the weights change at every position, however, the difference is that weights are dynamically generated by the model rather than ﬁxed after training (LeCun et al., 1998; Taigman et al., 2014; Chen et al., 2015).

Original: Our approach also bears similarity to location-based attention which does not access the context to determine attention weights, however, we do not directly take the attention weights from the previous time-step into account (Chorowski et al., 2015; Luong et al., 2015). 
Revised: Our approach also bears similarity to location-based attention which does not access the context to determine attention weights, however, we do not directly take the attention weights from the previous time-step into account (Chorowski et al., 2015; Luong et al., 2015). Shen et al.

Revised: (2018a) reduce complexity by performing attention within blocks of the input sequence and Shen et al.

Revised: (2017; 2018b) perform more ﬁne-grained attention over each feature.

Original: Our experiments show that lightweight convolutions perform competitively to strong self-attention results and that dynamic convolutions can perform even better.
Revised: Our experiments show that lightweight convolutions perform competitively to strong self-attention results and that dynamic convolutions can perform even better.

Original: On WMT English-German translation dynamic convolutions achieve a new state of the art of 29.7 BLEU, on WMT English-French they match the best reported result in the literature, and on IWSLT German-English dynamic convolutions outperform self-attention by 0.8 BLEU.
Revised: On WMT English-German translation dynamic convolutions achieve a new state of the art of 29.7 BLEU, on WMT English-French they match the best reported result in the literature, and on IWSLT German-English dynamic convolutions outperform self-attention by 0.8 BLEU.

Original: Dynamic convolutions achieve 20% faster runtime than a highly-optimized self-attention baseline.
Revised: Dynamic convolutions achieve 20% faster runtime than a highly-optimized self-attention baseline.

Original: For language modeling on the Billion word benchmark dynamic convolutions perform as well as or better than self-attention and on CNN-DailyMail abstractive document summarization we outperform a strong self-attention model.
Revised: For language modeling on the Billion word benchmark dynamic convolutions perform as well as or better than self-attention and on CNN-DailyMail abstractive document summarization we outperform a strong self-attention model.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We ﬁrst outline sequence to sequence learning and self-attention.
Revised: We ﬁrst outline sequence to sequence learning and self-attention.

Original: Our work builds on non-separable convolutions as well as depthwise separable convolutions.
Revised: Our work builds on non-separable convolutions as well as depthwise separable convolutions.

Original: Sequence to sequence learning
Revised: Sequence to sequence learning

Original: maps a source sequence to a target sequence via two separate networks such as in machine translation (Sutskever et al., 2014).
Revised: maps a source sequence to a target sequence via two separate networks such as in machine translation (Sutskever et al., 2014).

Original: The encoder network computes representations for the source sequence such as an English sentence and the decoder network autoregressively generates a target sequence based on the encoder output.
Revised: The encoder network computes representations for the source sequence such as an English sentence and the decoder network autoregressively generates a target sequence based on the encoder output.

Original: The self-attention module of Vaswani et al.
Revised: The self-attention module of Vaswani et al.

Original: (2017) applies three projections to the input X ∈ R n × d to obtain key (K), query (Q), and value (V) representations, where n is the number of time steps, d the input/output dimension (Figure 2a).
Revised: (2017) applies three projections to the input X ∈ R n × d to obtain key (K), query (Q), and value (V) representations, where n is the number of time steps, d the input/output dimension (Figure 2a).

Original: It also deﬁnes a number of heads H where each head can learn separate attention weights over d k features and attend to different positions.
Revised: It also deﬁnes a number of heads H where each head can learn separate attention weights over d k features and attend to different positions.

Original: The module computes dot-products between key/query pairs, scales to stabilize training, and then softmax normalizes the result.
Revised: The module computes dot-products between key/query pairs, scales to stabilize training, and then softmax normalizes the result.

Original: Finally, it computes a weighted sum using the output of the value projection (V):
Revised: Finally, it computes a weighted sum using the output of the value projection (V):

Original: [Equation]
Revised: [Equation]

Original: Depthwise convolutions perform a convolution independently over every channel.
Revised: Depthwise convolutions perform a convolution independently over every channel.

Original: The number of parameters can be reduced from d 2 k to dk where k is the kernel width.
Revised: The number of parameters can be reduced from d 2 k to dk where k is the kernel width.

Original: The output O ∈ R n × d of a depthwise convolution with weight W ∈ R d × k for element i and output dimension c is deﬁned as:
Revised: The output O ∈ R n × d of a depthwise convolution with weight W ∈ R d × k for element i and output dimension c is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original:  L IGHTWEIGHT C ONVOLUTIONS
Revised:  L IGHTWEIGHT C ONVOLUTIONS

Original: In this section, we introduce LightConv , a depthwise convolution which shares certain output channels and whose weights are normalized across the temporal dimension using a softmax.
Revised: In this section, we introduce LightConv , a depthwise convolution which shares certain output channels and whose weights are normalized across the temporal dimension using a softmax.

Original: Compared toself-attention, LightConv has a ﬁxed context window and it determines the importance of context elements with a set of weights that do not change over time steps. We will show that models equipped
Revised: Compared to

Original: [Figure]
Revised: [Figure]

Original: K
Revised: K

Original: [Figure]
Revised: [Figure]

Original: Linear
Revised: Linear

Original: Linear input
Revised: Linear input

Original: GLU
Revised: GLU

Original: [Figure]
Revised: [Figure]

Original: input
Revised: input

Original: LConv
Revised: LConv

Original: Linear Linear
Revised: Linear Linear

Original: Linear dynamic weights
Revised: Linear dynamic weights

Original: GLU
Revised: GLU

Original: [Figure]
Revised: [Figure]

Revised: self-attention, LightConv has a ﬁxed context window and it determines the importance of context elements with a set of weights that do not change over time steps.

Original:  with lightweight convolutions show better generalization compared to regular convolutions and that they can be competitive to state-of-the-art self-attention models ( § 6).
Revised: We will show that models equipped with lightweight convolutions show better generalization compared to regular convolutions and that they can be competitive to state-of-the-art self-attention models ( § 6).

Original: This is surprising because the common belief is that content-based self-attention mechanisms are crucial to obtaining state-of-theart results in natural language processing applications.
Revised: This is surprising because the common belief is that content-based self-attention mechanisms are crucial to obtaining state-of-theart results in natural language processing applications.

Original: Furthermore, the low computational proﬁle of LightConv enables us to formulate efﬁcient dynamic convolutions ( § 4).
Revised: Furthermore, the low computational proﬁle of LightConv enables us to formulate efﬁcient dynamic convolutions ( § 4).

Original: LightConv computes the following for the i -th element in the sequence and output channel c :
Revised: LightConv computes the following for the i -th element in the sequence and output channel c :

Original: [Equation]
Revised: [Equation]

Original: Weight sharing.
Revised: Weight sharing.

Original: We tie the parameters of every subsequent number of dH channels, which reduces the number of parameters by a factor of H .
Revised: We tie the parameters of every subsequent number of dH channels, which reduces the number of parameters by a factor of H .

Original: As illustration, a regular convolution requires 7,340, ( d 2 × k ) weights for d = 1024 and k = 7 , a depthwise separable convolution has 7,168 weights ( d × k ), and with weight sharing, H = 16 , we have only 112 ( H × k ) weights.
Revised: As illustration, a regular convolution requires 7,340, ( d 2 × k ) weights for d = 1024 and k = 7 , a depthwise separable convolution has 7,168 weights ( d × k ), and with weight sharing, H = 16 , we have only 112 ( H × k ) weights.

Original: We will see that this vast reduction in the number of parameters is crucial to make dynamic convolutions possible on current hardware.
Revised: We will see that this vast reduction in the number of parameters is crucial to make dynamic convolutions possible on current hardware.

Original: Softmax-normalization.
Revised: Softmax-normalization.

Original: We normalize the weights
Revised: We normalize the weights

Original: W ∈ R H
Revised: W ∈ R H

Original: × k across the temporal dimension k using a softmax operation:
Revised: × k across the temporal dimension k using a softmax operation:

Original: [Equation]
Revised: [Equation]

Original: Module.
Revised: Module.

Original: Figure 2b shows the architecture of the module where we integrate LightConv.
Revised: Figure 2b shows the architecture of the module where we integrate LightConv.

Original: We ﬁrst apply an input projection mapping from dimension d to 2 d , followed by a gated linear unit (GLU;
Revised: We ﬁrst apply an input projection mapping from dimension d to 2 d , followed by a gated linear unit (GLU;

Original: Dauphin et al. 2017), and the actual lightweight convolution.
Revised: Dauphin et al. 2017), and the actual lightweight convolution.

Original: The GLU uses half of the inputs as gates by applying sigmoid units and then computes a pointwise product with the other inputs.
Revised: The GLU uses half of the inputs as gates by applying sigmoid units and then computes a pointwise product with the other inputs.

Original: We also apply an output projection of size
Revised: We also apply an output projection of size

Original: W O ∈ R
Revised: W O ∈ R

Original: d × d to the output of LightConv.
Revised: d × d to the output of LightConv.

Original: Regularization.
Revised: Regularization.

Original: We found DropConnect to be a good regularizer for the LightConv module (Wan et al., 2013).
Revised: We found DropConnect to be a good regularizer for the LightConv module (Wan et al., 2013).

Original: Speciﬁcally, we drop every entry of the normalized weights softmax ( W ) with probability p and divide it by 1 − p during training.
Revised: Speciﬁcally, we drop every entry of the normalized weights softmax ( W ) with probability p and divide it by 1 − p during training.

Original: This amounts to removing some of the temporal information within a channel.
Revised: This amounts to removing some of the temporal information within a channel.

Original: Implementation.
Revised: Implementation.

Original: Existing CUDA primitives for convolutions did not perform very well to implement LightConv and we found the following solution faster on short sequences: We copy and expand the normalized weights
Revised: Existing CUDA primitives for convolutions did not perform very well to implement LightConv and we found the following solution faster on short sequences: We copy and expand the normalized weights

Original: W ∈ R H
Revised: W ∈ R H

Original: × k to a matrix of size BH × n × n , where B is the batch size.
Revised: × k to a matrix of size BH × n × n , where B is the batch size.

Original: We then reshape and transpose the inputs to size BH × n × dH , and perform a batch matrix multiplication to get the outputs.
Revised: We then reshape and transpose the inputs to size BH × n × dH , and perform a batch matrix multiplication to get the outputs.

Original: We expect a dedicated CUDA kernel to be much more efﬁcient.
Revised: We expect a dedicated CUDA kernel to be much more efﬁcient.

Original:  D YNAMIC CONVOLUTIONS
Revised:  D YNAMIC CONVOLUTIONS

Original: A dynamic convolution has kernels that vary over time as a learned function of the individual time steps.
Revised: A dynamic convolution has kernels that vary over time as a learned function of the individual time steps.

Original: A dynamic version of standard convolutions would be impractical for current GPUs due to their large memory requirements.
Revised: A dynamic version of standard convolutions would be impractical for current GPUs due to their large memory requirements.

Original: We address this problem by building on LightConv which drastically reduces the number of parameters ( § 3).
Revised: We address this problem by building on LightConv which drastically reduces the number of parameters ( § 3).

Original: DynamicConv takes the same form as LightConv but uses a time-step dependent kernel that is computed using a function f : R d → R H × k :
Revised: DynamicConv takes the same form as LightConv but uses a time-step dependent kernel that is computed using a function f : R d → R H × k :

Original: DynamicConv ( X, i, c ) =
Revised: DynamicConv ( X, i, c ) =

Original: LightConv ( X, f
Revised: LightConv ( X, f

Original: ( X i )  , i, c ) we model f with a simple linear module with learned weights W Q ∈ R H × k × d , i.e., f ( X i ) =
Revised: ( X i ) h, : , i, c ) we model f with a simple linear module with learned weights W Q ∈ R H × k × d , i.e., f ( X i ) =

Original: (cid:80) dc = [Equation] W 
Revised: (cid:80) dc =1 W Qh,j,c

Original: [Equation]

Original: X i,c .
Revised: X i,c .

Original: Similar to self-attention, DynamicConv changes the weights assigned to context elements over time.
Revised: Similar to self-attention, DynamicConv changes the weights assigned to context elements over time.

Original: However, the weights of DynamicConv do not depend on the entire context, they are a function of the current time-step only.
Revised: However, the weights of DynamicConv do not depend on the entire context, they are a function of the current time-step only.

Original: Self-attention requires a quadratic number of operations in the sentence length to compute attention weights, while the computation of dynamic kernels for DynamicConv scales linearly in the sequence length.
Revised: Self-attention requires a quadratic number of operations in the sentence length to compute attention weights, while the computation of dynamic kernels for DynamicConv scales linearly in the sequence length.

Original: Our experiments ( § 6) show that models using DynamicConv match or exceed the performance of state-of-the-art models that use context-based self-attention.
Revised: Our experiments ( § 6) show that models using DynamicConv match or exceed the performance of state-of-the-art models that use context-based self-attention.

Original: This challenges the typical intuitions about the importance of content-based self-attention in natural language processing applications.
Revised: This challenges the typical intuitions about the importance of content-based self-attention in natural language processing applications.

Original:  E XPERIMENTAL SETUP 5. M ODEL A RCHITECTURE
Revised:  E XPERIMENTAL SETUP 5. M ODEL A RCHITECTURE

Original: We use an encoder-decoder architecture for sequence to sequence learning (Sutskever et al., 2014) and we closely follow the architectural choices presented in Vaswani et al.
Revised: We use an encoder-decoder architecture for sequence to sequence learning (Sutskever et al., 2014) and we closely follow the architectural choices presented in Vaswani et al.

Original: Our self-attention baseline is the fairseq re-implementation of the Transformer Big architecture (Ott et al., 2018).
Revised: Our self-attention baseline is the fairseq re-implementation of the Transformer Big architecture (Ott et al., 2018).

Original: The encoder and decoder networks have N blocks each.
Revised: The encoder and decoder networks have N blocks each.

Original: Encoder blocks contain two sub-blocks:
Revised: Encoder blocks contain two sub-blocks:

Original: The ﬁrst is a self-attention module ( § 2), a LightConv module (3), or a DynamicConv module ( § 4).
Revised: The ﬁrst is a self-attention module ( § 2), a LightConv module (3), or a DynamicConv module ( § 4).

Original: The second sub-block is a feed-forward module: ReLU ( W 1 X + b 1 ) W 2 + b 2 where W 1 ∈ R d × d ff ,
Revised: The second sub-block is a feed-forward module: ReLU ( W 1 X + b 1 ) W 2 + b 2 where W 1 ∈ R d × d ff ,

Original: W 2 ∈ R
Revised: W 2 ∈ R

Original: d ff × d and d = 1024 , d ff = 4096 unless otherwise stated.
Revised: d ff × d and d = 1024 , d ff = 4096 unless otherwise stated.

Original: Sub-blocks are surrounded by residual connections (He et al., 2015) and layer normalization (Ba et al., 2016).
Revised: Sub-blocks are surrounded by residual connections (He et al., 2015) and layer normalization (Ba et al., 2016).

Original: Decoder blocks are identical except that they have an additional source-target attention sub-block between the self-attention and feed-forward module.
Revised: Decoder blocks are identical except that they have an additional source-target attention sub-block between the self-attention and feed-forward module.

Original: The source-target attention is equivalent to the self-attention module, except that the values and keys are projections over the encoder output for each source word.
Revised: The source-target attention is equivalent to the self-attention module, except that the values and keys are projections over the encoder output for each source word.

Original: Words are fed to the encoder and decoder networks in d dimensional embeddings.
Revised: Words are fed to the encoder and decoder networks in d dimensional embeddings.

Original: We add sinusoidal position embeddings to encode the absolute position of each word in the sequence (Kaiser et al., 2017; Vaswani et al., 2017).
Revised: We add sinusoidal position embeddings to encode the absolute position of each word in the sequence (Kaiser et al., 2017; Vaswani et al., 2017).

Original: The model computes a distribution over vocabulary V by transforming the decoder output via a linear layer with weights W V ∈ R d × V followed by softmax normalization.
Revised: The model computes a distribution over vocabulary V by transforming the decoder output via a linear layer with weights W V ∈ R d × V followed by softmax normalization.

Original: LightConv and DynamicConv are identical to Transformer Big, except that self-attention modules are swapped with either ﬁxed or dynamic convolutions.
Revised: LightConv and DynamicConv are identical to Transformer Big, except that self-attention modules are swapped with either ﬁxed or dynamic convolutions.

Original: These models also use fewer parameters per block (cf.
Revised: These models also use fewer parameters per block (cf.

Original: Figure 2b and Figure 2c) and we therefore increase the number of blocks to N = 7 for the encoder to roughly match the parameter count of Transformer Big.
Revised: Figure 2b and Figure 2c) and we therefore increase the number of blocks to N = 7 for the encoder to roughly match the parameter count of Transformer Big.

Original: We generally set H = 16 .
Revised: We generally set H = 16 .

Original: Both
Revised: Both

Original: LightConv and DynamicConv set the the encoder and decoder kernel sizes to 3, 7, 15, 31x4 for each block respectively; except for the decoder where we have only three top layers with kernel size 31.
Revised: LightConv and DynamicConv set the the encoder and decoder kernel sizes to 3, 7, 15, 31x4 for each block respectively; except for the decoder where we have only three top layers with kernel size 31.

Original: 5. D ATASETS AND E VALUATION
Revised: 5. D ATASETS AND E VALUATION

Original: To get a thorough understanding of the limitations of LightConv and DynamicConv we evaluate on three different tasks: machine translation, language modeling and abstractive summarization.
Revised: To get a thorough understanding of the limitations of LightConv and DynamicConv we evaluate on three different tasks: machine translation, language modeling and abstractive summarization.

Original: Machine Translation.
Revised: Machine Translation.

Original: We report results on four benchmarks: For WMT English to German (EnDe) we replicate the setup of Vaswani et al.
Revised: We report results on four benchmarks: For WMT English to German (EnDe) we replicate the setup of Vaswani et al.

Original: (2017), based on WMT’16 training data with 4.5M sentence pairs, we validate on newstest2013 and test on newstest2014.
Revised: (2017), based on WMT’16 training data with 4.5M sentence pairs, we validate on newstest2013 and test on newstest2014.

Original: The vocabulary is a 32K joint source and target byte pair encoding (BPE; Sennrich et al. 2016).
Revised: The vocabulary is a 32K joint source and target byte pair encoding (BPE; Sennrich et al. 2016).

Original: For WMT English to French (EnFr), we borrow the setup of Gehring et al.
Revised: For WMT English to French (EnFr), we borrow the setup of Gehring et al.

Original: (2017) with 36M training sentence pairs from WMT’14, validate on newstest2012+2013 and test on newstest2014.
Revised: (2017) with 36M training sentence pairs from WMT’14, validate on newstest2012+2013 and test on newstest2014.

Original: The 40K vocabulary is based on a joint source and target BPE factorization.
Revised: The 40K vocabulary is based on a joint source and target BPE factorization.

Original: For WMT English to Chinese (Zh-En), we pre-process the WMT’17 training data following Hassan et al.
Revised: For WMT English to Chinese (Zh-En), we pre-process the WMT’17 training data following Hassan et al.

Original: (2018) resulting in 20M sentence pairs.
Revised: (2018) resulting in 20M sentence pairs.

Original: We develop on devtest2017 and test on newstest2017.
Revised: We develop on devtest2017 and test on newstest2017.

Original: For IWSLT’14 German-English (De-En) we replicate the setup of Edunov et al.
Revised: For IWSLT’14 German-English (De-En) we replicate the setup of Edunov et al.

Original: (2018) for 160K training sentence pairs and 10K joint BPE vocabulary.
Revised: (2018) for 160K training sentence pairs and 10K joint BPE vocabulary.

Original: For this benchmark only, data is lowercased.
Revised: For this benchmark only, data is lowercased.

Original: For WMT En-De, WMT En-Fr, we measure case-sensitive tokenized BLEU.
Revised: For WMT En-De, WMT En-Fr, we measure case-sensitive tokenized BLEU.

Original: For WMT En-De only we apply compound splitting similar to Vaswani et al.
Revised: For WMT En-De only we apply compound splitting similar to Vaswani et al.

Original: For WMT Zh-En we measure detokenized BLEU to be comparable to Hassan et al.
Revised: For WMT Zh-En we measure detokenized BLEU to be comparable to Hassan et al.

Original: We train three random initializations of a each conﬁguration and report test accuracy of the seed which resulted in the highest validation BLEU.
Revised: We train three random initializations of a each conﬁguration and report test accuracy of the seed which resulted in the highest validation BLEU.

Original: Ablations are conducted on the validation set
Revised: Ablations are conducted on the validation set

Original: and we report the mean BLEU and standard deviation on this set.
Revised: and we report the mean BLEU and standard deviation on this set.

Original: WMT En-De, WMT En-Fr are based on beam search with a beam width of 5, IWSLT uses beam 4, and WMT Zh-En beam 8 following
Revised: WMT En-De, WMT En-Fr are based on beam search with a beam width of 5, IWSLT uses beam 4, and WMT Zh-En beam 8 following

Original: Hassan et al.
Revised: Hassan et al.

Original: For all datasets, we tune a length penalty as well as the number of checkpoints to average on the validation set.
Revised: For all datasets, we tune a length penalty as well as the number of checkpoints to average on the validation set.

Original: Language Modeling.
Revised: Language Modeling.

Original: We evaluate on the large-scale Billion word dataset (Chelba et al., 2013) which contains 768M tokens and has a vocabulary of nearly 800K types.
Revised: We evaluate on the large-scale Billion word dataset (Chelba et al., 2013) which contains 768M tokens and has a vocabulary of nearly 800K types.

Original: Sentences in this dataset are shufﬂed and we batch sentences independently of each other.
Revised: Sentences in this dataset are shufﬂed and we batch sentences independently of each other.

Original: Models are evaluated in terms of perplexity on the valid and test portions.
Revised: Models are evaluated in terms of perplexity on the valid and test portions.

Original: Summarization.
Revised: Summarization.

Original: We test the model’s ability to process long documents on the CNN-DailyMail summarization task (Hermann et al., 2015; Nallapati et al., 2016) comprising over 280K news articles paired with multi-sentence summaries.
Revised: We test the model’s ability to process long documents on the CNN-DailyMail summarization task (Hermann et al., 2015; Nallapati et al., 2016) comprising over 280K news articles paired with multi-sentence summaries.

Original: Articles are truncated to 400 tokens (See et al., 2017) and we use a BPE vocabulary of 30K types (Fan et al., 2017).
Revised: Articles are truncated to 400 tokens (See et al., 2017) and we use a BPE vocabulary of 30K types (Fan et al., 2017).

Original: We evaluate in terms of F1-Rouge, that is Rouge-1, Rouge-2 and Rouge-L (Lin, 2004).
Revised: We evaluate in terms of F1-Rouge, that is Rouge-1, Rouge-2 and Rouge-L (Lin, 2004).

Original: 6 When generating summaries, we follow standard practice in tuning the maximum output length, disallowing repeating the same trigram, and we apply a stepwise length penalty (Paulus et al., 2017;
Revised: 6 When generating summaries, we follow standard practice in tuning the maximum output length, disallowing repeating the same trigram, and we apply a stepwise length penalty (Paulus et al., 2017;

Original: Fan et al., 2017; Wu et al., 2016).
Revised: Fan et al., 2017; Wu et al., 2016).

Original: 5. T RAINING AND H YPERPARAMETERS
Revised: 5. T RAINING AND H YPERPARAMETERS

Original: Translation.
Revised: Translation.

Original: We use a dropout rate of 0.3 for WMT En-De and IWSLT De-En, 0.1 for WMT EnFr, and 0.25 for WMT Zh-En.
Revised: We use a dropout rate of 0.3 for WMT En-De and IWSLT De-En, 0.1 for WMT EnFr, and 0.25 for WMT Zh-En.

Original: WMT models are optimized with Adam and a cosine learning rate schedule (Kingma & Ba, 2015; Loshchilov & Hutter, 2016) where the learning rate is ﬁrst linearly warmed up for 10K steps from 10 − 7 to 10 − 3 and then annealed following a cosine rate with a single cycle.
Revised: WMT models are optimized with Adam and a cosine learning rate schedule (Kingma & Ba, 2015; Loshchilov & Hutter, 2016) where the learning rate is ﬁrst linearly warmed up for 10K steps from 10 − 7 to 10 − 3 and then annealed following a cosine rate with a single cycle.

Original: For IWSLT’14 De-En, we use a schedule based on the inverse square root of the current step (Vaswani et al., 2017).
Revised: For IWSLT’14 De-En, we use a schedule based on the inverse square root of the current step (Vaswani et al., 2017).

Original: We train the WMT models on 8 NVIDIA V100 GPUs for a total of 30K steps on WMT En-De, 40K steps for WMT Zh-En and 80K steps for WMT En-Fr.
Revised: We train the WMT models on 8 NVIDIA V100 GPUs for a total of 30K steps on WMT En-De, 40K steps for WMT Zh-En and 80K steps for WMT En-Fr.

Original: For IWSLT De-En we train for 50K steps on a single GPU.
Revised: For IWSLT De-En we train for 50K steps on a single GPU.

Original: We use ﬂoating point 16 precision and accumulate the gradients for 16 batches before applying an update (Ott et al., 2018), except for IWSLT where we do not accumulate gradients.
Revised: We use ﬂoating point 16 precision and accumulate the gradients for 16 batches before applying an update (Ott et al., 2018), except for IWSLT where we do not accumulate gradients.

Original: Batches contain up to 459K source tokens and the same number of target tokens for both WMT En-De and WMT
Revised: Batches contain up to 459K source tokens and the same number of target tokens for both WMT En-De and WMT

Original: Zh-En, 655K for En-Fr, and 4K for IWSLT De-En.
Revised: Zh-En, 655K for En-Fr, and 4K for IWSLT De-En.

Original: We use label smoothing with 0 .
Revised: We use label smoothing with 0 .

Original: 1 weight for the uniform prior distribution over the vocabulary (Szegedy et al., 2015; Pereyra et al., 2017).
Revised: 1 weight for the uniform prior distribution over the vocabulary (Szegedy et al., 2015; Pereyra et al., 2017).

Original: Language Modeling.
Revised: Language Modeling.

Original: We follow the same setup as for translation but remove the encoder module.
Revised: We follow the same setup as for translation but remove the encoder module.

Original: For the Billion word benchmark we use an adaptive softmax output layer to reduce the computational burden of the large vocabulary (Grave et al., 2016) and tie it with variable sized input word
Revised: For the Billion word benchmark we use an adaptive softmax output layer to reduce the computational burden of the large vocabulary (Grave et al., 2016) and tie it with variable sized input word

Original: We use the following parameters for ROUGE-1.
Revised: We use the following parameters for ROUGE-1.

Original: .pl : -
Revised: .pl : -

Original: -n
Revised: -n

Original: [Table] [Table]
Revised: [Table] [Table]

Original: embeddings (

Original: We train on 32 GPUs with batches of 65K tokens for 975K updates.
Revised: We train on 32 GPUs with batches of 65K tokens for 975K updates.

Original: As optimizer we use Nesterov’s accelerated gradient method (Sutskever et al., 2013) with a momentum value of 0 . 

Revised: As optimizer we use Nesterov’s accelerated gradient method (Sutskever et al., 2013) with a momentum value of 0 .

Original: and we renormalize gradients if their norm exceeds 0 .
Revised: and we renormalize gradients if their norm exceeds 0 .

Original: 1 (Pascanu et al., 2013).
Revised: 1 (Pascanu et al., 2013).

Original: The learning rate is linearly warmed up from 10 − 7 to 1 for 16K steps and then annealed using a cosine learning rate schedule (Loshchilov & Hutter, 2016) with one cycle.
Revised: The learning rate is linearly warmed up from 10 − 7 to 1 for 16K steps and then annealed using a cosine learning rate schedule (Loshchilov & Hutter, 2016) with one cycle.

Original: Summarization.
Revised: Summarization.

Original: We train with Adam using the cosine learning rate schedule with a warmup of 10K steps and a period of 20K updates.
Revised: We train with Adam using the cosine learning rate schedule with a warmup of 10K steps and a period of 20K updates.

Original: We use weight decay 1e-3 and dropout 0.3.
Revised: We use weight decay 1e-3 and dropout 0.3.

Original:  R ESULTS 6. M ACHINE T RANSLATION
Revised:  R ESULTS 6. M ACHINE T RANSLATION

Original: We ﬁrst report results on WMT En-De and WMT En-Fr where we compare to the best results in the literature, most of which are based on self-attention.
Revised: We ﬁrst report results on WMT En-De and WMT En-Fr where we compare to the best results in the literature, most of which are based on self-attention.

Original: Table 1 shows that LightConv performs very competitively and only trails the state of the art result by 0.1 BLEU on WMT En-Fr; the state of the art is based on self-attention (Ott et al., 2018).
Revised: Table 1 shows that LightConv performs very competitively and only trails the state of the art result by 0.1 BLEU on WMT En-Fr; the state of the art is based on self-attention (Ott et al., 2018).

Original: This is despite the simplicity of LightConv which operates with a very small number of ﬁxed weights over all time steps whereas self-attention computes dot-products with all context elements at every time-step.
Revised: This is despite the simplicity of LightConv which operates with a very small number of ﬁxed weights over all time steps whereas self-attention computes dot-products with all context elements at every time-step.

Original: DynamicConv outperforms the best known result on WMT En-De by 0.4 BLEU and achieves a new state of the art, whereas on WMT En-Fr it matches the state of the art.
Revised: DynamicConv outperforms the best known result on WMT En-De by 0.4 BLEU and achieves a new state of the art, whereas on WMT En-Fr it matches the state of the art.

Original: This shows that content-based self-attention is not necessary to achieve good accuracy on large translation benchmarks.
Revised: This shows that content-based self-attention is not necessary to achieve good accuracy on large translation benchmarks.

Original: IWSLT is a much smaller benchmark and we therefore switch to a smaller architecture:
Revised: IWSLT is a much smaller benchmark and we therefore switch to a smaller architecture:

Original: d ff = 1024 , d = 512 , and H = 4 .
Revised: d ff = 1024 , d = 512 , and H = 4 .

Original: The self-attention baseline on this dataset is the best reported result in the literature (Table 2).
Revised: The self-attention baseline on this dataset is the best reported result in the literature (Table 2).

Original: 7 LightConv outperforms this baseline by 0.4 BLEU and DynamicConv improves by 0.8 BLEU.
Revised: 7 LightConv outperforms this baseline by 0.4 BLEU and DynamicConv improves by 0.8 BLEU.

Original: We further run experiments on WMT Zh-En translation to evaluate on a non-European language.
Revised: We further run experiments on WMT Zh-En translation to evaluate on a non-European language.

Original: LightConv outperforms the baseline by 0.5 BLEU and DynamicConv by 0.6 BLEU.
Revised: LightConv outperforms the baseline by 0.5 BLEU and DynamicConv by 0.6 BLEU.

Original: [Table]
Revised: [Table]

Original: M ODEL ABLATION
Revised: M ODEL ABLATION

Original: In this section we evaluate the impact of the various choices we made for LightConv ( § 3) and DynamicConv ( § 4).
Revised: In this section we evaluate the impact of the various choices we made for LightConv ( § 3) and DynamicConv ( § 4).

Original: We ﬁrst show that limiting the maximum context size of self-attention has no impact on validation accuracy (Table 3).
Revised: We ﬁrst show that limiting the maximum context size of self-attention has no impact on validation accuracy (Table 3).

Original: Note that our baseline is stronger than the original result of Vaswani et al.
Revised: Note that our baseline is stronger than the original result of Vaswani et al.

Original: Next, we replace self-attention blocks with non-separable convolutions (CNN) with kernel size 3 and input/output dimension d = 1024 .
Revised: Next, we replace self-attention blocks with non-separable convolutions (CNN) with kernel size 3 and input/output dimension d = 1024 .

Original: The CNN block has no input and output projections compared to the baseline and we add one more encoder layer to assimilate the parameter count.
Revised: The CNN block has no input and output projections compared to the baseline and we add one more encoder layer to assimilate the parameter count.

Original: This CNN with a narrow kernel trails self-attention by 1 BLEU.
Revised: This CNN with a narrow kernel trails self-attention by 1 BLEU.

Original: We improve this result by switching to a depthwise separable convolution (CNN Depthwise) with input and output projections of size d = 1024 .
Revised: We improve this result by switching to a depthwise separable convolution (CNN Depthwise) with input and output projections of size d = 1024 .

Original: When we progressively increase the kernel width from lower to higher layers then this further improves accuracy.
Revised: When we progressively increase the kernel width from lower to higher layers then this further improves accuracy.

Original: This narrows the gap to self-attention to only 0.5 BLEU.
Revised: This narrows the gap to self-attention to only 0.5 BLEU.

Original: DropConnect gives a slight performance improvement and weight sharing does not decrease performance.
Revised: DropConnect gives a slight performance improvement and weight sharing does not decrease performance.

Original: Adding softmax normalization to the weights is only 0.3 BLEU below the accuracy of the baseline.
Revised: Adding softmax normalization to the weights is only 0.3 BLEU below the accuracy of the baseline.

Original: This corresponds to LightConv.
Revised: This corresponds to LightConv.

Revised: In Appendix A we compare softmaxnormalization to various alternatives.

Original: Finally, dynamic convolutions (DynamicConv) achieve the same validation accuracy as self-attention with slightly fewer parameters and at 20% higher inference speed.
Revised: Finally, dynamic convolutions (DynamicConv) achieve the same validation accuracy as self-attention with slightly fewer parameters and at 20% higher inference speed.

Original: Softmax-normalization is crucial for DynamicConv since training diverged in our experiments when removing it.
Revised: Softmax-normalization is crucial for DynamicConv since training diverged in our experiments when removing it.

Original: To make the models more comparable, we do not introduce GLU after the input projection.
Revised: To make the models more comparable, we do not introduce GLU after the input projection.

Original: For comparison, we re-implemented averaged attention networks (AAN; Zhang et al. 2018) which compute a uniform average over past model states instead of a weighted average as in self-attention.
Revised: For comparison, we re-implemented averaged attention networks (AAN; Zhang et al. 2018) which compute a uniform average over past model states instead of a weighted average as in self-attention.

Original: Our re-implementation is efﬁcient: we measure 129 sentences/sec for a base transformer-AAN on newstest2014 compared to 20 sentences/sec for Zhang et al.
Revised: Our re-implementation is efﬁcient: we measure 129 sentences/sec for a base transformer-AAN on newstest2014 compared to 20 sentences/sec for Zhang et al.

Original: Table 3 shows that our models outperform this approach.
Revised: Table 3 shows that our models outperform this approach.

Original: Note that AANs still use self-attention in the encoder network while as our approach does away with self-attention both in the encoder and decoder.
Revised: Note that AANs still use self-attention in the encoder network while as our approach does away with self-attention both in the encoder and decoder.

Original: 6. L ANGUAGE M ODELING
Revised: 6. L ANGUAGE M ODELING

Original: As second task we consider language modeling on the Billion word benchmark.
Revised: As second task we consider language modeling on the Billion word benchmark.

Original: The self-attention baseline has N = 16 blocks, each with a self-attention module and a feed-forward module using d ff = 4096 and d = 1024 .
Revised: The self-attention baseline has N = 16 blocks, each with a self-attention module and a feed-forward module using d ff = 4096 and d = 1024 .

Original: DynamicConv uses N = 17 blocks to assimilate the parameter count and we use kernel sizes 15x2, 31x4 and 63x11.
Revised: DynamicConv uses N = 17 blocks to assimilate the parameter count and we use kernel sizes 15x2, 31x4 and 63x11.

Original: Table 4 shows that DynamicConv achieves slightly better perplexity than our self-attention baseline which is very competitive.
Revised: Table 4 shows that DynamicConv achieves slightly better perplexity than our self-attention baseline which is very competitive.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 6. A BSTRACTIVE S UMMARIZATION
Revised: 6. A BSTRACTIVE S UMMARIZATION

Original: Finally
Revised: Finally

Original: [Table]
Revised: [Table]

Original: we evaluate on the CNN-DailyMail abstractive document summarization benchmark where we encode a document of up to 400 words and generate multi-sentence summaries.
Revised: we evaluate on the CNN-DailyMail abstractive document summarization benchmark where we encode a document of up to 400 words and generate multi-sentence summaries.

Original: This tests the ability of our model to deal with longer sequences.
Revised: This tests the ability of our model to deal with longer sequences.

Original: We reduce model capacity by setting d = 1024 , d ff = 2048 , H = 8 , similar to the Transformer base setup of Vaswani et al.
Revised: We reduce model capacity by setting d = 1024 , d ff = 2048 , H = 8 , similar to the Transformer base setup of Vaswani et al.

Original: [Table]
Revised: [Table]

Original: Table 5 shows that LightConv outperforms the self-attention baseline as well as comparable previous work and DynamicConv performs even better.
Revised: Table 5 shows that LightConv outperforms the self-attention baseline as well as comparable previous work and DynamicConv performs even better.

Original: We also show results for the specialized summarization architecture of Gehrmann et al.
Revised: We also show results for the specialized summarization architecture of Gehrmann et al.

Original: and note that our model is a standard sequence to sequence architecture.
Revised: and note that our model is a standard sequence to sequence architecture.

Original: Our model could also be trained with reinforcement learning (Celikyilmaz et al., 2018) to improve results.
Revised: Our model could also be trained with reinforcement learning (Celikyilmaz et al., 2018) to improve results.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented lightweight convolutions which perform competitively to the best reported results in the literature despite their simplicity.
Revised: We presented lightweight convolutions which perform competitively to the best reported results in the literature despite their simplicity.

Original: They have a very small parameter footprint and the kernel does not change over time-steps.
Revised: They have a very small parameter footprint and the kernel does not change over time-steps.

Original: This demonstrates that self-attention is not critical to achieve good accuracy on the language tasks we considered.
Revised: This demonstrates that self-attention is not critical to achieve good accuracy on the language tasks we considered.

Original: Dynamic convolutions build on lightweight convolutions by predicting a different kernel at every time-step, similar to the attention weights computed by self-attention.
Revised: Dynamic convolutions build on lightweight convolutions by predicting a different kernel at every time-step, similar to the attention weights computed by self-attention.

Original: The dynamic weights are a function of the current time-step only rather than the entire context.
Revised: The dynamic weights are a function of the current time-step only rather than the entire context.

Original: Our experiments show that lightweight convolutions can outperform a strong self-attention baseline on WMT’17 Chinese-English translation, IWSLT’14 German-English translation and CNNDailyMail summarization.
Revised: Our experiments show that lightweight convolutions can outperform a strong self-attention baseline on WMT’17 Chinese-English translation, IWSLT’14 German-English translation and CNNDailyMail summarization.

Original: Dynamic convolutions improve further and achieve a new state of the art on the test set of WMT’14 English-German.
Revised: Dynamic convolutions improve further and achieve a new state of the art on the test set of WMT’14 English-German.

Original: Both lightweight convolution and dynamic convolution are 20% faster at runtime than self-attention.
Revised: Both lightweight convolution and dynamic convolution are 20% faster at runtime than self-attention.

Original: On Billion word language modeling we achieve comparable results to self-attention.
Revised: On Billion word language modeling we achieve comparable results to self-attention.

Original: We are excited about the future of dynamic convolutions and plan to apply them to other tasks such as question answering and computer vision where inputs are even larger than the tasks we considered in this paper.
Revised: We are excited about the future of dynamic convolutions and plan to apply them to other tasks such as question answering and computer vision where inputs are even larger than the tasks we considered in this paper.

Revised: S UPPLEMENTARY M ATERIAL A C OMPARISON OF SOFTMAX - NORMALIZATION TO ALTERNATIVES

Revised: We compare our proposed softmax-normalization of weights to other alternatives in Table 6.

Revised: For each setting, we use three seeds and report the mean and the standard deviation of the BLEU score on WMT English-German newstest2013.

Revised: The softmax and norms are computed over the kernel dimension.

Revised: Simply using the absolute value of the weights or squaring them does not make the training more stable, which shows that having all non-negative weights is not critical.

Revised: Dividing the weights by the (cid:96) 2 -norm or bounding the weights with sigmoid or the hyperbolic tangent function also stablizes the training procedure; however, the softmax-normalization performs best.

Revised: [Table]