Original: S YNTHESIZER : R ETHINKING S ELF -A TTENTION FOR T RANSFORMER M ODELS
Revised: S YNTHESIZER : R ETHINKING S ELF -A TTENTION FOR T RANSFORMER M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The dot product self-attention is known to be central and indispensable to stateof-the-art Transformer models.
Revised: The dot product self-attention is known to be central and indispensable to stateof-the-art Transformer models.

Original: But is it really required?
Revised: But is it really required?

Original: This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models.
Revised: This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models.

Original: Via extensive experiments, we ﬁnd that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all.
Revised: Via extensive experiments, we ﬁnd that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all.

Original: To this end, we propose S YNTHESIZER , a model that learns synthetic attention weights without token-token interactions.
Revised: To this end, we propose S YNTHESIZER , a model that learns synthetic attention weights without token-token interactions.

Original: In our experiments, we ﬁrst show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks.
Revised: In our experiments, we ﬁrst show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks.

Original: When composed with dot product attention, we ﬁnd that Synthesizers consistently outperform Transformers.
Revised: When composed with dot product attention, we ﬁnd that Synthesizers consistently outperform Transformers.

Original: Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only 60% faster but also improves perplexity by a relative 3 .
Revised: Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only 60% faster but also improves perplexity by a relative 3 .

Original: Finally, we show that simple factorized
Revised: Finally, we show that simple factorized

Original: Synthesizers can outperform Linformers on encoding only tasks.
Revised: Synthesizers can outperform Linformers on encoding only tasks.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Transformer models (Vaswani et al., 2017) have demonstrated success across a wide range of tasks.
Revised: Transformer models (Vaswani et al., 2017) have demonstrated success across a wide range of tasks.

Original: This has resulted in Transformers largely displacing once popular auto-regressive and recurrent models in recent years.
Revised: This has resulted in Transformers largely displacing once popular auto-regressive and recurrent models in recent years.

Original: At the heart of Transformer models lies the query-key-value dot product attention.
Revised: At the heart of Transformer models lies the query-key-value dot product attention.

Original: The success of Transformer models is widely attributed to this self-attention mechanism since fully connected token graphs, which are able to model long-range dependencies, provide a robust inductive bias.
Revised: The success of Transformer models is widely attributed to this self-attention mechanism since fully connected token graphs, which are able to model long-range dependencies, provide a robust inductive bias.

Original: But is the dot product self-attention really so important?
Revised: But is the dot product self-attention really so important?

Original: Do we need it?
Revised: Do we need it?

Original: Is it necessary to learn attention weights via pairwise dot products?
Revised: Is it necessary to learn attention weights via pairwise dot products?

Original: This paper seeks to develop a deeper understanding of the role that the dot product self-attention mechanism plays in Transformer models.
Revised: This paper seeks to develop a deeper understanding of the role that the dot product self-attention mechanism plays in Transformer models.

Original: The fundamental role of dot product self-attention is to learn self-alignment, i.e., to determine the relative importance of a single token with respect to all other tokens in the sequence.
Revised: The fundamental role of dot product self-attention is to learn self-alignment, i.e., to determine the relative importance of a single token with respect to all other tokens in the sequence.

Original: To this end, there have been memory metaphors and analogies constructed to support this claim.
Revised: To this end, there have been memory metaphors and analogies constructed to support this claim.

Original: Indeed, the terms query , keys , and values imply that self-attention emulates a content-based retrieval process which leverages pairwise interactions at its very core.
Revised: Indeed, the terms query , keys , and values imply that self-attention emulates a content-based retrieval process which leverages pairwise interactions at its very core.

Original: Moving against convention, this paper postulates that we cannot only do without dot product self-attention but also content-based memory-like self-attention altogether.
Revised: Moving against convention, this paper postulates that we cannot only do without dot product self-attention but also content-based memory-like self-attention altogether.

Original: Traditionally, attention weights are learned at the instance or sample level, where weights are produced by instance-level pairwise interactions.
Revised: Traditionally, attention weights are learned at the instance or sample level, where weights are produced by instance-level pairwise interactions.

Original: As a result, these instance-speciﬁc interactions often ﬂuctuate freely across different instances as they lack a consistent global context.
Revised: As a result, these instance-speciﬁc interactions often ﬂuctuate freely across different instances as they lack a consistent global context.

Original: This paper proposes S YNTHESIZER , a new model that learns to synthesize the self-alignment matrix instead of manually computing pairwise dot products.
Revised: This paper proposes S YNTHESIZER , a new model that learns to synthesize the self-alignment matrix instead of manually computing pairwise dot products.

Original: We propose a diverse suite of synthesizing functions and extensively evaluate them.
Revised: We propose a diverse suite of synthesizing functions and extensively evaluate them.

Original: We characterize the source information that these synthesizing functions receive, i.e., whether they receive information from individual tokens, token-token interactions, and/or global task information.
Revised: We characterize the source information that these synthesizing functions receive, i.e., whether they receive information from individual tokens, token-token interactions, and/or global task information.

Original: Intuitively, different source inputs to the synthesizing functions should capture diverse views, which may be useful when employed in conjunction.
Revised: Intuitively, different source inputs to the synthesizing functions should capture diverse views, which may be useful when employed in conjunction.

Original: Aside from generalizing the standard Transformer model, we show that it is possible to achieve competitive results with fully global attention weights that do not consider token-token interactions or any instance-level (local) information at all.
Revised: Aside from generalizing the standard Transformer model, we show that it is possible to achieve competitive results with fully global attention weights that do not consider token-token interactions or any instance-level (local) information at all.

Original: More speciﬁcally, a random matrix S YNTHESIZER model achieves a 27 .
Revised: More speciﬁcally, a random matrix S YNTHESIZER model achieves a 27 .

Original: 27 BLEU score on WMT 2014 English-German 1 .
Revised: 27 BLEU score on WMT 2014 English-German 1 .

Original: Via a set of rigorous experiments, we observe that the popular and well-established dot-product content-based attention can be approximated with simpler variants such as random matrices or dense layers without sacriﬁcing much performance in some cases.
Revised: Via a set of rigorous experiments, we observe that the popular and well-established dot-product content-based attention can be approximated with simpler variants such as random matrices or dense layers without sacriﬁcing much performance in some cases.

Original: In our experiments, we also show that our relatively simple Synthesizer models also outperform
Revised: In our experiments, we also show that our relatively simple Synthesizer models also outperform

Original: Dynamic Convolutions (Wu et al., 2019) with a +3.5 % relative improvement in perplexity while being 60% faster.
Revised: Dynamic Convolutions (Wu et al., 2019) with a +3.5 % relative improvement in perplexity while being 60% faster.

Original: On encoding tasks, our factorized Synthesizers can outperform other low-rank efﬁcient Transformer models such as Linformers (Wang et al., 2020).
Revised: On encoding tasks, our factorized Synthesizers can outperform other low-rank efﬁcient Transformer models such as Linformers (Wang et al., 2020).

Original: While simple Synthesizer models are able to perform competitively, our experiments show that the pairwise dot product is still ultimately helpful.
Revised: While simple Synthesizer models are able to perform competitively, our experiments show that the pairwise dot product is still ultimately helpful.

Original: When composing our synthesizing functions with dot products, we ﬁnd that they consistently improve the performance of Transformers.
Revised: When composing our synthesizing functions with dot products, we ﬁnd that they consistently improve the performance of Transformers.

Original: In general, we believe our ﬁndings will spur further investigation and discussion about the true role and utility of the self-attention mechanism in Transformer models.
Revised: In general, we believe our ﬁndings will spur further investigation and discussion about the true role and utility of the self-attention mechanism in Transformer models.

Original: Our Contributions
Revised: Our Contributions

Original: Our key contributions are described as follows:
Revised: Our key contributions are described as follows:

Original: • We propose Synthetic Attention, a new way of learning to attend without explicitly attending (i.e., without dot product attention or content-based attention).
Revised: • We propose Synthetic Attention, a new way of learning to attend without explicitly attending (i.e., without dot product attention or content-based attention).

Original: Instead, we generate the alignment matrix independent of token-token dependencies and explore a potpourri of parameterized functions for synthesizing attention matrices.
Revised: Instead, we generate the alignment matrix independent of token-token dependencies and explore a potpourri of parameterized functions for synthesizing attention matrices.

Original: • We propose S YNTHESIZER , a new model that leverages Synthetic Attention.
Revised: • We propose S YNTHESIZER , a new model that leverages Synthetic Attention.

Original: The model performs competitive to state-of-the-art Transformer models on a wide range of language tasks, including machine translation and language modeling.
Revised: The model performs competitive to state-of-the-art Transformer models on a wide range of language tasks, including machine translation and language modeling.

Original: • Moreover, we show that (1) random learnable alignment matrices perform competitively and (2) token-token dependencies are not necessary to achieve good performance with Transformer models on certain tasks.
Revised: • Moreover, we show that (1) random learnable alignment matrices perform competitively and (2) token-token dependencies are not necessary to achieve good performance with Transformer models on certain tasks.

Original: • On large-scale masked language modeling on the C4 dataset (Raffel et al., 2019), we show that simple random Synthesizers can outperform/match Lightweight Dynamic convolutions (Wu et al., 2019) along with outperforming Transformers and Universal Transformers (Dehghani et al., 2018).
Revised: • On large-scale masked language modeling on the C4 dataset (Raffel et al., 2019) and ﬁnetuning on SuperGLUE and GLUE benchmarks, we show that simple random Synthesizers can outperform/match Lightweight Dynamic convolutions (Wu et al., 2019) along with outperforming Transformers and Universal Transformers (Dehghani et al., 2018).

Original: On two encoding tasks, factorized random Synthesizers outperform low-rank Linformers (Wang et al., 2020).
Revised: On two encoding tasks, factorized random Synthesizers outperform low-rank Linformers (Wang et al., 2020).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Attention-based models are used across a wide spectrum of problem domains.
Revised: Attention-based models are used across a wide spectrum of problem domains.

Original: Such models are especially popular, due to their effectiveness, in the language and vision domains.
Revised: Such models are especially popular, due to their effectiveness, in the language and vision domains.

Original: Attention models can be traced back to the machine translation models of (Bahdanau et al., 2014) and (Luong et al., 2015), where attention is employed to learn soft word alignments between language pairs.
Revised: Attention models can be traced back to the machine translation models of (Bahdanau et al., 2014) and (Luong et al., 2015), where attention is employed to learn soft word alignments between language pairs.

Original: The intuition behind the attention mechanism is deeply-rooted in the notion of memory-based retrieval (Graves et al., 2014;
Revised: The intuition behind the attention mechanism is deeply-rooted in the notion of memory-based retrieval (Graves et al., 2014;

Original: Weston et al., 2014), in which soft differentiable addressing of memory was initially proposed.
Revised: Weston et al., 2014), in which soft differentiable addressing of memory was initially proposed.

Original: The paradigm of learning self-alignments, also known as self-attention, has been largely popularized by Transformer models (Vaswani et al., 2017).
Revised: The paradigm of learning self-alignments, also known as self-attention, has been largely popularized by Transformer models (Vaswani et al., 2017).

Original: This technical narrative has also been explored by a number of other recent studies, including those on intra-attention (Parikh et al., 2016), selfmatching networks (Wang et al., 2017), and LSTMN (Cheng et al., 2016).
Revised: This technical narrative has also been explored by a number of other recent studies, including those on intra-attention (Parikh et al., 2016), selfmatching networks (Wang et al., 2017), and LSTMN (Cheng et al., 2016).

Original: To this end, Transformer models, which function primarily based on self-attention and feed-forward layers, generally serve as a reliable replacement for autoregressive recurrent models.
Revised: To this end, Transformer models, which function primarily based on self-attention and feed-forward layers, generally serve as a reliable replacement for autoregressive recurrent models.

Original: The self-attention layer itself has been the subject of many recent technical innovations.
Revised: The self-attention layer itself has been the subject of many recent technical innovations.

Original: For example, recent studies have investigated improving the layer’s overall efﬁciency via sparsiﬁcation and reducing the complexity of computing the alignment matrix (Child et al., 2019; Kitaev et al., 2020;
Revised: For example, recent studies have investigated improving the layer’s overall efﬁciency via sparsiﬁcation and reducing the complexity of computing the alignment matrix (Child et al., 2019; Kitaev et al., 2020;

Original: Huang et al., 2018; Tay et al., 2020; Beltagy et al., 2020).
Revised: Huang et al., 2018; Tay et al., 2020; Beltagy et al., 2020).

Original: These methods are tightly coupled with the query-key-value paradigm, employing a form of memory-based content retrieval as an attention mechanism.
Revised: These methods are tightly coupled with the query-key-value paradigm, employing a form of memory-based content retrieval as an attention mechanism.

Original: On the other end of the spectrum, there have been studies that advocate for replacing self-attention with convolution (Wu et al., 2019).
Revised: On the other end of the spectrum, there have been studies that advocate for replacing self-attention with convolution (Wu et al., 2019).

Original: The recent surge in interest in simplifying the attention mechanism raises important questions about the role and utility of the pairwise dot products, which are one the deﬁning characteristics of self-attention models.
Revised: The recent surge in interest in simplifying the attention mechanism raises important questions about the role and utility of the pairwise dot products, which are one the deﬁning characteristics of self-attention models.

Original: Meanwhile, in the image domain, (Cordonnier et al., 2019) shows connection of Transformers with CNNs.
Revised: Meanwhile, in the image domain, (Cordonnier et al., 2019) shows connection of Transformers with CNNs.

Original: Our work is a new take on the self-attention mechanism in Transformer models.
Revised: Our work is a new take on the self-attention mechanism in Transformer models.

Original: We delve deeper, starting with replacing the pairwise dot products with what we call synthesizing functions that learn attention matrices that may or may not depend on the input tokens.
Revised: We delve deeper, starting with replacing the pairwise dot products with what we call synthesizing functions that learn attention matrices that may or may not depend on the input tokens.

Original: The most closely related work is ((Raganato et al., 2020)), in which the authors propose using ﬁxed (i.e., not learned) attention patterns in Transformer encoders.
Revised: The most closely related work is ((Raganato et al., 2020)), in which the authors propose using ﬁxed (i.e., not learned) attention patterns in Transformer encoders.

Original: However, the scope of their work is limited to encoders and relies on manually deﬁned handcrafted patterns that seem to work well.
Revised: However, the scope of their work is limited to encoders and relies on manually deﬁned handcrafted patterns that seem to work well.

Original: Our work takes this intuition further and expands on this narrative.
Revised: Our work takes this intuition further and expands on this narrative.

Original:  T HE P ROPOSED M ETHOD
Revised:  T HE P ROPOSED M ETHOD

Original: This section introduces our proposed S YNTHESIZER model.
Revised: This section introduces our proposed S YNTHESIZER model.

Original: At its core, our model is essentially a Transformer model with self-attention modules replaced with our Synthetic Attention modules.
Revised: At its core, our model is essentially a Transformer model with self-attention modules replaced with our Synthetic Attention modules.

Original: Figure 3.1 illustrates the key ideas behind (a) Transformer (b) Dense Synthesizers and (c) Random
Revised: Figure 3.1 illustrates the key ideas behind (a) Transformer (b) Dense Synthesizers and (c) Random

Original: Synthesizers.
Revised: Synthesizers.

Original: 3. S YNTHESIZER M ODEL
Revised: 3. S YNTHESIZER M ODEL

Original: This section introduces Synthetic Attention, our proposed self-attention module.
Revised: This section introduces Synthetic Attention, our proposed self-attention module.

Original: Our model removes the notion of query-key-values in the self-attention module and directly synthesizes the alignment matrix instead.
Revised: Our model removes the notion of query-key-values in the self-attention module and directly synthesizes the alignment matrix instead.

Original: Dense Synthesizer Let us consider the simplest variation of the S YNTHESIZER model which is conditioned on each input token.
Revised: Dense Synthesizer Let us consider the simplest variation of the S YNTHESIZER model which is conditioned on each input token.

Original: Overall, our method accepts an input X ∈ R (cid:96) × d and produces an output of Y ∈ R (cid:96) × d .
Revised: Overall, our method accepts an input X ∈ R (cid:96) × d and produces an output of Y ∈ R (cid:96) × d .

Original: Here, (cid:96) refers to the sequence length and d refers to the dimensionality of the model.
Revised: Here, (cid:96) refers to the sequence length and d refers to the dimensionality of the model.

Original: We ﬁrst adopt F ( . ) , a parameterized function, for projecting input
Revised: We ﬁrst adopt F ( . ) , a parameterized function, for projecting input

Original: X i from d dimensions to (cid:96) dimensions.
Revised: X i from d dimensions to (cid:96) dimensions.

Original: [Equation]
Revised: [Equation]

Original: where F ( . ) is a parameterized function that maps R d to R (cid:96) and i is the i -th token of X .
Revised: where F ( . ) is a parameterized function that maps R d to R (cid:96) and i is the i -th token of X and is applied position-wise (to each vector in the sequence of length (cid:96) ).

Original: Intuitively, this can be interpreted as learning a token-wise projection to the sequence length (cid:96) .
Revised: Intuitively, this can be interpreted as learning a token-wise projection to the sequence length (cid:96) .

Original: Essentially, with this model, each token predicts weights for each token in the input sequence.
Revised: Essentially, with this model, each token predicts weights for each token in the input sequence.

Original: In practice, we adopt a simple two layered feed-forward layer with ReLU activations for F ( . ) :
Revised: In practice, we adopt a simple two layered feed-forward layer with ReLU activations for F ( . ) :

Original: [Equation]
Revised: [Equation]

Original: where σ R is the ReLU activation function and W 1 ∈ R d × d and W 2 ∈ R d × (cid:96) .
Revised: where σ R is the ReLU activation function and W 1 ∈ R d × d and W 2 ∈ R d × (cid:96) .

Original: Hence, B  is now of R (cid:96)× (cid:96) . Given B  , we now compute:
Revised: Hence, B i is now of R (cid:96) . Given B ∈ R (cid:96) × (cid:96) , we now compute:

Original: [Equation]
Revised: [Equation]

Original: where G ( . ) is another parameterized function of X that is analogous to V (value) in the standard
Revised: where G ( . ) is another parameterized function of X that is analogous to V (value) in the standard

Original: Transformer model.
Revised: Transformer model.

Original: This approach eliminates the dot product  altogether by replacing QK (cid:62) in standard Transformers with the synthesizing function F ( . ) .
Revised: This approach eliminates the dot product attention Y = Softmax ( QK (cid:62) ) V altogether by replacing QK (cid:62) in standard Transformers with the synthesizing function F ( . ) .

Original: Random Synthesizer
Revised: Random Synthesizer

Original: The previous variant learns synthetic attention by conditioning on each input of X and projecting to (cid:96) dimensions.
Revised: The previous variant learns synthetic attention by conditioning on each input of X and projecting to (cid:96) dimensions.

Original: Hence, the Dense Synthesizer conditions on each token 
Revised: Hence, the Dense Synthesizer conditions on each token independently, as opposed to pairwise token interactions in the vanilla Transformer model. We consider another variation of S YNTHESIZER where the attention weights are not conditioned on any

Original: [Figure]
Revised: [Figure]

Original: Key
Revised: Key

Original: Value
Revised: Value

Original: Dot Product Attention

Original: Output Input X
Revised: [Figure] X

Original: Value
Revised: Value

Original: Output Input X
Revised: [Figure]

Original: Value
Revised: Value

Original: Output

Original: Random Synthesizer

Original: [Figure]
Revised: [Figure]

Original: c [Figure] independently, as opposed to pairwise token interactions in the vanilla Transformer model. We consider another variation of S YNTHESIZER where the attention weights are not conditioned on any input tokens.
Revised: input tokens.

Original: Instead, the attention weights are initialized to random values.
Revised: Instead, the attention weights are initialized to random values.

Original: These values can then either be trainable or kept ﬁxed (denoted as Fixed ).
Revised: These values can then either be trainable or kept ﬁxed (denoted as Fixed ).

Original: Let R be a randomly initialized matrix.
Revised: Let R be a randomly initialized matrix.

Original: The Random Synthesizer is deﬁned as:
Revised: The Random Synthesizer is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where R ∈ R (cid:96) × (cid:96) .
Revised: where R ∈ R (cid:96) × (cid:96) .

Original: Notably, each head adds (cid:96) 2 parameters to the network.
Revised: Notably, each head adds (cid:96) 2 parameters to the network.

Original: The basic idea 2 of the
Revised: The basic idea 2 of the

Original: Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-speciﬁc alignment that works well globally across many samples.
Revised: Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-speciﬁc alignment that works well globally across many samples.

Original: This is a direct generalization of the recently proposed ﬁxed self-attention patterns Raganato et al.
Revised: This is a direct generalization of the recently proposed ﬁxed self-attention patterns Raganato et al.

Original: Factorized Models
Revised: Factorized Models

Original: The Dense Synthesizer adds d × (cid:96) parameters to the network.
Revised: The Dense Synthesizer adds d × (cid:96) parameters to the network.

Original: On the other hand, the Random Synthesizer adds (cid:96) × (cid:96) parameters.
Revised: On the other hand, the Random Synthesizer adds (cid:96) × (cid:96) parameters.

Original: Here, note that we omit the Q, K projections in the standard Transformer which results in further parameter savings.
Revised: Here, note that we omit the Q, K projections in the standard Transformer which results in further parameter savings.

Original: Despite these savings, synthesized models can be cumbersome to learn when (cid:96) is large.
Revised: Despite these savings, synthesized models can be cumbersome to learn when (cid:96) is large.

Original: Hence, we propose factorized variations of the S YNTHESIZER models and show that these variants perform comparably in practice.
Revised: Hence, we propose factorized variations of the S YNTHESIZER models and show that these variants perform comparably in practice.

Original: Factorized Dense Synthesizer Factorized outputs not only slightly reduce the parameter cost of the S YNTHESIZER but also aid in preventing overﬁtting.
Revised: Factorized Dense Synthesizer Factorized outputs not only slightly reduce the parameter cost of the S YNTHESIZER but also aid in preventing overﬁtting.

Original: The factorized variant of the dense synthesizer can be expressed as follows:
Revised: The factorized variant of the dense synthesizer can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where F A ( . ) projects input X i into a dimensions, F B ( . ) projects
Revised: where F A ( . ) projects input X i into a dimensions, F B ( . ) projects

Original: X i to b dimensions, and a × b = (cid:96) .
Revised: X i to b dimensions, and a × b = (cid:96) .

Original: The output of the factorized module is now written as:
Revised: The output of the factorized module is now written as:

Original: [Equation]
Revised: [Equation]

Original: where C = H A ( A ) ∗ H B ( B ) where H A , H B are tiling functions and C ∈ R (cid:96) × (cid:96) .
Revised: where C = H A ( A ) ∗ H B ( B ) where H A , H B are tiling functions and C ∈ R (cid:96) × (cid:96) .

Original: The tiling function simply duplicates the vector k times, i.e., R (cid:96) → R (cid:96) × k .
Revised: The tiling function simply duplicates the vector k times, i.e., R (cid:96) → R (cid:96) × k .

Original: In this case, H A ( · ) is a projection of R a → R a × b and H B ( · ) is a projection of R b → R b × a .
Revised: In this case, H A ( · ) is a projection of R a → R a × b and H B ( · ) is a projection of R b → R b × a .

Original: To avoid having similar values within the same block, we compose the outputs of H A and H B .
Revised: To avoid having similar values within the same block, we compose the outputs of H A and H B .

Original: Factorized Random Synthesizer Similar to Factorized Synthesizers, we are also able to factorize
Revised: Factorized Random Synthesizer Similar to Factorized Synthesizers, we are also able to factorize

Original: R into low rank matrices R 1 , R 2 ∈ R [Equation] × k .
Revised: R into low rank matrices R 1 , R 2 ∈ R (cid:96) × k .

Original: [Equation]
Revised: [Equation]

Original: Therefore, it is easy to see that, for each head, this reduces the parameter costs from (cid:96) 2 to 2( (cid:96)k ) where k << (cid:96) and hence helps prevent overﬁtting.
Revised: Therefore, it is easy to see that, for each head, this reduces the parameter costs from (cid:96) 2 to 2( (cid:96)k ) where k << (cid:96) and hence helps prevent overﬁtting.

Original: In practice, we use a small value of k = 8 .
Revised: In practice, we use a small value of k = 8 .

Original: Mixture of Synthesizers Finally, we note that all of the proposed synthetic attention variants can be mixed in an additive fashion.
Revised: Mixture of Synthesizers Finally, we note that all of the proposed synthetic attention variants can be mixed in an additive fashion.

Original: This can be expressed as:
Revised: This can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: where S ( . ) is a parameterized synthesizing function and the α (where (cid:80) α = 1 ) are learnable weights.
Revised: where S ( . ) is a parameterized synthesizing function and the α (where (cid:80) α = 1 ) are learnable weights.

Original: In the case of mixing Random Factorized with standard Dense Synthesizers, this is expressed as:
Revised: In the case of mixing Random Factorized with standard Dense Synthesizers, this is expressed as:

Original: [Equation]
Revised: [Equation]

Original: We investigate several Mixture of Synthesizers variants in our experiments.
Revised: We investigate several Mixture of Synthesizers variants in our experiments.

Original: On Parameters Depending on Sequence Length Random and dense Synthesizers both rely on parameters that depend on length (cid:96) .
Revised: On Parameters Depending on Sequence Length Random and dense Synthesizers both rely on parameters that depend on length (cid:96) .

Original: In general, we deﬁne a maximum length and dynamically truncate to the actual length of each batch.
Revised: In general, we deﬁne a maximum length and dynamically truncate to the actual length of each batch.

Original: We note that this is in similar spirit to trainable positional encodings which have been common practice in Transformer models.
Revised: We note that this is in similar spirit to trainable positional encodings which have been common practice in Transformer models.

Original: Hence, we do not forsee any issue here.
Revised: Hence, we do not forsee any issue here.

Original: In the case that this is really a problem, one potential solution is to project to a smaller value b and tile b to the maximum sequence length.
Revised: In the case that this is really a problem, one potential solution is to project to a smaller value b and tile b to the maximum sequence length.

Original: We leave this exploration to future work.
Revised: We leave this exploration to future work.

Original: 3. D ISCUSSION
Revised: 3. D ISCUSSION

Original: This paper asks fundamental questions about the attention matrix A and whether it is possible to synthesize A by alternate means other than pairwise attention.
Revised: This paper asks fundamental questions about the attention matrix A and whether it is possible to synthesize A by alternate means other than pairwise attention.

Original: It is worth noting that the regular dot product attention can also be subsumed by our S YNTHESIZER framework, i.e., S YNTHESIZER generalizes the Transformer model.
Revised: It is worth noting that the regular dot product attention can also be subsumed by our S YNTHESIZER framework, i.e., S YNTHESIZER generalizes the Transformer model.

Original: In the case of the Transformer, the synthesizing function in question is S ( X ) =
Revised: In the case of the Transformer, the synthesizing function in question is S ( X ) =

Original: F Q ( X ) F K ( X ) (cid:62) .
Revised: F Q ( X ) F K ( X ) (cid:62) .

Original: [Table]
Revised: [Table]

Original: On
Revised: S (

Revised: [Table]

Original: Sample
Revised: Sample

Original: [Table]
Revised: [Table]

Original:  Table 1 lists the different model variants explored within our S YNTHESIZER framework.
Revised: | θ [Table] Table 1 lists the different model variants explored within our S YNTHESIZER framework.

Original: The ’condition on’ column refers to whether the synthesized output is produced as a function of X i or every
Revised: The ’condition on’ column refers to whether the synthesized output is produced as a function of X i or every

Original: X i , X j pair.
Revised: X i , X j pair.

Original: The ‘sample‘ column indicates whether a given variant leverages local or global context.
Revised: The ‘sample‘ column indicates whether a given variant leverages local or global context.

Original: Random Synthesizers are global because they share the same global alignment patterns across all samples.
Revised: Random Synthesizers are global because they share the same global alignment patterns across all samples.

Original: Dense Synthesizers are considered to be local as they are conditioned on X i , which makes the alignment pattern dependent on each individual sample.
Revised: Dense Synthesizers are considered to be local as they are conditioned on X i , which makes the alignment pattern dependent on each individual sample.

Original: To this end, it is imperative for synthesized models to have multiple heads to be effective.
Revised: To this end, it is imperative for synthesized models to have multiple heads to be effective.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section outlines our experimental setup and results.
Revised: This section outlines our experimental setup and results.

Original: We ﬁrst conduct experiments on ﬁve tasks to evaluate the effectiveness 3 of different Synthesizer variants along with how they compare to the vanilla Transformer.
Revised: We ﬁrst conduct experiments on ﬁve tasks to evaluate the effectiveness 3 of different Synthesizer variants along with how they compare to the vanilla Transformer.

Original: Speciﬁcally, we conduct experiments on (1) machine translation (EnDe,
Revised: Speciﬁcally, we conduct experiments on (1) machine translation (EnDe,

Original: EnFr) (2) autoregressive language modeling (LM1B) (3) text generation (summarization and dialogue modeling and (4) multi-task natural language processing (GLUE/SuperGLUE).
Revised: EnFr) (2) autoregressive language modeling (LM1B) (3) text generation (summarization and dialogue modeling and (4) multi-task natural language processing (GLUE/SuperGLUE).

Original: Details of each experiments can be found in the appendix.
Revised: Details of each experiments can be found in the appendix.

Revised: Notation of Variants We use R to denote Random, D to denote Dense and V to denote vanilla dot product attention.

Revised: Fix to represent Fixed Random, FR to represent Factorized Random and FD to represent Factorized random.

Revised: For Mixture Synthesizers, we use + to denote that two methods are mixed.

Original: 4. C OMPARING S YNTHESIZER V ARIANTS AND T RANSFORMER M ODELS
Revised: 4. C OMPARING S YNTHESIZER V ARIANTS AND T RANSFORMER M ODELS

Original: This section dives into a detailed study of multiple Synthesizer variants and the base Transformer model.
Revised: This section dives into a detailed study of multiple Synthesizer variants and the base Transformer model.

Original: [Table]
Revised: [Table]

Original: Experimental Results on MT/LM First, we observe that our Random Synthesizer baseline achieves 27 .
Revised: Experimental Results on MT/LM First, we observe that our Random Synthesizer baseline achieves 27 .

Original: 27 on EnDe and 41 .
Revised: 27 on EnDe and 41 .

Original: 12 on EnFr.
Revised: 12 on EnFr.

Original: The non-trainable (i.e., ﬁxed) variant performs substantially worse, but still yields surprisingly strong ≈ 24 BLEU with ﬁxed random attention weights.
Revised: The non-trainable (i.e., ﬁxed) variant performs substantially worse, but still yields surprisingly strong ≈ 24 BLEU with ﬁxed random attention weights.

Original: Most other S YNTHESIZER variants achieve competitive performance, although with slight performance degradation compared to Transformers.
Revised: Most other S YNTHESIZER variants achieve competitive performance, although with slight performance degradation compared to Transformers.

Original: An interesting ﬁnding is that the Mixture model of Random + Dense synthesizer performs comparably to vanilla Transformers on EnDe.
Revised: An interesting ﬁnding is that the Mixture model of Random + Dense synthesizer performs comparably to vanilla Transformers on EnDe.

Original: When mixing the standard dot product attention, performance further increases by +0 .
Revised: When mixing the standard dot product attention, performance further increases by +0 .

Original: 8 BLEU points (EnDe).
Revised: 8 BLEU points (EnDe).

Original: In general, the performance of S YNTHESIZER variants are competitive with Transformers for this task.
Revised: In general, the performance of S YNTHESIZER variants are competitive with Transformers for this task.

Original: On LM1b, We ﬁnd that the Random
Revised: On LM1b, We ﬁnd that the Random

Original: Synthesizers perform within 1 - 2 PPL points away from the vanilla Transformer model.
Revised: Synthesizers perform within 1 - 2 PPL points away from the vanilla Transformer model.

Original: The best performing model is the Synthesizer (D+V), which achieves the best performance on this setting.
Revised: The best performing model is the Synthesizer (D+V), which achieves the best performance on this setting.

Original: Results on Text Generation For summarization, we ﬁnd that the (R) and (D) variants do not outperform
Revised: Results on Text Generation For summarization, we ﬁnd that the (R) and (D) variants do not outperform

Original: Transformers.
Revised: Transformers.

Original: The performance of the (D) model is ≈ 2 Rouge-L points below Transformers.
Revised: The performance of the (D) model is ≈ 2 Rouge-L points below Transformers.

Original: Hence, we postulate that the local sample-wise pairwise interactions are important for the summarization task.
Revised: Hence, we postulate that the local sample-wise pairwise interactions are important for the summarization task.

Original: On the other hand, the utility of synthesized attention can also be observed, i.e., the (R+V) and (R+D) models both outperform Transformers.
Revised: On the other hand, the utility of synthesized attention can also be observed, i.e., the (R+V) and (R+D) models both outperform Transformers.

Original: On the dialogue task, Synthesizers (R) and (D) both outperform vanilla Transformers by a reasonable margin ( ≈ 1-3) points across most/all metrics.
Revised: On the dialogue task, Synthesizers (R) and (D) both outperform vanilla Transformers by a reasonable margin ( ≈ 1-3) points across most/all metrics.

Original: The best performing model here is the (D) variant.
Revised: The best performing model here is the (D) variant.

Original: Surprisingly, unlike most other tasks, the (+V) variants do not perform well, signifying that dot product self-attention may actually be harmful for this task.
Revised: Surprisingly, unlike most other tasks, the (+V) variants do not perform well, signifying that dot product self-attention may actually be harmful for this task.

Original: [Table]
Revised: [Table]

Original: Results on GLUE and SuperGLUE Tables 4 and 5 report results on the GLUE and SuperGLUE benchmarks.

Original: We note that the (R) and (D) variants of S YNTHESIZER do not achieve reasonable performance.

Original: This can be largely attributed to the fact that the encoder self-attention in the T5 setting also functions as a cross-sentence attention.

Original: For example, in the entailment or reading comprehension tasks, the premise and hypothesis are concatenated together and self-attention effectively acts as cross-sentence attention 4 .

Original: On datasets like SST, a straightforward sentiment classiﬁcation task, this cross sentence attention is not necessary and therefore Syn (R) and Syn (D) both perform com- petitively.

Original: Optimistically, we observe that the mixture model Syn (R+V) outperforms the T5 model by a substantial margin (+1.9 points on SuperGLUE and +0.6 points on GLUE).

Original: [Table]

Original: [Table]

Original: 4. C OMPARING S YNTHESIZERS WITH D YNAMIC C ONVOLUTIONS
Revised: 4. C OMPARING S YNTHESIZERS WITH D YNAMIC C ONVOLUTIONS

Original: To ascertain the competitiveness of Synthesizers [Table] we  compare them with Dynamic convolutions (Wu et al., 2019).
Revised: To ascertain the competitiveness of Synthesizers, we also compare them with Dynamic convolutions (Wu et al., 2019).

Revised: We compare them on (1) pretraining perplexity using the masked language modeling objective on C4 and (2) downtream ﬁnetuning results on GLUE and SuperGLUE.

Revised: Results on Masked Language

Original:  We also benchmark the speed of these models.
Revised: Modeling We also benchmark the speed of these models.

Original: In order to do so [Table] we conduct additional experiments on the T5 adaptation of masked language modeling on the C4 dataset (Raffel et al., 2019) by comparing against lightweight dynamic convolutions (Wu et al., 2019) on a masked language modeling task.
Revised: In order to do so, we conduct additional experiments on the T5 adaptation of masked language modeling on the C4 dataset (Raffel et al., 2019) by comparing against lightweight dynamic convolutions (Wu et al., 2019) on a masked language modeling task.

Original: We also take this chance to benchmark the  speed of Synthesizers compared with Transformers.
Revised: We also take this chance to benchmark the [Table] [Table] speed of Synthesizers compared with Transformers.

Original: Experiments are conducted on Mesh Tensorﬂow (Shazeer et al., 2018) and ran on 2x2 TPU V3 Chips for approximately 524 K steps.
Revised: Experiments are conducted on Mesh Tensorﬂow (Shazeer et al., 2018) and ran on 2x2 TPU V

Original: [Table]
Revised: [Table]

Original: Results on MLM Table 6 reports the validation set log perplexity on masked language modeling 5 .
Revised: Results on MLM Table 4 reports the validation set log perplexity on masked language modeling 4 .

Original: We observe that Synthesizers (R) can outperform Dynamic Convolutions by a relative +3.5% while being +60% faster.
Revised: We observe that Synthesizers (R) can outperform Dynamic Convolutions by a relative +3.5% while being +60% faster.

Original: Against Lightweight Dynamic Convolutions, we match the performance while being +5% faster.
Revised: Against Lightweight Dynamic Convolutions, we match the performance while being +5% faster.

Original: Given that this is the simple random Synthesizer baseline, we ﬁnd this extremely interesting how it is able to outperform dynamic convolutions, a relatively complex model.
Revised: Given that this is the simple random Synthesizer baseline, we ﬁnd this extremely interesting how it is able to outperform dynamic convolutions, a relatively complex model.

Original: The
Revised: The

Original: Random Synthesizer also has less FLOPS compared to both convolution models.
Revised: Random Synthesizer also has less FLOPS compared to both convolution models.

Original: On the other hand, the Mixture Synthesizer models that use the dot product attention improves the performance of the base Transformer model with relatively an equal model speed.
Revised: On the other hand, the Mixture Synthesizer models that use the dot product attention improves the performance of the base Transformer model with relatively an equal model speed.

Original: Finally, similar to the earlier results, we see a consistent performance gain of Synthesizer (D+V) and Synthesizer (R+V) outperforming the base Transformer model.
Revised: Finally, similar to the earlier results, we see a consistent performance gain of Synthesizer (D+V) and Synthesizer (R+V) outperforming the base Transformer model.

Revised: Results on GLUE and SuperGLUE Tables 5 and 6 report results on the GLUE and SuperGLUE benchmarks.

Original: 4. C OMPARING S YNTHESIZERS WITH L INFORMERS
Revised: We note that the (R) and (D) variants of S YNTHESIZER do not achieve reasonable performance. This can be largely attributed to the fact that the encoder self-attention in the T5 setting also functions as a cross-sentence attention. For example, in the entailment or reading comprehension tasks, the premise and hypothesis are concatenated together and self-attention effectively acts as cross-sentence attention 5 . On datasets like SST, a straightforward sentiment classiﬁcation

Revised: On

Revised: , the perceived success of pairwise

Revised: -attention might also be attributed to the that these reality this is computationally prohibitive task, this cross sentence attention is not necessary and therefore Syn (R) and Syn (D) both perform competitively.

Revised: To this end, Dynamic Convolutions (Wu et al., 2019) also do not have this encoder ”cross-attention” and therefore also suffer on many of these pairwise matching tasks.

Revised: Notably, in this ‘no cross attention’ setting, the Random Synthesizers are are 4 to 5 percentage points higher in

Revised: GLUE/SuperGLUE score compared to Dynamic Convolutions.

Revised: Optimistically, we observe that the mixture model Syn (R+V) outperforms the T5 model by a substantial margin (+1.9 points on SuperGLUE and +0.6 points on GLUE).

Revised: Naturally, the hybrid mixture model also very substantially outperforms Dynamic Convolution.

Revised: Finally to ensure that the

Revised: Syn (+V) variations are not outperforming Transformers due to simply having more parameters, we also compared with T5 (Base+) which has equal number of parameters to Syn (+V) variants (approximately ≈ 10 M more parameters).

Revised: Our results show that Synthesizers (+V) still outperform T (Base+).

Revised: 4. C OMPARING S YNTHESIZERS WITH L INFORMERS

Original: We conduct more experiments comparing factorized random Synthesizers with Linformers.
Revised: We conduct more experiments comparing factorized random Synthesizers with Linformers.

Original: Since
Revised: Since

Original: Linformer cannot be used to decode, we compare them on two encoding tasks from tensorﬂow datasets (AGnews (Zhang et al., 2015) and movie reviews (Maas et al., 2011)).
Revised: Linformer cannot be used to decode, we compare them on two encoding tasks from tensorﬂow datasets (AGnews (Zhang et al., 2015) and movie reviews (Maas et al., 2011)).

Original: We use k = 32 for both factorized models.
Revised: We use k = 32 for both factorized models.

Original: We also benchmark Transformers on this task.
Revised: We also benchmark Transformers on this task.

Original: Note we do not use contextualized embeddings so results are not comparable with other work.
Revised: Note we do not use contextualized embeddings so results are not comparable with other work.

Original: [Table]
Revised: [Table]

Original: Results We notice that factorized Synthesizers (FR) are competitive with Linformers and Transformers on this task.
Revised: Results We notice that factorized Synthesizers (FR) are competitive with Linformers and Transformers on this task.

Original: The accuracy of Syn (FR) is competitive with Linformers while Syn (FR+V) outperforms both Transformers and Linformers.
Revised: The accuracy of Syn (FR) is competitive with Linformers while Syn (FR+V) outperforms both Transformers and Linformers.

Original: 4. O VERALL S UMMARY OF Q UANTITATIVE R ESULTS
Revised: 4. O VERALL S UMMARY OF Q UANTITATIVE R ESULTS

Original: This section summarizes our overall ﬁndings.
Revised: This section summarizes our overall ﬁndings.

Original: • Synthetic Attention is competitive even without Dot Product Attention On all evaluated tasks, we showed that synthesized attention functions competitively, i.e., it achieves performance reasonably close to the dot product self-attention.
Revised: • Synthetic Attention is competitive even without Dot Product Attention On all evaluated tasks, we showed that synthesized attention functions competitively, i.e., it achieves performance reasonably close to the dot product self-attention.

Original: On one task (dialogue generation), the dot product self-attention is found to actually degrade performance.
Revised: On one task (dialogue generation), the dot product self-attention is found to actually degrade performance.

Original: Amongst the other tasks, machine translation is the least affected by the removal of the vanilla dot product.
Revised: Amongst the other tasks, machine translation is the least affected by the removal of the vanilla dot product.

Original: These ﬁndings allow us to introspect about whether pairwise comparisons for self-attention are even necessary.
Revised: These ﬁndings allow us to introspect about whether pairwise comparisons for self-attention are even necessary.

Original: On the multi-task language understanding benchmark, the self-attention functions as a form of cross-attention by concatenating sentence pairs.
Revised: On the multi-task language understanding benchmark, the self-attention functions as a form of cross-attention by concatenating sentence pairs.

Original: Hence, synthesize attention performance is considerably worse than vanilla Transformers.
Revised: Hence, synthesize attention performance is considerably worse than vanilla Transformers.

Original: • Synthetic Attention and Dot Product Attention are highly complementary Overall, we also observe that the dot product attention is very helpful.
Revised: • Synthetic Attention and Dot Product Attention are highly complementary Overall, we also observe that the dot product attention is very helpful.

Original: To this end, synthetic attention is highly complementary to the pairwise dot product attention.
Revised: To this end, synthetic attention is highly complementary to the pairwise dot product attention.

Original: While Synthetic Attention can usually achieve competitive and fast performance on its own, synthetic attention boosts performs, composing multiple synthetic attention (and dot product attention) together shows gains on almost all tasks that we have investigated.
Revised: While Synthetic Attention can usually achieve competitive and fast performance on its own, synthetic attention boosts performs, composing multiple synthetic attention (and dot product attention) together shows gains on almost all tasks that we have investigated.

Original: Hence, we believe this to be a robust ﬁnding.
Revised: Hence, we believe this to be a robust ﬁnding.

Original: The simplest Synthesizers such as Random Synthesizers are fast competitive baselines
Revised: The simplest Synthesizers such as Random Synthesizers are fast competitive baselines

Original: Finally, we note that simple random Synthesizers are competitive with dynamic convolutions and Linformers, which are recently proposed models.
Revised: Finally, we note that simple random Synthesizers are competitive with dynamic convolutions and Linformers, which are recently proposed models.

Original: On two encoding task and a large-scale masked language modeling task, we show that random (or factorized random)
Revised: On two encoding task and a large-scale masked language modeling task, we show that random (or factorized random)

Original: Synthesizers remain competitive to other fast or efﬁcient Transformer models.
Revised: Synthesizers remain competitive to other fast or efﬁcient Transformer models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposed S YNTHESIZER , a new Transformer model that employs Synthetic Attention.
Revised: This paper proposed S YNTHESIZER , a new Transformer model that employs Synthetic Attention.

Original: We conducted a principled study to better understand and evaluate the utility of global alignment and local, instance-wise alignment (e.g., independent token and token-token based) in self-attention.
Revised: We conducted a principled study to better understand and evaluate the utility of global alignment and local, instance-wise alignment (e g., independent token and token-token based) in self-attention.

Original: We show that, on multiple tasks such as machine translation, language modeling, dialogue generation, masked language modeling and document classiﬁcation, synthetic attention demonstrates competitive performance compared to vanilla self-attention.
Revised: We show that, on multiple tasks such as machine translation, language modeling, dialogue generation, masked language modeling and document classiﬁcation, synthetic attention demonstrates competitive performance compared to vanilla self-attention.

Original: Moreover, for the dialogue generation task, pairwise interactions actually hurt performance.
Revised: Moreover, for the dialogue generation task, pairwise interactions actually hurt performance.

Original: Notably, we reemphasize that this study refers to self-attention.
Revised: Notably, we reemphasize that this study refers to self-attention.

Original: We found that we are not able to replace cross-attention with simpler variants in most cases.
Revised: We found that we are not able to replace cross-attention with simpler variants in most cases.

Original: Via a set of additional large-scale experiments, also ﬁnd that Synthesizers can outperform or match Dynamic Convolutions and Factorized Synthesizers can outperform other low rank Linformer models.
Revised: Via a set of additional large-scale experiments, also ﬁnd that Synthesizers can outperform or match Dynamic Convolutions and Factorized Synthesizers can outperform other low rank Linformer models.

Original: A A PPENDIX A. D ETAILED S ETUP FOR E XPERIMENTS
Revised: A A PPENDIX A. D ETAILED S ETUP FOR E XPERIMENTS

Original: Machine Translation We implement our models in Tensor2Tensor, using the standard base hyperparameter settings.
Revised: Machine Translation We implement our models in Tensor2Tensor, using the standard base hyperparameter settings.

Original: Speciﬁcally, we use byte-pair encoding (BPE), 6-layered Transformer networks with hidden size 512 , ﬁlter size of 2048 and 8 heads.
Revised: Speciﬁcally, we use byte-pair encoding (BPE), 6-layered Transformer networks with hidden size 512 , ﬁlter size of 2048 and 8 heads.

Original: We use label smoothing of0 . 1 .
Revised: We use label smoothing of

Original: The maximum sequence length is set to 256 .
Revised: The maximum sequence length is set to 256 .

Original: Training is performed using 8 x V100 GPUs.
Revised: Training is performed using 8 x V100 GPUs.

Original: We train all models for 250 K steps and report results at the last checkpoint.
Revised: We train all models for 250 K steps and report results at the last checkpoint.

Original: We use a length penalty of 0 .
Revised: We use a length penalty of 0 .

Original: 6 and beam size of 4 following the default settings.
Revised: 6 and beam size of 4 following the default settings.

Original: We also compare with standard Transformer models.
Revised: We also compare with standard Transformer models.

Original: In the interest of keeping a consistent, fair evaluation across all model settings, we do not use checkpoint averaging or tune the decoding hyperparameters although this generally leads to better performance.
Revised: In the interest of keeping a consistent, fair evaluation across all model settings, we do not use checkpoint averaging or tune the decoding hyperparameters although this generally leads to better performance.

Original: We evaluate BLEU scores using sacrebleu .
Revised: We evaluate BLEU scores using sacrebleu .

Original: Language Modeling We implement our models in Tensor2Tensor using the packed TPU setup of sequence length 256 .
Revised: Language Modeling We implement our models in Tensor2Tensor using the packed TPU setup of sequence length 256 .

Original: We train our models on 300 K steps on 16 TPU V2 chips.
Revised: We train our models on 300 K steps on 16 TPU V2 chips.

Original: We use the lmx base model setting for fair comparison across all model variations.
Revised: We use the lmx base model setting for fair comparison across all model variations.

Original: The model has 6 layers and 8 heads, along with a ﬁlter width of 2048 and hidden size of 512 .
Revised: The model has 6 layers and 8 heads, along with a ﬁlter width of 2048 and hidden size of 512 .

Original: We used conv relu for the positional feed-forward layers across all baselines since we ﬁnd them to perform slightly better.
Revised: We used conv relu for the positional feed-forward layers across all baselines since we ﬁnd them to perform slightly better.

Original: We report results (subword level perplexity scores) on thetest set at the ﬁnalcheckpoint.
Revised: We report results (subword level perplexity scores) on the ﬁnal

Original: Summarization For the summarization task, we train all models for 300 K steps and a batch size of 128 .
Revised: Summarization For the summarization task, we train all models for 300 K steps and a batch size of 128 .

Original: All models use the base size setting.
Revised: All models use the base size setting.

Original: For the dialogue generation task, due to the smaller dataset size, we train a small model for 20 K steps.
Revised: For the dialogue generation task, due to the smaller dataset size, we train a small model for 20 K steps.

Original: All results are reported on the test set.
Revised: All results are reported on the test set.

Original: For the summarization task, we use the well-established metrics, i.e., Rouge-1, Rouge-2 and Rouge-L.
Revised: For the summarization task, we use the well-established metrics, i.e., Rouge-1, Rouge-2 and Rouge-L.

Original: Experiments are conducted using Mesh Tensorﬂow.
Revised: Experiments are conducted using Mesh Tensorﬂow.

Original: Dialogue Generation For the dialogue generation task, we train our models on the small size for 20 K steps.
Revised: Dialogue Generation For the dialogue generation task, we train our models on the small size for 20 K steps.

Original: Experiments are conducted in Tensor2Tensor.
Revised: Experiments are conducted in Tensor2Tensor.

Original: We use NLG-Eval 6 (Sharma et al., 2017) and report BLEU-1, BLEU-4, Rouge-L, Meteor, CIDr and Embedding based similarity scores (Emb).
Revised: We use NLG-Eval 6 (Sharma et al., 2017) and report BLEU-1, BLEU-4, Rouge-L, Meteor, CIDr and Embedding based similarity scores (Emb).

Original: Multi-Task Language
Revised: Multi-Task Language

Original: Understanding Our experiments are based on the T5 repository 7 implemented in Mesh Tensorﬂow (Shazeer et al., 2018).
Revised: Understanding Our experiments are based on the T5 repository 7 implemented in Mesh Tensorﬂow (Shazeer et al., 2018).

Original: We pre-train the vanilla T5 models and our models for 524288 steps using the span denoising objective.
Revised: We pre-train the vanilla T5 models and our models for 524288 steps using the span denoising objective.

Original: We then co-train the model on multiple tasks.
Revised: We then co-train the model on multiple tasks.

Original: We co-train on the en mix mixture (SuperGLUE and GLUE) for 100 k steps with a constant learning rate of 10 − 3 .
Revised: We co-train on the en mix mixture (SuperGLUE and GLUE) for 100 k steps with a constant learning rate of 10 − 3 .

Original: Embedding and Softmax output layer parameters are kept ﬁxed.
Revised: Embedding and Softmax output layer parameters are kept ﬁxed.

Original: The maximum sequence length is set to 512 .
Revised: The maximum sequence length is set to 512 .

Original: We evaluate on the en mix mixture as deﬁned in the original codebase which is comprised of training GLUE, SuperGLUE and SQuAD in a single model.
Revised: We evaluate on the en mix mixture as deﬁned in the original codebase which is comprised of training GLUE, SuperGLUE and SQuAD in a single model.

Original: Pretraining experiments on C4 Experiments are conducted on Mesh Tensorﬂow.
Revised: Pretraining experiments on C4 Experiments are conducted on Mesh Tensorﬂow.

Original: We pretrain for 524288 steps and report the perplexity on the validation set.
Revised: We pretrain for 524288 steps and report the perplexity on the validation set.

Original: We use 2x2 TPU V3 chips for our experiments.
Revised: We use 2x2 TPU V3 chips for our experiments.

Original: The sequence length is 512 and optimizer is Adafactor.
Revised: The sequence length is 512 and optimizer is Adafactor.

Original: Experiments on Document Classiﬁcation We run experiments in JAX/FLAX ( https:// github.com/google/flax ) with base size models of 8 heads, 6 layers, MLP dimensions of 2048 and a hidden size of 512 .
Revised: Experiments on Document Classiﬁcation We run experiments in JAX/FLAX ( https:// github.com/google/flax ) with base size models of 8 heads, 6 layers, MLP dimensions of 2048 and a hidden size of 512 .

Original: We use the Adam optimizer with learning rate 0 . 05 and 8 K steps linear warmup.
Revised: We use the Adam optimizer with learning rate 0 . 05 and 8 K steps linear warmup.

Original: We train for 10 K steps and report evaluation results at 10 K step.
Revised: We train for 10 K steps and report evaluation results at 10 K step.

Original: We use a batch size of 128 .
Revised: We use a batch size of 128 .

Original: We build a new sentencepiece model for each new dataset comprising of 32 K tokens.
Revised: We build a new sentencepiece model for each new dataset comprising of 32 K tokens.

Original: No pretraining or contextualized embeddings are used.
Revised: No pretraining or contextualized embeddings are used.

Original: Experiments are run on 16 TPU v3 chips.
Revised: Experiments are run on 16 TPU v3 chips.

Original: A. A DDITIONAL V ARIANTS OF S YNTHESIZER
Revised: A. A DDITIONAL V ARIANTS OF S YNTHESIZER

Original: We report results of several additional variants of S YNTHESIZER , most of which we found to have marginal or no improvement over the simple dense/random variations.
Revised: We report results of several additional variants of S YNTHESIZER , most of which we found to have marginal or no improvement over the simple dense/random variations.

Original: • Convolution - Applying a 1D convolution instead of a 2 layer nonlinear network.
Revised: • Convolution - Applying a 1D convolution instead of a 2 layer nonlinear network.

Original: We vary the ﬁlter width in our experiments.
Revised: We vary the ﬁlter width in our experiments.

Original: • Bottleneck - Converting the 2 layered feed forward network to a bottleneck layer, e.g., 512 → 16 → 512 .
Revised: • Bottleneck - Converting the 2 layered feed forward network to a bottleneck layer, e.g., 512 → 16 → 512 .

Original: We also experiment with a convolutional variant of bottleneck,
Revised: We also experiment with a convolutional variant of bottleneck,

Original: i
Revised: i

Original: [Table]

Original: e., projecting to low dimension space and then projecting back to high dimensions.
Revised: e., projecting to low dimension space and then projecting back to high dimensions.

Original: • Gated Linear Units (GLU), applying the GLU units of (Dauphin et al., 2017) as the Synthesizing function.
Revised: • Gated Linear Units (GLU), applying the GLU units of (Dauphin et al., 2017) as the Synthesizing function.

Original: [Table]
Revised: [Table]

Original: A. E FFECT OF N UMBER OF H EADS
Revised: A. E FFECT OF N UMBER OF H EADS

Original: We also investigate the impact of the number of heads on performance.
Revised: We also investigate the impact of the number of heads on performance.

Original: We trained three Random
Revised: We trained three Random

Original: Synthesizer models for the small version of the machine translation tasks using the T5 framework without pretraining.
Revised: Synthesizer models for the small version of the machine translation tasks using the T5 framework without pretraining.

Original: For simplicity, evaluation is done via greedy decoding.
Revised: For simplicity, evaluation is done via greedy decoding.

Original: We report scores on the development set.
Revised: We report scores on the development set.

Original: We are mainly interested in relative performance and not absolute numbers.
Revised: We are mainly interested in relative performance and not absolute numbers.

Original: Table 9 reports the results on varying the number of heads on performance.
Revised: Table 9 reports the results on varying the number of heads on performance

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: A. A NALYSIS
Revised: Enc L

Original: [Table]
Revised: [Table]

Original: Distribution of Weights We are interested in investigating how the synthetically generated attention weights differ from the dot product attention weights.
Revised: Distribution of Weights We are interested in investigating how the synthetically generated attention weights differ from the dot product attention weights.

Original: Figure 2 shows the attention histograms on trained
Revised: Figure 2 shows the attention histograms on trained

Original: Transformer and S YNTHESIZER models.
Revised: Transformer and S YNTHESIZER models.

Original: We report histograms at layers 1 , 3 , and 5 of a 6 layered (Transformer or S YNTHESIZER ) model at 50 K steps.
Revised: We report histograms at layers 1 , 3 , and 5 of a 6 layered (Transformer or S YNTHESIZER ) model at 50 K steps.

Original: We found that the weight distributions remain relatively identical thereafter.
Revised: We found that the weight distributions remain relatively identical thereafter.

Original: Figure 3 shows the initialization state.
Revised: Figure 3 shows the initialization state.

Original: We observe that there are distinct differences in the weight distribution of
Revised: We observe that there are distinct differences in the weight distribution of

Original: S YNTHESIZER and Transformer models.
Revised: S YNTHESIZER and Transformer models.

Original: The variance of the S YNTHESIZER weights tends to be higher.
Revised: The variance of the S YNTHESIZER weights tends to be higher.

Original: On the other hand, the weights on the Transformer model tends to gravitate near 0 and have smaller variance.
Revised: On the other hand, the weights on the Transformer model tends to gravitate near 0 and have smaller variance.

Original: There are also notable differences across the (R) and (D)
Revised: There are also notable differences across the (R) and (D)

Original: S YNTHESIZER variants.
Revised: S YNTHESIZER variants.

Original: Speciﬁcally, the (D) model in general has greater max values with more values in the 0 .
Revised: Speciﬁcally, the (D) model in general has greater max values with more values in the 0 .

Original: 2 range while the values of the R model tends to stay closer to 0 .
Revised: 2 range while the values of the R model tends to stay closer to 0 .

Original: A. W HAT PATTERNS DO S YNTHESIZERS LEARN ?
Revised: A. W HAT PATTERNS DO S YNTHESIZERS LEARN ?

Original: In this section, we perform a deeper analysis of the S YNTHESIZER model.
Revised: In this section, we perform a deeper analysis of the S YNTHESIZER model.

Original: Synthesizer weights on LM1B.
Revised: Synthesizer weights on LM1B.

Original: Transformer weights on LM1B.
Revised: Transformer weights on LM1B.

Original: Analysis Finally, we are interested to understand what these
Revised: Analysis Finally, we are interested to understand what these

Original: Synthesizer models are learning.
Revised: Synthesizer models are learning.

Original: We inspect the random synthetic attention weights for language modeling task LM1B and visualise the differences compared to the vanilla attention.
Revised: We inspect the random synthetic attention weights for language modeling task LM1B and visualise the differences compared to the vanilla attention.

Original: We ﬁnd that, for the LM task, Synthesizers are capable of learning a local window, emulating the vanilla Transformer quite closely despite starting from completely random.
Revised: We ﬁnd that, for the LM task, Synthesizers are capable of learning a local window, emulating the vanilla Transformer quite closely despite starting from completely random.

Original: The weights, however, seem smoother and less coarse as compared to the Transformer.
Revised: The weights, however, seem smoother and less coarse as compared to the Transformer.

Original: This seems to reﬂect what we expect since the Synthesizer does not beneﬁt from token speciﬁc information.
Revised: This seems to reﬂect what we expect since the Synthesizer does not beneﬁt from token speciﬁc information.

Original: We provide additional analysis and visualisation of weights for the Machine Translation task in the supplementary material.[Figure] [Figure]
Revised: We provide additional analysis and visualisation of weights for the Machine Translation task in the supplementary material.

Original:  M ORE A TTENTION W EIGHTS A NALYSIS
Revised: A. M ORE A TTENTION W EIGHTS A NALYSIS

Original: This section illustrates the attention weights extracted from different variants of Synthesizer on the machine translation (En-De) task.
Revised: This section illustrates the attention weights extracted from different variants of Synthesizer on the machine translation (En-De) task.

Original: Weights are extracted from lower layers although we do not ﬁnd
Revised: Weights are extracted from lower layers although we do not ﬁnd

Original: any substantial difference in the patterns in early layers and deeper layers.
Revised: any substantial difference in the patterns in early layers and deeper layers.

Original: We extract them from
Revised: We extract them from

Original: Tensorboard midway during training.
Revised: Tensorboard midway during training.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Analysis We ﬁrst observe that these weights differ a lot from the LM weights shown in the main paper in Section 4.5.
Revised: Analysis We ﬁrst observe that these weights differ a lot from the LM weights shown in the main paper in Section 4.5.

Original: This shows that the Synthesizer learns very different weights for different tasks.
Revised: This shows that the Synthesizer learns very different weights for different tasks.

Original: Next, based on the weighs on MT, we observe a very different pattern in all variants of Synthesizer.
Revised: Next, based on the weighs on MT, we observe a very different pattern in all variants of Synthesizer.

Original: For the decoder weights, the main difference seems to be the overall magnitude and distribution values of the weights.
Revised: For the decoder weights, the main difference seems to be the overall magnitude and distribution values of the weights.

Original: However, we can easily observe the cracks and lines of the factorized variants.
Revised: However, we can easily observe the cracks and lines of the factorized variants.

Original: For the encoder weights, we observe that the Random and Dense variants are more uniform.
Revised: For the encoder weights, we observe that the Random and Dense variants are more uniform.

Original: On the other hand, there appears to be structural/regional clustering of values in the factorized variants.
Revised: On the other hand, there appears to be structural/regional clustering of values in the factorized variants.

Revised: A. C ONVERGENCE OF S YNTHESIZERS VS T RANSFORMERS Convergence of Synthesizers (Brown) vs Transformers (Blue)

Revised: Figure A.7 shows the convergence of Synthesizers vs Transformers on SuperGLUE ﬁnetuning.

Revised: In general, Synthesizers and Transformers have pretty similar convergence patterns and this is also observed across other tasks.