Original: DARTFormer: Finding The Best Type Of Attention
Revised: DARTFormer: Finding The Best Type Of Attention

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Given the wide and ever growing range of different efficient Transformer attentionmechanisms, it is important to identify which attention is most effective whengiven a task.
Revised: Given the wide and ever growing range of different efficient Transformer attention mechanisms, it is important to identify which attention is most effective when given a task.

Original: In this work, we are also interested in combining different attentiontypes to build heterogeneous Transformers.
Revised: In this work, we are also interested in combining different attention types to build heterogeneous Transformers.

Original: We first propose a DARTS-like Neural Architecture Search (NAS) method to find the best attention for a given task, inthis setup, all heads use the same attention (homogeneous models).
Revised: Focussing on NLP-based tasks, we first propose a DARTS-like Neural Architecture Search (NAS) method to find the best attention for a given task. In this setup, all heads use the same attention (homogeneous models).

Original: Our resultssuggest that NAS is highly effective on this task, and it identifies the best attentionmechanisms for IMDb byte level text classification and Listops.
Revised: Our results suggest that NAS is highly effective at this, and it identifies the best attention mechanisms for IMDb byte level text classification and Listops.

Original: We then extend ourframework to search for and build Transformers with multiple different attentiontypes, and call them heterogeneous Transformers.
Revised: We then extend our framework to search for and build Transformers with multiple different attention types, and call them heterogeneous Transformers.

Original: We show that whilst theseheterogeneous Transformers are better than the average homogeneous models, theycannot outperform the best .
Revised: We show that whilst these heterogeneous Transformers are better than the average homogeneous models, they cannot outperform the best .

Original: We explore the reasons why heterogeneous attentionmakes sense, and why it ultimately fails.
Revised: We explore the reasons why heterogeneous attention makes sense, and why it ultimately fails.

Original:  Introduction
Revised:  Introduction

Original: Since the first proposal of the Transformer architecture by Vaswani et al.
Revised: Since the first proposal of the Transformer architecture by Vaswani et al.

Original: (2017), many alternativeshave been proposed for the attention mechanism (Zaheer et al., 2020; Katharopoulos et al., 2020;
Revised: (2017) many alternatives have been proposed for the attention mechanism (Zaheer et al., 2020; Katharopoulos et al., 2020;

Original: Wang et al., 2020; Liu et al., 2018b; Beltagy et al., 2020; Choromanski et al., 2020; Kitaev et al. ,2020; Child et al., 2019; Tay et al., 2021) due to the original dot-product attention mechanism havingquadratic complexities in time and space with respect to the length of the input sequence.
Revised: Wang et al., 2020; Liu et al., 2018b; Beltagy et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Child et al., 2019; Tay et al., 2021) due to the original dot-product attention mechanism having quadratic complexities in time and space with respect to the length of the input sequence.

Original: Recentwork (Tay et al., 2020b) showed that these alternative architectures often perform well at differenttasks when there is no pretraining, thus there is no clear attention that is best at every type of task.
Revised: Recent work (Tay et al., 2020b) showed that these alternative architectures often perform well at different tasks when there is no pretraining, thus there is no clear attention that is best at every type of task.

Original: Therefore we ask: Can we efficiently learn the best attention for a given long range task?
Revised: Therefore we ask: Can we efficiently learn the best attention for a given long range task?

Original: It is thought that each attention head in the Transformer can learn a different relationship, muchlike how in Convolutional Neural Networks (CNNs) each kernel learns a different feature.
Revised: It is thought that each attention head in the Transformer can learn a different relationship, much like how in Convolutional Neural Networks (CNNs) each kernel learns a different feature.

Original: Tay et al.(2020b) hypothesizes that each attention mechanism represents a different functional bias for whichattention relationships should be learned, and that the utility of this bias is dependent on the task andits processing requirements.
Revised: Tay et al. (2020b) hypothesizes that each attention mechanism represents a different functional bias for which attention relationships should be learned, and that the utility of this bias is dependent on the task and its processing requirements.

Original: Thus if we had many different attention types in a Transformer, theycould each more easily learn different types of relationship, and thus make the Transformer moreeffective overall.
Revised: Thus if we had many different attention types in a Transformer, they could each more easily learn different types of relationship, and thus make the Transformer more effective overall.

Original: So is the optimal attention for a task a mixture of different attentions?
Revised: So is the optimal attention for a task a mixture of different attentions?

Original: In this paper we apply Neural Architecture Search (NAS) techniques to the Transformer attentionsearch space and propose DARTFormer, a DARTS-like method (Liu et al., 2018a) for finding thebest attention, a high level illustration of the method is presented in Figure 1.
Revised: In this paper we apply Neural Architecture Search (NAS) techniques to the Transformer attention search space and propose DARTFormer, a DARTS-like method (Liu et al., 2018a) for finding the best attention, a high level illustration of the method is presented in Figure 1.

Original: To do this we usemultiple candidate attention types in parallel and sum their outputs in the attention mechanism of the
Revised: To do this we use multiple candidate attention types in parallel and sum their outputs in the attention mechanism of the

Revised: [Figure]

Original: Transformer.
Revised: Transformer.

Original: Following Wang et al.
Revised: Following Wang et al.

Original: we use masked validation accuracy drop as our metric fordetermining the performance of each attention type.
Revised: we use masked validation accuracy drop as our metric for determining the performance of each attention type.

Original: [Figure]

Original: For clarity we refer to computation of the QKV linear projections, multi-head attention, and thenconcatenation and linear projection back down to the sequence features as the attention block.
Revised: For clarity we refer to computation of the QKV linear projections, multi-head attention, and then concatenation and linear projection back down to the sequence features as the attention block.

Original: In astandard Transformer there is only a single attention block that has multiple heads.
Revised: In a standard Transformer there is only a single attention block that has multiple heads.

Original: When trainingmixed attention models (either supernetworks or a final heterogeneous model) we use multipleattention blocks, each containing only a single attention type.
Revised: When training mixed attention models (either supernetworks or a final heterogeneous model) we use multiple attention blocks, each containing only a single attention type.

Original: Following Tay et al.
Revised: Following Tay et al.

Original: (2020b), we use a representative mixture of different attention mechanisms aspart of our search space in order to cover the main methods of achieving efficient attention.
Revised: (2020b), we use a representative mixture of different attention mechanisms as part of our search space in order to cover the main methods of achieving efficient attention.

Original: Thespecific attentions we use are: Bigbird (Zaheer et al., 2020), Linear Transformer (Katharopoulos et al.,2020), Linformer (Wang et al., 2020), Local attention (Liu et al., 2018b), Longformer (Beltagy et al.,2020), Performer (Choromanski et al., 2020), Reformer (Kitaev et al., 2020), Sparse Transformer(Child et al., 2019), and Synthesizer (Tay et al., 2021).
Revised: The specific attentions we use are: Bigbird (Zaheer et al., 2020), Linear Transformer (Katharopoulos et al., 2020), Linformer (Wang et al., 2020), Local attention (Liu et al., 2018b), Longformer (Beltagy et al., 2020), Performer (Choromanski et al., 2020), Reformer (Kitaev et al., 2020), Sparse Transformer (Child et al., 2019), and Synthesizer (Tay et al., 2021).

Original: We use our setup to investigate two key paradigms.
Revised: We use our setup to investigate two key paradigms.

Original: The first is learning the best attention for a newtask with a single layer Transformer, this means only one full-scale Transformer needs to be trainedafter a good attention mechanism is found.
Revised: The first is learning the best attention for a new task with a single layer Transformer, this means only one full-scale Transformer needs to be trained after a good attention mechanism is found.

Original: The second paradigm, illustrated in Figure 1, is using asingle layer Transformer to find the best head-wise heterogeneous attention mixture for that task, andthen using that mixture in each layer of a full Transformer model, making it layer-wise homogeneous.
Revised: The second paradigm, illustrated in Figure 1, is using a single layer Transformer to find the best head-wise heterogeneous attention mixture for that task, and then using that mixture in each layer of a full Transformer model, making it layer-wise homogeneous.

Original: We test these paradigms on three different tasks.
Revised: We test these paradigms on three different tasks.

Original: They are taken from Tay et al.
Revised: They are taken from Tay et al.

Original: (2020b) and werespecifically designed to test the capabilities of efficient long range Transformers.
Revised: (2020b) and were specifically designed to test the capabilities of efficient long range Transformers.

Original: In this paper we make the following contributions:
Revised: In this paper we make the following contributions:

Original: • We propose a DARTS-like framework to efficiently find the best attention for a task.
Revised: • We propose a DARTS-like framework to efficiently find the best attention for a task.

Original: • We extend this framework to building and searching for optimal heterogeneous attention Transformer models.
Revised: • We extend this framework to building and searching for optimal heterogeneous attention Transformer models.

Original: • We empirically show that heterogeneous Transformers cannot outperform the best homoge- neous Transformer for our selected long range NLP tasks.
Revised: • We empirically show that heterogeneous Transformers cannot outperform the best homogeneous Transformer for our selected long range NLP tasks.

Original:  Related Work 
Revised:  Related Work 2.1 Transformer Attention

Original: 2.1 Transformer Attention

Original: Since Vaswani et al.
Revised: Since Vaswani et al.

Original: (2017), a large variety of replacement attention mechanisms were proposed.
Revised: (2017), a large variety of replacement attention mechanisms were proposed.

Original: These use different approaches, such as, low rank approximations and kernel based methods
Revised: These use different approaches, such as, low rank approximations and kernel based methods

Original: (Wang et al., 2020; Choromanski et al., 2020; Katharopoulos et al., 2020; Tay et al., 2021),fixed/factorised/random patterns (Tay et al., 2021; Child et al., 2019; Zaheer et al., 2020; Beltagyet al., 2020; Tay et al., 2020a), learnable patterns (Kitaev et al., 2020; Tay et al., 2020a), recurrence(Dai et al., 2019) and more (Lee et al., 2019).
Revised: (Wang et al., 2020; Choromanski et al., 2020; Katharopoulos et al., 2020; Tay et al., 2021), fixed/factorised/random patterns (Tay et al., 2021; Child et al., 2019; Zaheer et al., 2020; Beltagy et al., 2020; Tay et al., 2020a), learnable patterns (Kitaev et al., 2020; Tay et al., 2020a), recurrence (Dai et al., 2019) and more (Lee et al., 2019).

Original: Tay et al.
Revised: Tay et al.

Original: (2020c) gives a detailed survey of differentattention mechanisms.
Revised: (2020c) gives a detailed survey of different attention mechanisms.

Original: Tay et al.
Revised: Tay et al.

Original: (2020b) compares the performance, speed and memory usage of many of these differentattention mechanisms on a variety of tasks, including NLP and image processing.
Revised: (2020b) compares the performance, speed and memory usage of many of these different attention mechanisms on a variety of tasks, including NLP and image processing.

Original: Their main findingis that the performance of each attention mechanism is highly dependent on the nature of the taskbeing learned when pretrained embeddings or weights aren’t used.
Revised: Their main finding is that the performance of each attention mechanism is highly dependent on the nature of the task being learned when pretrained embeddings or weights aren’t used.

Original: This motivates the initial part ofour research.
Revised: This motivates the initial part of our research.

Original: 2.2 NAS on Transformers
Revised: 2.2 NAS on Transformers

Original: Recent work of Liu et al.
Revised: Recent work of Liu et al.

Original: (2022) applies RankNAS (Hu et al., 2021) to cosFormer (Qin et al. , 2022)and standard Transformers (Vaswani et al., 2017).
Revised: (2022) applies RankNAS (Hu et al., 2021) to cosFormer (Qin et al., 2022) and standard Transformers (Vaswani et al., 2017).

Original: They perform a search of the hyperparameters on acosFormer network and compare these to the same method applied to the standard Transformer.
Revised: They perform a search of the hyperparameters on a cosFormer network and compare these to the same method applied to the standard Transformer.

Original: Tsaiet al.
Revised: Tsai et al.

Original: (2020) searches the hyperparameter space of a BERT (Devlin et al., 2018) model architectureheterogeneously to find an optimal efficient network.
Revised: (2020) searches the hyperparameter space of a BERT (Devlin et al., 2018) model architecture heterogeneously to find an optimal efficient network.

Original: These methods search hyperparameters, whereaswe fix hyperparameters and search over the space of attention mechanisms.
Revised: These methods search hyperparameters, whereas we fix hyperparameters and search over the space of attention mechanisms.

Original:  Method3.1 DARTS Style NAS
Revised:  Method 3.1 DARTS Style NAS

Original: We train a supernetwork that contains attention heads of each attention type.
Revised: We train a supernetwork that contains attention heads of each attention type.

Original: This is analogous tothe edges in a DARTS cell containing all possible operations for that edge.
Revised: This is analogous to the edges in a DARTS cell containing all possible operations for that edge.

Original: We also use ‘fixed α ’with masked validation accuracy as our metric for the strength of each edge as detailed in Wang et al.(2021).
Revised: We also use ‘fixed α ’ with masked validation accuracy as our metric for the strength of each edge as detailed in Wang et al.

Original: Standard DARTS assigns a weight for each edge in the computation graph and softmaxes thison the output of each cell (see Figure 2, left, in Appendix C), since we are using a ’fixed α ’ approachwe do not need to do this and simply average our edges without using learnable weights.
Revised: Standard DARTS assigns a weight for each edge in the computation graph and softmaxes this on the output of each cell (see Figure 2, left, in Appendix C), since we are using a ’fixed α ’ approach we do not need to do this and simply average our edges without using learnable weights.

Original: This allowsus to train the supernetwork until convergence and then select the best attention (or prune out theworst).
Revised: This allows us to train the supernetwork until convergence and then select the best attention (or prune out the worst).

Original: Note that this removes the need to carry out the bi-level optimisation that is required in theoriginal DARTS paradigm that uses validation performance to train the edge weights.
Revised: Note that this removes the need to carry out the bi-level optimisation that is required in the original DARTS paradigm that uses validation performance to train the edge weights.

Original: Our approachis illustrated in Figure 1.
Revised: Our approach is illustrated in Figure 1.

Original: The DARTS supernetwork cell and the standard Transformer architectureare given in Appendix C in Figure 2, to show how our architecture relates to them.
Revised: The DARTS supernetwork cell and the standard Transformer architecture are given in Appendix C in Figure 2, to show how our architecture relates to them.

Original: 3.2 Architecture and Search Space
Revised: 3.2 Architecture and Search Space

Original: Our Transformer encoder supernetwork consists of an embedding layer, an attention block supernetwork, a feed-forward network (FFN), and a linear classifier layer.
Revised: Our Transformer encoder supernetwork consists of an embedding layer, an attention block supernetwork, a feed-forward network (FFN), and a linear classifier layer.

Original: In normal Transformer attention,each attention block consists of: QKV linear projections from features to number of heads × headdimension, scaled dot product attention on each head, concatenation of heads, and a final denseprojection back to the feature dimension (Vaswani et al., 2017).
Revised: In normal Transformer attention, each attention block consists of: QKV linear projections from features to number of heads × head dimension, scaled dot product attention on each head, concatenation of heads, and a final dense projection back to the feature dimension (Vaswani et al., 2017).

Original: In theory, we want to search over thespace of alternatives for the scaled dot product attention operation independently for each head.
Revised: In theory, we want to search over the space of alternatives for the scaled dot product attention operation independently for each head.

Original: Thiswould replace scaled dot product attention in a normal Transformer with an average of candidateattention mechanisms in a DARTS-like paradigm.
Revised: This would replace scaled dot product attention in a normal Transformer with an average of candidate attention mechanisms in a DARTS-like paradigm.

Original: The averages from each head would then beconcatenated and linearly projected back to the feature dimension as normal.
Revised: The averages from each head would then be concatenated and linearly projected back to the feature dimension as normal.

Original: However, some attention mechanisms, such as Reformer (Kitaev et al., 2020), modify the linearprojections or concatenation operations.
Revised: However, some attention mechanisms, such as Reformer (Kitaev et al., 2020), modify the linear projections or concatenation operations.

Original: Because of this we instead search over candidate multi-headattention blocks where each one implements the projections, attention, and concatenation operations.
Revised: Because of this we instead search over candidate multi-head attention blocks where each one implements the projections, attention, and concatenation operations.

Original: This is illustrated in Figure 4 in Appendix C .
Revised: This is illustrated in Figure 4 in Appendix C .

Original: When the candidate blocks are single head and wehave H blocks per candidate attention mechanism, after the architecture search is complete, thecomputation graph is equivalent to if we searched over the attention mechanisms themselves with Hheads.
Revised: When the candidate blocks are single head and we have H blocks per candidate attention mechanism, after the architecture search is complete, the computation graph is equivalent to if we searched over the attention mechanisms themselves with H heads.

Original: With the search at the block level with single heads, each attention mechanism can learn its own linear projections, whereas in a search over just the attention mechanisms these would be sharedwithin each head.
Revised: With the search at the block level with single heads, each attention mechanism can learn its own linear projections, whereas in a search over just the attention mechanisms these would be shared within each head.

Original: The disadvantage of this is increased memory and computation during the search,since now heads do not share the linear projection.
Revised: The disadvantage of this is increased memory and computation during the search, since now heads do not share the linear projection.

Original: 3.3 Experiments 
Revised: 3.3 Experiments 3.3.1 Finding Optimal Homogeneous Attention

Original: 3.3.1 Finding Optimal Homogeneous Attention

Original: For the first experiment we train a single layer network with a block for each attention mechanism.
Revised: For the first experiment we train a single layer network with a block for each attention mechanism.

Original: These blocks are each initialized with the desired number of final heads for that task.
Revised: These blocks are each initialized with the desired number of final heads for that task.

Original: After trainingwe perform a single masked validation accuracy trial and pick the highest scoring mechanism as agood candidate mechanism for that task when used in a full Transformer model homogeneously.
Revised: After training we perform a single masked validation accuracy trial and pick the highest scoring mechanism as a good candidate mechanism for that task when used in a full Transformer model homogeneously.

Original: Thisparadigm is summarized in an algorithmic form in Algorithm 1 in Appendix D.
Revised: This paradigm is summarized in an algorithmic form in Algorithm 1 in Appendix D.

Original: 3.3.2 Finding Optimal Heterogeneous Attention
Revised: 3.3.2 Finding Optimal Heterogeneous Attention

Original: In this paradigm we try to learn an optimal single layer with a fixed number of heads and stack it,analogous to searching for an optimal cell in DARTS.
Revised: In this paradigm we try to learn an optimal single layer with a fixed number of heads and stack it, analogous to searching for an optimal cell in DARTS.

Original: We begin with H attention blocks of each attention type, each one a single head, where H is thedesired number of final attention heads in our heterogeneous Transformer.
Revised: We begin with H attention blocks of each attention type, each one a single head, where H is the desired number of final attention heads in our heterogeneous Transformer.

Original: We train the supernetworkto rough convergence and then alternate between removing the worst attention block and fine-tunetraining the remaining model.
Revised: We train the supernetwork to rough convergence and then alternate between removing the worst attention block and fine-tune training the remaining model.

Original: We do this until we have only H single-head attention blocks left, eachpotentially a different attention mechanism.
Revised: We do this until we have only H single-head attention blocks left, each potentially a different attention mechanism.

Original: Once we have the optimal layer architecture for the task, we reinitialize the Transformer network withmultiple layers where each one contains the mix of attentions found to be optimal.
Revised: Once we have the optimal layer architecture for the task, we reinitialize the Transformer network with multiple layers where each one contains the mix of attentions found to be optimal.

Original: Where possiblewe group attentions of the same type into the same attention block with multiple heads.
Revised: Where possible we group attentions of the same type into the same attention block with multiple heads.

Original: We thentrain this full Transformer model on the task until convergence.
Revised: We then train this full Transformer model on the task until convergence.

Original: This paradigm is summarized in
Revised: This paradigm is summarized in

Original: Algorithm 2 in Appendix D.
Revised: Algorithm 2 in Appendix D.

Original: We will refer to this method as ‘NAS Prune’.
Revised: We will refer to this method as ‘NAS Prune’.

Original: We also test a simpler method where we take the top 4 scoring attentions from the single layer to findthe best homogeneous attention, and then train a full Transformer network with an even mix of theseattentions at each layer.
Revised: We also test a simpler method where we take the top 4 scoring attentions from the single layer to find the best homogeneous attention, and then train a full Transformer network with an even mix of these attentions at each layer.

Original: This allows us to evaluate the general capabilities of heterogeneous attention.
Revised: This allows us to evaluate the general capabilities of heterogeneous attention.

Original: We will refer to this method as ‘NAS One-shot’.
Revised: We will refer to this method as ‘NAS One-shot’.

Original: We also experiment with a head-wise homogeneous, layer-wise heterogeneous attention Transformer.
Revised: We also experiment with a head-wise homogeneous, layer-wise heterogeneous attention Transformer.

Original: This is detailed in Appendix B.
Revised: This is detailed in Appendix B.

Original: 3.4 Attention Evaluation
Revised: 3.4 Attention Evaluation

Original: In order to evaluate the strength of each block we used a masked validation accuracy drop metric(Wang et al., 2021).
Revised: In order to evaluate the strength of each block we used a masked validation accuracy drop metric (Wang et al., 2021).

Original: First we get a baseline accuracy by running our model on the validation set.
Revised: First we get a baseline accuracy by running our model on the validation set.

Original: Then, for each block, we mask it out and record the new validation accuracy.
Revised: Then, for each block, we mask it out and record the new validation accuracy.

Original: This is illustrated in
Revised: This is illustrated in

Original: Figure 3 in Appendix C.
Revised: Figure 3 in Appendix C.

Original: The best block is the one which drops the accuracy the most, and the worsteither drops it the least or increases it the most.
Revised: The best block is the one which drops the accuracy the most, and the worst either drops it the least or increases it the most.

Original:  Evaluation
Revised:  Evaluation

Original: For each experiment we run the search on 3 different tasks taken from Long Range Arena (LRA)(Tay et al., 2020b).
Revised: For each experiment we run the search on 3 different tasks taken from Long Range Arena (LRA) (Tay et al., 2020b).

Original: These are: byte level binary text classification on IMDb dataset with 1k sequencelength, Listops - 10-way classification with 2k sequence length, and byte level document matchingwhich uses 4k sequence length.
Revised: These are: byte level binary text classification on IMDb dataset with 1k sequence length, Listops - 10-way classification with 2k sequence length, and byte level document matching which uses 4k sequence length.

Original: Hyperparameters typically match those used in LRA, details aregiven in the Appendix A. One notable difference is that we train each homogeneous model 3 timesand take an average.
Revised: Hyperparameters typically match those used in LRA, details are given in the Appendix A. One notable difference is that we train each homogeneous model 3 times and take an average.

Original: The Listops and document matching tasks are also trained for longer to allowthe models to better converge.
Revised: The Listops and document matching tasks are also trained for longer to allow the models to better converge.

Original: 4.1 Finding Optimal Homogeneous Attention
Revised: 4.1 Finding Optimal Homogeneous Attention

Original: Table 1 shows us that our selection process is effective at identifying the best attention mechanismfor the text classification and Listops tasks.
Revised: Table 1 shows us that our selection process is effective at identifying the best attention mechanism for the text classification and Listops tasks.

Original: For both it also picks out the third best performing modelwith either its own second or third best.
Revised: For both it also picks out the third best performing model with either its own second or third best.

Original: For the document matching task, however, the single layerfails to find one of the top 3 attention mechanisms within top 3 picks.
Revised: For the document matching task, however, the single layer fails to find one of the top 3 attention mechanisms within top 3 picks.

Original: One notable difference betweenits’ document matching scores and the scores for the other two are that they are much smaller.
Revised: One notable difference between its’ document matching scores and the scores for the other two are that they are much smaller.

Original: Thiscould indicate that when the score is low magnitude, the model is not confident in its selection.
Revised: This could indicate that when the score is low magnitude, the model is not confident in its selection.

Original: Thiscan then be used to identify if it made a poor choice in advance of training the full homogeneousmodel.
Revised: This can then be used to identify if it made a poor choice in advance of training the full homogeneous model.

Original: [Table]
Revised: [Table]

Original: 4.2 Finding Optimal Heterogeneous Attention
Revised: 4.2 Finding Optimal Heterogeneous Attention

Original: Table 2 shows the performance for the best homogeneous model, the fully trained model with anarchitecture chosen by the pruning NAS method (NAS Prune), and the fully trained model with anarchitecture chosen by taking the best four attentions from the single layer selection to find the bestattention (NAS One-shot). 
Revised: Table 2 shows the performance for the best homogeneous model, the fully trained model with an architecture chosen by the pruning NAS method (NAS Prune) [Table] and the fully trained model with an architecture chosen by taking the best four attentions from the single layer selection to find the best attention (NAS One-shot). [Table]

Revised: Listops

Original: [Table]
Revised: [Table]

Revised: DocumentMatching

Revised: [Table]

Original: This shows us that heterogeneous Transformer networks fail to beat the best homogeneous modelfor any task.
Revised: This shows us that heterogeneous Transformer networks fail to beat the best homogeneous model for any task.

Original: We also experimented with choosing a mixture of the best performing homogeneous attentions, and having more attention heads of the better attention mechanisms for that task, butneither of these two variations gave consistent improvement on the NAS One-shot method.
Revised: We also experimented with choosing a mixture of the best performing homogeneous attentions, and having more attention heads of the better attention mechanisms for that task, but neither of these two variations gave consistent improvement on the NAS One-shot method.

Original: This alsoshows us that the expensive pruning method poses no consistent advantages over the far cheaper NAS
Revised: This also shows us that the expensive pruning method poses no consistent advantages over the far cheaper NAS

Original: One-shot when it correctly identifies good attentions.
Revised: One-shot when it correctly identifies good attentions.

Original:  Theories For Heterogeneous Attention Sub-optimality
Revised:  Theories For Heterogeneous Attention Sub-optimality

Original: Whilst searching for the optimal homogeneous attention works well and validates our NAS method,our found heterogeneous attention Transformer models cannot beat the best homogeneous models.
Revised: Whilst searching for the optimal homogeneous attention works well and validates our NAS method, our found heterogeneous attention Transformer models cannot beat the best homogeneous models.

Original: This conflicts with the idea that the different mechanisms provide different biases which help the
Revised: This conflicts with the idea that the different mechanisms provide different biases which help the

Original: Transformer learn many different and useful relationships.
Revised: Transformer learn many different and useful relationships.

Original: Here we provide some theories to explainwhat we have observed.
Revised: Here we provide some theories to explain what we have observed.

Original: Different attention mechanisms do not bias the relationships learned.
Revised: Different attention mechanisms do not bias the relationships learned.

Original: It might be that the relationshipslearned are not affected by the attention type, and the performance differences noted in Tay et al.(2020b) are due to some other effect.
Revised: It might be that the relationships learned are not affected by the attention type, and the performance differences noted in Tay et al. (2020b) are due to some other effect.

Original: Thus the amount of functional bias by attention mechanisms issmall.
Revised: Thus the amount of functional bias by attention mechanisms is small.

Original: Each tasks is best solved with attention that has learned specific types of relationships.
Revised: Each tasks is best solved with attention that has learned specific types of relationships.

Original: Whilst learninglots of relationships might be useful, perhaps each tasks requires a narrow subset of relationships tobe learned, and thus a homogeneous attention well suited to this narrow subset is optimal.
Revised: Whilst learning lots of relationships might be useful, perhaps each tasks requires a narrow subset of relationships to be learned, and thus a homogeneous attention well suited to this narrow subset is optimal.

Original: Bad attention degrades performance.
Revised: Bad attention degrades performance.

Original: It might be that the different attention mechanisms do learndifferent relationships, and this generally helps, but a small number of poorly suited attention typesdegrade model performance.
Revised: It might be that the different attention mechanisms do learn different relationships, and this generally helps, but a small number of poorly suited attention types degrade model performance.

Original: This could be tested by more comprehensive studies on the effects ofcombining different attention types.
Revised: This could be tested by more comprehensive studies on the effects of combining different attention types.

Original: The training of other aspects of the Transformer is dependent on the attention mechanism.
Revised: The training of other aspects of the Transformer is dependent on the attention mechanism.

Original: It mightbe that when trained in homogeneous models, the attention mechanism influences how other parts ofthe model learn, such as the feed-forward network.
Revised: It might be that when trained in homogeneous models, the attention mechanism influences how other parts of the model learn, such as the feed-forward network.

Original: Thus if there were multiple attention mechanisms,these parts might have conflicting pressures on their learning which cause them to not be well suitedto any of the attention mechanisms.
Revised: Thus if there were multiple attention mechanisms, these parts might have conflicting pressures on their learning which cause them to not be well suited to any of the attention mechanisms.

Original:  Conclusion
Revised:  Conclusion

Original: For a given task it is often not clear which attention mechanism will perform best.
Revised: For a given task it is often not clear which attention mechanism will perform best.

Original: Using a DARTSlike method you can train a single layer mixed attention Transformer and use masked validationaccuracy drop to determine the best attention to use in a full Transformer model.
Revised: Using a DARTSlike method you can train a single layer mixed attention Transformer and use masked validation accuracy drop to determine the best attention to use in a full Transformer model.

Original: If the maskedvalidation drop scores are very low, the best attention may not have been correctly identified.
Revised: If the masked validation drop scores are very low, the best attention may not have been correctly identified.

Original: Heterogeneous attention Transformer networks typically perform better than the average homogeneous one.
Revised: Heterogeneous attention Transformer networks typically perform better than the average homogeneous one.

Original: Reasonable combinations of attention can be found via a DARTS-like method.
Revised: Reasonable combinations of attention can be found via a DARTS-like method.

Original: However,heterogeneous Transformers cannot beat a homogeneous model that is using the best attentionmechanism for that given task
Revised: However, heterogeneous Transformers cannot beat a homogeneous model that is using the best attention mechanism for that given task.

Original: A Experimental Details
Revised: A Experimental Details

Original: In all tasks we tested on, we used an Adam optimizer with linear warm-up and square root decay.
Revised: In all tasks we tested on, we used an Adam optimizer with linear warm-up and square root decay.

Original: Weused a base learning rate of 0.05, β 1 = 0 .
Revised: We used a base learning rate of 0.05, β 1 = 0 .

Original: 9 , and β 2 = 0 .
Revised: 9 , and β 2 = 0 .

Original: We used a batch size of 32 for every task.
Revised: We used a batch size of 32 for every task.

Original: All attention mechanisms used the [CLS] token for classification.
Revised: All attention mechanisms used the [CLS] token for classification.

Original: Task specific hyperparametersare given in table 3.
Revised: Task specific hyperparameters are given in table 3.

Original: In all Transformer models the feed-forward network (FFN) has a single hiddendimension.
Revised: In all Transformer models the feed-forward network (FFN) has a single hidden dimension.

Original: Before testing we take the model which had the best validation accuracy during training,the long train times are to ensure convergence.
Revised: Before testing we take the model which had the best validation accuracy during training, the long train times are to ensure convergence.

Original: [Table]
Revised: [Table]

Original: B Layer-wise Heterogeneous Attention
Revised: B Layer-wise Heterogeneous Attention

Original: Here we try another method of heterogeneous attention.
Revised: Here we try another method of heterogeneous attention.

Original: Instead of using layers with an identicalmixture of attentions at each layer, we make each layer homogeneous in its attention but use differentattentions for different layers.
Revised: Instead of using layers with an identical mixture of attentions at each layer, we make each layer homogeneous in its attention but use different attentions for different layers.

Original: Given we want each layer to have H attention heads of a single type, we initialize an attention blockwith H heads for each candidate attention mechanism in each layer.
Revised: Given we want each layer to have H attention heads of a single type, we initialize an attention block with H heads for each candidate attention mechanism in each layer.

Original: We train the supernetwork torough convergence and then alternate between removing an attention block in a layer and fine-tunetraining the remaining model.
Revised: We train the supernetwork to rough convergence and then alternate between removing an attention block in a layer and fine-tune training the remaining model.

Original: We go through each layer starting with the first, removing the worstattention block before moving on to the next layer.
Revised: We go through each layer starting with the first, removing the worst attention block before moving on to the next layer.

Original: Once we remove a block from the final layer webegin again at the first layer.
Revised: Once we remove a block from the final layer we begin again at the first layer.

Original: We keep passing through the network, removing and fine-tuning, untilonly one attention block remains in each layer.
Revised: We keep passing through the network, removing and fine-tuning, until only one attention block remains in each layer.

Original: We then take the attention blocks learned for each layer and train a new Transformer from scratchwith those attentions at each layer.
Revised: We then take the attention blocks learned for each layer and train a new Transformer from scratch with those attentions at each layer.

Original: This paradigm is summarized in algorithm form in the appendixas Algorithm 3.
Revised: This paradigm is summarized in algorithm form in the appendix as Algorithm 3.

Original: Due to the high memory requirements of this task, we use block sampling when training.
Revised: Due to the high memory requirements of this task, we use block sampling when training.

Original: This meansthat for each training batch we only use a randomly selected subset of the candidate blocks in eachlayer.
Revised: This means that for each training batch we only use a randomly selected subset of the candidate blocks in each layer.

Original: This reduces memory requirements as there are less intermediate values and computations, butcan affect the training dynamics.
Revised: This reduces memory requirements as there are less intermediate values and computations, but can affect the training dynamics.

Original: Using this method within a DARTS like paradigm is shown to workwell by Xu et al.
Revised: Using this method within a DARTS like paradigm is shown to work well by Xu et al.

Original: We test this method on the text classification task which has 1k input sequence length to further limitcomputation requirements.
Revised: We test this method on the text classification task which has 1k input sequence length to further limit computation requirements.

Original: As the Transformers used in LRA for this task had 6 layers, we searchedfor a 6 layer model with layer-wise attention heterogeneity.
Revised: As the Transformers used in LRA for this task had 6 layers, we searched for a 6 layer model with layer-wise attention heterogeneity.

Original: The resulting architecture was:
Revised: The resulting architecture was:

Original: Performer → Bigbird → Performer → Sparse → Performer → Sparse
Revised: Performer → Bigbird → Performer → Sparse → Performer → Sparse

Original: When trained with the same hyperparameters as the LRA models, it achieved a test accuracy of64.6%.
Revised: When trained with the same hyperparameters as the LRA models, it achieved a test accuracy of 64.6%.

Original: This puts this model around average for this task, unable to beat several attentions such as the
Revised: This puts this model around average for this task, unable to beat several attentions such as the

Original: Linear Transformer which had a test accuracy of 65.9%, or the Performer which achieved 65.4%.
Revised: Linear Transformer which had a test accuracy of 65.9%, or the Performer which achieved 65.4%.

Original: C Supplementary Diagrams
Revised: C Supplementary Diagrams

Original: Figures 2 to 4 help illustrate how our approach relates to DARTS and the original Transformer,attention heads being pruned during the search process in finding optimal heterogeneous attention,and an overview of the differences between the theoretical and actual supernetwork.
Revised: Figures 2 to 4 help illustrate how our approach relates to DARTS and the original Transformer, attention heads being pruned during the search process in finding optimal heterogeneous attention, and an overview of the differences between the theoretical and actual supernetwork.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: QKVLinear
Revised: QKVLinear

Original: [Figure]
Revised: [Figure]

Revised: Concatenation

Original: Linear
Revised: Linear

Original: Attn
Revised: Attn

Original: [Figure]
Revised: [Figure]

Original: QKVLinear
Revised: Attn

Original: [Figure]
Revised: [Figure]

Original: QKVLinear
Revised: QKVLinear

Original: Attn
Revised: Attn

Original: [Figure]
Revised: [Figure]

Revised: QKVLinear

Revised: Attn.

Revised: [Figure]

Original: Linear
Revised: Linear

Original: Head
Revised: Head

Original: QKVLinear
Revised: QKVLinear

Original: Attn
Revised: Attn.

Original: [Figure]
Revised: [Figure]

Original: Linear
Revised: Linear

Original: [Figure]
Revised: Head

Original: QKVLinear
Revised: QKVLinear

Original: Attn.
Revised: Attn.

Revised: [Figure]

Revised: Linear

Original: [Figure]
Revised: [Figure]

Original: QKVLinear
Revised: QKVLinear

Original: Attn.
Revised: Attn.

Revised: type

Original: [Figure]
Revised: [Figure]

Original: D Algorithms
Revised: D Algorithms

Original: Algorithms 1 and 2 correspond to the paradigms outlined in section 3.3.
Revised: Algorithms 1 and 2 correspond to the paradigms outlined in section 3.3.

Original: Let B i denote the set of attention blocks in layer i , with L denoting the set of layers.
Revised: Let B i denote the set of attention blocks in layer i , with L denoting the set of layers.

Original: We also use a torepresent accuracy and s to represent a score with S i representing the set of scores for the attentionblocks in layer i .
Revised: We also use a to represent accuracy and s to represent a score with S i representing the set of scores for the attention blocks in layer i .

Original: For single layer networks, the layer subscript is omitted.
Revised: For single layer networks, the layer subscript is omitted.