E IT : E NHANCED I NTERACTIVE T RANSFORMER
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we tackle the head degradation problem in attention.
We propose an
E nhanced I nteractive T ransformer (E IT ) architecture in which the standard multihead self-attention is replaced with the enhanced multi-head attention (EMHA).
EMHA removes the one-to-one mapping constraint among queries and keys in multiple subspaces and allows each query to attend to multiple keys.
On top of that, we develop a method to make full use of many-to-many mapping by introducing two interaction models, namely Inner-Subspace Interaction and CrossSubspace Interaction.
Extensive experiments on a wide range of tasks (e.g. machine translation, abstractive summarization, grammar correction, language modelling and brain disease automatic diagnosis) show its superiority with a very modest increase in model size.
I NTRODUCTION
[Figure]
Transformer (Vaswani et al., 2017) and its variants have yielded promising results on a wide range of natural language processing tasks (Devlin et al., 2019; Brown et al., 2020).
The strength lies in its ability to capture global dependencies among all positions of a given sequence, which is endowed by the multihead self-attention network (MHSA).
Just as its name implies, MHSA splits the hidden representation into several feature subspaces (namely head), and attention operations are performed within each subspace.
Researchers attribute the success of MHSA to the fact that different subspaces can capture distinct information, and further regularization of enlarging the head distance leads to good results (Li et al., 2018).
Except for the empirical evidence, Wang et al.
(2022) theoretically demonstrates that multihead strategy can degrade the rate of over-smoothing 1 and the effect is in proportion to the number of subspaces.
Since the benefits of MHSA, a natural question arises that “Can Transformer attain persistent and sufficient gains from larger subspaces?”
[Table]
For vanilla Transformer, simply enlarging subspaces suffers from performance deterioration since each query and key owns a lower dimension which can not guarantee precise attention maps (Shazeer et al., 2020).
To overcome this downside, one can apply a tiny model, e.g., a linear layer, to the attention maps to learn more accurate attention maps.
Along this research line, Zhou et al.
(2021b) expands the attention distribution space based on original attention maps without reducing the head dimension, namely
Attention Expansion mechanism.
As is shown in Figure 1, we plot the performance against the number of heads on WMT’14 En-De.
Obviously, the vanilla Transformer cannot make full use of the enlarging subspaces.
Meanwhile, both Talking-Head and Refiner cannot persist the benefits when the head is larger than 32 or 128.
Besides, their gains from enlarged subspaces are weak.
This phenomenon could be interpreted from three aspects summarized in Table 1.
First, in standard
Multi-Head Partition , the head dimension descends as the number of heads increases, resulting in inaccurate attention maps.
Second, the attention maps generated by Attention Expansion own high similarity, which deteriorates the quality of information contained in attention maps.
These issues root the main cause for the difficulty of attaining more gains from enlarged subspaces.
[Figure]
Many-to-Many can generate more diverse attention maps.
In this work, we design a new variant, namely E IT , to solve the aforementioned weakness simultaneously.
Concretely, we break the vanilla one-to-one mapping and instead adopt a many-to-many mapping strategy (Figure 2(b)) to acquire distinct subspace representation without sacrificing per head dimension.
Moreover, we further develop a two-stage enhanced interactive approach to better utilize the enlarged subspaces.
It involves two interaction models - call them inner-subspace interaction and cross-subspace interaction .
The inner-subspace interaction model (ISI) encourages the learning of unique and refined features for each subspace, while the cross-subspace interaction model (CSI) focuses more on the learning of universal features over different subspaces.
Our contributions can be summarized as follows:
• We propose a many-to-many mapping that generates numerous attention maps efficiently. • We propose an E IT architecture that can outperform Transformer on a wide range of tasks including machine translation, abstractive summarization, grammar error correction, language modelling and brain disease automatic diagnosis.
• E IT can attain persistent gains from enlarged heads as well as keep high expressiveness.
It can also simulate some property of attention with numerous heads, e g., antioversmoothing. For example, features learned by E IT have a lower token correlation. • The final attention maps generated by E IT are more confident in their decision though these attention maps own high similarity among each other.
A TTENTION IN T RANSFORMER
The core component of Transformer is the MHSA module.
Given an embedded input sequence
X ∈ R T × d , where T and d represent the sequence length and embedding dimension respectively.
MHSA first generates three key components, namely queries, keys and values, via three linear transformation matrices W Q , W K ,
W V ∈ R d × d as: Q = XW Q , K = XW K , V = XW V .
It then divides the Q , K , V into three sets U Q , U K and U V , e.g., U K denotes the set { K 1 , · · · , K M } and M is the number of heads.
The attention distribution of i -th head is computed via: A i =
Softmax( Q i ( K i ) T √ d k ) ∈ R T × T , where d k is the head dimension that equals dM .
Then it calculates the final output O as follows:
[Equation]
where O i ∈ R T × d k is the output of i -th subspace in MHSA and
W O ∈ R d × d is a learnable matrix.
Such a way to obtain attention maps has an important property: the head dimension is inversely proportional to the number of subspaces.
Given this, when the number of subspaces is too large, the queries and keys with low head dimensions can not generate accurate attention maps (Shazeer et al., 2020).
Consequently, Transformer suffers performance degradation.
Previous work such as refiner (Zhou et al., 2021b) adopts attention expansion to generate more attention maps from a small number of original attention maps as follows: (cid:2)
A 1 , A 2 , . . . , A M e [Equation] = f
[Equation]
A
[Equation] , A [Equation] A M
[Equation]
) where M e is the number of heads after attention expansion and f () is the expansion function which can be linear transformation or convolution. Note that the input attention maps are calculated via query-key multiplication. There is no guarantee to keep the diversity of the expanded attention maps, which will prevent the self-attention attaining more gains from enlarged subspaces (See Figure 6(a)).
[Figure]
E NHANCED I NTERACTIVE T RANSFORMER
In this work we propose a novel Enhanced Interactive Transformer in which we replace the multihead self-attention with Enhanced Multi-Head Attention mechanism (EMHA) that can efficiently benefit from more heads.
The proposed EMHA augments the MHSA from two aspects: 1) It adopts the many-to-may mapping schema to generate a large number of diverse attention maps.
2) It introduces comprehensive interactions to unearth the merits of the enlarged subspaces.
3. D EFINITION OF I NTERACTION S PACE AND R ECEPTIVE F IELD
metrics.
Here we set M to 8 as usual. Module | Ψ | | Φ | MHSA 8 EMHA
We first start with the definition of interaction space.
Focusing on the attention operation, the interaction space, namely Φ , consists of several query-key mapping pairs, e.g., { (cid:10) Q 1 , K 1 (cid:11) , . .
, (cid:10) Q M , K M (cid:11) } .
In this way, the size of the interaction space ( | Φ | ) describes how many mapping pairs are there during the attention calculation.
To investigate how to enlarge the interaction space effectively, another important concept, the receptive field of the attention mechanism, is defined.
The receptive field refers to the set of keys each query can attend to.
In MHSA, the receptive field of each query Q i is { K i } .
However, in EMHA, without special declaration, the receptive field of each query Q i is the whole key set { K 1 , . . .
, K M } .
The comparison of these two variants in terms of the receptive field and the interaction space is listed in Table 2.
The connection between Φ and Ψ is formulated as: | Φ | =
M · | Ψ | , where | Ψ | denotes the size of receptive field.
We also extend EMHA to a universal one with any size of receptive field for efficiency-performance trade-offs in Appendix H.
3. R ECEPTIVE F IELD E XPANSION
Through Section 3.1, we can easily connect the strength of interaction space with the number of attention maps since in MHSA, attention maps are always computed by applying sim( · ) to querykey mappings, e.g., dot-multiplication. Thus, it is crucial to efficiently enlarge the interaction space.
Many-to-Many Mapping Scheme To efficiently enlarge the interaction space, we propose to expand the receptive field of each query from the single key to M keys.
As illustrated in Figure 3(a), four queries and four keys can be served as two components in a bipartite graph and each element, e.g., Q 1 , in a component can interact with any elements, e.g., K 1 , . .
, K 4 , in another component. When the interaction operation is the scaled dot multiplication, we can obtain 16 attention maps, which is four times as much as that in standard MHSA.
Formally, supposing one with M heads, the i -th attention map can be formally calculated as:
[Equation]
where S i ∈ R T × T is the attention maps without softmax, ⌊⌋ is the round down operation and % is the mod operation.
For example, S 5 is computed by Q 2 and K 1 when M equals to 4.
3. D UAL E NHANCED I NTERACTION
[Figure] The previous section has introduced how to efficiently enlarge the interaction space.
However, directly enlarging the receptive field results in useless signals, e.g., noisy edges in attention maps.
Besides, directly utilizing them decreases the dimension of value, degrading the expressiveness of features (Shazeer et al., 2020).
A feasible solution is to distinguish the useful information and abandon the useless parts and fully leverage the useful ones.
The inner-subspace interaction (ISI) relationship describes the connection among the attention maps generated by the same query. These attention maps have a closer relationship with each other since their queries are the same.
This provides a suitable solution to distinguish the useful information and abandon the useless parts, since from the perspective of clustering, similar information can be grouped into one. As illustrated in Figure 3(a), the ISI relationship exists in the attention maps in the same column.
Inner-Subspace Interaction Modeling
In ISI modeling, one thing we need to consider is to keep the diversity among the output attention maps when extracting useful information from these materials obtained by RFE. Given this, we adopt two-layer group convolutions (Krizhevsky et al., 2012) accompanied by the ReLU activation to consist this sub-module, since group convolution owns several separate convolutions to process features from different groups and subspace-specific functions (Cui et al., 2019) are of great benefit to enlarge head distances in the MHSA.
Denote f ( · ) as a single layer group convolution.
As illustrated in Figure 4(a), given the output of RFE, namely S , as input, ISI sub-module is computed as:
[Equation]
where ˆ˙ S ∈ R M Hisi × T × T is the hidden output and ˙ S ∈ R M × T × T is the final output of the ISI sub-module. We use M H isi to represent the intermediate head size in ISI sub-module and set the number of groups in group convolutions to M .
For other configurations, please see Appendix A. 3.3. C ROSS -S UBSPACE I NTERACTION
Definition of Cross-Subspace Interaction Relationship The cross-subspace interaction (CSI) relationship describes the collaboration among different subspaces, which aims to fully leverage the useful information.
It exists in the attention maps generated by different queries. As illustrated in Figure 3(a), the CSI relationship exists in the attention maps in the different columns.
Cross-Subspace Interaction Modeling
In CSI modeling, we want to leverage the advantages of attention maps of different heads to generate final precise attention maps. we adopt two-layer convolutions accompanied by the ReLU activation to consist this sub-module. Let us denote g ( · ) as a single layer convolution.
As illustrated in Figure 4(a), given the output of ISI, namely ˙ S , as input, CSI sub-module is computed as:
[Equation] where ˆ¨ S ∈ R M Hcsi × T × T is the hidden output and ¨ S ∈ R M × T × T is the final output of the CSI sub-module.
We use M H csi to represent the intermediate head size in CSI sub-module.
3. E FFICIENT V ERSION OF E IT
Despite the theoretically computational efficiency and parametric efficiency of group convolutions, they slow down the training in practice (Ma et al., 2018).
To alleviate this issue, we provide another efficient version of E IT , namely E-E IT , by simplifying the design of ISI and CSI.
In this way, EE IT avoids parts of memory consumption and somehow speeds up the computational efficiency.
Specifically, we adopt a single layer in each sub-module, as illustrated in Figure 4(b).
In E-E IT , the dual enhanced interactions are computed as: [Equation] where ˙ S ∈ R M H × T × T and ¨ S ∈ R M × T × T and M H is a hyper-parameter, e.g. we set it as 32 for the base configuration.
E XPERIMENTS 4. R ESULTS ON M ACHINE T RANSLATION
We evaluated E IT on both WMT’14 En-De and WMT’16 En-Ro machine translation tasks with
BLEU and sacreBLEU 2 .
More details about the experimental setup are included in Appendix A.
Main Results Table 3 and Table 4 display the results of baselines and our methods on En-De and En-Ro tasks.
For the En-De task, we select baselines from three aspects: head modification (Zhou et al., 2021b; Shazeer et al., 2020; Wang & Tu, 2020; Li et al., 2019; 2018; Zhang et al., 2022; Nguyen et al., 2022), localness modelling (Li et al., 2022; Fan et al., 2021; Yang et al., 2019), and deep transformer (Liu et al., 2020b; Wang et al., 2019; Wei et al., 2020).
A similar selection of baselines is also for the En-Ro task.
For a fair comparison, we re-implement some of these models in our codebase. We keep the training setups of these methods the same as ours. Through the broad comparison, we can get the following observations: Our E IT variants outperform the vanilla Transformer with negligible added parameters against different configurations on both datasets.
To be honest, E IT adopts frequent feed-forward-like mapping strategies, which is time consuming.
To alleviate this, we also provide another choice, namely E-E IT .
Beyond our expectations, it not only delivers competitive results compared with the full version, but also saves redundant computations (we show the comparison in Section 5.4).
Besides, we also provides a lot of parameter analysis (Appendix H) and trade-off strategy (Appendix I).
Our E IT can beat all selected methods of head modification and localness modelling, including the latest methods such as MoA (Zhang et al., 2022), Fishformer (Nguyen et al., 2022), UMST (Li et al., 2022), on both datasets.
This indicates that our methods can better unearth the potential of the multi-head strategy.
Our EIT can beat all selected deep transformers on the En-De task.
For example, a 48-layer EIT can beat the 48-layer baseline over 0.69 BLEU points.
Besides, compared to M SC and A DMIN , E IT beats them with fewer parameters.
4. R ESULTS ON O THER S EQUENCE G ENERATION T ASKS
We also conducted experiments on the other two sequence generation tasks: the abstractive summarization task and the grammar error correction task.
Due to the limited page, please refer to the Appendix A for the details of experimental setups.
Table 5 shows that both E IT and E-E IT can outperform the standard Transformer by a large margin on both tasks.
For example, E IT achieves scores of 41.62 ROUGE-1 points, 18.70 ROUGE-2 points and 38.33 ROUGE-L points on the CNN-DailyMail dataset and scores of 57.05 F 0 . 5 on CONLL [Table]
More details about the settings of thees models are given in Appendix [Table] [Table] [Table]
[Table] dataset. Compared with other strong baselines, our E IT and E-E IT can still show superiority on these datasets in terms of ROUGE-1 points and F 0 . 5 value, e
[Table]
g., E IT surpasses S URFACE by 0.
and 0.45 in terms of ROUGE-1 points and F 0 .
5 value, respectively.
We also evaluate our E IT on an automatic disease diagnosis task with four metrics: Accuracy, AUROC, Sensitivity and Specificity. We select a widely used real-world fMRI dataset: Autism Brain Imaging Data Exchange (ABIDE).
ABIDE consists of 1009 brain networks from 17 international sites
[Table]
of which 516 samples are autism spectrum disorder patients.
Due to the strong heterogeneity of this data, we follow the preprocessing process in Kan et al.
Moreover, we adopt the
CC200 (Craddock et al., 2012) as the Regin-of-Interest (ROI) partition template.
We select two latest methods, the Mvs-GCN (Wen et al., 2022) and BrainNetTF (Kan et al., 2022), as our comparison.
The experiment setups and configurations of our BrainNetE IT F are the as in Kan et al.
Each experiment is conducted 5 times and we report the mean and standard deviation of the four metrics.
The results are exhibited in Table 6.
We can see that our BrainNetE IT F can outperform all the baselines in terms of all metrics.
4. R ESULTS ON L ANGUAGE M ODELING
We also evaluate our E IT on a language modeling task: wiki103-text with Perplexity (PPL).
We follow the official preprocessing procedure (Ott et al., 2019), and select Adaptive Input Transformer (Baevski & Auli, 2019) as our baselines.
Both the baseline and our E IT are all 8-layer big models with 8 heads.
More details are given in Appendix A. Table 7 displays the results.
We can see that our E IT can also outperform the baseline with 1.16 in terms of testPPL.
D ISCUSSION
In this section [Table] we present a detailed analysis to provide some insights into why EMHA is superior to vanilla MHSA.
Without the special declaration, we conduct experiments on E IT .
5. A BLATION S TUDIES
[Table]
Table 8 summarizes the impacts of removing each module on En-De and En-Ro tasks, respectively.
Notably, when removing the RFE module (#2 vs. #3), we observe an obvious decline in performance on two translation tasks.
This indicates that the RFE module plays an important role in E IT , as it can generate plentiful diverse attention maps.
Besides, we can see there is a vast drop in BLEU when removing the ISI submodule (#2 vs. #4).
This is because directly enlarging subspaces leads to the existence of some useless information.
However, the ISI sub-module provides a suitable way to abandon it while retaining the benefits of the former.
This unique design differs from the head expansion (Zhou et al., 2021b).Moreover, applying a single module (# 6, 7, 8 v.s #1, 2) is inferior to E IT , which indicates E IT is well-designed, e.g., ISI efficiently utilizes the enlarged heads and CSI fully learns universal features across all subspaces.
Note that the additional primary cost comes from the ISI sub-module due to the unfriendly support for the implementation of group convolution in PyTorch (Paszke et al., 2019).
5. D OES I MPROVEMENTS C OME F ROM THE H EAD M ODIFICATION
[Figure]
[Figure]
In our dual enhanced interaction modeling, we apply convolution operation to attention maps, which has a potential to introduce local biases, a kind of useful information for Transformer.
To validate this, inspired by Fan et al.
(2021), we define a localness metric, namely C , that measures the localness of attention maps since if there is a local bias, each token will distribute larger attention weights on their neighbouring tokens.
We plot the C value of our E IT and Transformer on the test set of En-De task and CNN-DailyMail task in Figure 5.
Note that due to the limit time of rebuttal, we only use a subset of test set consists of 1000 samples for CNN-DailyMail task.
Through the results (mean), we can see that there is no significant local enhancement phenomenon on both tasks.
Note that the attention maps in the first layer of EIT on the abstractive summarization have a strong local pattern, but the kernel sizes are all set to 1 on this task.
So we conclude the improvements do not come from explicitly introducing local biases.
5. W HAT CAN E IT BRING
In this section, we aim to unearth why E IT is superior to standard transformers.
To be specific, we trace the variation of head distance among different heads throughout the calculation process, and investigate the connection between head distance and quality of features.
The results are exhibited in the Figure 6.
For more details, please refer to Appendix B.
We obtain the following observations: [Figure]
Diverse Initial Attention Maps
As plotted in Figure 6(a), the initial numerous attention maps in our E IT (generated by RFE) own the lowest similarity, compared to that in Talking-head and Refiner (generated by attention expansion). This is within our expectation.
When we further replace RFE with attention expansion in our E IT system, the performance on En-De task suffers a vast drop from 28.00 to 27.41. This indicates when one benefits from enlarged subspaces, he first needs to generate diverse initial numerous attention maps.
They are just like raw materials, the level of diversity among them determines how much information they contain. High-Similarity Final Attention Maps As plotted in Figure 6(b), E IT owns the higher average similarity among attention maps from different heads, compared to standard Transformer. Different from normal redundant attention maps, the high similarity is the result of head interaction and the high-similarity attention maps have a property: they are more confident in their decisions (see Section 5.5 and Appendix L).
When we explicitly break this kind of high similarity via Li et al.
(2018), the performance suffers from degradation (from 28.00 to 27.65 under base configuration). Thus, we say that the high similarity among attention maps resulting from head interaction is a good pattern.
Lower Token Correlation
Figure 6(c) plots the token correlation of our E IT and Transformer models against encoder depths.
The token correlation is measured by the pearson correlation coefficient (Benesty et al., 2009).
More details are given in Appendix B. We can see that the features from E IT have a lower token correlation than that of vanilla Transformer.
This indicates E IT has a more powerful learning capacity since high correlation among token representations reduces the learning capacity of Transformer (Gong et al., 2021; Shi et al., 2022; Wang et al., 2022).
More Efficient utilization of Functions
Figure 6(d) plots the ratio of function output and final output of our E IT and Transformer models.
We can see that the EMHA and FFN in E IT contribute more.
This is unusual since Attention and FFN have a tendency to learn similar token representations and the residual connection is of great benefit to resist this tendency (Wang et al., 2022).
A reasonable explanation is that EMHA and FFN in E IT has a tendency to learn lower similar token representation.
So they do not rely too much on the residual connection.
5. M ODEL S CALABILITY AND E FFICIENCY C OMPARISON
E IT is well scalable to deep configurations.
The results are exhibited in Figure 7(left).
E IT outperforms the Transformer at all depths with up to 0.69 BLEU points on average.
Similar phenomena could be observed in E-E IT .
Moreover, Figure 7(middle and right) also displays the memory consumption and computational cost.
E IT costs 8.5% more memory consumption and 44.4% more training costs than the baseline with a depth of 6.
The E-E IT only costs 9.4% more memory consumption and 21.7% more training costs than the baseline under all the configurations on average.
This demonstrates E-E IT is efficient.
Moreover, as aforementioned, the main extra costs come from the group convolution.
We ascribe it to the unfriendly support by PyTorch.
More analyses for this are included in the Appendix E. We will release an efficient implementation in future.
5. V ISUALIZATION
We visualize the attention maps (Figure 8) for an example on the En-De task, and get the following observations: 1) Attention maps obtained by E IT attend to certain patterns, e.g., word boundary information (a@@, gh@@ and ast) (Li et al., 2022).
Our attention maps are more confident in their decision, e.g., each token pays more attention to a few positions rather than learning a smoothing view over all positions.
These demonstrate that our E IT can generate more precise attention maps.
Besides, though the nearly consistent attending results lead to the high similarity among attention [Figure] aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead maps, we find that if the patterns are useful, high-similarity attention maps are not a drawback.
More results of other cases in other tasks are included in Appendix L.
R ELATED W ORK
Improved Multi-Head Mechanism How to effectively improve the multi-head attention has always been a hot research topic in recent years.
Previous work has revealed that multi-head attention can be further enhanced by encouraging individual attention heads to extract distinct information (Li et al., 2018;
Cui et al., 2019; Sukhbaatar et al., 2019; Guo et al., 2020; Hao et al., 2019).
Another branch of research is designing more complex interactive modeling to make better use of the multiple subspace information (Shazeer et al., 2020; Wang & Tu, 2020; Li et al., 2019).
Besides, Voita et al.
(2019) empirically demonstrates that some heads in attention are useless and can be pruned without performance degradation.
Along this line, researchers investigate how to efficiently cut off redundant heads (Michel et al., 2019; Behnke & Heafield, 2020).
Different from these work, Wang et al.
(2022) theoretically demonstrates the benefits of multi-head attention.
Motivated by it, our work investigates how to benefit from enlarged heads efficiently.
Convolution + Attention Convolution and attention mechanism have been dominated paradigms for local modelling and global modelling, respectively.
Collaboration of convolution and attention mechanism has become an interesting topic.
Recently, many researchers have shifted their attention to incorporating the convolution into the transformer (Yang et al., 2019;
Zhou et al., 2021b;
Zhao et al., 2019; Pan et al., 2021; Wu et al., 2021; Xiao et al., 2021; Peng et al., 2021).
Their core idea is to fully leverages the advantages of these two paradigms.
Our work follows this thread of research but is quite different from them in two aspects: 1) We mainly adopt the convolutions and group convolutions for flexibly modelling the cross-head interaction but not for injecting localness.
The convolution of our work is directly applied to the attention map.
Note that Zhou et al.
(2021b) is similar to our work.
However, they mainly focus on modelling the localness bias.
C ONCLUSIONS
In this paper, we propose E IT , an alternative to the standard multi-head attention network.
It further advances the multi-head schema by breaking the standard one-to-one mapping constraint.
Meanwhile, E IT employs the inner-subspace interaction and cross-subspace interaction to make full use of the expanded attention maps.
In addition, E-E IT can be served as another choice considering the trade-off between performance and computation efficiency.
Experimental results on five widelyused tasks demonstrate the effectiveness of E IT -variants, which deliver consistent improvements to the standard Transformer.
A S ETUP OF E XPERIMENTS A. M ACHINE T RANSLATION T ASK
Dataset We evaluated our approach on two widely used machine translation datasets: WMT’
En-De and WMT’16 En-Ro.
The En-De dataset contains approximately 4.5M tokenized training sentence pairs.
We selected newstest2013 and newstest2014 as the validation and test data, respectively.
As for the En-Ro dataset, it consists of 0.6M tokenized training sentence pairs.
We performed shared BPE operations on both datasets to overcome the out-of-vocabulary (OOV) problem.
Concretely, we set the size of BPE operations to 32K and 20K for En-De and En-Ro datasets, resulting in a shared vocabulary with sizes of 34040 and 19064, respectively.
Model Configuration Our model architectures are based on Transformer (Vaswani et al., 2017).
We provided three basic configurations, namely base , deep , and big which follow the configurations in Vaswani et al.
We adopted a pre-normalization strategy (Wang et al., 2019) considering training stability under different configurations.
The newly added hyper-parameters follow the settings in Table 10.
Training & Evaluation
Our implementations are based on Fairseq (Ott et al., 2019).
Our experiments are performed on the GEFORCE RTX 3090 cards.
We use 8 GEFORCE RTX 3090 cards to train models for the WMT’14 En-De task.
As for the models on the WMT’16 En-Ro task, we train them on 4 GEFORCE RTX 3090 cards.
The batch sizes for En-De and En-Ro tasks areand 16384, respectively.
The total updates are 50K, 50K and 100K for base , deep and big in En-De task, respectively.
We adopt Adam (Kingma & Ba, 2015) as an optimizer with an adam β of (0.9, 0.997).
The learning rate scheduler is invert sqrt with a learning rate of 0.002 and warmup updates of 16000.
We also adopt label smoothing with a ratio of 0.1 in all the experiments.
More details are exhibited in Table 11.
During the evaluation process, we set the beam number to 4 and the length penalty to 0.6 for the En-De task.
As for the En-Ro task, the number of beams is 5 and the length penalty is 1.3.
[Table]
A. A BSTRACTIVE S UMMARIZATION T ASK
Dataset For abstractive summarization, we conduct experiments on a widely used corpus, e.g.,
CNN/DailyMail dataset.
It consists of 287K training documents.
Shared BPE operations with a size of 30K are performed on all the training data, resulting in a vocabulary of 32584.
Model Configuration We only provide the base configuration of our E IT and E-E IT for abstractive summarization.
The details are presented in Table 10.
Training
Evaluation We train models for an abstractive summarization task on 8 GEFORCE
RTX 3090 cards with a batch size of 131072 and total updates of 30K. We adopt a weight decay strategy with a ratio of 0.0001.
Other hyper-parameters are the same as that in machine translation tasks.
You can find their settings in Table 11.
During testing, the number of beams is set to 4 and the length penalty is set to 2.0.
Besides, we set the minimal length and maximum length to 55 and 140, respectively.
[Table] [Table]
A. G RAMMAR E RROR C ORRECTION T ASK
Dataset For the grammar error correction task, we select the CONLL dataset to evaluate our approach.
The CONLL dataset consists of 827K training sentences.
We replicate the setup in Chollampatt & Ng (2018) and adopt the word-level dropout technique (Sennrich et al., 2016) to alleviate the overfitting problem.
More details are listed in Table 9.
Model Configuration For grammar error correction task, we only provide the base configuration of our E IT and E-E IT .
The details are presented in Table 10.
Notice that the models on this task adopt a post-normalization strategy.
Training & Evaluation We train models for the grammar error correction task on 8 GEFORCE
RTX 3090 cards.
The batch size is 65536 and the total updates are 14K. More training details are shown in Table 11.
During testing, the beams and length penalty are set to 6 and 0.6, respectively.
A. A UTOMATIC D ISEASE D IAGNOSIS T ASK
Dataset For the automatic disease diagnosis task, we select the ABIDE dataset to evaluate our approach. The ABIDE dataset consists of 1009 brain networks from 1009 real samples of 17 international sits. Due to the heterogeneity of this data, we adopt the shared data with re-standardized data splitting in Kan et al.
Specifically, 70%, 10% and 20% samples are served as the training, validation and test sets, respectively.
Model Configuration For ABIDE task, we still follow the model configuration in Kan et al.
Specifically, we build our BrainNetE IT F with two-layer encoder.
The number of heads
M are set to 4 for each layer.
Training & Evaluation We train all models including the BrainNetTF and BrainNetE IT F fro 200 epochs on a single GEFORCE RTX 3090 card.
Each model is trained by 5 times.
We adopt
Adam (Kingma & Ba, 2015) as an optimizer with an initial learning rate of 10 − 4 and a weight decay of 10 − 4 .
The bath size is set to 64.
We adopt the checkpoint of the final epoch is used for evaluating the test set.
A. L ANGUAGE M ODELING T ASK
Dataset For the language modeling task, we select the WikiText-103 dataset to evaluate our approach.
The training set consists of 103M words from 28K articles.
While for the validation and test sets, they are made up of 218K and 246K words, respectively.
In details, we follow the instructions in Fairseq (Ott et al., 2019) to obtain and preprocess the data.
Model Configuration For WikiText-103 task, Both baseline and our model are all 8-layer big model with 8 heads.
Note that the baseline we adopted are adaptive input transformer (Baevski &
Auli, 2019).
In this task, the kernel sizes in DEI are all set to 1.
Training & Evaluation
The training and evaluation settings are all follow the standard instructions for language modeling in PyTorch (Ott et al., 2019).
Note that due to the limit time of rebuttal, we only train both baseline and E IT with 169692 updates.
All these configurations can be easily found in your fairseq codebase.
B D ETAILS OF A NALYSIS B. C ALCULATION OF H EAD D ISTANCE
Inspired by the attention metrics in Zhou et al.
(2021a)
and Wang et al.
(2022), we measure the distance between different subspaces by calculating cosine similarity among attention maps.
Notice that our metric focuses on the diversity of attention maps, which is quite different from them.
Given two attention maps A 1 , A 2 ∈ R T × T , Sim ( A 1 , A 2 ) is calculated as follows:
[Equation]
Then we can obtain the head similarity by applying Eq.
(6) to attention maps from every two subspaces and average them:
[Equation]
The obtained head similarity ranges from [0, 1].
The larger the head similarity is, the lower the distances between different subspaces are.
[Table] C ALCULATION OF T OKEN C ORRELATION
We define a metric T C , which measures the correlation among the representations of different tokens.
Given a sequence representation X ∈ R T × d , which consists of T tokens, the token correlation of X can be formally defined as:
[Equation]
where ρ ( · ) is the pearson correlation coefficient (Benesty et al., 2009), which measures the correlation between two vectors:
[Equation]
Intuitively, the larger the T C is, the higher the token correlation is, degrading the model’s learning capacity (Gong et al., 2021).
B. C ALCULATION OF U TILIZATION R ATIO OF F UNCTIONS
Inspired by Liu et al.
(2020a), we define a metric, namely UR , to measure the utilization ratio of functions.
Given a function f , its utilization ratio UR ( f ) is calculated as follows:
[Equation]
(10) where std( f ) denotes the standard deviation of the output of function f and std( f + res ) denotes the standard deviation of the output of function f added the residual.
C V ISUALIZATION OF T RAINING AND V ALIDATION P ERPLEXITY
[Figure]
We plot the training and validation perplexity of Transformer and our E IT on the WMT’14 task in
Figure 9.
We can see that our E IT owns lower training and validation perplexity than Transformer.
D D ETAILED ADDED PARAMETERS OF OUR METHODS
The detailed parameters of our methods are listed in Table 12.
[Table]
E T HEORETICAL C OMPLEXITY A NALYSIS OF G ROUP C ONVOLUTION
Given an input representation, namely H ∈ R d i × s h × s w , we process it with a standard convolution and a group convolution to obtain the output representations
ˆ H s ∈ R d
o × s
oh × s ow and ˆ
H g ∈ R d
o × s
oh × s ow , respectively, where d i is the dimension of the input feature, d o is the dimension of the output feature, s h is the height of the input feature, s w is the width of the input feature, s
oh is the height of the output feature and s ow is the width of the output feature.
When processing by a group convolutions with an output dimension d og , group g and a kernel size ( K h , K w ), we first reshape H to obtain H g ∈ R g ×
d g × s h × s w and generate the corresponding output, namely ˆ H g ∈ R d
o × s
oh × s ow , after reshaping the original output
ˆ H g m ∈ R g × d og × s
oh × s ow , where g is the number of groups, d g , equal to d i g , is the dimension of each group in the input feature and d og equals d o g which denotes the dimension of each group in the output feature.
During this process, we can obtain that the number of its parameters, namely N g , is d og × d g
× g × K
h × K w and its computational complexity, namely C g , is O
( g × d og × s
oh × s ow ×
K h
× K w × d g ) .
Correspondingly, when processing H by a standard convolution with an output dimension d o and a kernel size ( K h , K w ), the number of its parameters, namely N s , is d o × d
i × K h × K w and its computational complexity, namely C s , is O ( d o × s
oh × s ow ×
K h
× K w × d i ) .
By simplifying these expressions, we can get the final equations as follows: N g =
d o g ×
d i × K h
× K w =
N s g ,
C g
=
O
( d o g ×
d i × s
oh × s ow ×
K h × K w )
≈ C s g , which demonstrates that the group convolutions have theoretically computational efficiency and parametric efficiency. F A NALYSIS ON P LACEMENT OF DEI
Table 13 compares the impacts on several placements of DEI module, e.g., ISI → Softmax → CSI.
First, Softmax operation is insensitive to the placement of the CSI sub-module, which results in negligible BLEU degradation (#1 vs. #2).
Moreover, by comparing #2 and #3, we find that when placing the ISI sub-module behind the Softmax, the performance suffers a dramatic BLEU drop on both tasks.
A possible explanation is that the ISI module destroys the normalized attributes of the attention distributions since the convolutions with large kernel sizes introduce more noise in the original distributions.
This also indicates that a soft rectification of attention distributions can boost the quality of attention distributions, which highly correlates with the findings of previous work Zhou et al. (2021b). G W HETHER AND H OW M UCH EIT VARIANTS B ENEFIT FROM E NLARGED S UBSPACES [Table]
Intuitively, E IT can benefit from more heads under the same configuration, as it adopts the many-to-many scheme.
To validate this, we select only one attention map and share it across all the heads.
To be more specific, we still follow the calculation process to obtain the attention maps.
The main difference is that we modify the Eq.
(1) as follows:
O
[Equation]
A
[Equation]
A
[Equation]
O .
The results are exhibited in Table 14.
We find that both E IT and EE IT with selected attention map are still superior to the standard Transformer on two translation tasks.
This indicates that the expressiveness of one head in E MHA is comparable to that of eight heads in standard MHSA, which also demonstrates E IT and E-E IT have high efficiency in attaining gains from the enlarged subspaces.
H E FFECT OF S IZE OF R ECEPTIVE F IELD AND N UMBER OF H EADS
We also investigate the connection between performance and two important hyper-parameters: Respective Field and Number of Heads .
We first give a general definition of receptive field with any size r as follows:
Ψ ( i , r , U K ) =
{ K k |
K k ∈
U K
∧ k
=
( i + j − 1)% M, j = 1 , 2 , . . . , r } . With such definition, we conduct experiments with different r and different M . The results are exhibited in Figure 10.
Intuitively, the larger the size of the receptive field and the number of heads, the more attention maps we can obtain.
We observe that the performance gain increases as the sizes of these parameters grow.
This reveals that E IT can indeed benefit from more attention maps.
[Figure]
I E FFICIENCY /P ERFORMANCE T RADE - OFFS I. N
[Figure]
We further dig out the effect of the number of E IT encoder layers on the performance.
This can also be served as a way to trade-off between computational efficiency and performance.
Figure 11 exhibited the results.
We can see our E IT can maintain a high-level performance even when the number of E IT layers is one. The reason is that, as illustrated in Figure6(c), the standard Transformer has a higher token correlation even in the first layer, which does harm to the learning of later layers, while our E IT in the shallow layers can maintain a low level token correlation. Thus, they can collaborate well. Motivated by this phenomenon, we further apply such design to the big configuration.
We set the number of layers as 2, and keep the training strategy the same as that in Appendix A. Surprisingly,
We can train a big model with a performance of 29.29 within 7.12 hours, which cut off 31% training costs while improving 1.7% performance.
I.
[Figure]
Since there are several hyper-parameters in both ISI and CSI sub-modules, it is necessary to figure out how they affect performance. Figure 12 (a-d) plots the performance of E IT against the kernel size and the increasing rate.
We find that the ISI sub-module prefers a larger kernel size than the
CSI sub-module.
A reliable explanation is that extracting the representative information from sufficient attention maps requires a large receptive field.
Moreover, we find that E IT achieves the best performance with the M ψh and M ϕh of 2 r and 8 M , respectively.
Here M ψh and M ϕh represent the intermediate head size in ISI and CSI, respectively.
These observations further help us trade off efficiency and performance well.
J I S P ERFORMANCE HIGHLY RELATED TO DIVERSITY OF ATTENTION MAPS ?
[Figure]
We conduct detailed experiments on the WMT’14 En-De translation task to answer this question.
We exhibit the results in Figure 13, where we show the performance of Transformer with different ratios of restriction (left) as well as their corresponding similarity among attention maps learnt from different heads (right).
In most cases, models with this restraint are inferior to the standard
Transformer.
Moreover, the level of diversity among attention maps is not related to performance improvement.
For instance, the R-1 model owning the lowest head similarity, achieves the lowest performance.
All these indicate that similar attention positions are not a drawback for MHSA.
In contrast, similar attention maps may result from learning universal features.
All heads learn the most critical information for tasks, which results in high similarity.
This hypothesis can also be validated by Figure 8(below) and Figure 18.
K M ORE D ISCUSSION ABOUT O VERSMOOTHING AND F EATURES C OLLAPSE
We plot the cross-layer similarity of features in Figure 14.
We follow the metric defined in Zhou et al.
(2021a)
to calculate the cross-layer similarity of features.
We can see that the final representation learnt by E IT encoder is more distinct from that learnt by the first layer of encoder than
Transformer.
According to the findings in (Zhou et al., 2021a), we can conclude that E IT can extract more expressive representation.
Moreover, combining Figure 14 with Figure 6(middle & right), we can observe following trends:
• As the token correlation increases, the utilization of function like attention or FFN decreases.
• As the utilization of function like attention and FFN decreases, the slop of cross-layer similarity curve becomes to flat.
All these seem to reveal that the feature collapse problem attributes to the oversmoothing of functions to some degree.
L M ORE V ISUALIZATION
Detailed Visualizations of Attention Maps In this section, we visualize attention maps of each period, e.g., RFE, ISI and CSI.
The results are exhibited in Figure 15.
We can see that RFE can indeed generate a large number of diverse attention maps.
We also plot the attention maps generated by Attention Expansion in Figure 16.
The attention maps generated by Attention Expansion own high similarity.
Some attention maps even degrade to a nearly uniform distribution, such as the attention map placed in row 1 and column 5.
These attention maps may lead to some inefficiency since they seem to be useless. [Figure]
Visualization of Examples in Abstractive Summarization
Task We also plot the attention distributions of a real case in the abstractive summarization task in Figure 17 (Transformer) and Figure 18 (E IT ).
We can see that the attention maps of all subspaces generated by E IT have a strong weight within the diagonal region.
As for Transformer, only a few attention maps can capture such information.
aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead aaaaHead
[Figure]