EIT : Enhanced Interactive Transformer (2024)

Abstract: Two principles: the complementary principle and the consensus principle are widely acknowledged in the literature of multi-view learning. However, the current design of multi-head self-attention, an instance of multi-view learning, prioritizes the complementarity while ignoring the consensus. To address this problem, we propose an enhanced multi-head self-attention (EMHA). First, to satisfy the complementary principle, EMHA removes the one-to-one mapping constraint among queries and keys in multiple subspaces and allows each query to attend to multiple keys. On top of that, we develop a method to fully encourage consensus among heads by introducing two interaction models, namely inner-subspace interaction and cross-subspace interaction. Extensive experiments on a wide range of language tasks (e.g., machine translation, abstractive summarization and grammar correction, language modeling), show its superiority, with a very modest increase in model size. Our code would be available at: https://github.com/zhengkid/EIT-Enhanced-Interactive-Transformer.

## introduction transformer architectures @xcite have yielded promising results on a wide range of natural language processing tasks @xcite . a key factor contributing to their success is the multi-head self-attention network (mhsa), which enables efficient modeling of global dependencies among tokens in parallel. notably, instead of utilizing a single attention mechanism, mhsa uses an ensemble of attention models, each models a small subspace, and finally aggregates these results to the final one. the core idea is similar to subspace learning @xcite or multi-view learning @xcite . in the realm of multi-view learning, two fundamental principles guide the research: the complementary principle and the consensus principle @xcite . the complementary principle emphasizes that each data view may possess unique knowledge not present in other views, prompting the use of multiple views for a comprehensive and accurate data description. on the contrary, the consensus principle aims to maximize the agreement on multiple distinct views. however, in the context of mhsa design, most studies predominantly focus on the complementary principle. this oversight is evident in their encouragement of diverse information capture by different heads @xcite @xcite and the adoption of complex aggregation operations @xcite . some studies @xcite @xcite even consider the high similarity among attention heads as a significant problem referred to as attention redundancy. although diversity is crucial in multi-view learning, @xcite has shown that simply fusing diverse outputs does not guarantee improved results: the probability of a disagreement of two independent hypotheses upper bounds the error rate of either hypothesis. the consensus principle highlights the need to minimize disagreement among views to achieve better outcomes. in response to the consensus principle, several studies @xcite have focused on minimizing disagreement among views to achieve better outcomes. however, in the context of mhsa research, there is a tendency to prioritize complementarity over consensus among different attention heads. here we ask a question: can balancing these two principles benefit the design of mhsa mechanisms? however, encouraging such a consensus in multihead self-attention is challenging. in our preliminary experiments, we found that directly utilizing regularization terms can achieve this goal but cannot improve performance. drawing inspirations from human behavior where group discussions and interactions foster consensus, we propose introducing interactions among different subspaces in mhsa to achieve consensus. to this end, we propose a new multi-head self-attention variant: enhanced multi-head self-attention, which encourages consensus among attention heads while guaranteeing to contain sufficient information. to ensure information sufficiency, we propose a novel many-to-many mapping scheme to generate numerous high-quality initial attention maps. this can generate more attention maps without suffering low-bottleneck problems @xcite . on top of these sufficient attention maps, we propose two interaction components: inner-subspace interaction (isi) and cross-subspace interaction (csi). these hierarchical interaction modules fully encourage consensus among attention maps of different heads. the outcome of this work is an enhanced interactive transformer (eit) architecture in that mhsa is replaced with enhanced multi-head attention (emha). our proposed eit has been demonstrated to be simple to implement and highly parameter efficient, yet it produces consistent performance improvements across a diverse set of tasks, including machine translation, grammar error correction, abstractive summarization, and language modeling. in addition, we have developed a computationally efficient variant of eit, which, while still maintaining strong performance on several tasks, is better suited for low-latency industrial applications. ## preliminary: multi-head self-attention multi-head self-attention (mhsa) is an efficient operation that can capture the interactions among tokens. given an embedded input sequence @xmath0, mhsa is defined as follows: a i represents the attention distribution of i-th head and o represents the output of multi-head self-attention mechanism. without the special declaration, we use q i , k i , v i to refer to xw i q , xw i k , xw i v , respectively, which denotes the query, key and value in i-th head. ## enhanced interactive transformer we design a novel enhanced interactive transformer (eit) in which we replace the multi-head self-attention with an enhanced multi-head attention mechanism (emha) that encourages consensus among different attention heads. our method mainly modified eq. ( 1 ) but otherwise follows the standard transformer. ## experiment settings we evaluated our eit on four widely used benchmarks foot_0 : 1) machine translation, 2) grammar error correction, 3) abstractive summarization, and 4) language modeling. the detailed architecture setups, training setups and evaluation setups were presented in appendix a. ## ablation studies settings. we gave detailed description about the settings of ablation studies. results. ## analysis of computational efficiency macs comparison. resource comparison. in addition to theoretically exploring the efficiency of eit variants, we also measured the practical computational consumption during the training process. without losing generality, we focused on the model for the base configuration. we can see that eit consumes 8% more memory consumption and 45% more training costs than the baseline with a depth of 6. to mitigate this, we have proposed the e-eit which only costs 5% more memory consumption and 10% more training costs than the baseline but delivered similar performance compared to eit. notably, as shown in ## related work low bottleneck in multi-head attention the "low bottleneck" issue in multi-head self-attention (mhsa) occurs when adding more heads to transformers, which does not correspondingly improve performance. @xcite first identified this issue, attributing it to the diminishing head dimension as the number of heads increases, which limits the creation of precise attention maps. in response, @xcite introduced "talking-head attention," using two linear transformations around the softmax function to address this bottleneck. later, @xcite proposed a framework involving "ghost heads" to enrich attention patterns, differentiating from talkinghead attention in the positioning of linear transformations and the number of ghost heads. our approach introduces a many-to-many mapping in mhsa, using existing queries and keys for more attention maps through direct query-key multiplication. improved multi-head mechanism previous work has shown that multi-head attention can be further enhanced by encouraging individual attention heads to extract distinct information @xcite @xcite @xcite . another branch of research is designing more complex interactive modeling to make better use of the multiple subspace information @xcite @xcite . besides, @xcite empirically demonstrates that some heads in attention are useless and can be pruned without performance degradation. along this line, researchers investigate how to efficiently cut off redundant heads @xcite . different from theirs, our study utilized the benefits of both diversity and consistency. ## conclusions in this paper, we propose eit, an alternative to the transformer architecture. it further advances the multi-head schema by fully leveraging two principles in multi-view learning: the complementary principle and the consensus principle. in addition, e-eit can serve as another choice considering the trade-off between performance and computation efficiency. experimental results on four widely-used tasks demonstrate the effectiveness of eit-variants, which deliver consistent improvements to the standard transformer.