Enlivening Redundant Heads in Multi-head Self-attention for Machine Translation (2021)

Abstract: Multi-head self-attention recently attracts enormous interest owing to its specialized functions, significant parallelizable computation, and flexible extensibility. However, very recent empirical studies show that some self-attention heads make little contribution and can be pruned as redundant heads. This work takes a novel perspective of identifying and then vitalizing redundant heads. We propose a redundant head enlivening (RHE) method to precisely identify redundant heads, and then vitalize their potential by learning syntactic relations and prior knowledge in the text without sacrificing the roles of important heads. Two novel syntax-enhanced attention (SEA) mechanisms: a dependency mask bias and a relative local-phrasal position bias, are introduced to revise self-attention distributions for syntactic enhancement in machine translation. The importance of individual heads is dynamically evaluated during the redundant heads identification, on which we apply SEA to vitalize redundant heads while maintaining the strength of important heads. Experimental results on widely adopted WMT14 and WMT16 English to German and English to Czech language machine translation validate the RHE effectiveness.

## introduction recently, self-attention network (san) @xcite has been applied to various natural language processing tasks. instead of drawing distanceaware dependencies like recurrent neural network @xcite and convolutional neural network @xcite , san captures short-and long-range relations between elements. san involves all signals with a weighted averaging operation, which may incorporate too many unrelated elements to concentrates on specific relations. recent work has modified san to enhance specific relation learning. for example, in (shen et al., 2018), a directional self-attention network (disan) uses one to multiple positional masks to model the asymmetric attention between two elements and capture context-aware relations for all tokens. @xcite modeled the local information by revising the attention distribution with a learnable gaussian bias to focus on neighboring relations. @xcite extended san to efficiently consider distinct representations of the relative linear position relations between sequence elements. however, the above approaches consider the multi-head san as a whole but ignore unbalanced contribution distributions between heads. furthermore, multi-head san combines different attentions from multiple subspaces to construct transformer @xcite and achieves the state-of-the-art results in recent neural machine translation (nmt) tasks @xcite . the very recent work @xcite shows that the encoder-side individual heads in transformer make different contributions, multi-heads can be classified into important heads and redundant heads and pruning redundant heads does not seriously affect performance. they also assume that important heads play various roles which influence the generated translations to different extents, including syntactic function (focusing on dependent relations), positional function (focusing on neighboring words), and rare words-based function. to date, our understanding of the roles of dis-tinct multi-heads is very limited, with no systematic analysis available of the roles of different heads. in this paper, we precisely identify redundant heads at the encoder-side of transformer and demonstrate the potential of syntactically reactivating the redundant heads to improve the multi-head san performance. fig. 1 illustrates the different rationales of existing work against ours in multi-head san. the left part represents those approaches that directly enhance overall heads as a whole w.r.t. their designed functions but do not differentiate their roles. such approaches may downplay the functions of important heads and the diversity of the multi-head mechanism. the middle part represents the methods that analyze contributions and functions of multi-head san and then prune the determined redundant heads but rely on those important heads only. as shown in the right part, this paper proposes a dynamic and unified strategy to identify redundant heads and then enliven them to fulfill their potential. by enlivening the redundant heads, our approach enhances the performance of redundant heads without sacrificing the essential functions of important heads. in addition, our method further increases the scale of important heads. specifically, we take nmt as an example to illustrate our method of identifying and reactivating the redundant heads in multi-head san. we firstly propose two novel syntax-enhanced attention (sea) mechanisms for machine translation: 1) the dependency-enhanced attention to use a dependent matrix as mask to model the intensive attention between dependent elements and filter elements without direct dependent relations; and 2) local-phrase-enhanced attention to incorporate a distinct and learnable relative local-phrasal position matrix as bias, which is transformed from a constituency tree under the rules of local-phrase. these syntax-enhanced attention mechanisms simulate the specific functions of important heads but differ from the existing self-attention improvement approaches. compared to the dependency tree, there is distinct syntactic layer information for each word in the constituency tree, which is extracted to calculate the relative phrasal position to reflect syntactic relations between elements. to this end, we define a novel phrase type local-phrase to only extract syntactically related words as phrase by leveraging the constituency tree, regardless of sequence distance. further, we propose a dynamic and lightweight redundant heads enlivening (rhe) strat-egy for multi-head san to reactivate and enhance the roles of redundant heads. lastly, a dynamic function gate is designed, which is transformed from the average of maximum attention weights to compare with syntactic attention weights and identify redundant heads which do not capture meaningful syntactic relations in the sequence. we test the above design on three widelyused translation tasks wmt14 and wmt16 @xmath0. extensive analyses reveal that enlivening redundant heads in multi-head san beats improving overall heads, and the proposed syntax-enhanced attention mechanisms with dependency and local phrases further effectively improve the translation performance. ## related work one popular extension to the san is to revise attention distribution by static and dynamic biases. different dimensions of biases have been considered, including directional relation @xcite and localness @xcite @xcite . @xcite improves san with directional masks and multidimensional features by explicitly revising attention distribution. in this paper, we focus on the explicit syntactic biases by proposing dependencyenhanced attention and local-phrase-enhanced attention. several papers show that explicitly modeling dependency @xcite or phrase @xcite @xcite is useful for tasks such as nmt. related to our work, @xcite and @xcite also modify parts of self-attention heads with syntactic information. however, they randomly assign heads instead of analysing the importance and function of each head in advance. @xcite restricts san with the neighboring elements and performs better for longer sequences in acoustic modeling and natural language inference tasks. @xcite leverages gaussian bias predicted by the query vector to dynamically model the localness for san. other work analyzes the attention weights of different nmt models @xcite @xcite . @xcite considers how different heads correspond to specific relations and proves that redundant heads can be pruned without greatly decreasing translation performance. however, they disregard the full potential of redundant heads as in our sea. @xcite realizes the diversity of multiple attention heads and introduces a disagreement regularization to explicitly encourage the diversity. nevertheless, they do not realize that only partial individual heads are redundant, which is a prerequisite for optimizing multi-head diversity. in summary, while some of the related work recognizes the approach of revising attention distribution with bias, our work represents the first to propose a complement and precise strategy to analyze individual heads, identify redundant heads and then enliven them with syntactic bias. ## background 3.0.1 multi-head self-attention multi-head san @xcite @xcite projects the input sequence to multiple subspaces (h attention heads), applies the scaled dot-product attention to the hidden states in each head, and then concatenates the output. for each self-attention head head i (1 ≤ @xmath1, given an input sequence @xmath2, ..., x n }, each hidden state in the l-layer is constructed by attending to the states in the (l -1)-th layer. specifically, the hidden states of (l -1)-th layer h l-1 ∈ r n×d h are firstly transformed into the queries @xmath3, the keys @xmath4, and the values @xmath5, where d h represents the dimensionality of each head. the hidden state h i of the l-th layer is calculated as: ## the rhe design fig. 2 shows the architecture of our proposed redundant heads enlivening (rhe) approach to identify redundant heads and then enliven them by revising self-attention distributions with a syntactic bias. rhe takes full advantage of the multi-head san by capturing both dependent and distinct phrasal relations. first, two syntax-enhanced attention (sea) mechanisms: dependency enhanced attention (dea) and local-phrase enhanced attention (lpea), are proposed. dea disables the attention between elements without dependencies by leveraging the dependency mask, and lpea precisely regulates the self-attention distribution by a distinct and learnable local-phrase bias. the bias represents relative local-phrasal position transformed from a constituency tree. lpea precisely captures both short-and long-term syntactic relations. second, the redundant head identification module dynamically determines the importance and function of each head during the training process per the average sum of syntactic attention weights. lastly, the self-attention of redundant heads is replaced by sea to enliven their full potential and roles. ## conclusions while multi-head self-attention networks show a significant potential in improving learning tasks such as nmt, an open challenging topic is to quantify the redundancy and importance of each head and further improve the weak heads. this paper makes one step forward by not only precisely analyzing and identifying redundant heads but introducing a dynamic redundant heads enlivening (rhe) mechanism to identify and enliven each redundant head toward full potential without affecting the function of other important heads as in alternatively enhancing all heads. the proposed dependency-enhanced attention and local-phraseenhanced attention effectively capture the different syntactic relations between elements. we'll work on strategies to integrate dea and lpea in future. metric task pcfg neural f1 wsj parsing 91.20 93.55 bleu en-de (wmt16) 25.98 26.28 en-de (wmt14) 28.02 28.27 en-cs (wmt16) 15.92 16.27 table a3: performance of two classical constituency tree parser tools on the penn treebank wsj test set (f1 score) and its corresponding effect on the lpea+rhe nmt model (bleu score).