Highway Transformer: Self-Gating Enhanced Self-Attentive Networks (2020)

Abstract: Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, standing on the multi-headed dot product attention by attending to all the global contexts at different locations. Through a pseudo information highway, we introduce a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic importance within the multi-dimensional latent space of individual representations. The subsidiary content-based SDU gates allow for the information flow of modulated latent embeddings through skipped connections, leading to a clear margin of convergence speed with gradient descent algorithms. We may unveil the role of gating mechanism to aid in the context-based Transformer modules, with hypothesizing that SDU gates, especially on shallow layers, could push it faster to step towards suboptimal points during the optimization process.

## introduction self-attention mechanism has lately attracted extensive interests due to its remarkable achievement on a wide range of sequence modeling applications, including natural language processing such as neural machine translation @xcite @xcite , language modeling (lm) @xcite , self-supervised pretraining @xcite @xcite ; image generation @xcite ; deep reinforcement learning @xcite , etc. holding the great promise of deep neural networks in language and images, transformer capitalizes on the stacked multi-headed self-attention mechanism based on the conventional encoderdecoder architecture in a sequence-to-sequence (seq2seq) manner to learn the global soft signals without explicit recurrence mechanism. multi-head dot product attention (mhdpa) not only underpins the parallel training of multiple heads but captures long-term dependencies across an arbitrarily long distance within the same context. in which separated multiple heads independently draw sub-level attentions within the latent semantic sub-space of a fixed dimension, where different heads are presumed to signal different meaning aspects implicitly @xcite . additionally, residual connections between layers allow the deep tandem stack of multiple identical modules by impeding degradation problem during training @xcite . thus transformer architectures take the place of recurrent neural networks (rnns), especially long short-term memory (lstm) networks @xcite to be the model solution to learning sequential data. recently, there have been plenty of works contending that gating mechanisms could play a vital role or even entirely substitute rnns or transformers to model language sequences. @xcite firstly claimed that non-recurrent networks are also highly competitive with conventional rnndominated models in lm. they proposed the hierarchical gated temporal convolution neural networks (cnns) with gated linear units (glu) to replace the recurrent connections in rnns and achieved strong performance with faster training speed. @xcite integrated absolute positional embedding, multi-step attention, glu, and residual connections into entirely convolutional models to outperform strong lstm models in nmt and abstractive summarization tasks. @xcite applied dynamic convolutions using shared softmax-normalized filters of depth-wise on glu-regulated inputs within a fixed reception field rather than global contexts, challenging the common self-attention-dominated intuition. however, all of the models, as mentioned earlier, adopt stacked cnns rather than self-attention networks (san) to attend to the global contexts. it is well-known that cnns are good at learning localregion features rather than long-term dependency, while sans are adept in attending global dependencies. context-based self-attention can capture the importance of relative relations under a valid context and is thus location-unaware. it focuses on the object-wise attention distributions between any two words but ignores the fundamental importance of feature-wise information. intuitionally, people need to consider not only the global contextual dependency but the meaning of individual words to comprehend the reading materials better. grounding on this, we apply self-gating approaches on transformer blocks for seq2seq modeling that combines gating units with skip-connections and transformers to jointly take into account both the inner feature-wise importance and the relation-aware content-based attention distribution. we adopt the self-dependency gating approach to intrinsically draw a binary importance ratio of itself and decide how much information of each feature to retain or remove. our key contributions are: ## preliminaries this section briefly introduces the related background of transformer and highway networks. san has been dominant in most sota sequence learning models, whose basic components consist of stacked transformers modules. we conduct comparison experiments on the transformer and its two variants, transformer-xl @xcite and r-transformer @xcite . ## experiments and results we apply the gating mentioned above on transformer variants described in section 2 on lm tasks and respectively make comparisons in terms of both the convergence process and the final performance. for fairness, we apply sdu components based on the same hyperparameters as the original paper foot_0 . our code is available foot_1 . ## conclusion and future work gating-enhanced architecture enjoys both the advantage of mhdpa and self-regulated gating mechanism, allowing for the pseudo-highway information flow for better convergence by elastically intro-ducing a few trainable parameters. it outperforms or matches the performance of common transformer variants without hyperparameter tuning. it is empirically proved that self-gating units on shallow layers could provide more internal representations of importance and significantly benefit for convergence. this also supports the argument that different levels of transformer components attend to different semantic aspects while lower levels pay more attention to local regions. in the future, it is necessary to interpret the semantics that transformer layers in different depths can convey, which is beneficial for the computing-efficiency.