On Orthogonality Constraints for Transformers (2021)

Abstract: Orthogonality constraints encourage matrices to be orthogonal for numerical stability. These plug-and-play constraints, which can be conveniently incorporated into model training, have been studied for popular architectures in natural language processing, such as convolutional neural networks and recurrent neural networks. However, a dedicated study on such constraints for transformers has been absent. To fill this gap, this paper studies orthogonality constraints for transformers, showing the effectiveness with empirical evidence from ten machine translation tasks and two dialogue generation tasks. For example, on the large-scale WMT’16 En→De benchmark, simply plugging-and-playing orthogonality constraints on the original transformer model (Vaswani et al., 2017) increases the BLEU from 28.4 to 29.6, coming close to the 29.7 BLEU achieved by the very competitive dynamic convolution (Wu et al., 2019).

## introduction transformers @xcite are a class of neural architectures that have made a tremendous transformative impact on modern natural language processing research and applications. transformers have not only served as a powerful inductive bias for general-purpose sequence transduction @xcite but also lived as the core of large pretrained language models @xcite @xcite . that said, the study of more effective training for this class of models is still an open research question, bearing great potential to impact a myriad of applications and domains. to improve numerical stability during training, the trick of enforcing orthogonality constraints has surfaced recently. in the analysis of numerical stability, enforcing orthogonality constraints can upper-bound the lipschitz constant of linear transformations. the lipschitz constant is a measure that approximates the rate of change (variation) of representations. theoretically, controlling the lipschitz constant, which may be achieved via orthogonality constraints, yields representations that are robust and less sensitive to perturbations. in view of this, orthogonality constraints have been studied for convolutional neural networks (cnns) @xcite and recurrent neural networks (rnns) @xcite @xcite . such plug-and-play constraints can be incorporated into model training without additional hassle. for example, cnn-based models incorporating orthogonality constraints have demonstrated empirical effectiveness for tasks such as person reidentification @xcite and keyword spotting @xcite , while rnn-based models that enforce such constraints have shown promising empirical results for response generation @xcite ) and text classification @xcite . however, a dedicated study on orthogonality constraints for transformers has been absent so far. to fill this research gap, we study orthogonality constraints for transformers, which are imposed on (i) linear transformations in self-attention and position-wise feed-forward networks and (ii) the affinity matrix in self-attention. mathematically, orthogonality constraints on the weights of these linear transformations can be motivated by bounded lipschitz constants. we also formally analyze the self-attention mechanism by bounding perturbations to the affinity matrix in the face of input changes. furthermore, we conduct extensive experiments on ten neural machine translation (both subword-level and character-level) tasks and two dialogue generation tasks. our experimental results are promising, demonstrating that the performance of transformers can be consistently boosted with orthogonality constraints. for example, on the large-scale wmt'16 @xmath0, simply plugging-and-playing orthogonality constraints on the original transformer model @xcite increases the bleu from 28.4 to 29.6, coming close to the 29.7 bleu achieved by the very competitive dynamic convolution @xcite . notation for any vector x and any matrix x, x and x denote their l 2 -norm and spectral norm, respectively. ## orthogonality constraints for transformers recall that in the transformer architecture, keys, queries, and values all come from the same place in the self-attention module. they are linearly transformed for computing multiple attention heads, where all the heads are aggregated by another linear transformation. the position-wise feed-forward network is also built on two linear transformations with activations. in the following, we will describe orthogonality constraints for (i) linear transformations in self-attention and position-wise feedforward networks and (ii) the affinity matrix in self-attention. ## experiments we evaluate the effectiveness of orthogonality constrained transformers (oc-transformers for brevity) on ten neural machine translation tasks and two dialogue generation tasks. specifically, we assess three variants, largely pertaining to where orthogonality constraints are applied, i.e., (i) am (for the affinity matrix in self-attention), (ii) la (for the linear transformations in self-attention), and (iii) lf (for the linear transformations in position-wise feed-forward networks). we evaluate them in an incremental fashion with three main model labels: var-i (am only), var-ii (am + la), and var-iii (am + la + lf). the scaling factor λ is tuned amongst {10 -6 , 10 -8 , 10 -10 }. ## conclusion we studied orthogonality constraints for transformers, which are imposed on (i) linear transformations in self-attention and position-wise feedforward networks and (ii) the affinity matrix in self-attention. we showed that such plug-and-play constraints, which can be conveniently incorporated, consistently boost performance of transformers on ten different machine translation tasks and two dialogue generation tasks. for example, on the large-scale wmt'16 @xmath1, simply plugging-and-playing orthogonality constraints on the original transformer model @xcite increases the bleu from 28.4 to 29.6, coming close to the 29.7 bleu achieved by the very competitive dynamic convolution @xcite . broader impact given widespread adoptions of transformer models, the proposed plug-and-play orthogonal constraints could also be useful to computer vision, automatic speech recognition, time series analysis, and biological sequence analysis.