On the Locality of Attention in Direct Speech Translation (2022)

Abstract: Transformers have achieved state-of-the-art results across multiple NLP tasks. However, the self-attention mechanism complexity scales quadratically with the sequence length, creating an obstacle for tasks involving long sequences, like in the speech domain. In this paper, we discuss the usefulness of self-attention for Direct Speech Translation. First, we analyze the layer-wise token contributions in the self-attention of the encoder, unveiling local diagonal patterns. To prove that some attention weights are avoidable, we propose to substitute the standard self-attention with a local efficient one, setting the amount of context used based on the results of the analysis. With this approach, our model matches the baseline performance, and improves the efficiency by skipping the computation of those weights that standard attention discards.

## introduction recently, transformer-based models have gained popularity and have revolutionized natural language processing (nlp) @xcite @xcite . in the speechto-text setting, the transformer works with audio features like the mel-spectrogram @xcite . these features provide longer input sequences compared to their raw text counterparts. this can be a problem when regarding complexity, since the transformer's attention matrix computational cost is o(n 2 ), where n is the sequence length. in speech, a common approach used to overcome this issue and reduce the input sequence length is to employ convolutional layers with stride before the transformer encoder. however, even with the addition of convolutional layers, time and memory complexity is still an issue. we build upon recent advances in the explainability of the transformer to analyze the amount of context used by self-attention when dealing with speech features. recent interpretability works have moved beyond raw attention weights as a measure of layer-wise input attributions and have integrated other modules in the self-attention, such as the norm of the vectors multiplying the attention weights @xcite , the layer normalization, and the residual connection @xcite . in the automatic speech recognition (asr) domain, the usefulness of the self-attention has been argued @xcite , showing that its exposure to the full context might not be necessary, especially in the top layers. we carry out this analysis for direct speech translation (st) systems, which are capable of translating between languages from speech to text with a single model. the encoder of these systems needs to jointly perform acoustic and semantic modeling, while in asr the latter is not that relevant @xcite . to the best of our knowledge, this is the first work that uses interpretability methods to understand how the transformer's self-attention behaves in the direct st task. in this work, we use the layer-wise contributions proposed by @xcite to analyze the patterns of self-attention in direct st in en-de, en-es and en-it tasks, unveiling their strong local nature. consequently, using self-attention might not be entirely useful, but it is computationally costly. to verify this hypothesis, based on our analysis, we propose a new architecture designed to maximize the efficiency of the model while minimizing the information loss, and demonstrate no hinder in the model's performance in any of the three directions. we achieve this by substituting regular self-attention with local attention in those layers where the contributions are placed around the diagonal. finally, we analyze the performance of the proposed model. ## speech-to-text transformer recent works have attempted to adapt the transformer to speech tasks @xcite . in the direct st domain, a usual approach is adding two convolutional layers with a stride of 2 before the transformer @xcite . by doing this, the sequence length is reduced to a fourth of the initial one. after the two convolutional layers, the speech-to-text transformer (s2t transformer) consists of a regular transformer model, composed of 12 encoder layers and 6 decoder layers. the main component of the transformer is the multi-head attention mechanism, in particular, the self-attention is in charge of mixing contextual information. given a sequence of token representations {x 1 , • • • , x n }, each of the h heads projects these vectors to queries q @xmath0, keys k @xmath1, with head dimension d @xmath2, where d is the model embedding dimensionality. the self-attention attention (sa) computes: where w h @xmath3, and training details. we reproduce the s2t transformer training with fairseq @xcite . the training procedure consists of two phases. first, we pre-train the model in the asr setting @xcite . then, we substitute the decoder with a randomly initialized one, and both are finally trained in the st task (see appendix c for more details on the hyperparameters). for the trainings, we use the must-c english-german, english-spanish and english-italian subsets @xcite . ## model analysis in this section, we present the analysis of the encoder self-attention in the s2t transformer. interpretability method. @xcite propose an interpretability method that measures the impact of each layer input, i.e token representations (x j ), to the output of the layer, considering also the layer normalization and the residual connection. they provide the formulation for the attention block of the original transformer architecture, which has layer normalization on top of the self-attention module. in this work, we give an adaptation to the group of models that normalize before the multi-head attention (pre-ln), such as the s2t transformer. the complete chain of computations in the pre-ln attention block can be reformulated as a simple expression of the layer inputs: we can now express the attention block output as a sum of transformed input vectors (f i (x j )): where f i (x j ) is defined as: kobayashi et al. (2021) measure the contribution c i,j of each input vector x j to the layer output xi with the euclidean norm of the transformed vector: layer-wise analysis. we analyze the contribution scores obtained with eq. 5 from the encoder layers in both asr (pre-training) and st tasks. from the results shown in figure 4 (see also appendix e) we observe that most layers' contributions are dense around the diagonal. to measure the degree of diagonality in the contribution matrices at each layer l, we build upon the attention diagonality proposed by @xcite , originally defined with attention weights and proportions of the sequence length. we reformulate it with the obtained contributions, and token ranges w (see appendix a for more details on the differences): in figure 3 we show the ccd results for asr and st across layers, where we can observe a strong diagonal pattern. we can see that, surprisingly, ccd is very similar in both tasks. this contradicts the belief that, because of the need for deeper semantic processing when translating, st needs more context than asr. furthermore, we see different behaviors along the encoder, and a trend towards uniformly distributed contributions in the first layers. moreover, in figure 4 , we can see differences between those layers that show local patterns. layers 4, 5 and 6 attend to close context. instead, those layers at the end of the encoder, as 11 or 12, need larger context, and we can see how the contributions create patterns corresponding to words in the spectrogram, enabling us to see interactions between them (figure 1 ). additionally, we see that contribution matrices reveal silences in the speech sequence. however, we believe that further research is needed to fully understand the meaning of these patterns. ## efficient speech-to-text transformer from the previous analysis, we hypothesize that suitable local attention patterns may potentially avoid the computation of unused attention scores. note that if a token does not contribute to the output of a layer, its attention score can be canceled. our objective is to maximize the efficiency of the model while minimizing the performance drop. window size selection. ccd could serve as a starting point to obtain optimal window lengths. however, it requires predefining the amount of total contribution required inside the window, which makes it fragile to properly detect local patterns. on one hand, it can be too sensitive to a strong main diagonal. on the other, it may overestimate random distant contributions. we propose an al-algorithm 1: window size selection input: c l : contribution matrix, n : number of tokens, t: min diagonal contribution threshold output: w l : optimal window size ## conclusions transformer-based models are the current stateof-the-art in many different fields. however, the quadratic complexity of the self-attention module usually hinders the usefulness of the model in real-life applications. this problem worsens when working with long sequences, as is the case with speech. in this paper, we have questioned the need of computing all attention weights in st. we have analyzed the contribution matrices, and we have seen that, in many layers, the relevant scores are placed in a diagonal pattern. therefore, we have hypothesized that these weights do not need to be calculated. to verify our hypothesis, we have trained a model that substitutes regular self-attention with local attention, with a suitable window size for each layer. we have seen that as we expected, the results are almost equal to the ones obtained with the baseline model, but the complexity has been lowered significantly. regarding interpretability, we have found how the transformer establishes connections between words in speech sequences. furthermore, we have seen that, in contrast to what was expected, diagonality scores are similar in both st and asr tasks, meaning that they use the same amount of context.