Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation (2022)

Abstract: Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as long sequence lengths and redundancy between adjacent tokens. Therefore, we believe that regular self-attention mechanism might not be well suited for it. Different approaches have been proposed to overcome these problems, such as the use of efficient attention mechanisms. However, the use of these methods usually comes with a cost, which is a performance reduction caused by information loss. In this study, we present the Multiformer, a Transformer-based model which allows the use of different attention mechanisms on each head. By doing this, the model is able to bias the self-attention towards the extraction of more diverse token interactions, and the information loss is reduced. Finally, we perform an analysis of the head contributions, and we observe that those architectures where all heads relevance is uniformly distributed obtain better results. Our results show that mixing attention patterns along the different heads and layers outperforms our baseline by up to 0.7 BLEU.

## introduction conventionally, speech-to-text translation (st) task has been addressed through cascade approaches @xcite , which consists of the concatenation of an automatic speech recognition block (asr), for the audio transcription, with another machine translation block (mt), for the translation of such transcription into the desired language. however, this approach ignores some information present in the audio, since it translates from the audio transcript, and is also vulnerable to error propagation, since an error in the asr block automatically causes a mistranslation @xcite . consequently, end-to-end alternatives based on an encoder-decoder structure and attention mechanisms have become increasingly popular in recent years @xcite @xcite . these are capable of translating the audio without the explicit need for transcription, thus avoiding the problems of the cascade approach and allowing unified optimization of the training parameters. the advent of the transformer @xcite revolutionized the mt field, enabling models based on this architecture to achieve the stateof-the-art results. nowadays, transformer-based models are used to process all types of data, such as images @xcite or speech @xcite . however, due to its self-attention mechanism, the vanilla transformer scales quadratically with the input sequence length, which makes it extremely inefficient when processing long sequences. in speech tasks, it is common to extract audio features every 10 ms to build the input sequences, which causes them to be considerably longer than text sequences. moreover, since the representation of a single phoneme requires several tokens @xcite , the presence of redundancy among the audio tokens is inferred. therefore, state-of-the-art architectures propose the implementation of down sampling strategies prior to the model collapsing adjacent vectors in a fixed way @xcite @xcite . similarly, some studies propose to extract more informative sequences using pretrained compression modules @xcite @xcite , obtaining considerable translation quality gains. while these achieve good results, we propose another approach, questioning the use of multi-head self-attention (mhsa), originally proposed for text, for the information extraction from speech sequences. the closest work to ours was done by @xcite , who used longformer's @xcite local attention pattern as an efficient alternative to self-attention for speech processing. however, they observed that, due to the scarcity of global context in the encoder output, the quality of the translations was slightly hindered. recently, inspired by linformer's @xcite attention, @xcite proposed convattention as an attention mechanism that, by compressing keys and values, is more efficient and therefore able to directly process long sequences. however, this mechanism is not used as a replacement of the encoder self-attention, but as an extra input processing before a ctc-based compression module @xcite . our contribution to st field is a new transformer variant, the multiformer, an architecture based on the s2t transformer by @xcite . our architecture enables the use of different attention mechanisms in the same encoder layer, by configuring individually the pattern of each head. with this approach, the multiformer is able to apply efficient attention mechanisms, while maintaining the ability to learn both local and global content from speech sequences. this diversity among heads in a layer is also meant to stimulate a more varied information extraction and, therefore, reduce the presence of low-relevant heads @xcite @xcite . furthermore, we explore the use of different head configurations for each encoder layer. this could help to adapt the attention mechanisms to the needs of each layer. to the best of our knowledge, this is the first study that allows this kind of head-wise configuration. ## model in this section, we first introduce a new selfattention module that allows the use of multiple attention mechanisms in parallel ( §2.1). next, we explain the multiformer ( §2.2), which replaces the transformer encoder mhsa by the new proposed module. ## heads contribution analysis mhma allows the use of different attention mechanisms in parallel, therefore we wanted to evaluate the head contribution in each of the encoder layers. in general, given an input sequence of n tokens {x 1 , ..., x n } ∈ r n×d and a model with an embeddings dimension d (head dimension d h ), the output of each attention-head z h @xmath0: figure 3 : head relevance for each layer of the proposed models. they have been computed using the median of n contributions (equation 4) from 500 random samples of the en-de training partition. heads marked in orange use local attention while those in purple are using convattention. with @xmath1, and b @xmath2. as previously done for interpretability research @xcite , the above expression (equation 2) can be rewritten as follows: inspired by @xcite , for each layer, we define the contribution of each head to the attention output y i as the euclidean norm of the projected output vector of heads: in this section we first explain the training details ( §4.1) in order to ensure reproducibility of experiments. 2 then we briefly describe multiformer architectures and the procedure we followed ( §4.2). ## results first, it can be observed from secondly, analyzing the heads contribution of the baseline architecture, we can observe that the heads contribution tends to accumulate in few heads. this means we obtain similar conclusions than @xcite , but for the st setting. furthermore, our work goes one step further, showing that those architectures where the heads contribution is uniformly distributed, obtain a higher performance. this finding confirms that, in st, some heads on a regular transformer tend to learn irrelevant information. this shows that mhsa might not be as capable as expected of extracting different kinds of patterns, unless it is biased on purpose towards doing so. in particular, all multiformer variants improve the results obtained by the baseline and the local_attention and conv_attention architectures. however, these improvements are not equal in all languages pairs, and go from 0.15 to 0.57 bleu for multif ormer_lc, from 0.13 to 0.69 bleu for multif ormer_v1 and from 0.29 to 0.45 bleu for multif ormer_v2, becoming the latter the architecture with the most robust gains. ## conclusions in this paper, we present the multiformer, the first transformer-based model that allows to combine different attention mechanisms in the mhsa module. this helps the model extracting different types of token interactions from each head, hence preventing the appearance of irrelevant heads. by applying this diversity of attention patterns with efficient mechanisms, the model is able to maintain both local and global context across encoder layers while being more efficient. experiments on 3 language pairs show that all multiformer architectures outperform the results achieved by the s2t transformer in the st task, with an improvement up to 0.69 bleu for the english-spanish direction in the multif ormer_v1.