Speechformer: Reducing Information Loss in Direct Speech Translation (2021)

Abstract: Transformer-based models have gained increasing popularity achieving state-of-the-art performance in many research fields including speech translation. However, Transformer’s quadratic complexity with respect to the input sequence length prevents its adoption as is with audio signals, which are typically represented by long sequences. Current solutions resort to an initial sub-optimal compression based on a fixed sampling of raw audio features. Therefore, potentially useful linguistic information is not accessible to higher-level layers in the architecture. To solve this issue, we propose Speechformer, an architecture that, thanks to reduced memory usage in the attention layers, avoids the initial lossy compression and aggregates information only at a higher level according to more informed linguistic criteria. Experiments on three language pairs (en→de/es/nl) show the efficacy of our solution, with gains of up to 0.8 BLEU on the standard MuST-C corpus and of up to 4.0 BLEU in a low resource scenario.

## introduction speech-to-text translation (st) has been traditionally approached with cascade architectures consisting of a pipeline of two sub-components @xcite : an automatic speech recognition (asr), which transforms the audio input into a textual representation, and a machine translation (mt) model, which projects the transcript into the target language. a more recent approach consists in directly translating speech into target text using a single model @xcite . this direct solution has interesting advantages @xcite ): i) it can better exploit audio information (e.g. prosody) during the translation phase, ii) it has lower latency, and iii) it is not affected by error propagation. the authors contributed equally. thanks to these advantages, the initially huge performance gap with cascade systems has gradually closed @xcite , motivating research towards further improvements. direct st models are fed with features extracted from the audio with high frequency (usually every 10ms). this, on average, makes the resulting input sequence of vectors ∼10 times longer than the corresponding text, leading to an intrinsically redundant (i.e. long and repetitive) representation. for this reason, it is not possible to process speech data with a vanilla transformer encoder @xcite , whose self-attention layers have quadratic memory complexity with respect to the input length. state-of-the-art architectures tackle the problem by collapsing adjacent vectors in a fixed way, i.e. by mapping a predefined number of vectors (usually 4) into a single one, either using strided convolutional layers @xcite @xcite or by stacking them @xcite . as a positive side effect, these length reduction solutions lower input redundancy. as a negative side effect, they disregard the variability over time of the amount of linguistic and phonetic information in audio signals (e.g. due to pauses and speaking rate variations) by giving equal weight to all features. in doing this, relevant features are penalized and considered equally important to the irrelevant ones, resulting in an information loss. recently, @xcite obtained considerable translation quality gains by collapsing consecutive vectors with the same phonetic content instead of compressing them in a fixed way. @xcite also showed that selecting a small percentage (∼16%) of input time steps based on their informativeness improves st quality. on the downside, these approaches respectively require adding a model that performs phoneme classification and a pre-trained adaptive feature selection layer on top of an asr encoder, losing the compactness of direct solutions at the risk of error propagation. in direct st, @xcite and @xcite addressed the problem with a transcript/phoneme-based compression leveraging connectionist temporal classification (ctc - @xcite . however, since these methods are applied to the representation encoded by transformer layers, the initial content-unaware downsampling of the input is still required for memory reasons, at the risk of losing important information. to avoid initial fixed compression, we propose speechformer: the first transformer-based architecture that processes full audio content maintaining the original dimensions of the input sequence. inspired by recent work on reducing the memory complexity of the attention mechanism @xcite , we introduce a novel attention layer -the convattention -whose memory requirements are reduced by means of convolutional layers. as the benefits of avoiding the initial lossy compression might be outweighed by the increased redundancy of the encoded audio features, we aggregate the high-level representation of the input sequence in a linguistically informed way, as in @xcite . in other words, we collapse vectors representing the same linguistic atomic content (words, sub-words, pauses) into a single element, since they express the same linguistic information. the usage of the convattention and of the linguistically motivated compression produces a considerably shorter, yet informative, sequence that fits the memory requirements of vanilla transformer encoder layers. experiments on three language directions (@xmath0.8 bleu points on the standard must-c corpus and obtains significantly larger gains (up to 4.0 bleu) in a low resource setting where the amount of training data is reduced to 100 hours. ## model in this section, we first introduce a novel attention layer that enables to process raw audio features without downsampling ( §2.1). then, we present an architecture that leverages this attention mechanism in the first encoder layers and reduces the redundancy of the more informative but longer resulting sequences with ctc compression ( §2.2). ## experimental settings we experimented on three languages of must-c @xcite : english-german, english-spanish, and english-dutch. to ensure the reproducibility of our work, all training details are provided in the appendix and the code -based on fairseq @xcite -is released open source. 1 following @xcite , we share the convolution parameters of the convattention layers both among k and v and among the attention heads. we select the compression factor and the 1d convolution kernel size with a set of preliminary experiments on the en-de validation set. the compression factor (χ) is chosen among 4, 8, and 16, since 4 is the minimum value that avoids out-ofmemory issues. the kernel size is set either equal to or twice as the value of χ. ## results we compare our proposed model to a strong baseline represented by a transformer-based model with initial fixed sub-sampling @xcite and its baseline+compression variant that includes the average ctc compression strategy, as per @xcite . we choose to also develop the second baseline to make the comparison with speechformer fair since they both use the ctc compression strategy. ## conclusion in the wake of previous works showing the benefits of a content-informed compression over fixed downsampling of the audio features, we proposed speechformer: the first st transformer-based model able to encode the whole raw audio features without any sub-optimal initial subsampling typical of current state-of-the-art models. our solution is made possible by the introduction of a modified attention mechanism -the convattention -that reduces the memory complexity to o(( n χ ) 2 ). as the plain application of convattention layers leads to redundant sequences, high-level hidden states are compressed with a ctc-based strategy to obtain a compact, yet informative representation that can be processed by vanilla transformer encoder layers. experiments on three language pairs show that speechformer significantly outperforms a stateof-the-art st model by 0.5-0.8 bleu, reaching a peak of 4 bleu points in a low resource scenario.