Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders (2021)

Abstract: Encoder pre-training is promising in end-to-end Speech Translation (ST), given the fact that speech-to-translation data is scarce. But ST encoders are not simple instances of Automatic Speech Recognition (ASR) or Machine Translation (MT) encoders. For example, we find that ASR encoders lack the global context representation, which is necessary for translation, whereas MT encoders are not designed to deal with long but locally attentive acoustic sequences. In this work, we propose a Stacked Acoustic-and-Textual Encoding (SATE) method for speech translation. Our encoder begins with processing the acoustic sequence as usual, but later behaves more like an MT encoder for a global representation of the input sequence. In this way, it is straightforward to incorporate the pre-trained models into the system. Also, we develop an adaptor module to alleviate the representation inconsistency between the pre-trained ASR encoder and MT encoder, and develop a multi-teacher knowledge distillation method to preserve the pre-training knowledge. Experimental results on the LibriSpeech En-Fr and MuST-C En-De ST tasks show that our method achieves state-of-the-art BLEU scores of 18.3 and 25.2. To our knowledge, we are the first to develop an end-to-end ST system that achieves comparable or even better BLEU performance than the cascaded ST counterpart when large-scale ASR and MT data is available.

## introduction end-to-end speech translation (e2e st) has become popular recently for its ability to free designers from cascading different systems and shorten setting model bleu restricted cascaded 23.3 e2e+pre-training 23.1 unrestricted cascaded 28.1 e2e+pre-training 25.6 the pipeline of translation @xcite @xcite . promising results on small-scale tasks are generally favorable. however, speech-to-translation paired data is scarce. researchers typically use pre-trained automatic speech recognition (asr) and machine translation (mt) models to boost st systems @xcite . for example, one can initialize the st encoder using a large-scale asr model @xcite . but we note that, despite significant development effort, our end-to-end st system with pre-trained models was not able to outperform the cascaded st counterpart when the asr and mt data size was orders of magnitude larger than that of st (see in this paper, we explore reasons why pretraining has been challenging in st, and how pretrained asr and mt models might be used together to improve st. we find that the st encoder plays both roles of acoustic encoding and textual encoding. this makes it problematic to view an st encoder as either an individual asr encoder or an individual mt encoder. more specifically, there are two problems. the assumption here is that the upstream component is an mt-like encoder, whereas the st encoder actually behaves more like an asr encoder. we address these problems by marrying the world of asr encoding with the world of mt encoding. we propose a stacked acoustic-and-textual encoding (sate) method to cascade the asr encoder and the mt encoder. it first reads and processes the sequence of acoustic features as a usual asr encoder. then an adaptor module passes the acoustic encoding output to an mt encoder with two principles: informative and adaptive. in this way, pre-trained asr and mt encoders can work for what we would originally design them, and the incorporation of pre-trained models into st is more straightforward. in addition, we develop a multi-teacher knowledge distillation method to robustly train the st encoder and preserve the pretrained knowledge during fine-tuning @xcite . we test our method in a transformer-based endto-end st system. experimental results on the lib-rispeech en-fr and must-c en-de speech translation benchmarks show that it achieves the stateof-the-art performance of 18.3 and 25.2 bleu points. under a more challenging setup, where the large-scale asr and mt data is available, sate achieves comparable or even better performance than the cascaded st counterpart. we believe that we are the first to present an end-to-end system that can beat the strong cascaded system in unrestricted speech translation tasks. ## related work speech translation aims at learning models that can predict, given some speech in the source language, the translation into the target language. the earliest of these models were cascaded: they treated st as a pipeline of running an asr system and an mt system sequentially @xcite @xcite . this allows the use of off-the-shelf models, and was (and is) popular in practical st systems. however, these systems were sensitive to the errors introduced by different component systems and the high latency of the long pipeline. as another stream in the st area, end-to-end methods have been promising recently @xcite @xcite . the rise of end-to-end st can be traced back to the success of deep neural models @xcite . but, unlike other well-defined tasks in deep learning, annotated speech-to-translation data is scarce, which prevents well-trained st models. a simple solution to this issue is data augmentation @xcite . this method is model-free but generating large-scale synthetic data is time consuming. as an alternative, researchers used multi-task learning (mtl) to robustly train the st model so that it could benefit from additional guide signals @xcite @xcite @xcite . generally, mtl requires a careful design of the loss functions and more complicated architectures. in a similar way, more recent work pre-trains different components of the st system, and consolidates them into one. for example, one can initialize the encoder with an asr model, and initialize the decoder with the target-language side of an mt model @xcite @xcite . more sophisticated methods include better training and fine-tuning @xcite , the shrink mechanism @xcite , the adversarial regularizer (alinejad and sarkar, 2020), and etc. although pre-trained models have quickly become dominant in many nlp tasks, they are still found to underperform the cascaded model in st. this motivates us to explore the reasons why this happens and methods to solve the problems accordingly. ## why is st encoding difficult? following previous work in end-to-end models @xcite , we envision an encoding-decoding process in which an input sequence is encoded into a representation vector, and the vector is then decoded into an output sequence. in such a scenario, all end-to-end st, asr and mt systems can be viewed as instances of the same architecture. then, components of these systems can be pre-trained and re-used across them. an underlying assumption here is that the st encoder is doing something quite similar to what the mt (or asr) encoder is doing. however, @xcite find that the asr model benefits from a small attention window, which is inconsistent with the mt model @xcite . to verify this, we compare the behavior of st, asr and mt encoders. we choose transformer as the base architecture @xcite and run experiments on the must-c en-de corpus. we report the results on the must-c en-de tst-common test data. for stronger systems, we use connectionist temporal classification (ctc) @xcite as the auxiliary loss on the encoders when we train the asr and st systems @xcite @xcite . the ctc loss forces the encoders to learn alignments between speech and transcription. it is necessary for the state-of-the-art performance @xcite . here we define the localness of a word as the sum of the attention weights to the surrounding words (or features) within a fixed small window foot_0 . the window size is 10% of the sequence length. figure 1(a) shows the localness of the attention weights for different layers of the encoders. we see that the st and asr encoders prefer local attention which indicates a kind of short-distance dependencies in processing acoustics feature sequences. whereas the mt encoder generates a more global distribution of attention weights for word sequences, especially when we stack more layers. this result arises a new question: is local attention sufficient for speech translation? then, we design another experiment to examine if the high localness in attention weights of the asr and st encoders is due to the bias imposed by ctc. in figure 1 (b), we use the ctc loss in the intermediate layer and show the average localness of the layers above or below ctc. the ctc loss demonstrates strong preference for locally attentive models. the upper-level layers act more like an mt encoder, that is, the layers with no ctc loss generates more global distributions. taking this further, figure 1(c ) demonstrates a slightly higher bleu score when we free more upper-level layers from the guide of ctc. meanwhile, the word error rate (wer) increases because only lower parts of the model are learned in a standard manner of asr. now we have some hints: the st encoder is not a simple substitution of the asr encoder or the mt encoder. rather, they are complementary to each other, that is, we need the asr encoder to deal with the acoustic input, and the mt encoder to generate the representation vector that can work better with the decoder. ## the method in speech translation, we want the encoder to represent the input speech to some sort of decoderfriendly representations. we also want the encoder to be "natural" for pre-training. in the following, we describe, stacked acoustics-and-textual encoding (sate), a new st encoding method to meet these requirements, and improvements of it. ## conclusion in this paper, we investigate the difficulty of speech translation and shed light on the reasons why pretraining has been challenging in st. this inspires us to propose a stacked acoustic-and-textual encoding method, which is straightforward to incorporate the pre-trained models into st. we also introduce an adaptor module and a multi-teacher knowledge distillation method for bridging the gap between pre-training and fine-tuning. results on the librispeech and must-c corpora demonstrate the superiority of our method. furthermore, we achieve comparable or even better performance than the cascaded counterpart when large-scale asr and mt data is available.