Effectively pretraining a speech translation decoder with Machine Translation data (2020)

Abstract: Directly translating from speech to text using an end-to-end approach is still challenging for many language pairs due to insufficient data. Although pretraining the encoder parameters using the Automatic Speech Recognition (ASR) task improves the results in low resource settings, attempting to use pretrained parameters from the Neural Machine Translation (NMT) task has been largely unsuccessful in previous works. In this paper, we will show that by using an adversarial regularizer, we can bring the encoder representations of the ASR and NMT tasks closer even though they are in different modalities, and how this helps us effectively use a pretrained NMT decoder for speech translation.

## introduction automatic speech translation (ast) aims to directly translate audio signals in the source language into the text words in the target language. for many years, the pipeline of transcribing speech with asr and then translating with the mt component was a standard method to address the speech translation problem. having access to lots of data in many language pairs, the cascaded model for speech translation can benefit from well-trained asr and mt components and generate high-quality translations. in recent years, it has shown that we can remove the transcription step and build an end-to-end model that is strong enough to compete with the cascaded model @xcite . such models not only have lower inference latency, but they also do not suffer from the problem of errors that propagate from one component to the next. however, the scarcity of available resources is the main challenge in this task, and a variety of methods are proposed to address this problem. one of the most effective approaches to increase the performance of ast systems is to pretrain the encoder using an asr model @xcite . while pretraining the encoder by an asr model even in different languages shows promising results @xcite , using a pretrained mt decoder is not beneficial @xcite or slightly improve the result @xcite and even in some cases may worsen the results @xcite . one explanation for this phenomenon is that the decoder works well only if its input comes from an encoder that it was trained with @xcite . to solve the problem of invariant encoder representations, we make use of an adversarial regularizer in our loss function to bring the output of the asr encoder closer to the input of mt decoder. we show that this modification can improve the bleu score by +2.0 bleu points. ## related work the cascaded pipeline of transcribing speech signals and then translating them using an mt component @xcite was for many years the standard design of speech translation systems @xcite . the idea of having an end-to-end structure for this task showed promising results in the works of @xcite @xcite @xcite . after the success of @xcite in creating a powerful model for st systems, more recent studies focused on exploring their power, and one of the main approaches to boost the performance of such models is to make use of available data from other tasks, such as asr and nmt. @xcite @xcite show that multitask learning can be effective and @xcite @xcite investigate various data augmentation techniques. the impact of pretraining the encoder with asr model is also studied in @xcite @xcite . in experiments of @xcite the performance gain of pretraining the decoder with an mt model was marginal. @xcite addresses the asr encoder and mt decoder gap problem by proposing a "transcoder" and use smooth-l1 loss to bring asr hidden representation close to mt encoder hidden representation. the idea of modifying loss function in ast models was also discussed in @xcite . their formulation of the additional loss is different from ours, and they use their additional loss function in a different nmt architecture from ours. the idea of adding adversarial regularizer was discussed in other tasks such as unsupervised mt @xcite or zero-shot translation @xcite . the closest research to our work is @xcite , which uses a similar adversarial network to bring encoder representations closer together. however, they apply their model to the zero-shot machine translation task, with a different architecture. they also apply their regularizer to the representations of the different languages with the same modalities. ## conclusion in this paper, we study the impact of pretraining an ast decoder using an mt model and propose a method to make the pretraining step more effective. we show that we can align the latent representations of different modalities by using adversarial loss and make the asr encoder more compatible with the mt decoder. our experiments demonstrate that we can improve the performance by around 1.5 bleu points on two language pairs compared to conventional pretraining methods.