End-to-End Speech Translation with Adversarial Training (2020)

Abstract: End-to-End speech translation usually leverages audio-to-text parallel data to train an available speech translation model which has shown impressive results on various speech translation tasks. Due to the artificial cost of collecting audio-to-text parallel data, the speech translation is a natural low-resource translation scenario, which greatly hinders its improvement. In this paper, we proposed a new adversarial training method to leverage target monolingual data to relieve the low-resource shortcoming of speech translation. In our method, the existing speech translation model is considered as a Generator to gain a target language output, and another neural Discriminator is used to guide the distinction between outputs of speech translation model and true target monolingual sentences. Experimental results on the CCMT 2019-BSTC dataset speech translation task demonstrate that the proposed methods can significantly improve the performance of the End-to-End speech translation system.

## introduction typically, a traditional speech translation (st) system usually consists of two components: an automatic speech recognition (asr) model and a machine translation (mt) model. firstly, the speech recognition module transcribes the source language speech into the source language utterances @xcite . secondly, the machine translation module translates the source language utterances into the target language utterances @xcite . due to the success of end-to-end approaches in both automatic speech recognition and machine translation, researchers are increasingly interested in end-to-end speech translation. and, it has shown impressive results on various speech translation tasks @xcite @xcite . however, due to the artificial cost of collecting audio-to-text parallel data, speech translation is a natural low-resource translation scenario, which greatly hinders its improvement. actually, the audio-to-text parallel data has only tens to hundreds of hours which are equivalent to about hundreds of thousands of bilingual sentence pairs. thus, it is far from enough for the training of a high-quality speech translation system compare to bilingual parallel data of millions or even tens of millions for training a high-quality text-only nmt. recently, there have some recent works that explore to address this issue. @xcite pre-trained an asr model on high-resource data, and then finetuned the asr model for low-resource scenarios. @xcite and @xcite proposed multi-task learning methods to train the st model with asr, st, and nmt tasks simultaneously. @xcite proposed a knowledge distillation approach which utilizes a text-only mt model to guide the st model because there is a huge performance gap between end-toend st and mt model. despite their success, these approaches still need additional labeled data, such as the source language speech, source language transcript, and target language translation. in this paper, we proposed a new adversarial training method to leverage target monolingual data to relieve the low-resource shortcoming of end-to-end speech translation. the proposed method consists of a generator model and a discriminator model. specifically, the existing speech translation model is considered as a generator to gain a target language output, and another neural discriminator is used to guide the distinction between outputs of speech translation model and true target monolingual sentences. in particular, the generator and the discriminator st model discriminator generator loss speech st loss st output quality score discriminator st output real text discriminator loss low quality score high quality score st model training step discriminator training step figure 1: proposed end-to-end speech translation with adversarial training are trained iteratively to challenge and learn from each other step by step to gain a better speech translation model. experimental results on ccmt 2019-bstc dataset speech translation task demonstrate that the proposed methods can significantly improve the performance of the endto-end speech translation system. ## proposed method the framework for the method of adversarial training consists of a generator and a discriminator. in this paper, generator is the existing end-to-end st model, which is based on the encoder-decoder model with an attention mechanism @xcite . the discriminator is a model based on a convolutional neural network, and the output is a quality score. the discriminator is aiming to get higher quality scores for real text and lower quality scores for the output of the st model in the discriminator training step. in other words, the discriminator is expected to distinguish the input text as much as possible. meanwhile, our method can not only leverage the ground truth to supervise the training of st model，but also make use of the discriminator to enhance the output of the st model by using target monolingual data, as shown in figure 1 . ## conclusion in this paper, we present the adversarial training approach to improve the end-to-end speech translation model. we applied gan to the speech translation task and achieved good results in the experimental results. since gan's structure is used, this method can be applied to any end-toend speech translation model. unlike the multitask, pre-trained, and knowledge distillation previously proposed, this method requires the use of additional parallel corpus, which is very expensive to collect. in the future, we will experiment with unpaired text in order to be able to use this method to utilize an infinite amount of spoken text.