End-to-End Speech-Translation with Knowledge Distillation: FBK @ IWSLT 2020 (2020)

Abstract: This paper describes FBK’s participation in the IWSLT 2020 offline speech translation (ST) task. The task evaluates systems’ ability to translate English TED talks audio into German texts. The test talks are provided in two versions: one contains the data already segmented with automatic tools and the other is the raw data without any segmentation. Participants can decide whether to work on custom segmentation or not. We used the provided segmentation. Our system is an end-to-end model based on an adaptation of the Transformer for speech data. Its training process is the main focus of this paper and it is based on: i) transfer learning (ASR pretraining and knowledge distillation), ii) data augmentation (SpecAugment, time stretch and synthetic data), iii)combining synthetic and real data marked as different domains, and iv) multi-task learning using the CTC loss. Finally, after the training with word-level knowledge distillation is complete, our ST models are fine-tuned using label smoothed cross entropy. Our best model scored 29 BLEU on the MuST-CEn-De test set, which is an excellent result compared to recent papers, and 23.7 BLEU on the same data segmented with VAD, showing the need for researching solutions addressing this specific data condition.

## introduction the offline speech translation task consists in generating the text translation of speech audio recordings into a different language. in particular, the iwslt2020 task @xcite evaluates german translation of english recordings extracted from ted talks. the test dataset is provided to participants both segmented in a sentence-like format using a voice activity detector (vad) and in the original unsegmented form. although custom segmentation of the data can provide drastic improvements in the final scores, in our work we have not addressed it, participating only with the provided segmentation. two main approaches are possible to face the speech translation task. the classic one is the cascade solution, which includes automatic speech recognition (asr) and machine translation (mt) components. the other option is an end-to-end (e2e) solution, which performs st with a single sequence-to-sequence model. both of them are allowed for the iwslt2020 task, but our submission is based on an e2e model. e2e st models gained popularity in the last few years. their rise is due to the lack of error propagation and the reduced latency in generating the output compared to the traditional cascaded approach. despite these appealing properties, they failed so far to reach the same results obtained by cascade systems, as shown also by last year's iwslt campaign @xcite . one reason for this is the limited amount of parallel corpora compared to those used to separately train asr and mt components. moreover, training an e2e st system is more difficult because the task is more complex, since it deals with understanding the content of the input audio and translating it into a different language directly and without recurring to intermediate representations. the above-mentioned observations have led researchers to focus on transferring knowledge from mt and asr systems to improve the st models. a traditional approach consists in pretraining components: the st encoder is initialized with the asr encoder and the st decoder with the mt decoder. the encoder pretraining has indeed proved to be effective @xcite , while the decoder pretraining has not demonstrated to be as effective, unless with the addition of adaptation layers @xcite . a more promising way to transfer knowledge from an mt model is to use the mt as a teacher to distill knowledge for the st training @xcite . this is the approach we explore in the paper. despite its demonstrated effectiveness, asr pretraining has been replaced in some works by multitask learning @xcite . in this case, the model is jointly trained with two (or more) loss functions and usually the model is composed of 3 components: i) a shared encoder, ii) a decoder which generates the transcription, and iii) a decoder which generates the translation. we adopt the slightly different approach introduced by @xcite , which does not introduce an additional decoder but relies on the ctc loss in order to predict the transcriptions @xcite . as this multi-task learning has been proposed for speech recognition and has demonstrated to be useful in that scenario, we also include the ctc loss in asr pretraining. another topic that received considerable attention is data augmentation. many techniques have been proposed: in this work we focus on specaugment @xcite , time stretch and subsequence sampling @xcite . moreover, we used synthetic data generated by automatically translating the asr datasets with our mt model. this process can also be considered as a sequence-level knowledge distillation technique, named sequence kd @xcite . in this paper, we explore different ways to combine synthetic and real data. we also check if the benefits of the techniques mentioned above are orthogonal and joining them leads to better results. our experiments show that: to summarize, our submission is characterized by tagged synthetic data, multi-task with ctc loss on the transcriptions, data augmentation and wordlevel knowledge distillation. ## training data this section describes the data used to build our models. they include: i) mt corpora (english-german sentence pairs), for the model used in knowledge distillation; ii) asr corpora (audio and english transcriptions), for generating a pretrained encoder for the st task; iii) st corpora (audios with corresponding english transcription and german translation), for the training of our st models. for each task, we used all the relevant datasets allowed by the evaluation campaign foot_0 . ## experimental settings for our experiments, we used the described training sets and we picked the best model according to the perplexity on must-c en-de validation set. we evaluated our models on three benchmarks: i) the must-c en-de test set segmented at sentence level, ii) the same test set segmented with a vad @xcite , and iii) the iwslt 2015 test set @xcite . we trained with adam (kingma and ba, 2015) (betas (0.9, 0.98)). unless stated otherwise, the learning rate was set to increase linearly from 3e-4 to 5e-4 in the first 5,000 steps and then decay with an inverse square root policy. for fine-tuning, the learning rate was kept fixed at 1e-4. a 0.1 dropout was applied. each gpu processed mini-batches containing up to 12k tokens or 8 samples and updates were performed every 8 mini-batches. as we had 8 gpus, the actual batch size was about 512. in the case of multi-domain training, a batch for each domain was processed before an update: since we have three domains, the overall batch size was about 1,536. moreover, the datasets in the different domains had different sizes, so the smaller ones were oversampled to match the size of the largest. as the truncation of the output values of the teacher model to the top 8 leads to a more peaked distribution, we checked if contrasting this bias is beneficial or not. hence, we tuned the value of the temperature at generation time in the interval 0.8-1.5. the temperature t is a parameter which is used to divide the logits before the sof tmax and determines whether to output a softer (if @xmath0. by default t is 1, returning an unmodified distribution. the generation of the results reported in this paper was performed using @xmath1.3 for the models trained on word kd. this usually provided a 0.1-0.5 bleu increase on our benchmarks compared to @xmath2, confirming our hypothesis that a compensation of the bias towards a sharper distribution is useful. instead, the t was set to 1 during the generation with models trained with label smoothed cross entropy, as in this case a higher (or lower) temperature caused performance losses up to 1 bleu point. all experiments were executed on a single machine with 8 tesla k80 with 11gb ram. our implementation is built on top of fairseq @xcite , an open source tool based on pytorch @xcite . ## results the mt model used as teacher for sequence kd and word kd scored 32.09 bleu on the must-c en-de test set. we trained also a smaller mt model to initialize the st decoder with it. moreover, we trained two asr models. one without the multitask ctc loss and one with it. they scored respectively 14.67 and 10.21 wer. all the st systems having ctc loss were initialized with the latter, while the others were initialized with the former. ## conclusions we described fbk's participation in iwslt2020 offline speech translation evaluation campaign @xcite . our work focused on the integration of transfer learning, data augmentation, multi-task training and the training scheme used to combine real and synthetic data. based on the results of our experiments, our submission is characterized by a multi-domain training scheme, with additional ctc loss on the transcriptions and wordlevel knowledge distillation, followed by a finetuning on label smoothed cross entropy. overall, the paper demonstrates that the combination of the above-mentioned techniques can improve the performance of end-to-end st models so that they can be competitive with cascaded solutions. moreover, it shows that i) tagged synthetic data leads to more robust models than a pretraining on synthetic data followed by a fine-tuning on datasets with ground-truth targets and ii) finetuning on label smoothed cross entropy after a training with knowledge distillation brings significant improvements. the huge gap (5.5 bleu) between data segmented in sentences and data segmented with vad highlights the need of custom solutions for the latter. in light of these considerations, our future research will focus on techniques to improve the results when the audio segmentation is challenging for st models.