Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021)

Abstract: Pretraining and multitask learning are widely used to improve the speech translation performance. In this study, we are interested in training a speech translation model along with an auxiliary text translation task. We conduct a detailed analysis to understand the impact of the auxiliary task on the primary task within the multitask learning framework. Our analysis confirms that multitask learning tends to generate similar decoder representations from different modalities and preserve more information from the pretrained text translation modules. We observe minimal negative transfer effect between the two tasks and sharing more parameters is helpful to transfer knowledge from the text task to the speech task. The analysis also reveals that the modality representation difference at the top decoder layers is still not negligible, and those layers are critical for the translation quality. Inspired by these findings, we propose three methods to improve translation quality. First, a parameter sharing and initialization strategy is proposed to enhance information sharing between the tasks. Second, a novel attention-based regularization is proposed for the encoders and pulls the representations from different modalities closer. Third, an online knowledge distillation is proposed to enhance the knowledge transfer from the text to the speech task. Our experiments show that the proposed approach improves translation performance by more than 2 BLEU over a strong baseline and achieves state-of-the-art results on the MuST-C English-German, English-French and English-Spanish language pairs.

## introduction end-to-end methods have achieved significant progress in speech to text translation (st) and even surpassed the traditional pipeline-based methods in some applications @xcite . however, the success of endto-end methods relies on large amounts of training data, which is quite expensive to obtain and relatively small in practice. building st systems from pretrained models with multitask learning (mtl) is widely used to overcome the limited training data issue @xcite @xcite @xcite . nevertheless, little prior work has been devoted to understanding the interactions between different tasks. @xcite conduct an empirical study on computer vision tasks for mtl. they find many "assumptions" for mtl may not be held for specific applications. for example, "similar" tasks do not necessarily train better together. in this study, we focus on training the st model along with an auxiliary text to text machine translation (mt) task. we are interested in the task interactions with different modalities and in improving the primary st task with the help from the auxiliary mt task. the model is initialized with pretrained modules from automatic speech recognition (asr) and mt. two types of analysis are conducted on the fine-tuned multitask learned models. the first focuses on the model variation by comparing fine-tuned models with pretrained models for different tasks. the second aims to measure internal representation differences due to different modalities. the analysis leads to three main findings. first, the analysis confirms that mtl tends to generate similar model representations for different input modalities and preserves more information from the pretrained mt modules. second, we do not observe significant negative transfer effect from the mt task to the corresponding st task. sharing more parameters is helpful to transfer knowledge to the primary st task. finally, the top layers in the st decoder are more critical to the translation performance and they are also more sensitive to the modality difference. the model representations from different modalities demonstrate larger difference for the top layers in our analysis. inspired by these findings, we propose three techniques to enhance the performance of the primary st task. first, we propose to maximize parameter sharing between the st and mt tasks, i.e. the entire decoder and the top encoder layers. those shared parameters are initialized with the corresponding mt models. second, a cross-attentive regularization is introduced for the encoders. it minimizes the l2 distance between two reconstructed encoder output sequences and encourages the encoder outputs from different modalities to be closer to each other. finally, an online knowledge distillation learning is introduced for mtl in order to enhance knowledge transfer from the mt to the st task. our contributions are summarized as follows: ## related work multitask learning aims to improve generalization by leveraging domain-specific information contained in the training signals of related tasks @xcite . compared with single task, mtl has many advantages, such as the potential to improve performance by sharing complementary information or acting as a regularizer. many previous works focus on learning a good model for all tasks. chen et al. ( 2018 ) study the gradients from different tasks and conduct task dependent gradient normalization to encourage different tasks to learn at similar speed. maninis et al. (2019); liu et al. (2019a); pfeiffer et al. (2020) introduce task-dependent components to enhance individual task performance. @xcite explore different multitask training strategies for st, and they find the oneto-many strategy, in which an encoder is shared between the st and asr tasks, is more effective. anastasopoulos and chiang (2018) further extend it to a triangle structure by concatenating asr and st models. @xcite compare different multitask strategies for the st task, and they confirm many-to-one strategy, in which mt and st are trained together and the decoder is shared between two tasks, is effective if extra bitext data is used. in this work, we carefully study the relation between co-trained tasks in the many-to-one strategy, and the analysis results guide us to propose three techniques to learn more from the auxiliary mt task and enhance the st performance further. @xcite propose criticality analysis to measure the importance of different modules from the trained model. parameters in the selected module or layer are partially rolled back to the initial values, and the module criticality or importance is measured by the performance drop after modification. larger performance drops indicate a more critical module. inspired by their work, we extend it to the analysis on the jointly trained models with different pretrained modules and schemes. @xcite ; @xcite propose to employ canonical correlation to measure the similarity between different models given the same input. we extend their work to study a model with inputs from different modalities. ## methods the proposed st system is co-trained with the mt task as depicted in figure 1 . the modules in the primary st task are connected with dark gray lines and the auxiliary mt task is illustrated with light gray lines. the parameters in the blue modules are shared between the two tasks. during inference with speech input, only modules related to the st task are used. the model has two encoders, a text encoder and a speech encoder, to take text and speech input respectively. the decoder is shared between the two tasks. to encourage knowledge sharing between the two tasks, the top encoder layers are also shared. the parameters of the shared modules are initialized with a pretrained mt model. a novel crossattentive regularization is proposed to reduce the distance between encoder outputs from different input modalities. we also introduce a novel online knowledge distillation method where the output from the auxiliary mt task is used to guide the st model training. the cross-attentive regularization and online knowledge distillation are illustrated as orange modules in figure 1 and the details are presented in the following two subsections. ## experimental setup experiments are conducted on three must-c (gangi et al., 2019a) language pairs: en-de, en-es and en-fr. the models are developed and analyzed on the dev set and the final results are reported on the tst-common set. we use wmt parallel data from different years, 2013 for spanish, 2014 for german, and 2016 for french, as extra text training corpus for mtl. case-sensitive detokenized bleu is reported by sacrebleu with default options @xcite . we use the "t-md" configuration from @xcite in all experiments. the speech encoder has 12 transformer layers while the decoder is with 6 transformer layers. for the mtl model, the text encoder has 6 transformer layers. the transformer layer has an input embedding size of 512 and middle layer dimension 2048. we share parameters of all 6 text encoder transformer layers with the top 6 transformer layers in the speech encoder, hence both encoders use the same modules to generate the encoder outputs. the adam optimizer (kingma and ba, 2014) with a learning rate 0.002 is employed in the experiments. label smoothing and dropout rate are both set to 0.1. we choose α = 0.8 and λ = 0.02 in equation 6 through grid search ([0.1, 1.0] for α and [0.01, 0.05] for λ). input speech is represented as 80d log melfilterbank coefficients computed every 10ms with a 25ms window. global channel mean and variance normalization is applied. the specaugment @xcite data augmentation with the lb policy is applied in all experiments. the input text tokens are converted into their corresponding pronunciation form as phoneme sequences @xcite . the grapheme to phoneme conversion is done through the "g2p en" python package @xcite . the leading phoneme in a word is appended with an extra " " to mark word boundaries. in total, the vocabulary size for the input phonemes is 134. the target vocabulary consists of 10k "unigram" subword units learned by sentencepiece @xcite with full character coverage of all training text data. all st or jointly trained models are initialized with pretrained asr and mt modules. the asr model is trained on the same english speech training data from must-c with the "t-md" configuration too. the pretrained mt models are trained for each language pair with the aforementioned wmt data. the mt encoder and decoder configurations are the same as the text encoder and decoder in the mtl model mentioned above. the models are fine-tuned to 100 epochs using 8 v100 gpus for approximate one day. the batch size is 10,000 frames for speech to text translation samples and 10,000 tokens for parallel text samples per gpu. the model parameters are updated every 4 batches. speech training samples and text input samples are used to update the model alternatively. the models are trained with fairseq @xcite . the last 10 checkpoints are averaged for inference with beam size 5. 1 . ## conclusions in this study, we focus on understanding the interactions between the st and mt tasks under the mtl framework, and on boosting the performance of the primary st model with the auxiliary mt task. two types of analysis on model variation and modality variation, are conducted on the mtl models. the analysis demonstrates mtl helps to preserve information from the mt task and generates similar model representations for different modalities. we observe a minimal negative transfer effect between the two tasks. sharing more parameters can further boost the information transfer from the mt task to the st model. the analysis also reveals that the model representation difference due to modality difference is nontrivial, especially for the top decoder layers, which are critical for the translation performance. inspired by the findings, we propose three techniques to increase knowledge transfer from the mt task to the st task. these techniques include parameter sharing and initialization strategy to improve the information sharing between tasks, car and online kd to encourage the st system to learn more from the auxiliary mt task and then generate similar model representations from different modalities. our results show that the proposed methods improve translation performance and achieve state-of-the-art results on three must-c language pairs.