On Knowledge Distillation for Translating Erroneous Speech Transcriptions (2021)

Abstract: Recent studies argue that knowledge distillation is promising for speech translation (ST) using end-to-end models. In this work, we investigate the effect of knowledge distillation with a cascade ST using automatic speech recognition (ASR) and machine translation (MT) models. We distill knowledge from a teacher model based on human transcripts to a student model based on erroneous transcriptions. Our experimental results demonstrated that knowledge distillation is beneficial for a cascade ST. Further investigation that combined knowledge distillation and fine-tuning revealed that the combination consistently improved two language pairs: English-Italian and Spanish-English.

## introduction speech translation (st) converts utterances in a source language into text in another language. conventional st systems called cascade or pipeline st consist of two components: automatic speech recognition (asr) and machine translation (mt). in the cascade st, the error propagation from asr to mt seriously degrades the st performance. on the other hand, a new st system called end-to-end or direct st uses a single model to directly translate the source language speech into target language text @xcite . such an end-to-end approach is a new paradigm in st and is attracting much research attention. however, a naive end-toend st without additional training, such as asr tasks, remains inferior to a cascade st @xcite . additionally, it requires parallel data of the source language speech and the target language text, which cannot be obtained easily in practice. recent st studies have incorporated the techniques of cascade st to end-to-end sts. multitask training with an asr subtask has been used successfully in end-to-end st @xcite @xcite . initializing an end-to-end st with a pretrained asr or mt has also become a common approach @xcite @xcite @xcite . in this work, we focus on the cascade approach due to its performance advantage against end-toend sts. another reason is that cascade st models can be incorporated into end-to-end sts, as shown in previous studies. during the training of an mt model for a cascade st, we can use clean human transcripts for the source language speech as input. however, since the mt in a cascade st always receives asr output during inferences, asr errors should be propagated to the mt model to cause translation errors. what if we use erroneous speech transcriptions by asr for training? that approach means the mt model is trained to translate erroneous transcriptions into correct text, which would not generally be appropriate. one possible solution is to use both types of input (clean and erroneous transcriptions) for training, not just one. the question is how to use them. what is the proper training strategy for cascade sts? this is what we want to learn. in this work, we address such problems by applying knowledge distillation to cascade sts. we distill the knowledge of a teacher model based on clean transcriptions to a student model based on erroneous transcriptions. we also investigate the joint use of knowledge distillation and fine-tuning. experimental results revealed that the knowledge distillation improved the robustness against asr errors and that the knowledge distillation after the fine-tuning provided more significant improvement. ## related work some st studies have tackled the problem of asr error propagation. n-best hypotheses @xcite , confusion networks @xcite , and lattices @xcite were used to include asr ambiguity in the st process. @xcite used the weighted sum of embedding vectors for asr word hypotheses based on their posterior probabilities. sperber et al. (2017b) and xue et al. (2020) showed that translation accuracy against erroneous speech transcriptions can be improved by introducing pseudo asr errors in the training data of mt. knowledge distillation (kd) @xcite is a method of transferring knowledge from a teacher to a student model. typically, the student model is trained by minimizing the kl-divergence @xcite loss between the output probability distributions of the teacher and student models (wordlevel kd). sequence-level knowledge distillation (sequence-level kd) @xcite targets the token-sequence generated by the teacher model using beam search. in our experiments, sequence-level kd outperformed word-level one, and @xcite showed similar trends. therefore, in our experiments, we call it kd. the kd technique is prevalent in many applications of machine learning, including mt (non-autoregressive machine translation @xcite , simultaneous translation @xcite , etc.). typically, it is used to distill knowledge from a larger teacher model to a smaller or faster student model. recent works @xcite have shown that the student model's accuracy exceeds that of the teacher model, even if its size is identical as the student model. kd has also been applied to st. @xcite applied kd to an end-to-end st using an mt model based on clean transcriptions as the teacher of the end-toend st model. our work focuses on the application of kd to a cascade st using a teacher model based on clean transcripts for the student model that takes erroneous inputs. @xcite proposed distillation as a remedy for the effective use of noisy parallel data for machine translation. they first trained the teacher model only on high-quality, clean data. then they fed the source-side of the noisy parallel data into the teacher model and trained the student model to translate from the noisy source to the teacher's output. the main difference between their work and ours is that we have loosely equivalent source sentences (clean or erroneous transcription), which can be paired with the same target sentence. therefore, the student model can be trained with more reliable objectives obtained by feeding clean transcriptions to the teacher model. ## cascade st suppose triplet @xmath0, ..., w j ), @xmath1, ..., x k ), and @xmath2, ..., y l ), where w (1 ≤ @xmath3, x(1 ≤ @xmath4, and y (1 ≤ @xmath5, the corresponding transcribed source language tokens, and translated target language tokens. in a cascade st, first the asr model is trained by the w and x pair. then the mt model is trained to translate from x to y . the loss function of mt model l m t is defined using cross entropy: when training an mt model, we can also use x instead of x, which is the output of the asr model. we call the model trained with clean input x mt clean (fig. 1(a) ) and the one trained with asr-based input x mt asr (fig. 1(b) ). ## discussion we analyzed the results with the spanish to english models to discuss how erroneous transcriptions affect translation results and how kd and ft work. ## conclusion we presented and discussed the benefits of using two machine learning techniques in cascade st: knowledge distillation and fine-tuning. our experimental results showed the advantages of the proposed method in two different conditions. our results also suggest that combining knowledge distillation and fine-tuning is more beneficial than using either one because they have different roles. in future work, we will incorporate our findings into an end-to-end st to grow speech translation.