Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcripts (2023)

Abstract: End-to-end automatic speech translation (AST) relies on data that combines audio inputs with text translation outputs. Previous work used existing large parallel corpora of transcriptions and translations in a knowledge distillation (KD) setup to distill a neural machine translation (NMT) into an AST student model. While KD allows using larger pretrained models, the reliance of previous KD approaches on manual audio transcripts in the data pipeline restricts the applicability of this framework to AST. We present an imitation learning approach where a teacher NMT system corrects the errors of an AST student without relying on manual transcripts. We show that the NMT teacher can recover from errors in automatic transcriptions and is able to correct erroneous translations of the AST student, leading to improvements of about 4 BLEU points over the standard AST end-to-end baseline on the English-German CoVoST-2 and MuST-C datasets, respectively. Code and data are publicly available: https://github.com/HubReb/imitkd_ast/releases/tag/v1.1

## introduction the success of data-hungry end-to-end automatic speech translation (ast) depends on large amounts of data that consist of speech inputs and corresponding translations. one way to overcome the data scarcity issue is a knowledge distillation (kd) setup where a neural machine translation (nmt) expert (also called oracle) is distilled into an ast student model @xcite . the focus of our work is the question of whether the requirement of high-quality source language transcripts, as in previous applications of kd to ast, can be relaxed in order to enable a wider applicability of this setup to ast scenarios where no manual source transcripts are available. examples for such scenarios are low-resource settings (e.g., for languages without written form for which mostly only audio-translation data are available), or settings where one of the main uses of source transcripts in ast -pre-training the ast encoder from an automatic speech recognition (asr) system-is replaced by a large-scale pre-trained asr system (which itself is trained on hundreds of thousands hours of speech, but the original training transcripts are not available @xcite ). relaxing the dependence of pre-training ast encoders on manual transcripts has recently been studied by @xcite . our focus is instead to investigate the influence of manual versus synthetic transcripts as input to the student model in an imitation learning (il) approach @xcite , and to lift this scenario to ast. to our knowledge, this has not been attempted before. we present a proof-ofconcept experiment where we train an asr model on a few hundred hours of speech, but discard the manual transcripts in il training, and show that this asr model is sufficient to enable large nmt models to function as error-correcting oracle in an il setup where the ast student model works on synthetic transcripts. focusing on the il scenario, we show that one of the key ingredients to make our framework perform on synthetic asr transcripts is to give the ast student access to the oracle's full probability distribution instead of only the expert's optimal actions. furthermore, when comparing two il algorithms of different powereither correcting the student output in a single step, or repairing outputs till the end of the sequencewe find that, at least in the setup of a referenceagnostic nmt teacher, the single-step correction of student errors is sufficient. one of the general reasons for the success of our setup may be a reduction of data complexity and an increase of variations of outputs, similar to applications of kd in nmt @xcite . to investigate the special case of imitation-based kd on synthetic speech inputs, we provide a manual analysis of the nmt expert's behavior when faced with incorrect synthetic transcripts as input, or when having to correct a weak student's translation in the il setting. we find that the nmt oracle can correct errors even if the source language input lacks semantically correct information, by utilizing its language modeling capability to correct the next-step token. this points to new uses of large pre-trained asr and nmt models (besides initialization of encoder and decoder, respectively) as tools to improve non-cascading end-to-end ast. ## related work imitation learning addresses a deficiency of sequence-to-sequence learning approaches, nicknamed exposure bias @xcite , that manifests as the inference-time inability to recover from own errors, leading to disfluent or hallucinated translations @xcite . il aims to replace the standard learning paradigm of teacher forcing @xcite (which decomposes sequence learning into independent per-step predictions, each conditioned on the golden truth context rather than the context the model would have produced on its own) by enriching the training data with examples of successful recovery from errors. we build upon two previous adaptations of il to nmt @xcite and lift them to ast. knowledge distillation @xcite ) transfers the knowledge encoded in a large model, called teacher, to a far smaller student model by using the teacher to create soft labels and train the student model to minimize the cross-entropy to the teacher. kd has been successfully used for machine translation @xcite , speech recognition (wong and gales, 2016) and speech translation @xcite . synthetic speech translation training datasets have been used previously to train ast models: @xcite used an asr-nmt model cascade to translate unlabeled speech data for augmentation. to obtain more machine translation (mt) training data, @xcite ; @xcite generated synthetic speech data with a textto-speech model. @xcite applied kd between an nmt expert and an ast student with manual transcriptions as expert input to improve ast performance. @xcite improved upon this by increasing the available training data by utilizing a mt model to translate the audio transcripts of asr datasets into another language, yet they still use manual transcripts for distillation in the following finetuning phase. further attempts focused on improving ast models by utilizing mt data for multitask learning with speech and text data @xcite @xcite , such as xstnet @xcite and fat-mlm @xcite . a question orthogonal to ours, concerning the influence of pre-training encoder and/or decoder on source transcripts, has been investigated by @xcite . they achieved competitive results without any pretraining via the introduction of parameterized distance penalty and neural acoustic feature modeling in combination with ctc regularization with translations as labels. their question and solutions are orthogonal to ours and are likely to be yield independent benefits. ## imitation-based knowledge distillation we view an auto-regressive nmt or ast system as a policy π that defines a conditional distribution over a vocabulary of target tokens @xmath0: π(v|@xmath1; x). this policy is instantiated as the output of the softmax layer. when training with teacher-forcing, the cross-entropy (ce) loss ℓ(•) is minimized under the empirical distribution of training data d: l ce (π) = e (y,x)∼d [ t @xmath2, π)]. to perform well at test time we are interested in the expected loss under the learned model distribution: l(π) = e (y,x)∼π [ t @xmath3, π)]. as shown by @xcite , the discrepancy between l and l ce accumulates quadratically with the sequence length t , which in practice could manifest itself as translation errors. they proposed the dagger algorithm which has linear worst-case error accumulation. it, however, relies on the existence of an oracle policy π * that, conditioned on the same input x and the partially generated π's prefix @xmath4, can produce a single next-step correction to @xmath5. @xcite further proposed the aggrevate algorithm which relies on an even more powerful oracle that can produce a full continuation in the task-loss optimal fashion: for nmt, this means continuing the @xmath6, as done for example in hormann and ## experiments we experiment with english-german ast on the covost2 @xcite (430 hours) and the must-c (di gangi et al., 2019) datasets (408 hours) foot_1 . as expert model, we use the transformer from facebook's submission to wmt19 @xcite , which is based on the big transformer architecture proposed by @xcite . trained by teacher forcing on ground truth targets with a label smoothing @xcite factor of 0.1. kd + @xcite denotes word-level knowledge distillation between the expert's and student's full output probability. ikd and ikd + denote imitation knowledge distillation, where student model is corrected by the empirical distribution of the optimal expert actions or the full expert policy @xcite , respectively. synthikd and synthikd + are our variants with synthetic transcripts. we used the same same exponential decay schedule (β = 1 t ) used by @xcite as early experiments showed that this performed best in our setup. all ast models' encoders are initialized with the encoder of the corresponding asr model, trained on the respective datasets with crossentropy and the label-smoothing factor of 0.1. because of the relatively small size of these datasets, our experiments should seen as proof-of-concept, showing that asr models trained on a few hundred hours of audio provide synthetic transcripts of sufficient quality to enable imitation-based kd for ast. the standalone performance of our asr models is listed in ## conclusion we showed that a pretrained nmt model can successfully be used as an oracle for an ast student, without requiring gold source language transcripts as in previous approaches to imitation learning for ast. this widens the applicability of imitation learning approaches to datasets that do not contain manual transcripts or to pre-trained asr models for which training transcripts are not available. our qualitative analysis suggests an explanation of the fact that the nmt oracle is robust against mismatches between manual and synthetic transcripts by its large language model capabilities that allow it to continue the prefix solely based on its learned contextual knowledge.