Consistency is Key: On Data-Efficient Modality Transfer in Speech Translation (2023)

Abstract: End-to-end approaches have shown promising results for speech translation (ST), but they suffer from its data scarcity compared to machine translation (MT). To address this, progressive training has become a common practice, of using external MT data during the fine-tuning phase. Despite of its prevalence and computational overhead, its validity is not extensively corroborated yet. This paper conducts an empirical investigation and finds that progressive training is ineffective. We identify learning-forgetting trade-off as a critical obstacle, then hypothesize and verify that consistency learning (CL) breaks the dilemma of learning-forgetting. The proposed method, which combines knowledge distillation (KD) and CL, outperforms the previous methods on MuST-C dataset even without additional data, and our proposed consistency-informed KD achieves additional improvements against KD+CL. Code and models are availble at https://github.com/hjlee1371/consistency-s2tt.

## introduction while traditional speech-to-text translation (st) systems are built by pipelining automatic speech recognition (asr) and machine translation (mt), end-to-end (e2e) approach recently emerges as a promising direction to ameliorate error propagation and model complexity problems @xcite . however, e2e st models encounter data scarcity due to the need for cross-modal annotations, which are less abundant compared to datasets used in related tasks such as machine translation. our goal is to enable effective cross-modal transfer from machine translation (mt) models, which have ample training data, to st models with limited data. in pursuit of this goal, we investigate the widely used progressive training technique in st @xcite @xcite , where external mt data is continuously integrated during the fine-tuning phase. however, our evaluation brings forth surprising results, as we find that progressive training is inadequate and leads to suboptimal outcomes. this inadequacy is due to the dilemma in adding mt data for st training. using external mt data may incur interference between the two tasks, but having less external mt data leads to forgetting. to break the knot in figure 1 , we first shed light on the overlooked relationship between consistency and forgetting. in addition, we introduce a novel approach called consistency-informed knowledge distillation (ckd). our findings and proposed methods are thoroughly evaluated on the must-c benchmark, encompassing various language pairs. the results demonstrate the superior performance and enhanced data efficiency of our approach compared to previous methods. our contributions are as follows. ## motivation: is progressive training effective? problem statement and baseline to motivate, we define our problem and describe a progressive learning baseline by @xcite , that we use as strong baseline throughout the paper. the speech translation (st) corpus, denoted as d @xmath0, x, y)}, consists of s (source language speech), x (source language text or transcription), and y (target language text or translation). due to the scarcity of speech translation datasets, it is common practice to train st models jointly with mt or asr subtasks @xcite . similarly, in our approach, we train our model jointly on st and mt using multitask cross-entropy losses, denoted as l ce (s, y, θ) and l ce (x, y, θ). in progressive training, the mt training is continued during the fine-tuning phase using external data source d @xmath1, y)}. specifically, at each epoch, d ext is randomly downsampled to d ′ ext , and during training, st triplets (s, x, y) or mt pairs (x, y) are sampled from the union of d st and d ′ ext . in addition to joint training, we incorporate mtto-st online knowledge distillation (kd) proposed by @xcite . for the data triplet (s, x, y), the kd loss is computed as: progressive training is ineffective despite its widespread adoption, we propose that the efficacy of progressive training has been accepted without sufficient empirical evidence and followings highlight our empirical findings against common beliefs. from based on our observation of catastrophic forgetting, it might be expected that cross-modal kd would benefit from progressive training, as augmented mt data provides a more reliable teacher signal. however, the inclusion of an extra kd objective in their main idea is to restrict the gradient of new tasks within the low-curvature "valleys" of the loss landscape of previous task. while traditional ewc approximates the hessian as the diagonals of the fisher information matrix i f ii (θ iθ * i ) 2 , we can remove diagonal approximation from ewc through well-known relation between kl divergence and fisher as while θ * is fixed parameter obtained from prior tasks in original ewc, we can interpret it as current parameter and θ = θ * + ∆θ as the parameter of submodel induced by arbitrary perturbations. it recalls recently proposed cl losses such as r-drop @xcite , and provokes the question that whether the cl framework can ameliorate the intrinsic dilemma of st. we thus adopt r-drop as our cl framework, and weighted loss α cl l cl is added to the final loss. consistency-informed kd if the concept of consistency is indeed crucial for preventing mt forget-ting, as we hypothesized, we can expect substantial improvements by incorporating cross-modal kd compared to not using it. to achieve further enhancements, we can go beyond the basic combination of kd and cl. in particular, we introduce a novel loss called consistency-informed kd (ckd), which leverages more detailed information on consistency, as outlined below. it augments vanilla kd with token-level weighting based on mt consistency matrix c m t . concretely, c m t ij represents the bidirectional kl divergence between two forward pass probabilities for i-th token and j-th vocabulary: p 1 (y @xmath2, x) and p 2 (y @xmath3, x). intuitively, ckd can be understood as ignoring inconsistent mt teacher probabilities at token-level. ## results and analysis consistency learning remedies catastrophic forgetting we begin by examining the hypothesis that cl data-efficiently remedies catastrophic forgetting. as illustrated in figure 2 , cl demonstrates a remarkable ability to retain mt knowledge even in the absence of additional mt data, thereby confirming our hypothesis. the final st bleu scores, presented in cl provides a more reliable teacher for kd thanks to our data-efficient solution for catastrophic forgetting, we can confidently predict larger gains from knowledge distillation (kd), as explained in 3. to empirically demonstrate this, we train our st model using a simple combination of kd and cl. the results in furthermore, we observe that the performance gain from kd is more pronounced when combined with cl compared to kd alone (+0.29 bleu with cl vs. +0.21 bleu without cl). this suggests that the improvements achieved through cl are not limited to intra-modal regularization, but rather have a broader cross-modal impact. thus, we can attribute the enhanced performance to a better mt teacher originating from the non-forgetting effect of cl. additional improvement from ckd and comparison of methods from simple kd+cl is comparable to well-chosen d ext some previous works have observed that introducing the spoken domain d ext , such as open-subtitles @xcite , improves st bleu @xcite . to compare our data-efficient method with more competitive models, we also conducted intensive experiments for en-de using opensubtitles as our new d ext during finetuning. without cl, models trained with opensubtitles achieve higher bleu scores, which aligns with previous works and demonstrates the importance of domain knowledge. however, with cl, training with a well-chosen external mt becomes worthless. considering the difficulties and high computation costs of determining the best d ext , our suggested approach provides a more practical and efficient way of training st models. further detailed analysis of the relationship between cl and forgetting can be found in the appendix e. ## conclusion in this paper, we conduct thorough experiments to reexamine the effectiveness and efficiency of progressive training. we identify the key challenge of balancing st and mt tasks and discover that, when kd and cl are combined with a balance, adding data plays a deterimental role and thus can be omitted for higher data efficiency. our findings lead us to propose ckd, which dynamically utilizes intra-modal consistency for cross-modal kd. our experiments demonstrate the effectiveness of ckd in terms of both performance and data efficiency. we also provide additional analysis to support our findings.