CLAD - ST : Contrastive Learning with Adversarial Data for Robust Speech Translation (2023)

Abstract: The cascaded approach continues to be the most popular choice for speech translation (ST). This approach consists of an automatic speech recognition (ASR) model and a machine translation (MT) model that are used in a pipeline to translate speech in one language to text in another language. MT models are often trained on the well-formed text and therefore lack robustness while translating noisy ASR outputs in the cascaded approach, degrading the overall translation quality significantly. We address this robustness problem in downstream MT models by forcing the MT encoder to bring the representations of a noisy input closer to its clean version in the semantic space. This is achieved by introducing a contrastive learning method that leverages adversarial examples in the form of ASR outputs paired with their corresponding human transcripts to optimize the network parameters. In addition, a curriculum learning strategy is then used to stabilize the training by alternating the standard MT log-likelihood loss and the contrastive losses. Our approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text.

## introduction neural machine translation (nmt) has made significant advancements over the past several years with claims of achieving 'human parity' @xcite , and 'super-human performance' @xcite . however, despite making tremendous progress in quality and coverage @xcite , nmt models have been identified to lack robustness in dealing with noisy inputs @xcite . robustness is especially important in cascaded speech translation (st) systems, where an nmt model works on the output of the upstream automatic speech recognition (asr) system. in this scenario, significant mt performance degradation has human transcript: i'm not sure that's wise, given the importance of the problem, but there's now the geoengineering discussion about: should that be in the back pocket in case things happen faster, or this innovation goes a lot slower than we expect? asr output: i'm not sure that's why it's given the importance of the problem. but now that the geoengineering discussion about should that be in the back pocket in case things happen faster or this innovation goes a lot slower than we expect. prior research has tried to tackle the robustness problem in nmt models independently by (1) synthetic noise injection @xcite , including mimicking asr errors for the st task @xcite @xcite or (2) by adding noise to the continuous-space representations learned by the nmt model @xcite @xcite . some of the synthetic noise injection methods replace words with probable words from asr confusion lattices @xcite but fail to consider the full context and to realistically replicate asr errors, like the phrase-level errors and segmentation errors that asr models often make (see on the other encoder … [cls] x 1 x 2 x 3 x |x| s x e 1 e 2 e 3 e |x| … decoder … y 1 y 2 y 3 y |y| d 1 d 2 d 3 d |y| … softmax + ℒ nmt encoder … [cls] x̅ 1 x̅ 2 x̅ 3 x̅ |x̅ | s x̅ ē 1 ē 2 ē 3 ē |x̅ | … ℒ ctl encoder … [cls] x 1 x 2 x 3 x |x| s x e 1 e 2 e 3 e |x| … asr hypothesis human transcript source text translated text figure 1: proposed model and the two alternating training objectives. l nmt is the standard nmt negative log-likelihood loss and l ctl is the contrastive loss for improving robustness to asr outputs. to address the robustness problem, particularly in the context of cascaded st, we propose to combine the best of the two approaches. this is obtained by training the nmt model with adversarial examples generated from the asr outputs and encouraging the encoder representations of both the asr outputs and their corresponding human transcripts to be closer to each other. this is done via contrastive learning @xcite , in addition to the standard log-likelihood nmt training, and through curriculum learning @xcite by pretraining on log-likelihood loss followed by iteratively alternating between the two objectives. our method leads to significant improvements in the st task on two language pairs (en-de, en-fr) without hurting the performance on mt of well-formed texts. our approach also follows the standard cascade approach that does not rely on any end-to-end st data (speech aligned with their transcription and translation) unlike that of @xcite and @xcite , making it easier to apply to a wider range of language pairs. 2 method our nmt model is a transformer model @xcite with several encoder and decoder layers, with self-attention modules within each block and a cross-attention module to link the encoder and decoder components. training the network parameters θ is done by minimizing the mean negative log-likelihood over a batch of n source-target text translation pairs (x, y ) = (x (1) , y (1) ), . . . , (x (n) , y (n) ). this objective encourages producing a target sentence y with a sequence of tokens y 1 , y 2 , . . . , y |y| given the source sentence x with sequence of tokens to improve the robustness of the nmt model on noisy asr outputs for cascaded speech translation, we use a contrastive learning method @xcite that is aimed to bring the representations of a noisy sentence generated by the upstream asr closer to its clean version (the human transcript for the same audio) in the semantic space modeled by the nmt encoder. to get the encoder sentence representations efficiently for contrastive learning, a [cls] token is prepended to the input sentences similar to the bert model @xcite . we found that using encoder output corresponding to the [cls] token is slightly better than using the mean of encoder outputs as the sentence representation. s x , which also gets added to each of the decoder outputs before softmax (see figure 1 ). contrastive learning uses speech transcription corpora (i.e., speech paired with human transcripts). the speech input is passed through the upstream asr model to obtain the asr outputs. given the noisy asr output x paired with its corresponding clean human transcript x, we minimize the contrastive objective l ctl , which is an average of two symmetric sentence-level contrastive loss functions. d(u, v) denotes the cosine distance between two vectors u and v. ŝx ′ represents a negative example constructed for every other sentence x ′ in the batch. given x ′ ∈ (x ∪ x) \ {x, x} and its sentence embedding s the above linear interpolation in eq 2 with exponentially decaying λ x is implemented following @xcite to increase the hardness of negative examples by not selecting uninformative negative examples as training progresses. l c (x) follows similar formulation as l c (x) with x replaced by x, and vice versa. to stabilize the nmt parameters before applying contrastive learning, we also use curriculum learning @xcite . we first pre-train the model with nmt loss l nmt only for the first n batches, followed by iteratively training alternate batches with l ctl and l nmt losses, respectively, until convergence. with this model and training regime, which alternates between training with translation and speech recognition datasets independently, we are able to build robust nmt models for cascaded st without any requirement for speech translation data. ## experimental setup we experiment with english-german (en-de) and english-french (en-fr) language directions. training data: for parallel text translation data, we use wmt'16 en-de @xcite and wmt'14 en-fr @xcite news translation task data for the corresponding language directions. to get the asr data for contrastive learning, we use open ai whisper base asr @xcite to transcribe the speech in the training sets from mozilla common voice 12.0 @xcite , voxpopuli @xcite , and the english set of multilingual librspeech (pratap et al., 2020) for both language directions. as an additional experiment, we also try adding in-domain must-c training data. the full statistics of the datasets are in appendix a.2 ( ## conclusion we improve the robustness of mt to asr outputs using contrastive learning to bring the representations of clean and noisy examples closer in the semantic space. our approach does not require any speech translation corpus. we significantly improve the translation accuracy on noisy asr outputs without degrading translation accuracy on clean text. we also show that the approach is scalable to better-quality asr models in the cascade other than the one used during training. the proposed approach is generic and is applicable beyond the context of speech translation alone, such as translating user-generated chat text or non-native text if paired noisy-clean data is available.