Duplex Diffusion Models Improve Speech-to-Speech Translation (2023)

Abstract: Speech-to-speech translation is a typical sequence-to-sequence learning task that naturally has two directions. How to effectively leverage bidirectional supervision signals to produce high-fidelity audio for both directions? Existing approaches either train two separate models or a multitask-learned model with low efficiency and inferior performance. In this paper, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer, so that either end can simultaneously input and output a distinct language’s speech. Our model enables reversible speech translation by simply flipping the input and output ends. Experiments show that our model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of state-of-the-art baselines.

## introduction direct speech-to-speech translation (s2st) @xcite , transforming a source language's speech to the target language's speech, is essential for online international communications and is friendly to numerous languages that do not have their own writing systems or textual vocabularies. s2st circumvents a cascaded architecture @xcite @xcite of combining automatic speech recognition (asr) of the source speech, textual source-to-target machine translation (mt), and target text-to-speech (tts) synthesis where multiple types of datasets are required, error propagates, latency is high, and unavailable for thousands of (spoken) languages who do not have a writing system. for s2st, speech-to-speech parallel data is required, and it is costly to collect a comparable size dataset with textual counterparts. to alleviate the data scarcity problem, self-supervised pre-training and data augmentation techniques were used by @xcite , and unsupervised and weaklysupervised speech and text data under translatotron 2 @xcite were leveraged by @xcite . techniques such as multi-task learning @xcite , pseudo labeling @xcite , and knowledge distillation @xcite have also been adapted and achieved promising results. from s2st architecture's point of view, @xcite describes four categories, (1) translatotron @xcite style which includes a speech encoder and a spectrogram decoder, (2) translatotron2+ @xcite style which inserts a first-pass text decoder followed by a tts encoder between the two modules of translatotron, (3) speech-to-unit translation (s2ut) @xcite that uses discrete clustered units of the target language speech instead of spectrogram, and (4) unity @xcite that inserts a firstpass text decoder followed by a text-to-unit (t2u) encoder between the two modules in s2ut. in this paper, following the motivations of textual duplex machine translation @xcite , we leverage s2st's two directions: effectively utilizing supervision signals from both directions is estimated to both relieve the pain of data scarcity and bring novel architectures of training and inferencing. existing architectures (e.g., translatotron1/2, s2ut, and unity) either train two separate models or a multitask-learned model with low efficiency and inferior performance. in contrast, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex conformer, so that either end can simultaneously input and output a distinct language's speech. our model enables reversible speech translation by simply flipping the input and output ends. experiments show that our model achieves the first success of reversible speech translation with significant improvements of asr-bleu scores compared with a list of strong baselines. our contributions are concluded as follows: the architecture of reder is a stack of l reversible duplex transformer layers where the 1-st to l/2-th layers are mirror of the (l/2+1)-th to l-th layers to ensure the whole model being symmetric. in particular, each layer contains a multi-head selfattention (mhsa) module and a feed-forward network (ffn) module with a novel reversible design to ensure duplex behavior, where the input and output tensors of such a layer are split into two halves, the reverse form f -1 l can be computed by subtracting the residuals: here, noise ϵ t ∼ n (0, i) shares the same shape with x 0 and x t . in order to reconstruct from a gaussian noise input x t ∼ n (0, i), we need to learn a model p θ to approximate the conditional probabilities to run the reverse diffusion (reconstruction) process: note that the reverse conditional probability is tractable by first applying bayes' rule to three gaussian distributions and then completing the "quadratic component" in the exp(•) function: here, variance βt is a scalar and mean μt depends on x t and noise ϵ t : intuitively, q(x t-1 |x t , x 0 ) acts as a reference to learn p θ (x t-1 |x t ). we can use the variational lower bound (vlb) to optimize the negative loglikelihood: we further reparameterize the gaussian noise term instead to predict ϵ t from time step t's input x t and use a simplified objective that ignores the weighting term: ## duplex diffusion model cycle consistency has been utilized in textual neural machine translation @xcite and image-to-image translation @xcite . in this paper, we propose a duplex diffusion model that alternatively optimizes both directions by two diffusion processes. the training algorithm is described in algorithm 1. generally, we borrow ddpm @xcite , 2020)'s architecture and extend it to a duplex scenario where sequences of two ends are diffused alternatively during training. at the beginning, the source sequence x and target sequence y are encoded into dense representations by pretrained wav2vec models e x , e y through self-supervised learning on monolingual datasets, respectively. then, time t and two normal gaussian noise signals ϵ x , ϵ y are sampled. note that the lengths of the source and target sequences are diverse. we pre-define two variance schedules {β t,@xmath0, 1)} t @xmath1,@xmath2, 1)} t @xmath3, for the source and target languages, respectively. thus, we have α t,@xmath4,x , ᾱt,@xmath5,x , α t,@xmath6,y and ᾱt,@xmath7,x , as used in algorithm 1. the variance schedules, initial sequence representations and normal gaussian noises work together to give us diffused representations, x t and y t , respectively. they are then sent to the reversible duplex conformer architecture m θ (figure 3 ) to predict the noises. originally in figure 3 , we are intended to produce x 0 from y 0 in the reverse process of ← --m θ . now, we have two additional inputs, t and x t . the output also changes from predicting x 0 to estimating ϵ θ since the lengths of the source and target sequences are diverse, we follow textual duplex translation @xcite and double the source end's length by a upsampling convolutional network. we only describe the reverse process ← --m θ and the forward process --→ m θ shares the similar strategies. to achieve a full cycle consistency, predicting the target gaussian noise from source sequence by taking target noisy sequence as the "conditional memory" is more appropriate in current scenario setting so that both translation directions are achieved in one duplex diffusion model. after the reverse and forward processes, we can compute the mse losses of between the two pairs of reference and predicted noises. they are interpolated together by hyper-parameter weights λ 1 (=0.5) and λ 2 (=0.5) to the final loss l ddm to be optimized. ## training our reversible duplex conformer is largely inspired by reder @xcite . the novel parts are that (1) we select and reconstruct convolutionenhanced conformer @xcite to synthetically capture global information by attentions and local context by convolutions, and (2) we extend from textual duplex machine translation to (dense) duplex speech-to-speech translation. when training our reversible duplex conformer, we borrow and adapt the losses that are used in reder to fit our scenario. in reder, three types of losses were used. the first loss is to model the variable-length of source and target sequences by a latent alignment approach, i.e., the connectionist temporal classification (ctc) @xcite . starting from the conditional independence assumption, ctc is capable of efficiently (by dynamic programming) finding all valid (yet monotonic) alignments a which derives from the target y by allowing consecutive repetitions and inserting blank tokens. the ctc loss is defined by: we adapt this loss for speech translation when the target are clustered unit sequences. we use the mse loss instead when the target is a sequence of mel-spectrogram. also, to ensure the source sequence is always longer than the target sequence, we upsample the source sequences by convolutional layers before sending them to the reversible duplex conformer. the second loss measures the layer-wise forward-backward agreement (fba, measured by cosine similarity) of between the forward l-th layer's representation -→ h @xmath8 when predicting mel-spectrograms, the final loss function is: we reuse the default hyper-parameter values described in reder @xcite for setting weights w 1 to w 6 . in our experiments, we first train the reversible duplex conformer architecture by a predefined k 1 (=200,000) iterations and then apply the duplex diffusion training algorithm as shown in algorithm 1. after another predefined k 2 (=200,000) iterations, we fix the diffusion processes and focus on updating the reversible duplex conformer only so that traditional search algorithms such as beam search can be used for seeking target hypotheses. ## conclusion aiming at effectively leveraging bidirectional supervision signals of speech-to-speech translation (s2st), we have proposed two models for duplex s2st, a reversible duplex conformer and a duplex diffusion model. we compare with cascaded s2st models, single/multi-pass speech-tospectrogram/unit models and report significantly better or comparable asr-bleu and humanevaluated scores, with fewer training time and faster inference speed.