U nit Y : Two-pass Direct Speech-to-speech Translation with Discrete Units (2023)

Abstract: Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, UnitY, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.

## introduction automatic speech translation to another language is an indispensable technology for international communications, with the spread of social media and virtual communications nowadays. a traditional approach of speech-to-speech translation (s2st) is to cascade automatic speech recognition (asr), machine translation (mt), and textto-speech (tts) components, each of which is optimized separately on different data @xcite @xcite . with the emergence of sequence-to-sequence models @xcite @xcite , however, it is getting prevailing to adopt a direct approach foot_0 . this approach consists in translating input speech into the other language based on a single architecture with fewer components than the cascaded systems @xcite @xcite . the direct approach is attractive for building a lowlatency system with a simplified pipeline, thus reducing developing costs. however, direct s2st models suffer from poor performance due to data scarcity, similar to direct speech-to-text translation (s2tt) models @xcite . in the field of s2tt, data shortage has been addressed by leveraging pre-training @xcite @xcite , multi-task learning @xcite , pseudo labeling @xcite , knowledge distillation @xcite . consequently, the translation quality of direct s2tt models is approaching that of cascaded s2tt models @xcite . these techniques have also shown the effectiveness for direct s2st models and led to a decent performance @xcite @xcite . recent works @xcite propose to model discrete acoustic units, extracted from hu-bert @xcite , instead of a continuous speech signal that enables usage of a standard crossentropy loss during training. this speech-to-unit translation (s2ut) model significantly shortens the target sequence length and thus makes training and inference more efficient. the discrete units are directly converted to the waveform with a unit-based neural vocoder @xcite bypassing spectrogram representation. on the other hand, trans-latotron2 @xcite decomposes the target representations into linguistic and acoustic counterparts explicitly. the former predicts a phoneme @xcite defines it as a model that directly predicts the target spectrogram. in this paper, we use a more general definition that the entire architecture is optimized jointly and the translation is conducted in a more direct way. we do not include a vocoder in the training pipeline of all direct models. sequence first, and the latter synthesizes the target spectrogram conditioned on the continuous representation of the linguistic part. this paper presents a novel two-pass direct s2st architecture, dubbed unity, which takes the best of both worlds of the s2ut model and translatotron2. unlike translatotron2, unity models linguistic sequences using subwords (first pass) instead of phonemes, and it models speech as a discrete sequence of acoustic units (second pass). to achieve better translation quality and decoding efficiency, unity consists of a deep text decoder and a shallow unit decoder and enables better generalization to the first-pass decoder. we further introduce a textto-unit (t2u) encoder between the two decoders to bridge the gap between textual and acoustic representations. following the success of large-scale pre-training, we leverage unlabeled text effectively to pre-train the first pass text decoder with multilingual bart (mbart) @xcite at the subword level. extensive experiments show the superiority of the unity s2st system measured by both translation quality and runtime efficiency. first, unity achieves 4.2, 3.7, and 2.5 asr-bleu improvements over the s2ut model on the fisher @xmath0, cvss-c @xcite , and multi-domain en↔es @xcite corpora, respectively. the improvement holds even with high-resource data and pre-training. in addition, our proposed design improves trans-latotron2 as well, indicating its versatility for twopass direct s2st architectures regardless of the choice of the target. second, unity achieves 2.83× and 2.51× decoding speed-ups over the s2ut and improved translatotron2 models, respectively. a combination of the aforementioned improvements suggests the unity design as a starting point for further improvements in direct s2st. 2 ## unity in this section, we propose unity, a two-pass direct s2st model that generates subwords and discrete acoustic units subsequently. hereafter, we refer to discrete acoustic units as discrete units for brevity. let x denote a source speech input, and @xmath1, . . . , y m ) and @xmath2, . . . , u l ) be the corresponding reference text translation and discrete unit sequences in the target language, respectively. note that there is no duration information ## experimental results in this section, we present the experimental results on three corpora. we study various modeling choices from the perspective of target representation (spectrogram v.s. discrete unit) and decoder architectures (single pass v.s. two pass) in supervised and semi-supervised settings. we also benchmark the decoding efficiency of direct s2st models. ## analysis in this section, we conduct analyses to shed light on the source of improvements in unity. we also study whether the same techniques used for unity are helpful for s2spect2. we use the multi-domain @xmath4, but pseudo-labeled asr data is excluded for quick exploration, resulting in 196hour source speech. we report average dev scores over three runs with different random seeds. foot_6 ## conclusion we proposed unity, a novel efficient two-pass direct s2st model that subsequently generates both text and discrete unit outputs. we improved the model performance by predicting subwords in the first pass, bridging decoder representations by an additional encoder, deep-shallow two-pass decoders, regularizing the training with r-drop, and pre-training the first-pass decoder with text-based mbart. experimental evaluations demonstrated that unity outperformed a single-pass s2ut model consistently in translation quality and inference speed. we showed that the proposed methods improve the two-pass direct speech-to-spectrogram model as well, confirming their versatility. still, unity achieved 2.51× decoding speed-up over the case.