CTC -based Non-autoregressive Textless Speech-to-Speech Translation (2024)

Abstract: Direct speech-to-speech translation (S2ST) has achieved impressive translation quality, but it often faces the challenge of slow decoding due to the considerable length of speech sequences. Recently, some research has turned to non-autoregressive (NAR) models to expedite decoding, yet the translation quality typically lags behind autoregressive (AR) models significantly. In this paper, we investigate the performance of CTC-based NAR models in S2ST, as these models have shown impressive results in machine translation. Experimental results demonstrate that by combining pretraining, knowledge distillation, and advanced NAR training techniques such as glancing training and non-monotonic latent alignments, CTC-based NAR models achieve translation quality comparable to the AR model, while preserving up to 26.81 \times decoding speedup.

## introduction direct speech-to-speech translation (s2st) refers to the process of generating target speech directly from the source speech, without the need for generating intermediate source or target text. distinguished from traditional cascaded s2st @xcite , which involves cascading automatic speech recognition (asr), machine translation (mt), and text-to-speech (tts) models, direct s2st can mitigate error accumulation and may be applicable to low-resource languages without written forms @xcite . early direct s2st models often require text supervision during training @xcite , which is not feasible for numerous unwritten languages. to address this issue, researchers have proposed textless s2st based on discrete units @xcite , which does not rely on any text supervision during training. specifically, they first extract the discrete representation (i.e., discrete units) of the target speech based on the speech pretraining model hubert @xcite . then, they train a speech-to-unit translation (s2ut) model to generate target discrete units from the source speech and subsequently employ a unit-based vocoder to synthesize the target speech. however, due to the considerable length of discrete unit sequences, these models often exhibit higher decoding latency. to address this problem, @xcite propose the use of non-autoregressive (nar) model to generate discrete units in parallel. despite its faster decoding speed, it suffers from a significant decrease in translation quality compared to s2ut limited by the expressive ability of nar models. in this paper, we aim to explore the performance of more powerful nar models in s2st. we opt for models based on connectionist temporal classification (ctc; @xcite , which improves the model's expressive ability by expanding the output space. previous work has demonstrated that ctcbased nar models exhibit excellent performance in machine translation @xcite and speech-to-text translation @xcite . in this paper, we build upon ctc and incorporate multiple techniques including pretraining, knowledge distillation, and advanced nar training techniques such as glancing training and non-monotonic latent alignments. experimental results show that ctc-based models can attain translation quality comparable to the ar model, while maintaining up to 26.81× decoding speedup. ## related work speech-to-speech translation extends speech-totext translation (s2tt) @xcite @xcite that further synthesizes the target speech. ## conclusion in this paper, we propose ctc-s2ut, an nar textless s2st model based on ctc. by incorporating several training techniques including encoder pretraining, knowledge distillation, glancing training, and non-monotonic latent alignment, ctc-s2ut achieves comparable translation quality to the autoregressive s2ut model, while achieving 26.81× decoding speedup. further ablation studies validate the effectiveness of each training technique.