Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation (2024)

Abstract: In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST).Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S) generator to the speech-to-unit translation model. However, these systems are vulnerable to the presence of noise in input speech, which is an assumption in real-world translation scenarios. To address this limitation, we propose a U2S generator that incorporates a distillation with no label (DINO) self-supervised training strategy into it’s pretraining process.Because the proposed method captures noise-agnostic expressivity representation, it can generate qualified speech even in noisy environment.Objective and subjective evaluation results verified that the proposed method significantly improved the performance of the expressive S2ST system in noisy environments while maintaining competitive performance in clean environments.

## introduction speech-to-speech translation (s2st), which translates speech in one language into speech in other language is indispensable technique for breaking down language barriers in naturalistic communication among international communities @xcite @xcite . recently, the direct speech-to-unit translation (s2ut) approach that translates source speech into a discretized semantic unit of target speech has been gaining a lot of attention @xcite @xcite . thanks to their ability in modeling semantic units using a single network and the success of large-scale pre-training and data augmentation @xcite , their latest models can achieve state-of-the-art translation quality @xcite . on the other hand, it is also important to preserve source speech's expressivity features foot_1 such as vocal style, emotion, or tone during translation process to realize natural conversation with speech translator. however, it is non-trivial for a direct s2ut system to preserve the expressivity due to its design. specifically, since the target discrete units contain linguistic information besides expressivity information @xcite , the expressivity of source speech is hard to be captured by s2ut model. as a results, the translated speech provides monotonic and robotic sound where the source speech's expressivity doesn't exist. to achieve an expressivity-preserved s2st system, several studies propose to cascade an additional unit-to-speech (u2s) generator, also known as a textless acoustic model, on top of the s2ut model. this approach proposes the u2s model by replacing the input of the acoustic model used in tts systems from phonemes to discrete units. for instance, recently proposed pretssel-based s2st system @xcite generates speech by receiving target language's discrete units and source language speech's expressivity embedding. the s2st system with pretssel can achieve high-quality cross-lingual expressivity transfer and content translation performance because it effectively disentangles the linguistic and paralinguistic information using discrete units and expressivity embedding, respectively. however, this expressive s2st framework exhibit issues when applied in the real-world translation scenarios, where the recording environment is noisy. note that the expressivity encoder embedding space is trained to represent all information except linguistic one; that means a channel informa-tion such as background noise also exists in expressivity embedding along with vocal style, emotion, or tone information. consequently, when the input speech is recorded in the noisy environment, the u2s model tries to transfer background noise to output speech, which critically affects both contents and expressivity preservation performances. to address aforementioned problem, we propose a textless acoustic model that utilizes the selfdistillation with no label (dino) strategy to its pretraining @xcite . following the success of self-supervised speaker representation learning @xcite , the proposed method introduces two teacher-student encoders and optimizes these encoders using a self-distillation training strategy. specifically, the student encoder is updated to minimize its output probabilistic distance to the teacher encoder's output. then, the teacher encoder weights are iteratively updated by the exponential moving averaged (ema) weights of the student encoder. in addition, random noise augmentation is applied to the input of both expressivity encoders to learn noise-agnostic expressivity representations. we applied the proposed training strategy to the pretssel u2s generator, which we refer to as dino-pretssel. experimental results verified that the expressive s2st system with dino-pretssel outperformed conventional s2st models in noisy recording environments while still achieving compatitive performance in clean recording environments. specifically, the objective evaluation demonstrated that dino-pretssel achieved more noise-robust content and prosody preservation performance than other systems based on their asr-bleu @xcite and @xcite scores. additionally, the subjective evaluation confirmed its superior performance in generating natural speech sound with robust vocal style preservation compared to conventional systems through the mean opinion score (mos) and speaker-mos (s-mos) tests. ## background: expressive s2st with pretssel pretssel is a unit-based textless acoustic model for expressive s2st system (seamless communication, 2023). specifically, pretssel is pretrained to reconstruct 80-dimensional mel-spectrogram with 10-ms interval of input speech from the deduplicated (or reduced) xls-r units @xcite with 10k k-means clustering and the same mel-spectrograms. ## objective evaluation 6.1 metrics in the objective evaluation, we measured asr-bleu, autopcp, and snr to measure contents preservation, prosody preservation, and noise suppression performances, respectively. we also included s2t-bleu scores to specify the upperbound of asr-bleu score. we averaged the scores of individual utterances to obtain single representative score. we detailed the evaluation metrics in appendix e. ## expressive s2st in-the-wild to test the robustness of dino-pretssel in realworld noisy recording environments, we conducted a subjective evaluation using noisy samples of the voxlingua107 dataset @xcite , which were collected from en videos on youtube. when choosing source speech, we first measured snr, and randomly selected 100 noisy speeches having snr values from 5db to 15db. we then applied silero voice activity detection @xcite to remove leading and trailing silence. then, we conducted mos and s-mos tests for the @xmath0. the evaluation results verified the proposed dino-pretssel presented significantly higher performance to the noisy samples, especially compared to both pretssel and denoiser-based prets-sel. specifically, it achieved 3.59 mos and 3.11 s-mos results, which were 0.58 and 0.48 scores higher than those of original pretssel, respectively. the expressivity encoder of proposed dino-pretssel can effectively extract consistent vocal style and prosody information from source speech even in noisy environment. to show this, we extracted expressivity embeddings from clean and noisy speeches, and drew the t-distributed stochastic neighbor embedding (t-sne) plots (van der @xcite . in particular, we chose confused, happy and sad speech samples from expresso en benchmarking data (seamless communication, 2023), which showed clear differences in their speaking style. to simulate noisy environment, we randomly obtained noise from dns-5 dataset @xcite , and mixed those with the speeches by 10db snr. when drawing the t-sne plot, we marked clusters by following {speaker id, style id} labels. as illustrated in figure 3 , both expressivity embeddings from pretssel and dino-pretssel could be distinguishable by following the speaker and style labels when the source speech was clean. however, as illustrated in figure 4 , the prets-sel lost its ability to distinguish speaker and style, whereas the dino-pretssel still preserved the clusters when the input speech was corrupted by noise. this verified the robustness of proposed dino-pretssel's expressivity encoder in the noisy environment. ## conclusion in this paper, we proposed dino-pretssel, which incorporated dino training strategy into the pretssel-based u2s generator for the noiserobust s2st system. as our method employed strong self-distillation method in learning expressivity representation, the u2s generator could robustly transfer source speech's expressivity at s2st in noisy recording environment. the objective and subjective evaluations conducted on noisy source speech consistently verified that the proposed dino-pretssel outperformed other systems through its high asr-bleu, autopcp, mos, and s-mos results. we also verified that dino-pretssel has robustness in real-world noisy speech. future works include improving the robustness of prosody unity2 by utilzing this robust expressivity embedding. additionally, we plan to explore additional data augmentation methods such as reverberation for the training of dino-pretssel.