Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units (2023)

Abstract: We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people’s unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.

## introduction imagine hearing a famous catchphrase of your favourite television character spoken in their voice, but uncharacteristically fast, while atypically emphasising the end. this would immediately raise suspicion that something is "wrong". as humans we learn to recognise familiar people and voices, not only by their voice texture (timbre), but also by their typical speaking style @xcite . therefore, a true vc method should convert both voice texture and speaking style (rhythm, f0, etc.). figure 1 describes this visually. traditional vc methods mainly focused on changing the timbre of a given speaker while leaving the speaking style unchanged @xcite @xcite @xcite . recent methods propose to additionally convert speaking style @xcite @xcite . however, these mainly use only a single target utterance which does not fully capture speaker prosody. figure 1 : comparing traditional vc to speaking style conversion. white lines on the spectrograms show the pitch contour. while vc methods change the spectral features so the new utterance sounds like the new speaker, they do not change the rhythm and pitch. conversely, ssc matches the faster speaking style, and the target speaker's tendency to finish sentences with a pitch increase. this a real sample converted by dissc. due to variance within speaking style we do not expect the converted speech to exactly match the target. another line of work uses text transcription supervision and shows the potential benefit to vc @xcite . however, this limits the application to high resource languages and requires large scale data labelling. some recent prosody aware vc methods use spectrogram representations as input and output @xcite , rather than operating in the waveform domain. thus, they involve another phase of converting from the spectral domain to the time domain using a vocoder. moreover, using discrete self-supervised speech representations and generating waveforms from these was demonstrated to provide superior performance on plenty of downstream tasks such as speech and audio language modelling @xcite @xcite , multi-stream processing @xcite , speech emotion conversion @xcite , spoken dialogue @xcite , speech-tospeech translation @xcite , and audio generation @xcite . inspired by this line of work, we propose a simple and effective method for speaking style conversion (ssc) based on discrete self-supervised and partially disentangled speech representations. curating samples of different speakers pronouncing the same utterances is challenging, thus we follow the setup of unpaired data, similarly to @xcite ; @xcite ; @xcite ; @xcite . we give exact details of our approach in section 3. we formalise the setup of ssc and introduce an extensive evaluation suite in section 4. results suggest the proposed approach is greatly superior to the evaluated baselines (see section 5), while utilising a simple and effective method which can be trained on a single gpu within a couple of hours. our contributions: (i) we introduce a novel, simple, and effective method which uses discrete selfsupervised speech representations to perform textless speaking style conversion in the waveform domain; (ii) we formalise the task of speaking style conversion and propose diverse evaluation methods to analyze different speech characteristics; (iii) we empirically demonstrate the efficacy of the proposed approach using extensive empirical evaluation on several benchmarks considering both objective and subjective metrics. ## our approach -dissc we operate under the following setup: we perform any-to-many ssc which means we take any recording and convert it to any of the training set speakers. our training set contains only unpaired utterances -e.g. no training utterance is said by more than one speaker. the learning process is completely textless with no form of text supervision. our approach uses discrete units for speaking style conversion, and is denoted dissc. we decompose speech to few representations to synthesise speech in the target speaking style. we consider three components in the decomposition: phonetic-content, prosodic features (i.e., f0 and duration) and speaker identity, denoted by z c , (z dur , z f 0 ), z spk respectively. we propose a cascaded pipeline: (i) extract z c from the raw wave-form using a ssl model; (ii) predict the prosodic features of the target speaker from z c and z spk ; (iii) synthesise the waveform speech from the content, predicted prosody and target speaker identity. see figure 2 for a visual description of dissc. this modelling approach means moving from a continuous space to a discrete space, hence resulting in easier optimisation and better quality generations. ## ssc evaluation suite in this section we describe our setup which uses several datasets and fine-grained offline metrics for each part of the conversion -timbre, pitch, rhythm. as no standardised framework for evaluating ssc currently exists, we hope the proposed metrics, will help further advance research in this field. a detailed description of training configurations and hyper-parameters can be found in the appendix. ## results we compare dissc to several any-to-many (or many-to-many) baselines. notably, we evaluate speech resynthesis (sr), and autovc @xcite which are well known vc methods. in addition, we compare to autopst and bne-mol-seq2seqvc @xcite which are prominent prosodic-aware vc (i.e., ssc) methods. we use the official github implementations and train the models on our datasets using the authors recommended configuration. for a fair comparison, all baselines were evaluated under the any-to-many setup, i.e., generating only seen speakers. only seq2seqvc supports new speakers, because only that approach was published. however, we evaluate them on known speakers, with the entire training set as reference recordings. we show three variants of our method: dissc_rhythm, dissc_pitch and dissc_both which predict duration, pitch contour and both respectively. objective results for vctk and esd appear in all other baselines show rhythm errors comparable or worse than sr which does not convert rhythm. we can see a minor decrease in content quality (by wer and cer) of dissc compared to sr which does not convert prosody, but still superior to autopst. in addition, seq2seq also has better wer, at the expense of harming the rhythm conversion. potentially, the supervised phoneme recogniser is too fine, thus does not impose enough of a rhythm bottleneck. the harm in quality of dissc may be due to additional information bottlenecks by de-duplicating the speech units, and predicting f0. we additionally provide subjective evaluation considering both naturalness of the converted samples and their similarity to the target recording considering speaking style. these results are shown in figure 4 . we found that the naturalness of both dissc variants was comparable to sr, and not far from the reference recordings while far outperforming autopst. in addition, we see a noticeable increase in speaking style similarity when converting prosody with dissc compared to sr. however, also converting the pitch did not noticeably improve the results. we believe that this has to do with the lack of distinct speaking style which people can notice, especially from a single recording. ablation study. in order to properly evaluate the ability of dissc to learn the pitch speaking style, when such exists, we use the synthetic vctk data (as explained in section 4.1) with known pitch trends. these results ( figure 5 : dissc corrects abnormal rhythm patterns in specific words, compared to autopst. here we take the utterance "people look, but no one ever finds it" and stretch the duration of the word "finds" times three. irregular rhythm. lastly, to demonstrate that dissc predicts rhythm depending on content, and not only on average -we create a sample with abnormal rhythm patterns. we take an utterance and artificially stretch one of the words using speech resynthesis. we then attempt to "convert" the sample to the original speaker thus checking the ability to only change the rhythm of the abnormal part. we compare dissc and autopst in figure 5 , and see that dissc manages to mostly correct the irregular rhythm, while hardly changing the other parts. by contrast, autopst only slightly changes the rhythm. if e c performs optimally, we are guaranteed perfect results, as z c would be identical for irregular and original after dedup. however, hu-bert may introduce some errors specifically when considering signal variations @xcite . the current model suffers from slight content loss when using the deduped units. in addition, repeated utterances get slightly different units (in the abnormal rhythm experiment). therefore, in future work we aim to improve the robustness and disentanglement of the speech representation considering speaker and prosodic information.