Enhancing Polyglot Voices by Leveraging Cross-Lingual Fine-Tuning in Any-to-One Voice Conversion (2024)

Abstract: The creation of artificial polyglot voices remains a challenging task, despite considerable progress in recent years. This paper investigates self-supervised learning for voice conversion to create native-sounding polyglot voices. We introduce a novel cross-lingual any-to-one voice conversion system that is able to preserve the source accent without the need for multilingual data from the target speaker. In addition, we show a novel cross-lingual fine-tuning strategy that further improves the accent and reduces the training data requirements. Objective and subjective evaluations with English, Spanish, French and Mandarin Chinese confirm that our approach improves on state-of-the-art methods, enhancing the speech intelligibility and overall quality of the converted speech, especially in cross-lingual scenarios. Audio samples are available at: https://giuseppe-ruggiero.github.io/a2o-vc-demo/

## introduction in recent years, advancements in speech synthesis @xcite @xcite have significantly improved the quality of artificial polyglot voices. several noteworthy studies in cross-lingual voice conversion (xvc) merit attention, especially some any-to-one (a2o vc) foot_1 variants @xcite . the aim of xvc is to convert source voices into target voices without changing the linguistic content, even when source and target speak different languages @xcite . from an information perspective, vc can be accomplished by initially disentangling the source speech into representations describing the linguistic content and speaker characteristics and then synthesizing the converted speech from the extracted linguistic content while incorporating the identity of the target speaker @xcite . to disentangle content and speaker information, an asr model trained on a labeled dataset is frequently employed to extract the supervised spoken content representation, such as text @xcite or phonetic posteriorgram (ppg) @xcite . however, despite its promising results in vc, this approach encounters several limitations. first, the assembly of labeled datasets can be prohibitively expensive, especially in low-resource settings like the xvc scenario @xcite . second, the system's sensitivity to asr errors can lead to potential mispronunciations or inaccuracies in conversion. finally, text and ppg do not capture all linguistic nuances such as accent, expressions and style resulting in neutral-sounding synthesized speech @xcite . in several recent works @xcite @xcite , self-supervised learning (ssl) is used to train models for speech synthesis. this approach is primarily driven by the following advantages offered by ssl: recently, a variety of more versatile systems have emerged. one such example is knn-vc @xcite , an any-to-any vc (a2a vc) approach that extracts frame-wise wavlm representations for both source and target speech. it then converts by directly replacing each frame of the source representation with its nearest neighbors in the target representation set. this strategy allows ssl features to be used without any further disentanglement step. we propose an a2o vc system similar to that of @xcite except it learns to disentangle content information without the need for supplementary steps like discretization, and it can both enhance cross-lingual generation and reduce the training data requisites. our main contributions are as follows: the system consists of three components: a content encoder, an acoustic model, and a vocoder (figure 1 ). the content encoder extracts speech representations from an audio of any speaker, the acoustic model translates these representations into a mel spectrogram of the target speaker, and the vocoder converts the resulting mel spectogram into a time-domain waveform. for both training and inference, only audio data is required. ## experiments to evaluate the effectiveness of our vc approach, experiments across a range of different tasks and datasets were conducted. all the experiments were run on a linux machine with a single nvidia titan rtx gpu with 24 gb of ram. the proposed system was tested as two variants: proposed, our vc method using the conventional training strategy, and proposed-f, the alternative system employing the cross-lingual finetuning strategy. we selected two baseline models for comparison: soft-vc foot_4 @xcite and knn-vc foot_5 @xcite . both excel in cross-lingual voice conversion tasks, demonstrating strong content intelligibility, accent preservation, and speech quality. ## conclusion with this work we have developed an a2o vc system capable of generating human-level nativesounding polyglot voices without relying on multilingual data from the target speaker. we showed how several content and speaker disentanglement strategies applied to ssl features do not perform well in cross-lingual scenarios. to overcome this, we have demonstrated that by directly leveraging ssl features and performing cross-lingual finetuning, we are able to enhance the intelligibility and accent of the converted voices. furthermore, we have illustrated that our models can preserve source emotions throughout the conversion process. our experiments showcased improvements over strong baseline methods across all the examined metrics. moreover, mos tests conducted with multiple native participants revealed that our outcomes closely approximate human recordings in terms of overall quality. for future research, our aim is to explore the creation of high-quality polyglot voices in an a2a setup to further reduce data requirements.