O _ O - VC : Synthetic Data-Driven One-to-One Alignment for Any-to-Any Voice Conversion (2025)

Abstract: Traditional voice conversion (VC) methods typically attempt to separate speaker identity and linguistic information into distinct representations, which are then combined to reconstruct the audio. However, effectively disentangling these factors remains challenging, often leading to information loss during training. In this paper, we propose a new approach that leverages synthetic speech data generated by a high-quality, pretrained multispeaker text-to-speech (TTS) model. Specifically, synthetic data pairs that share the same linguistic content but differ in speaker identity are used as input-output pairs to train the voice conversion model. This enables the model to learn a direct mapping between source and target voices, effectively capturing speaker-specific characteristics while preserving linguistic content. Additionally, we introduce a flexible training strategy for any-to-any voice conversion that generalizes well to unseen speakers and new languages, enhancing adaptability and performance in zero-shot scenarios. Our experiments show that our proposed method achieves a 16.35% relative reduction in word error rate and a 5.91% improvement in speaker cosine similarity, outperforming several state-of-the-art methods. Voice conversion samples can be accessed at: https://oovc-emnlp-2025.github.io/

## introduction voice conversion specifically aims to transform a source speaker's voice to match a target speaker while preserving the original linguistic content. this is typically done by disentangling speech into content and speaker identity representations, which are combined during training to reconstruct the audio. at inference time, the source content is paired with a target speaker embedding to generate the converted speech. ## literature review the goal of voice conversion (vc) is to transform the voice of a source speaker into that of a target speaker while preserving the original linguistic content. achieving this requires an effective decomposition of speech signals into distinct components such as linguistic content, speaker timbre, and prosodic characteristics. early vc systems were typically trained as speech-to-speech models on parallel datasets, where multiple speakers uttered the same sentences, defining the task as a sequence-to-sequence problem @xcite . recent vc approaches have focused on reconstructing speech using disentangled representations of linguistic content and speaker identity. ## methodology in text-free vc systems, content and speaker identity are often not fully disentangled. as a result, speaker information can leak into the content representation, which undermines the system's ability to perform clean speaker conversion. this leakage reduces the system's generalization to unseen voices or speaking styles and often results in converted speech that retains characteristics of the source speaker. in text-based vc, asr-derived content representations are highly sensitive to transcription errors, mispronunciations, and noisy labels, which can compromise their reliability and degrade the quality of converted speech @xcite @xcite . some approaches attempt to leverage knowledge transfer from hidden representations of text encoders in multispeaker tts models @xcite . however, mapping audio representations directly to these text-based features is a difficult task. in addition, this process typically requires an explicit alignment mechanism between speech and text, which introduces further complexity. as an alternative solution, synthetic data offers several advantages for voice conversion. when both source and target audio are generated from the same linguistic content, it provides a clean and direct supervisory signal. this shared content allows precise frame-level alignment between source and target audio, enabling more stable and finegrained learning of the conversion function. unlike traditional approaches that rely on symbolic representations (e.g., phonemes or characters), synthetic data eliminates the need for such intermediates and avoids issues like label noise commonly found in real-data training. furthermore, with controlled or predefined durations, speaker-independent features are inherently aligned across domains, removing the need for forced alignment algorithms used in previous work @xcite . unlike methods that map audio to hidden text representations from multispeaker tts models, our approach directly maps source audio to target audio, simplifying the learning process. this enables one-to-one frame alignment, allowing the model to focus more effectively on speaker trans-formation while preserving linguistic content. finally, synthetic data enables the creation of diverse speaker pairs with uniform content, supporting the learning of generalizable speaker conversion mappings. motivated by these advantages, this work is the first to propose using synthetic data as a training paradigm for voice conversion models. ## conclusion we presented a robust voice conversion framework based on synthetic data and a two-phase training strategy. our method enhances speaker similarity, speech quality, and content consistency, particularly in zero-shot scenarios with unseen target speakers. experiments and ablation studies confirm the effectiveness of our approach and demonstrate its ability to prevent speaker information leakage from the source audio. additionally, we showed that the model generalizes well to unseen languages without requiring labeled data, making it highly suitable for low-resource settings.