EZ - VC : Easy Zero-shot Any-to-Any Voice Conversion (2025)

Abstract: Voice Conversion research in recent times has increasingly focused on improving the zero-shot capabilities of existing methods. Despite remarkable advancements, current architectures still tend to struggle in zero-shot cross-lingual settings. They are also often unable to generalize for speakers of unseen languages and accents. In this paper, we adopt a simple yet effective approach that combines discrete speech representations from self-supervised models with a non-autoregressive Diffusion-Transformer based conditional flow matching speech decoder. We show that this architecture allows us to train a voice-conversion model in a purely textless, self-supervised fashion. Our technique works without requiring multiple encoders to disentangle speech features. Our model also manages to excel in zero-shot cross-lingual settings even for unseen languages. We provide our code, model checkpoint and demo samples here: https://github.com/ez-vc/ez-vc

## introduction zero-shot voice conversion (vc) is the task of transforming a source speaker's voice characteristics into that of a target speaker while preserving linguistic content and prosodic attributes, even for speakers unseen during training. over the years with the advancement of modern deep learning techniques and substantial improvements in speech encoders and speech generation systems, numerous and vastly different approaches have been proposed to address this challenge. textless vc architectures have become the primary area of research in this domain since cascaded asr+tts systems are known to lose the nonverbal characteristics of the source speech such as laughs, whispers and other filler sounds. they also lead to cascaded errors. to overcome this, many textless vc systems these days employ either selfsupervised speech encoders (ssl) or neural audio codecs (nac) to extract speaker features or linguistic content before feeding them to a speech generation decoder. these speech representations are also often disentangled to obtain certain composite characteristics such as timbre or style. sometimes quantized speech representations are used which form as the input for a speech generation or language model. speech synthesis systems, which are a key component of vc architectures, have of late greatly benefited from the advancements in diffusion and continuous normalizing flow (cnf) based techniques. voicebox @xcite and its successors that use these methods are able to produce high quality audio outputs that are almost undistinguishable from real speech. these models thus show great promise for zero-shot vc tasks and yet architectures based on these methods remain under-explored. in this work we contribute the following, ## related work early research in vc focused on disentangling speaker and content information. works like yourtts @xcite focused on using speaker embeddings to extract speaker features from target speech but usually required reference text to be provided as well. recent works like sef-vc @xcite now prefer textless, speakerembedding free vc which is also able to perform better. since the advent of ssl speech models like hubert @xcite and wavlm @xcite , vc research has quickly learned to leverage them for their high correlation with both acoustic and linguistic content. knn-vc @xcite works by replacing representations of source speech with the nearest neighbour from the reference speech. vec2wav 2.0 @xcite on the other hand, uses a combination of discrete representations from vq-wav2vec for source content and wavlm features for capturing the timbre of the target speaker. at the same time, another school of approach has emerged that utilizes neural audio encoders and combines them with language models for high quality vc. unfortunately, these systems suffer from slow inference speeds due to their autoregressive nature. diffusion based techniques also have been explored by diffvc @xcite and similar works. these models are able to demonstrate natural and robust outputs. conditional flowmatching (cfm) based speech generation methods have also begun to appear in voice conversion literature. latest works such as adaptvc @xcite , stablevc @xcite , seed-vc @xcite and pflow-vc @xcite employ this technique for their speech decoders and generally couple them with ssl encoders. adaptvc uses speaker and content encoder adapters on top of hubert while stablevc includes three feature extractors for style, linguistic content, and mel-spectrograms. seed-vc on the other hand requires a timber shifter module and speakerembeddings besides a semantic feature extractor. pflow-vc proposes a slightly different approach by using a timbre encoder for target speaker and semantic encoder for source speech. in contrast, with our architecture we wish to eliminate the need for multiple encoders or adapters for voice conversion while still being able to achieve state-of-the-art results for any-to-any vc. ## ez-vc ez-vc is a simple architecture that only requires one pre-trained speech encoder and a trainable speech decoder. unlike most other works, we do not need multiple encoders for disentanglement of speech features. our architecture also benefits from using an off-the-shelf encoder. other than training a simple k-means model, we do not train our speech encoding module. this helps reduce the compute and training time requirements compared to existing methods that usually ask for training both the encoder and decoder modules. figure 1 provides a description of our model's architecture for both training and inference. at the time of training, our model does not require any supervised or labeled data. to prepare our training set, we extract the mel-spectrogram for every speech sample. these are then passed through the speech encoder first and then the resultant speech features from the 14th layer are taken and quantized using a k-means clustering model. the features are extracted at 75% of the model depth consistent with previous work @xcite . we also de-duplicate adjacent discrete units for all samples. the mel-spectrogram and the corresponding discrete units become the input data for our speech decoder training. with this, the model learns to produce a mel-spectrogram from the given discrete speech representation and is also able to condition the output on the provided unmasked speech. during inference, we pass both the source and target speech through our speech encoder system. the mel-spectrogram and discrete units of the target speech form the reference for our cfm model and the discrete units of the source speech form the prompt required to generate the corresponding mel. in practice, the target and source units are concatenated and given as input to the model. the target mel is then discarded upon inference. this generated mel inherits the speaker attributes from the reference target mel while the content and style is obtained from the source units. ## evaluation subjective and objective measures are equally important for evaluating voice conversion systems. in our test we use naturalness mean opinion score (nmos) and similarity mean opinion score (smos) as our subjective evaluations. for objectivity, we utilize speaker similarity (ssim) and utmos @xcite scores for comparing our models. we measure speaker similarity by using cosine similarity scores between our target speech and that of our output speech by using embeddings from a speaker verification model called ecapa-tdnn @xcite . for our baselines, we select few of the most recent and best performing open-source voice conversion models. this makes sure that we evaluate our model against the current state-of-the-art architectures available. we select seedvc, vec2wav 2.0, diff-hiervc @xcite and knn-vc as our baselines. vec2wav and knn-vc use primarily units-to-speech vocoders, while diff-hiervc employs diffusion based methods. seedvc and our work meanwhile uses cfm based speech models. we choose 10 samples for our evaluations. these samples are selected from various languages and accents. we prepare a variety of source and target speech combinations based on gender, interlingual and cross-lingual speech. we also include combinations of seen and unseen languages to test the robustness and generalization capabilities of these models. all audios are resasmpled to 16khz to ensure fair comparison. for our subjective evaluation, we provided these 10 samples to 20 student volunteers for comparison. each volunteer was asked to evaluate each sample based on it's naturalness which evaluates for mainly intelligibility, style preservence, and sound quality of the output speech in comparison to the source speech. in contrast, the similarity mean opinion score judges the similarity of the speaker in the output speech to that of the target speaker. we take the average of all the samples from all the volunteers which becomes the results of our nmos and smos scores. we further objectively compare our model with seed-vc on a seen language (english) and 2 unseen languages (german and spanish). the results, as shown in analyzing the naturalness and similarity mos scores from ## conclusion ez-vc hopes to make a substantial advancement in the field of zero-shot voice conversion, demonstrating that high-quality voice conversion can be achieved with a minimal architecture. our method balances both naturalness and speaker similarity without the need for complex feature disentanglement or multiple encoders. the model's ability to generalize across diverse linguistic settings highlights its robustness in crosslingual contexts. our findings may also suggest that discrete representations capture deeper, more universal representations of speech. our comprehensive evaluations show that ez-vc achieves significantly improved capabilities for zero-shot voice conversion. we hope that our work inspires further efforts to simplify voice conversion techniques.