Paper: SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (ICLR-2024)

Reviewer comment (Evaluation):
The authors should perhaps explain why in Table 1, 2 and 3, the PER is high on the real data but it goes down after processing through the voice conversion module. Is the overall pipeline achieving some sort of speech enhancement too?

Author response:
A. The PER for real data is the PER between the ground-truth transcription and the ASR transciption of real audio. The Real data PER is a reference to judge the accuracy of the ASR model. The PER of different speech synthesis techniques is between the ASR transcription of real audio and ASR transcription of generated audio. Therefore, the discrepancy between Real data PER and generated audio PER is expected.

Reviewer comment (Novelty):
The claim of the framework's efficiency in scaling to other languages through the introduction of SSL features is not unique to the authors' proposed model since similar approaches have been explored elsewhere.

Author response:
A. The main contribution of our paper is using self-synthesized training examples to improve voice conversion models by generating challenging training examples and iteratively improving the model. Besides, while prior work utilizes SSL features for voice conversion, they do not offer pitch and duration control over synthesized speech in a textless manner. Our approach allows duration and pitch control by deriving target duration and pitch in an unsupervised text-free manner during training and modeling the prediction in our synthesizer model. Finally, our approach does not require transcriptions, hence facilitating scaling up to larger datasets and more languages.

Reviewer comment (Novelty):
The concept of data augmentation using self-synthesized examples has been discussed in the literature, particularly in the context of speaker verification [5].

Author response:
A. The concept of generating self-synthesized examples can be applied to analogous problems, where a controllable media synthesis model is trained with imperfectly disentangled representations. There have been prior efforts in using neural network generated examples as data-augmentation in the recognition domain [1][2]. In contrast, we propose a technique to use self-synthesized examples in iteratively improving a voice conversion model and demonstrate its efficacy in improving controllable synthesis models. We further demonstrate the efficacy of our proposed technique by achieving significant improvements across 10 different languages, as evidenced by the comprehensive evaluations provided in Table 3 and detailed further in Appendix A.5. 
We recommend future work in exploring the efficacy of this technique for tasks like image attribute manipulation and domain translation.

[1] A simple framework for contrastive learning of visual representations, ICML, 2020

[2] What makes for good views for contrastive learning?, NeurIPS, 2020

Reviewer comment (Novelty):
The authors should clarify what specific aspects of their framework, beyond the use of SSL features, contribute to this scalability to strengthen the claim of originality in this regard.

Author response:
A. The main contribution of our paper is using self-synthesized training examples to improve voice conversion models by generating challenging training examples and iteratively improving the model. Besides, while prior work utilizes SSL features for voice conversion, they do not offer pitch and duration control over synthesized speech in a textless manner. Our approach allows duration and pitch control by deriving target duration and pitch in an unsupervised text-free manner during training and modeling the prediction in our synthesizer model. Finally, our approach does not require transcriptions, hence facilitating scaling up to larger datasets and more languages.

Reviewer comment (Experiments):
The introduction of the self-synthesized examples as a data augmentation method is intriguing. However, its efficacy and broader applicability remain questions. Can the authors provide experimental evidence or analysis showcasing the generalizability of this data augmentation technique across various methods?

Author response:
A. The concept of generating self-synthesized examples can be applied to analogous problems, where a controllable media synthesis model is trained with imperfectly disentangled representations. There have been prior efforts in using neural network generated examples as data-augmentation in the recognition domain [1][2]. In contrast, we propose a technique to use self-synthesized examples in iteratively improving a voice conversion model and demonstrate its efficacy in improving controllable synthesis models. We further demonstrate the efficacy of our proposed technique by achieving significant improvements across 10 different languages, as evidenced by the comprehensive evaluations provided in Table 3 and detailed further in Appendix A.5. 
We recommend future work in exploring the efficacy of this technique for tasks like image attribute manipulation and domain translation.

[1] A simple framework for contrastive learning of visual representations, ICML, 2020

[2] What makes for good views for contrastive learning?, NeurIPS, 2020