Paper: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis (ICLR-2024)

Reviewer comment (Writing):
The proposed method is not presented very clearly, specifically regarding why μ is chosen for clustering.

Author response:
Since the posterior fit on Gaussian distribution, the mean value represents the highest probability point of the feature distribution. Therefore, using the mean value(μ) rather than using other points is a good representative point for expressing the feature distribution.
We described our method for clustering and constructing the codebook in Section 3.2.
In our work, the codebook consists of centroids of 512 clusters, and k-means++ is used to cluster.

Reviewer comment (Experiments):
As mentioned in W.1, many previous zero-shot TTS methods also represent reference speech to a set of vectors and fuse them with text via attention. Though I can understand the proposed system and comparison system all use VITS as the backbone for fair comparison, I think it would be better to compare your speaker modeling method with the others for a better understanding of the superiority, since the ELF is the main claimed contribution.

Author response:
Although it can be valuable to make comparisons in the manner mentioned by the reviewer, the lack of sufficient implementation details to reproduce the performance claimed by previous works makes such comparisons difficult. We believe that our comparison with state-of-the-art works using the same backbone has sufficiently shown the effectiveness of our method. Additionally, to the best of our knowledge, our work represents a significant advance in that it is the first to show superior performance over the multi-speaker model trained on the target speakers dataset in a non-training approach.

Reviewer comment (Novelty):
The connection between the method itself and earlier works are not very clear. The author should have a short review of the related literature.

Author response:
**W1, W2 and Q1** \
Our method aims to express the overall speech characteristics of a target speaker rather than only the speech characteristics presented in a short reference audio. Examples of it are available in the second section of our demo page (newly added in the middle of the review period). \
In order to enable the TTS model to synthesize speech according to the overall speech characteristics of an unseen speaker and a given content, it is necessary to show the speaker's overall speech characteristics to the TTS model in training and to design the TTS model to combine the given speech characteristics according to the given content. It cannot be resolved with previous speaker encoders that take short reference audio as input and the TTS models [1,2,3] inputted with the output of the speaker encoders. \
To address it, our method consists of two stages: 1) encoding speakers' speech characteristics and storing the encoded features with discretization; 2) conditioning them to synthesize a target speaker's speech with a given content. \
Considering that the goal of our work is to synthesize a target speaker's speech similarly, the representation of speech characteristics for this should be sufficiently informative to reconstruct the target speaker's speech completely. An autoencoder serves the above purpose adequately by converting the input into a latent representation and reconstructing it back to the original input. We have summarized and supplemented this in the manuscript.

Reviewer comment (Writing):
The methods acquired themselves are sometimes a bit abrupt and lacks motivation.

Author response:
**W1, W2 and Q1** \
Our method aims to express the overall speech characteristics of a target speaker rather than only the speech characteristics presented in a short reference audio. Examples of it are available in the second section of our demo page (newly added in the middle of the review period). \
In order to enable the TTS model to synthesize speech according to the overall speech characteristics of an unseen speaker and a given content, it is necessary to show the speaker's overall speech characteristics to the TTS model in training and to design the TTS model to combine the given speech characteristics according to the given content. It cannot be resolved with previous speaker encoders that take short reference audio as input and the TTS models [1,2,3] inputted with the output of the speaker encoders. \
To address it, our method consists of two stages: 1) encoding speakers' speech characteristics and storing the encoded features with discretization; 2) conditioning them to synthesize a target speaker's speech with a given content. \
Considering that the goal of our work is to synthesize a target speaker's speech similarly, the representation of speech characteristics for this should be sufficiently informative to reconstruct the target speaker's speech completely. An autoencoder serves the above purpose adequately by converting the input into a latent representation and reconstructing it back to the original input. We have summarized and supplemented this in the manuscript.

Reviewer comment (Evaluation):
The prototype of the model does not show very notable improvements in metrics other than MOS scores for audio. Speaker blending should be applied to reach better performance.

Author response:
**W3** \
As shown in Table 1, our method is superior to the trained multi-speaker model VITS in the SMOS evaluation. It shows a more significant SMOS difference than the previous zero-shot approach model YourTTS. These differences are significantly larger than the differences in MOS evaluation results. To the best of our knowledge, our work is the first that shows superior results to the best-performing multi-speaker model trained on a target speaker's dataset with a non-training method. To enable intuitive confirmation in the table, we marked "Seen" for models trained with a target speaker's data and "Unseen" for models that were not trained on the dataset.

Reviewer comment (Experiments):
The study also lacks comparison with other fixed/earlier speech encoders or speaker encoders.

Author response:
**W4** \
We compared our method with YourTTS, which uses a different speaker encoder. This work has been cited as a crucial comparative work in recent works [4,5,6] and was recently recognized as state-of-the-art. We increase the fairness of comparison by using the same backbone. Although comparisons with a wider range of other works would also be valuable, we believe that we have sufficiently demonstrated the effectiveness and contribution of our method by showing its superiority over the model VITS trained with a target speaker's dataset.

Reviewer comment (Novelty):
Possibly because of the first point in weaknesses, in Section 2.2, I do not see strong motivation of acquiring autoencoders, despite there are multiple speaker encoding methods available (e.g. speaker encoders; speech factorization methods). Could you please clarify the motivation?

Author response:
**W1, W2 and Q1** \
Our method aims to express the overall speech characteristics of a target speaker rather than only the speech characteristics presented in a short reference audio. Examples of it are available in the second section of our demo page (newly added in the middle of the review period). \
In order to enable the TTS model to synthesize speech according to the overall speech characteristics of an unseen speaker and a given content, it is necessary to show the speaker's overall speech characteristics to the TTS model in training and to design the TTS model to combine the given speech characteristics according to the given content. It cannot be resolved with previous speaker encoders that take short reference audio as input and the TTS models [1,2,3] inputted with the output of the speaker encoders. \
To address it, our method consists of two stages: 1) encoding speakers' speech characteristics and storing the encoded features with discretization; 2) conditioning them to synthesize a target speaker's speech with a given content. \
Considering that the goal of our work is to synthesize a target speaker's speech similarly, the representation of speech characteristics for this should be sufficiently informative to reconstruct the target speaker's speech completely. An autoencoder serves the above purpose adequately by converting the input into a latent representation and reconstructing it back to the original input. We have summarized and supplemented this in the manuscript.

Reviewer comment (Evaluation):
Why speaker blending is needed to reach better CER than the ground truth? And is there any alternative to compensate?

Author response:
**Q2** \
Since the ASR model is trained with data from numerous speakers, it learns pronunciation and speech characteristics commonly observed across many speakers at a higher frequency. If the speaker space we trained is fitted according to the numerous speakers' real distribution, blending multiple speakers is equivalent to approaching speech characteristics with a higher probability of occurrence. Therefore, theoretically, it is natural that blending more speakers results in better CER. It is also evidence that the speaker space we trained represents the real distribution of speakers well.

Reviewer comment (Theory):
Final SFEN objective (Eqs. 2-3) comes out of thin air, it would be good to somehow try to explain theoretically how this can be derived, noting that VAE loss is derived from trying to model log p(x).

Author response:
Our work presents a method combining a variational autoencoder(VAE) and adversarial learning. It significantly improves the reconstruction performance of SFEN. Therefore, the objective we present consists of terms for optimizing generative adversarial networks (GAN) and VAE. In the previous work [1, 2], the authors proved that these objectives are to model a real data distribution. Other widely cited works [3, 4] showed that modeling a real data distribution from an unknown input distribution is possible with the same theoretical background as the first work [1]. Thus, it is also possible to model the real data distribution by applying the adversarial learning method with the output distribution of the SFEN encoder. In conclusion, both the VAE and adversarial learning terms are aligned to model a real data distribution. Additionally, we follow LSGAN [5] as the objective of adversarial learning, which presents a solid theoretical background that allows GAN to model real data distribution better.

Reviewer comment (Experiments):
Continuing the above critizism, elements in the loss are by necessity weighted somehow. How you decide the proper weighting?

Author response:
We followed the coefficient 45 for l1 reconstruction loss presented in the previous work [4] and did not use weight for other terms. We agree that details were omitted from the manuscript and have revised it.

Reviewer comment (Evaluation):
Plotting cosine scores as objective speajer recognition is ok, but proper speaker recognition results are needed. As in cosine score you only compare against targer speaker and totally miss the confusion with the non-target speaker. So presenting EER and minDCF values with the accomanying DET plots are necessary.

Author response:
We believe that the SMOS and SECS results have sufficiently demonstrated the effectiveness of our method. Also, subjective evaluation still carries more weight than objective evaluation in evaluating synthesized speech. The speaker vector plot we presented in the appendix is auxiliary data that demonstrates the superiority of our method. Referring to the related previous works [6,7,8,9,10], we can confirm that the metrics mentioned by the reviewer are unnecessary. In addition, as described in the manuscript, 21 test speakers are in our experimental settings. Measuring speaker verification performance with such few speakers does not provide valid evidence.

Reviewer comment (Writing):
Paper neeeds more thorough language editing but this can be performed in rebuttal stage. Abstract was badly written, but some other parts of the paper are quite ok.

Author response:
We have made minor corrections to the abstract. We will thoroughly review it again to enhance readability for readers.

Reviewer comment (Experiments):
I see that MT was used to obtain MOS and SMOS scores. How did you clean MT results as those can be sometimes extremely noisy. MT workers sometimes use scripting to speed up the work and so those those workers should be removed from the results.

Author response:
We agree that removing the mentioned noisy data is important when utilizing the crowd-sourced MOS method via Amazon Mturk. We removed noisy data by adding multiple evaluation samples that were not related to the comparisons, considering workers who gave clearly incorrect evaluations as random workers, and excluding all evaluation results from those workers.

Reviewer comment (Evaluation):
How did you measure CI for MOS and SMOS?

Author response:
The mean and CI were calculated by aggregating all scores for each comparative method.
We followed the commonly used method of measuring 95% confidence intervals in statistics. Detailed explanations are in the materials below.
http://www.stat.yale.edu/Courses/1997-98/101/confint.htm

Reviewer comment (Experiments):
Were all samples finally voceded using the same voceder before objective speaker recognition? If not, then you can easily add (inaudible) vocoder artifacts that your TDNN model can then pick up. Vocode all samples, even ground truth ones, using the same vocoder.

Author response:
Our method and the comparative methods do not use vocoders. Entire modules in the models are jointly trained; therefore, there are no separately operating vocoders.

Reviewer comment (Evaluation):
Why in Table 1, proposed CER is lower in #20 than in all audio?

Author response:
The difference between the two CER results is minimal. This degree of difference can be caused due to the randomness of model training. We see that they are almost identical results, and it is difficult to state that the robustness of the two methods is different.