Paper: NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR-2024)

Reviewer comment (Experiments):
They also conducted an ablation study well. However, it would be better if the authors could add the results according to the dataset and model size. The model size of NaturalSpeech 2 is too complex compared to VITS. In my personal experience, VITS with speaker prompt could achieve significantly better performance than YourTTS.

Author response:
$\textbf{Q1}$: It would be better if the authors could add the results according to the dataset and model size.

$\textbf{A1}$: This is a great suggestion. We conduct serial experiments by varying model sizes and data sizes. The results are shown as follows:
| Model Size | Data Size | CMOS |
| -------------- | ----- | ----- |
| 400M | 44,000 h | 0.00 |
| 400M | 10,000 h | -0.06 |
| 400M | 1,000 h | -0.41 |
| 60M | 1,000 h | -0.52 |

We have the following observations. 
* When we decrease the data size from 44k hours to 1k hours while keeping the model size unchanged, we can find the CMOS drops consistently. It indicates that the data size is important to generate high-quality speech. 
* When we further decrease the model size from 400M to 60M while keeping the data size unchanged, the CMOS drops 0.11, which indicates the model size is important as well.

Reviewer comment (Experiments):
For inference speed, NaturalSpeech 2 still has a low latency for its iterative generation. Although this discussion is included in the Appendix, it would be better if the authors could add the discussion of inference speed in the main text. This is just a limitation of diffusion models so I acknowledge the trade-off between quality and inference speed. Furthermore, I hope to know other metrics of NaturaSpeech 2 according to Diffusion Steps (WER or Similarity metric). Recently, Flow matching using optimal transport is utilized for fast speech synthesis. This could be adopted to this work.

Author response:
$\textbf{Q2}$: Add discussion of inference speed in the main text. Furthermore, I hope to know other metrics of NaturaSpeech 2 according to Diffusion Steps (WER or Similarity metric).  And suggestions for flow matching.

$\textbf{A2}$: Thank you for your valuable suggestion.  Firstly, we incorporate this discussion in the revised version as suggested.

Secondly, for additional metrics, we report SMOS and WER results in the table below. These results are in line with CMOS, further illustrating the trade-off between quality and inference speed.
| Diffusion Step | RTF   | CMOS  | SMOS  | WER  |
| -------------- | ----- | ----- | --------------- | ---- |
| 150            | 0.366 | 0.00  | 4.06 | 2.01 |
| 100            | 0.244 | -0.02 | 4.04 | 2.04 |
| 50             | 0.124 | -0.08 | 3.98 | 2.09 |
| 20             | 0.050 | -0.21 | 3.88 | 2.15 |

Finally, as suggested, we would like to explore efficient strategies such as the consistency model[1] and flow matching model[2] to speed up the diffusion model.

[1] Song, Yang, et al. Consistency models.
 
[2] Lipman, Yaron, et al. Flow matching for generative modeling.

Reviewer comment (Experiments):
This work utilizes a quantized latent vector for latent representation. In my experience, the quality of the model with the continuous latent representation before quantization showed a better performance in latent diffusion model for singing voice synthesis. Have you tried to train your model with the pre- or post-quantized representation for latent representation?

Author response:
$\textbf{Q3}$: This work utilizes a quantized latent vector for latent representation. Have you tried to train your model with the pre- or post-quantized representation for latent representation?

$\textbf{A3}$: Thanks for your careful reading. In the preliminary experiment, we did not observe performance improvements when replacing the post-quantized representation with the pre-quantized representation. 
 
Moreover, using post-quantized representation is driven by engineering considerations. For example, rather than directly storing 256-dimensional FP32 (256 x 32 bits) latent representations, we only needed to store 16 INT16 quantized IDs (16 x 16 bits), leading to a 32x boost in storage efficiency.

Reviewer comment (Reproducibility):
The details of singing voice synthesis are missing. It would be better if you could add the details for pre-processing of musical scores. How do you extract the duration of phonemes in this work?

Author response:
$\textbf{Q4}$:  How do you extract the duration of phonemes in this work?

$\textbf{A4}$: Thanks for your query. We would like to clarify that our singing voice data comes in the form of recorded audio, not musical scores. We use the same alignment tool for extracting duration information from singing voice data. We have updated the paper and please refer to the revision in Appendix E.

Reviewer comment (Reproducibility):
How do you extract the pitch information? This significantly affects the performance so the details should be included. (about F0 min, F0 max, resolution, and pitch extraction algorithm).

Author response:
$\textbf{Q5}$: How do you extract the pitch information?

$\textbf{A5}$: We appreciate your question. The pitch of each frame is extracted using the pyworld[1] with the dio + stonemask algorithm. The parameters are set to default.
 
[1] https://github.com/JeremyCCHsu/Python-Wrapper-for-World-Vocoder

Reviewer comment (Experiments):
The authors may train the audio codec with their speech dataset. I think it is important to utilize a high-quality speech codec for high-quality speech synthesis. In this regard, I hope that the authors will mention about this part by comparing your model with the same model utilizing an official Soundstream codec as a latent representation.

Author response:
$\textbf{Q6}$: The authors may train the audio codec with their speech dataset. I hope that the authors will mention about this part by comparing your model with the same model utilizing an official SoundStream codec as a latent representation.

$\textbf{A6}$: We appreciate your thorough understanding and insightful comments. Since we could not find the official pytorch implementation of SoundStream,  we attempted to reproduce it and the evaluation results are shown as follows (in the 12.8kbps setting):
| Implementation | VISQOL | 
| --- | --- |
| Ours                  |  4.38      |
| SoundStream Paper | 4.26 | 

From the results, we can find that our implementation is competitive with the original performance.

Reviewer comment (Experiments):
Why did the authors not include any experiments with single speaker data like LJSpeech.

Author response:
$\textbf{Q2}$: Why did the authors not include any experiments with single-speaker data like LJSpeech?

$\textbf{A2}$: Thank you for your question. We would like to highlight that our work focuses on diverse speech synthesis in terms of speaker identity, prosody, and style, especially the zero-shot text-to-speech synthesis (i.e., on unseen speakers).

For single-speaker datasets such as LJSpeech, firstly, they lack speech diversity since there is only one speaker. Thus, it is not aligned with our focus. Secondly, some previous works, such as [1], have already achieved human-level voice quality. This essentially narrows the scope for further enhancements using this dataset.

[1] Tan, Xu, et al. "Naturalspeech: End-to-end text to speech synthesis with human-level quality."

Reviewer comment (Experiments):
As an ablation what would be the shortest prompt in seconds that you can give for zero-speech synthesis?

Author response:
$\textbf{Q3}$: As an ablation what would be the shortest prompt in seconds that you can give for zero-speech synthesis? 

$\textbf{A3}$:  This is a great suggestion to investigate the shortest prompt length. We vary the length of the prompt and report the corresponding SMOS on the LibriSpeech test set. The table below shows that even when reducing the prompt length from 3.0 seconds to 1.0 seconds, NaturalSpeech 2 is still capable of generating high-quality speech. It further demonstrates the robustness and stability of our system despite the reduction in prompt length.
| Prompt Length| SMOS|
| ------- | ---- | 
| 3.0 s  |  4.06  |
| 2.0 s  |  3.98  |
| 1.0 s  |  3.79  |

Reviewer comment (Experiments):
After the phoneme Encoder you have a Duration and Pitch predictor. Why didn't you also include an Energy Predictor like the authors did in FastSpeech2 since the idea seems to be derived form there?

Author response:
$\textbf{Q4}$: Why didn't you also include an Energy Predictor like the authors did in FastSpeech2 since the idea seems to be derived from there? 

$\textbf{A4}$: Thank you for your insightful question. The energy predictor is indeed a versatile module. In practice, we have added the energy predictor and did not observe any improvement. For the sake of simplicity in our current model, we let the diffusion model implicitly predict energy information. However, we acknowledge that incorporating an energy predictor could further enhance controllability. We value this suggestion and will take it into consideration for our future research.

Reviewer comment (Writing):
The paper is sometimes unclear with regards to what the model components represent and how the components fit together. For example, the use of SoundStream and wavenet is not obvious. These are previously published approaches, that are used in novel ways here. It took multiple readings to understand how they are being used in this paper, and even still I’m not 100% sure that my understanding is correct. Broadly, the paper relies too heavily on Figure 1.0 to describe how the model fits together.

Author response:
Thanks for your careful reading. We have updated the paper and highlighted the updates as follows.

For neural audio codec, firstly, we adopt the SoundStream architecture because: 1) it provides good reconstruction quality and low bitrate, 2) it can convert waveforms into continuous vectors while retaining fine-grained details, and 3) based on the quantized token IDs, we can design the regularization loss, which further enhances the prediction precision. Secondly, we adapt the SoundStream by 1) we use the post-quantized latent $z$, i.e., the output from the RVQ, as the representation for waveform $x$, and as the training target for the diffusion model, 2) we develop the regularization loss $L_{ce-rvq}$.

We adopt WaveNet as the architecture of the diffusion model. Firstly, in our preliminary experiments, we compare various architectures including the vanilla transformer, conformer, UNet and WaveNet. Among these, WaveNet exhibits superior performance in terms of audio quality (CMOS), hence our choice for the diffusion model's architecture. Secondly, to facilitate in-context learning, we incorporate a Q-K-V attention and a FiLM layer in the block. Please refer to Appendix B for more details.
 
Regarding the interrelation between the components, our system is composed of three components: a prior model, a diffusion model, and an audio codec.  Similar to [1], the prior model (a phoneme encoder and a duration/pitch predictor) encodes the text $y$ into a prior $c$. Then, the diffusion model predicts the speech representation $z$ using prior $c$ as a condition. Finally, the speech representation $z$ is input into the decoder of the audio codec to generate the speech signal $x$.
 
[1]Ren, Yi, et al. Fastspeech 2: Fast and high-quality end-to-end text to speech.

Reviewer comment (Writing):
The argumentation around continuous vs discrete tokens is very hard to follow. It’s not clear why the discrete token sequence must necessarily be longer than a continuous sequence (Introduction). The first three pages spend a lot of effort describing why a continuous representation is a better fit for this task. Then in Section 3.1, the paper states “However, for regularization and efficiency purposes we use residual vector quantizers with a very large number of quantizers and codebook tokens to approximate the continuous vectors. This provides two benefits…” This is a particularly surprising turn of the argument to then go on to describe why discrete tokens are useful here.

Author response:
Firstly, we would like to clarify that our paper doesn't assert that the discrete token sequences must necessarily be longer than a continuous sequence. What we claim is that previous works often use multiple RVQ discrete token sequences for speech representation. When these sequences are flattened, it results in a multiple-fold increase in length.

Secondly, it's true that some attempts have been made to address this issue, but they still have their own problems. 
AudioLM[1] predicts the first two RVQ token sequences in the first stage, and predicts the remaining RVQ token sequences in the second stage. However, this approach has two drawbacks: 1) it requires two separate AM models and a two-stage generation process; 2) it still needs the flattening of the code sequences, which also extends the sequence length. VALL-E [2] first predicts the first RVQ token sequence in an AR manner, and then predicts the other RVQ token sequences in a NAR manner, which makes the modeling much more complex compared with one-stage generation. Moreover, MusicGen[3] claims that such parallel modeling may impact system performance.

In contrast, NaturalSpeech 2 uses a single continuous latent instead of multiple discrete tokens for each speech frame. It can prevent the increase in sequence length and support one-stage generation, which helps enhance overall performance.
 
[1] Borsos, Zalán, et al. Audiolm: a language modeling approach to audio generation.

[2] Wang Chengyi, et al. Neural codec language models are zero-shot text to speech synthesizers.

[3] Copet, Jade, et al. Simple and Controllable Music Generation.

Reviewer comment (Writing):
The diffusion formulation is too compact to be clearly followed. Page 5 contains the following sentence with a number of ambiguities: “Then we calculate the L2 distance between the residual vector with each codebook embedding in quantizer j and get a probability distribution with a softmax function, and then calculate the cross-entropy loss between the ID of the ground-truth quantized embedding ej and this probability distribution. Lce−rvq is the mean of the cross-entropy loss in all R residual quantizers, and λce−rvq is set to 0.1.” It is recommended to include an appendix entry or describe each clause separately in place.

Author response:
Thanks for your advice. We have provided more details in Appendix C.3 in the revised version. We also provide the detailed formulation of CE-RVQ loss as follows:
 
Moreover, to clarify, for each residual quantizer $j \in [1, R]$, we first get the residual vector: 
\begin{equation} 
z_j = z_0 - \sum^{j-1}_{m=1}\hat{e}^{m},
\end{equation}
 
where $\hat{e}^m$ is the ground-truth quantized embedding in the $m$-th residual quantizer. Then we calculate the L2 distance between the residual vector with each codebook embedding in quantizer $j$ and get a probability distribution as follows: 
\begin{equation} 
l_i = ||z_j - e^j_i||_{2}, 
\end{equation}

\begin{equation} 
s_i = \frac{e^{-l_i}}{\sum^{N_j}_{k=1}{e^{-l_k}}},
\end{equation}

where $N_j$ is the code number of residual quantizer $j$, and $s_i$ is the probability of code $i$ in codebook $j$. Finally, we can calculate the cross-entropy loss of residual quantizer $j$ given the ground-truth code index which is denoted as $L_{ce, j}$. The final CE-RVQ loss is shown as follows: \begin{equation}
L_{ce-rvq} = \sum_{j=1}^{R}{L_{ce, j}} 
\end{equation}

Reviewer comment (Writing):
Introduction: “the zero-shot capability that is important to achieve diverse speech synthesis” – why is zero-shot necessary for diverse speech synthesis? Also, for what contexts and use-cases is diverse speech synthesis necessary?

Author response:
Thanks for your careful reading.

Firstly, diverse speech synthesis aims to generate natural and human-like speech with rich diversity in speaker identity (e.g., gender, accent, timbre), prosody, style, et.al. However, it is difficult and expensive to collect all kinds of speeches to capture such diversity. And for zero-shot synthesis, it is valuable for generating speech with a variety of elements like timbre, prosody, and emotion, without the need for explicit training for these attributes, which significantly eases the difficulty of data collection and broadens the potential diversity in speech synthesis.
 
Secondly, diverse speech synthesis is necessary in various contexts and use-cases.  For instance,
1) Voice Assistants: To make interactions more personalized and engaging, voice assistants need to be able to generate a variety of voices.

2) Entertainment: In industries like animation or gaming, diverse speech synthesis can be used to create a wide array of unique character voices.

3) Language Learning: Diverse speech synthesis can help language learners by providing a variety of accents and pronunciations to learn from.

Reviewer comment (Novelty):
Why use a diffusion model instead of any other NAR model?

Author response:
Compared to NAR methods, the diffusion model stands out as it incorporates multiple iterations in inference. This allows for a progressive refinement over multiple iterations, which helps enhance the modeling capability, and makes it particularly effective high-quality generation.
 
To demonstrate this effectiveness, we employ FastSpeech2[1] as a NAR baseline in our paper (in Sec. 4.2). We adapt it by adding cross-attention on speech prompts for zero-shot synthesis. Furthermore, we also change the prediction target from the mel-spectrogram to the latent representation. For a fair comparison, we scale it to 400M parameters and trained it on the same MLS dataset. As shown in Table 2, the significant performance gap indicates limitations in its modeling capability.
 
[1] Ren, Yi, et al. Fastspeech 2: Fast and high-quality end-to-end text to speech.

Reviewer comment (Reproducibility):
What is the unit used in the L_pitch loss? Hz? log Hz? something else?

Author response:
Thanks for your careful reading. We normalize the pitch to zero mean and unit variance.

Reviewer comment (Presentation):
The variable z is used in a number of different ways; could this be clarified (e.g., in Figure 2 between the prompt, input to diffusion model, and output)?

Author response:
Thanks for your careful reading. We would like to clarify that the notation 'z' is used throughout the article to consistently represent the latent audio representation encoded by the codec.

Reviewer comment (Reproducibility):
Section 4.1, Page 6: What is the value of sigma in the sigma-second audio segment as a prompt?

Author response:
Thanks for your careful reading. As mentioned in Sec. 4.2 'Generation Similarity', we set $\sigma$ to $3$ in Sec.4.

Reviewer comment (Evaluation):
How much loss is incurred by filtering the output by a speech scoring model? For example, are 99% of utterances accepted, or 1%?

Author response:
There may be some misunderstanding about the “loss incurred by filtering”. We use a speech scoring model for reranking. We sample 10 candidates and select the best one.

Reviewer comment (Theory):
“When we disable the speech prompt in diffusion, the model cannot converge” – this seems remarkable. Why does the model require a speech prompt to learn?

Author response:
In practice, we find that the absence of a speech prompt, which supplies the speaker identity, may cause instability during the training phase. This may be attributed to the massive diversity of speech data given the large number of speakers in the training dataset.