Paper: PromptTTS 2: Describing and Generating Voices with Text Prompt (ICLR-2024)

Reviewer comment (Novelty):
The authors claim that the proposed variational method solves the one-to-many problem, but I doubt if this is the case. The proposed module basically learns to predict a melspec encoder's output from text prompts, in addition to conditoning the TTS backbone. I don't see how this solves the one-to-many problem as in the end we are still predicting from text (one) to different variability (many) in the voice. The only difference is that there is an additional auxiliary loss from the melspec encoder. This seems to be in principle similar to InstructTTS's approach to regularize melspec encoder and text encoder output to close in the embedding space.

Author response:
**Q1: About one-to-many problem**

As mentioned in your review, previous work such as InstructTTS has attempted to address the one-to-many problem by regularizing the melspec encoder and text encoder outputs to converge in the embedding space using L2 loss. However, this approach is limited in its ability to resolve the one-to-many problem, as the output of the melspec encoder contains much more information than that of the text encoder.

In contrast, we approach the one-to-many problem as a generation task and utilize generative models (specifically, diffusion models) to generate the output of the melspec encoder based on the output of the text encoder. We believe that VoiceGen differs from previous frameworks, and is the first framework capable of generating and fixing virtual speakers. This is a significant contribution to both future research and the real-world application of prompt-based TTS systems.

---

Dear Reviewer dNEb,

We are very delightful to see that most of the issues are addressed. For the Q1, we agree that "solve" the one-to-many problem is a kind of over-claim, as VoiceGen propose a better framework to "alleviate" the one-to-many problem (by using a generative model to predict the missing information about voice variability) than the previous works (using L2 regularization). We have changed the statements in our paper from "solve" the problem to "alleviate" the problem. Thanks again for your comments.

Best, Authors

Reviewer comment (Novelty):
The proposed prompt generation method, while the first of its kind, seems to be still quite limited by the very limited SLU output set, and the template used for LLM generation.

Author response:
**Q2: About the limitation of SLU output set and template**

The template is written by LLM and can be various by changing the prompt. However, the current limitations of the SLU output set can restrict the capabilities of VoiceGen. We anticipate that more robust SLU models will become available in the future, and we plan to continue our efforts to improve the performance of VoiceGen in this regard.

Reviewer comment (Evaluation):
The proposed method performs only marginally better than PromptTTS and InstructTTS.

Author response:
**Q3: About the performance**

Firstly, our results demonstrate that VoiceGen outperforms previous methods such as PromptTTS and VoiceGen by a significant margin. Further analyses (see Q1 in [Response to Reviewer g23M](https://openreview.net/forum?id=NsCXDyv2Bn&noteId=j7nT1dF7U1) and Q7 in [Response to Reviewer Ghqv (2/2)](https://openreview.net/forum?id=NsCXDyv2Bn&noteId=w6K4vz3QGP)) have confirmed that this improvement is consistent across different datasets and TTS backbones.

Secondly, the main contribution of our paper lies in the development of a general framework on modeling voice variability (first work to support virtual speaker creation) and text prompt writting (to save labeling cost while achieving good quality). We sincerely hope that the significance of our contribution is recognized.

Reviewer comment (Reproducibility):
Will the author open source the generated dataset or source code? I would consider the open sourcing effort as a contribution.

Author response:
**Q4: About the code and dataset**

We have updated the supplementary material to include the code for VoiceGen and the prompt generation pipeline, along with the generated dataset. It is worth noting that this pipeline can be utilized to generate prompts for other attributes by providing the necessary attributes, classes, and ChatGPT keys (further details can be found in the README file located in the data_pipeline folder).

Reviewer comment (Experiments):
After listening to the generated voices on the demo page, audio quality is still an issue and further improvements are required, especially for certain text prompts such as "Please speak at a fast speed, gentleman". The reason could be missing or few audio samples for corresponding prompts in training datasets.

Author response:
**Q1: About imperfect sample**

Yes, we keep this imperfect sample to showcase that the sampling result can be occasionally imperfect (although this can be alleviated by sampling for another time). We will keep working on further improving the stability and quality of model in future work.

Reviewer comment (Experiments):
What's the necessity of concatenating both text prompt representation and speech prompt representation for TTS backbone? Will the speech prompt representation itself be enough to guide the TTS backbone through cross attention?

Author response:
**Q2: About the concatenating of both text prompt representation and speech prompt representation**

As you noted in your review, using only speech prompt representation is sufficient in terms of information completeness. However, concatenating both text prompt representation and speech prompt representation can enhance the model's awareness of the information contained within the text prompt, thereby leading to a slight improvement in model accuracy.

Reviewer comment (Reproducibility):
So many details are left out, as this would not really be possible to fit into the paper. So without the exact code + recipe to produce all the results, it will be almost impossible to reproduce the results. I think having code + recipe available here is very important.

Author response:
**Q4: About the code and dataset**

We have updated the supplementary material to include the code for VoiceGen and the prompt generation pipeline, along with the generated dataset. It is worth noting that this pipeline can be utilized to generate prompts for other attributes by providing the necessary attributes, classes, and ChatGPT keys (further details can be found in the README file located in the data_pipeline folder).

Reviewer comment (Experiments):
All the experiments basically just show that the proposed model works well and solves the outlined problem. However, there is almost no analysis or ablation studies, etc. E.g. how important is it to use a diffusion model here? What about other model types? What about other smaller model details, and hyper parameters, etc?

Author response:
**Q7: About the further ablations**

We further conduct two ablations on the TTS backbone and the length of prompt and reference representations.

Besides the TTS backbone used in the submission (i.e., latent diffusion in NaturalSpeech 2 [1]), we further apply our method (as well as the baseline methods) on another TTS backbone based on the token prediction of codec results (as proposed in SoundStorm [2]). The results are as follows:

|  Model   | Gender | Speed | Volume | Pitch | Mean |
|  ----  | ----  |  ----  | ----  |  ----  | ----  |
| PromptTTS  | 97.47 | 89.96 | 92.41 | 85.06 | 91.23 |
| InstructTTS  | 96.55 | 89.66 | 92.18 | 84.14 | 90.63 |
| VoiceGen  | 98.92 | 92.18 | 93.95 | 90.73 | 93.95 |

From the result, we observe that VoiceGen consistently outperform baseline methods in the condition that TTS backbone in SoundStorm [2] is leveraged.

Moreover, we conduct experiments on a unique hyper-parameter in VoiceGen, i.e., the length of prompt and reference representations. The results are shown as follows:

|  Representation Length   | Gender | Speed | Volume | Pitch | Mean |
|  ----  | ----  |  ----  | ----  |  ----  |  ----  |
| 4  | 98.54 | 92.49 | 93.26 | 88.66 | 93.24 |
| 8 (Default Choice)  | 98.23 | 92.64 | 92.56 | 89.89 | 93.33 |
| 16  | 98.77 | 92.18 | 93.95 | 90.27 | 93.79 |
| 32  | 98.39 | 91.80 | 91.19 | 90.27 | 93.09 |

From the results, we observe the length of representation is not a sensitive hyper-parameter in terms of performance.

Reviewer comment (Writing):
I don't exactly understand the difference between speech prompts and text prompts. A text prompt is really like a description of what should be generated, like “Please generate a voice of a boy shouting out”. A speech prompt is the same but as speech? Or is this the text of the generated speech? Or sth else?

Author response:
**Q1: About the speech prompts and text prompts**

Similar to your comments, a text prompt is a description of the desired output voice, such as “Please generate a voice of a boy shouting out”.

For the speech prompt, it is **not** the speech version of text prompt. The speech prompt can be a speech of target style (maybe a speech of a famous person). The speech prompt based TTS model can learn from the speech prompt to generate speech in the same style (or voice) as the speech prompt. However, identifying an appropriate speech prompt can be a time-consuming task and may raise ethical concerns. Therefore, text description can be a flexiable and safe way for voice generation.

Reviewer comment (Reproducibility):
The input of variation network comprises the prompt representation (P1, ..., PM ), noised reference representation (R1t , ..., PMt ), and diffusion step t — how exactly? P and R are concatenated sequences? But the diffusion process only runs on R?

Author response:
**Q2: About the input of variation network**

To provide a more precise analysis, we will discuss the problem with numerical details. The variation network we employed is a Transformer-based model, with the prompt representation, noised reference representation, and reference representation all set to a length of 8. The prompt representation and noised reference representation both have a shape of [batchsize, 8, hiddensize], which are concatenated to form a shape of [batchsize, 8+8, hiddensize]. This concatenated representation is then fed into the Transformer model, with the output also having a shape of [batchsize, 8+8, hiddensize]. The latter 8 outputs, corresponding to the noised reference representation, with a shape of [batchsize, 8, hiddensize], are used as the predicted results, along with the reference representation, for loss calculation. 

It should be noted that although we only consider the latter 8 outputs, the prompt representation can also influence the output through self-attention. Our approach of using a single model for both prompt and noised reference representations was inspired by the implementation in DALLE-2 (https://github.com/lucidrains/DALLE2-pytorch).

Reviewer comment (Experiments):
Clarification on text attributes for text prompt generalization: There is only gender (via the existing SLU model), pitch, volume, and speed, nothing else? I would expect some more attributes, e.g. different emotion categories, etc.

Author response:
**Q3: About the clarification on text attribute and setting**

In our submission, all results presented in Table 1 are based on a fair comparison of text data, with all models utilizing LLM-written data. Additionally, the SLU and DSP models are fixed for metric calculation. We also conducted experiments on all models using text data from PromptTTS, which has lower quality. The results of these experiments can be found in Q1 in [Response to Reviewer g23M](https://openreview.net/forum?id=NsCXDyv2Bn&noteId=j7nT1dF7U1) and the conclusion is similar with that in Table 1 of the submission. 

Regarding the classes, we used two classes for gender (male and female) and three classes (high, normal, low) for other attributes such as pitch, speed, and volume. We plan to continue our work by adding more attributes, such as emotion, in our future research.

Reviewer comment (Experiments):
Section 5.1, table 1: I don't exactly understand what is measured under what conditions on what data. So, each TTS model (VoiceGen vs the others) generates some speech, then some attributes are given to produce some prompt, and then, for the given fixed SLU models and digital signal processing tools, the accuracy is measured? That uses the generated prompts via LLM as mentioned before? All of them, so 20K text prompts? How many classes are there for each of the attributes?

Author response:
**Q3: About the clarification on text attribute and setting**

In our submission, all results presented in Table 1 are based on a fair comparison of text data, with all models utilizing LLM-written data. Additionally, the SLU and DSP models are fixed for metric calculation. We also conducted experiments on all models using text data from PromptTTS, which has lower quality. The results of these experiments can be found in Q1 in [Response to Reviewer g23M](https://openreview.net/forum?id=NsCXDyv2Bn&noteId=j7nT1dF7U1) and the conclusion is similar with that in Table 1 of the submission. 

Regarding the classes, we used two classes for gender (male and female) and three classes (high, normal, low) for other attributes such as pitch, speed, and volume. We plan to continue our work by adding more attributes, such as emotion, in our future research.

Reviewer comment (Reproducibility):
Text prompt dataset: So this is released? Where? I did not find it.

Author response:
**Q4: About the code and dataset**

We have updated the supplementary material to include the code for VoiceGen and the prompt generation pipeline, along with the generated dataset. It is worth noting that this pipeline can be utilized to generate prompts for other attributes by providing the necessary attributes, classes, and ChatGPT keys (further details can be found in the README file located in the data_pipeline folder).

Reviewer comment (Experiments):
I don't exactly understand Section 5.1, table 3. This is always with the same phoneme sequence as input to the TTS model? But what is the text context here? Why are the results so different between PromptTTS, InstructTTS and VoiceGen? Are they really comparable? Do they have the same TTS backbone? Does the TTS backbone use diffusion in all cases?

Author response:
**Q6: About the clarification on Table 3 and Table 1**

Regarding Table 3, as you have pointed out, the phoneme sequence is always the same for the "Text Prompt", "Variation Network", and "TTS Backbone" columns. And all the models utilize the same diffusion-based TTS backbone.

The "Text Content" column is included for the sake of completeness, as some readers may argue that an extremely overfit TTS system can synthesize different voices or styles with different text content. However, this is not an issue for any of the baseline systems or VoiceGen. 

It is important to note that Table 3 (as well as Table 6) primarily demonstrates two key findings: 1) the variation network is capable of sampling different voices that correspond to the intention of a text prompt, and 2) the variability in voice is primarily controlled by the sampling results of the variation network, rather than other factors such as text prompt with the same meaning, text content to synthesize, or TTS backbone.

Regarding Table 1, as you have noted, the synthesized results can occasionally be inconsistent with the text prompts. Increasing the number of samples can increase the probability of obtaining inconsistent results, although this can be alleviated through engineering techniques (such as utilizing a selection model to choose a result that is more consistent with the prompt, as leveraged in DALLE-2 [3]).

Reviewer comment (Evaluation):
When you add variability through the variation network, how well are the properties of the text prompt for the speech style actually preserved? I guess this is table 1? But how much variance do you get when you add sampling results of the variation network?

Author response:
**Q6: About the clarification on Table 3 and Table 1**

Regarding Table 3, as you have pointed out, the phoneme sequence is always the same for the "Text Prompt", "Variation Network", and "TTS Backbone" columns. And all the models utilize the same diffusion-based TTS backbone.

The "Text Content" column is included for the sake of completeness, as some readers may argue that an extremely overfit TTS system can synthesize different voices or styles with different text content. However, this is not an issue for any of the baseline systems or VoiceGen. 

It is important to note that Table 3 (as well as Table 6) primarily demonstrates two key findings: 1) the variation network is capable of sampling different voices that correspond to the intention of a text prompt, and 2) the variability in voice is primarily controlled by the sampling results of the variation network, rather than other factors such as text prompt with the same meaning, text content to synthesize, or TTS backbone.

Regarding Table 1, as you have noted, the synthesized results can occasionally be inconsistent with the text prompts. Increasing the number of samples can increase the probability of obtaining inconsistent results, although this can be alleviated through engineering techniques (such as utilizing a selection model to choose a result that is more consistent with the prompt, as leveraged in DALLE-2 [3]).

Reviewer comment (Experiments):
I am not certain if the proposed model and the baseline models are trained on the same data, and hence I cannot draw conclusions that whether the proposed model outperforms the baselines because of the additional LLM generated data or because of the introduction of the variational network to address the one-to-many problem. It would be good to show how well the baseline performs with and without LLM-augmented text prmopts.

Author response:
**Q1: About the text data (Weakness 1 and 2)**

Since the ablation study on text data shows the superiority of LLM-written data over the prompts in PromptTTS (as reported in Table 4 of the submission), the results in Table 1 of the submission are conducted when all the models using the LLM-written data. Thus, the results are a fair comparison in terms of text data. 

To strengthen the conclusions of our paper, we conducted additional experiments on all models (i.e., PromptTTS, InstructTTS, and VoiceGen) using the text data in PromptTTS. The results are in the following table:

|  Model   | Text Dataset  | Gender | Speed | Volume | Pitch | Mean |
|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  |
| PromptTTS  | Prompts in PromptTTS | 98.93 | 87.43 | 89.35 | 85.44 | 90.29 |
| PromptTTS  | LLM-written | 98.01 | 89.66 | 92.49 | 85.95 | 91.54 |
|  ----  | ----  |  ----  | ----  |  ----  |  ----  |  ----  |
| InstructTTS  | Prompts in PromptTTS | 96.55 | 86.13 | 88.43 | 85.52 | 89.16 |
| InstructTTS  | LLM-written | 97.24 | 90.57 | 91.26 | 86.82 | 91.47 |
|  ----  | ----  |  ----  | ----  |  ----  |  ----  |  ----  |
| VoiceGen  | Prompts in PromptTTS | 98.77 | 90.80 | 90.57 | 89.58 | 92.43 |
| VoiceGen  | LLM-written | 98.23 | 92.64 | 92.56 | 89.89 | 93.33 |

From the Table, we observe that 1) VoiceGen outperforms baseline methods on both text prompt datasets; 2) using LLM-written data consistently improves the quality of prompt-based TTS on all models, compared to using the prompts in PromptTTS.

Reviewer comment (Experiments):
Given that the number of attribute combinations is rather small (2 x 3 x 3 x 3 = 54), I am suspicious about how useful it is to increase the number of text prompts. The author could have conducted ablation studies comparing using only the PromptTTS prompts vs those + x LLM-augmented prompts.

Author response:
**Q1: About the text data (Weakness 1 and 2)**

Since the ablation study on text data shows the superiority of LLM-written data over the prompts in PromptTTS (as reported in Table 4 of the submission), the results in Table 1 of the submission are conducted when all the models using the LLM-written data. Thus, the results are a fair comparison in terms of text data. 

To strengthen the conclusions of our paper, we conducted additional experiments on all models (i.e., PromptTTS, InstructTTS, and VoiceGen) using the text data in PromptTTS. The results are in the following table:

|  Model   | Text Dataset  | Gender | Speed | Volume | Pitch | Mean |
|  ----  | ----  |  ----  | ----  |  ----  | ----  |  ----  |
| PromptTTS  | Prompts in PromptTTS | 98.93 | 87.43 | 89.35 | 85.44 | 90.29 |
| PromptTTS  | LLM-written | 98.01 | 89.66 | 92.49 | 85.95 | 91.54 |
|  ----  | ----  |  ----  | ----  |  ----  |  ----  |  ----  |
| InstructTTS  | Prompts in PromptTTS | 96.55 | 86.13 | 88.43 | 85.52 | 89.16 |
| InstructTTS  | LLM-written | 97.24 | 90.57 | 91.26 | 86.82 | 91.47 |
|  ----  | ----  |  ----  | ----  |  ----  |  ----  |  ----  |
| VoiceGen  | Prompts in PromptTTS | 98.77 | 90.80 | 90.57 | 89.58 | 92.43 |
| VoiceGen  | LLM-written | 98.23 | 92.64 | 92.56 | 89.89 | 93.33 |

From the Table, we observe that 1) VoiceGen outperforms baseline methods on both text prompt datasets; 2) using LLM-written data consistently improves the quality of prompt-based TTS on all models, compared to using the prompts in PromptTTS.

Reviewer comment (Evaluation):
The authors did not give sufficient background on InstructTTS. That model also deploys a diffusion model and in principle would be capable of modeling one-to-many mapping. Why does the VoiceGen model perform better than InstructTTS?

Author response:
**Q2: About the diffusion in InstructTTS**

We apologize for not discussing InstructTTS in our paper. InstructTTS addresses the one-to-many challenge by leveraging speaker ID (once the speaker id is provided, the variability on voice becomes smaller). However, this approach has two limitations. First, it restricts the application of InstructTTS to the speakers in the training dataset. Second, it prevents the voice generation model from modeling timbre, which is an important attribute of voice generation. In contrast, VoiceGen can create different voices, including variability in timbre, for the same text prompt.

In addition, InstructTTS has also attempted to address the one-to-many problem by regularizing the reference encoder and text encoder outputs to converge in the embedding space using L2 loss. However, this approach is limited in its ability to resolve the one-to-many problem, as the output of the reference encoder contains much more information than that of the text encoder.

In contrast, we approach the one-to-many problem as a generation task and utilize generative models (specifically, diffusion models) to generate the output of the reference encoder based on the output of the text encoder. VoiceGen is the first framework capable of generating and fixing virtual speakers. This is a significant contribution to both future research and the real-world application of prompt-based TTS systems.

Reviewer comment (Experiments):
Given that the conditional attributes are all categorical, it can be conditioned with a lookup table for each attribute straightforwardly. How does that method compare with the proposed method? The paper does not really showcase the strength of free-form text description for conditioning attributes that are hard to categorize.

Author response:
**Q3: About the categorical attributes (Weakness 4 and Weakness 5)**

Firstly, regarding the effectiveness of text descriptions, we found that it outperforms the categorical lookup table method when controlling the voice with text description, as reported in PromptTTS[1], and we also confirmed this in our preliminary study.

Secondly, for conditioning attributes that are difficult to categorize, we believe that the extension of Face2Voice to our model partially demonstrates its ability to generate voice based on unspecific information.

Thirdly, we acknowledge that using more specific labels and descriptions for hard-to-categorize attributes would be beneficial. We plan to address this in our future work. We sincerely hope that our paper's contribution in presenting a general framework for modeling voice variability and text prompt writing can be recognized.

Reviewer comment (Experiments):
For step 1 when generating phrases, the authors show in Table 5 that male can be mapped to man/boy/guy/dude/gentlement. However, one would expect quite different voices between boy and man. This shows the weakness of such pipeline where text descriptions are created from underspecified labels.

Author response:
**Q3: About the categorical attributes (Weakness 4 and Weakness 5)**

Firstly, regarding the effectiveness of text descriptions, we found that it outperforms the categorical lookup table method when controlling the voice with text description, as reported in PromptTTS[1], and we also confirmed this in our preliminary study.

Secondly, for conditioning attributes that are difficult to categorize, we believe that the extension of Face2Voice to our model partially demonstrates its ability to generate voice based on unspecific information.

Thirdly, we acknowledge that using more specific labels and descriptions for hard-to-categorize attributes would be beneficial. We plan to address this in our future work. We sincerely hope that our paper's contribution in presenting a general framework for modeling voice variability and text prompt writing can be recognized.