Paper: Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis (ICLR-2024)

Reviewer comment (Experiments):
I just wonder why the authors do not state ProsoSpeech which has the same structure as P-LLM without word-level prosody modeling. It would be better if the authors add the ablation study for phoneme and word-level prosody modeling.

Author response:
**[About the missing statement of ProsoSpeech and the missing ablation study for phoneme-level and word-level prosody modeling]**  
We are sorry for the missing statement of ProsoSpeech. Here, we provide detailed descriptions of the differences between ProsoSpeech and our method. We also conduct ablation studies for the prosody modeling strategies. These descriptions and results are also added to Appendix H.

*Differences:* We agree that ProsoSpeech [1] has the same structure as P-LLM proposed in this paper (both of us use the auto-regressive Transformer for prosody modeling). However, ProsoSpeech aims to improve the **naturalness** of prosody modeling for traditional TTS, which is only conditioned on text and global speaker information. Compared with it, our P-LLM aims at improving the **similarity** of speaker-relevant prosodic patterns for zero-shot TTS, which extracts fine-grained prosodic information from latent prosodic prompts by leveraging the powerful in-context learning capability of LLM. We also propose a novel training strategy that enables P-LLM to capture the useful prosody-level information contained in the multi-sentence contextual prompts. Moreover, our work mainly aims at boosting the timbre and prosody prompting mechanisms for zero-shot TTS.

*Ablation study for prosody modeling strategy:* We use stride-8-level prosody modeling (the stride of the pooling layer inside the VQ encoder is set to 8) and ProsoSpeech uses word-level prosody modeling. We agree that the ablation studies in this aspect are very important. Therefore, we conducted the following experiments to verify the performance of the phoneme-level, word-level, and stride-8-level prosody modeling. In this experiment, all of the systems use 3 seconds of prompts. The results are shown in the following table. It can be seen that the stride-8-level prosody modeling (used in our paper) achieves the best performance. Intuitively speaking, the phoneme-level prosody modeling provides finer-grained information for better reconstruction while word-level prosody modeling provides more semantic information. Both of these methods would be easily affected by the alignment accuracy of the speeches in training and inference stages. In order to enhance the stability and performance of the proposed model, we use stride-8-level prosody modeling.

| Setting        |  WER  |  SIM  |  DTW  | CMOS-Q | CMOS-S |
| -------------- | :---: | :---: | :---: | :----: | :----: |
| Stride-8-level | 2.46% | 0.898 | 34.39 | 0.000  | 0.000  |
| Phoneme-level  | 2.94% | 0.897 | 35.78 | -0.107 | -0.052 |
| Word-level     | 3.12% | 0.892 | 34.42 | -0.120 | -0.078 |

Reviewer comment (Novelty):
Multi-reference Style Transfer methods were already utilized in many speech papers such as Attentron [S. Choi, 2020].

Author response:
**[About the multi-reference style transfer methods]**  
We apologize for not discussing the multi-reference style transfer methods in the previous version of the paper. In the revised edition, we discussed these methods in detail and highlighted the main differences between them and Mega-TTS in Section 2 (marked in red).

Reviewer comment (Experiments):
Are there any failure cases for the auto-regressive duration predictor? In my opinion, there are many components of this work. I think AR duration predictor is one of the important changes and it could be utilized for other TTS model easily so I hope you to analyze this predictor with additional ablation studies. Do not need to train the model additionally. I just suggest to infer the baseline TTS model with the predicted duration of your model.

Author response:
**[About the proposed auto-regressive duration predictor]**  
*Question:* Are there any failure cases for the auto-regressive duration predictor?

*Answer:* In our experiments, the robustness of the proposed auto-regressive duration predictor is comparable to that of the one used in the baseline TTS model (the duration predictor proposed in FastSpeech 2 [8]), and both models have very few failure cases. As shown in Table 1 in our paper, our Mega-TTS 2 achieves the lowest WER score, demonstrating its robustness.



*About additional experiments:* Thanks for your valuable and constructive suggestions about our auto-regressive duration predictor! We conduct additional experiments to demonstrate that our auto-regressive duration predictor can be utilized for other TTS model easily as a plugin module. Here we infer the zero-shot version of the baseline TTS models with the predicted duration of our auto-regressive duration predictor. The results are shown in the following table. It can be seen that both the speech naturalness and the prosody similarity are improved by our auto-regressive duration predictor.

| Setting                            | Duration Error | CMOS-Q | CMOS-S |
| ---------------------------------- | :------------: | :----: | :----: |
| Duration Predictor in FastSpeech 2 |     27.39      | 0.000  | 0.000  |
| Auto-regressive Duration Predictor |     25.78      | +0.071 | +0.103 |



*About the duration predictor in VITS2 [7]:* Thank you for your suggestions about adopting the adversarial duration predictor proposed in VITS2. Intuitively, both the autoregressive duration predictor in our work and the adversarial duration predictor in VITS2 are powerful generative models suitable for duration modeling (a speech attribute with rich diversity). We will try the adversarial duration predictor in VITS2 in our future work.

Reviewer comment (Experiments):
Does the model synthesize the speech robustly with the noisy reference prompt? It would be better if you could add the results on test-other set.

Author response:
**[About the results on conditioning noisy reference prompts]**  
Here we conduct zero-shot TTS experiments on LibriSpeech test-other set to verify our model's robustness with noisy reference prompts. From the following table, we can see that Mega-TTS maintains excellent performance with noisy reference prompts. We have added these results to Appendix L in the new version of the paper (marked in red). The generated speech samples can be accessed in the "Additional Examples for Rebuttal" Section of our demo page: [https://boostprompt.github.io/boostprompt/](https://boostprompt.github.io/boostprompt/). 

| Model       |    WER    |    SIM    |    DTW    |       QMOS        |       SMOS        |
| ----------- | :-------: | :-------: | :-------: | :---------------: | :---------------: |
| GT          |   3.17%   |     -     |     -     |   3.95$\pm$0.09   |   3.98$\pm$0.08   |
| VALL-E-3s   |   6.21%   |   0.889   |   36.13   |   3.64$\pm$0.12   |   3.70$\pm$0.11   |
| Mega-TTS-3s | **2.73%** | **0.904** | **32.50** | **3.77$\pm$0.10** | **3.80$\pm$0.09** |

Reviewer comment (Experiments):
Results on conditioning noisy reference prompts.

Author response:
**[About the results on conditioning noisy reference prompts]**  
Here we conduct zero-shot TTS experiments on LibriSpeech test-other set to verify our model's robustness with noisy reference prompts. From the following table, we can see that Mega-TTS maintains excellent performance with noisy reference prompts. We have added these results to Appendix L in the new version of the paper (marked in red). The generated speech samples can be accessed in the "Additional Examples for Rebuttal" Section of our demo page: [https://boostprompt.github.io/boostprompt/](https://boostprompt.github.io/boostprompt/). 

| Model       |    WER    |    SIM    |    DTW    |       QMOS        |       SMOS        |
| ----------- | :-------: | :-------: | :-------: | :---------------: | :---------------: |
| GT          |   3.17%   |     -     |     -     |   3.95$\pm$0.09   |   3.98$\pm$0.08   |
| VALL-E-3s   |   6.21%   |   0.889   |   36.13   |   3.64$\pm$0.12   |   3.70$\pm$0.11   |
| Mega-TTS-3s | **2.73%** | **0.904** | **32.50** | **3.77$\pm$0.10** | **3.80$\pm$0.09** |

Reviewer comment (Experiments):
Comparisons with other prosody modeling methods such as Prosody-TTS and CLAPSpeech.

Author response:
**[About the missing comparisons with other prosody modeling methods such as Prosody-TTS and CLAPSpeech]**  
Although our Mega-TTS, Prosody-TTS, and CLAPSpeech are all prosody modeling methods, the objectives and tasks of our Mega-TTS are different from theirs:

- Prosody-TTS [2], and CLAPSpeech [3] aim at improving the overall prosody naturalness for traditional TTS, while our work focuses more on improving the prosody similarity for Zero-shot TTS. Prosody-TTS [1] measures the MOS-P (**naturalness** of pitch, energy, and duration) as one of their metrics. While, as illustrated in Section 4.1 and Appendix A.2, we use SMOS to measure the prosody and timbre **similarity** for each speaker.  CLAPSpeech aims to improve the prosody prediction by encouraging the text encoder to extract prosody variance from the text context with contrastive loss in the joint multi-modal space. The text encoder proposed in CLAPSpeech can be seen as speaker-irrelevant. Moreover, the text encoder proposed in CLAPSpeech can be also used in our work as a plugin module.

Since the objectives and tasks of Prosody-TTS and CLAPSpeech are different from the task of our Mega-TTS, we did not make comparisons with Prosody-TTS, and CLAPSpeech in our paper.

Reviewer comment (Experiments):
Comparison with word-level Prosody modeling (ProsoSpeech).

Author response:
**[About the missing statement of ProsoSpeech and the missing ablation study for phoneme-level and word-level prosody modeling]**  
We are sorry for the missing statement of ProsoSpeech. Here, we provide detailed descriptions of the differences between ProsoSpeech and our method. We also conduct ablation studies for the prosody modeling strategies. These descriptions and results are also added to Appendix H.

*Differences:* We agree that ProsoSpeech [1] has the same structure as P-LLM proposed in this paper (both of us use the auto-regressive Transformer for prosody modeling). However, ProsoSpeech aims to improve the **naturalness** of prosody modeling for traditional TTS, which is only conditioned on text and global speaker information. Compared with it, our P-LLM aims at improving the **similarity** of speaker-relevant prosodic patterns for zero-shot TTS, which extracts fine-grained prosodic information from latent prosodic prompts by leveraging the powerful in-context learning capability of LLM. We also propose a novel training strategy that enables P-LLM to capture the useful prosody-level information contained in the multi-sentence contextual prompts. Moreover, our work mainly aims at boosting the timbre and prosody prompting mechanisms for zero-shot TTS.

*Ablation study for prosody modeling strategy:* We use stride-8-level prosody modeling (the stride of the pooling layer inside the VQ encoder is set to 8) and ProsoSpeech uses word-level prosody modeling. We agree that the ablation studies in this aspect are very important. Therefore, we conducted the following experiments to verify the performance of the phoneme-level, word-level, and stride-8-level prosody modeling. In this experiment, all of the systems use 3 seconds of prompts. The results are shown in the following table. It can be seen that the stride-8-level prosody modeling (used in our paper) achieves the best performance. Intuitively speaking, the phoneme-level prosody modeling provides finer-grained information for better reconstruction while word-level prosody modeling provides more semantic information. Both of these methods would be easily affected by the alignment accuracy of the speeches in training and inference stages. In order to enhance the stability and performance of the proposed model, we use stride-8-level prosody modeling.

| Setting        |  WER  |  SIM  |  DTW  | CMOS-Q | CMOS-S |
| -------------- | :---: | :---: | :---: | :----: | :----: |
| Stride-8-level | 2.46% | 0.898 | 34.39 | 0.000  | 0.000  |
| Phoneme-level  | 2.94% | 0.897 | 35.78 | -0.107 | -0.052 |
| Word-level     | 3.12% | 0.892 | 34.42 | -0.120 | -0.078 |

Reviewer comment (Novelty):
Some of the zero-shot TTS categories are missing from the related works. One of the most related is the attention-base adaptation method. Early in year 2020, Attentron [1] is proposed, which can adapt to unseen speakers with multiple reference utterances, just like the MRTE in MegaTTS. Such strategy is also used in zero-shot voice conversion domain [2]. Later, methods like [3] introduces cross-attention based model and compress the reference into a fixed-length sequence before decoder attention, which is more close to the one in NaturalSpeech2.

Author response:
**[About the missing zero-shot TTS categories in the related works]**  
We are sorry that the attention-base adaptation method is missing in the related works. In the revised version of the paper, we have included detailed discussions on the attention-based adaptation methods and highlighted the main differences between them and our own approach in Section 2 (marked in red). We would like to express our gratitude for your valuable suggestions once again.

Reviewer comment (Experiments):
Some ablation studies are missing. Since the primary design of the training strategy is using multiple reference instead of only one utterance, it is important to show the performance difference between using long context vs short context during training to verify the necessity of this training strategy.

Author response:
**[About the performance difference between using long context vs short context during training]**  
Yes, the ablation studies of the context length during training are essential. We have separately trained our Mega-TTS with different numbers of the contextual VQ codes for P-LLM and trained our Mega-TTS with different numbers of the contextual mel-spectrogram frames for MRTE. After evaluating the reconstruction quality for our compressive acoustic autoencoder and the cross-entropy loss for our P-LLM, we find that the performance of the model significantly **improves when we increase the length of context**, demonstrating the effectiveness of our multi-reference training strategy. We have added the experimental results and analysis to Appendix J in the new version of the paper (marked in red).

Reviewer comment (Theory):
Is there a justification of the assumption in section 3.1 "the mutual information between $y^t$ and $\tilde{y}$ only contains timbre information"? This assumption is not very obvious, since in the audiobook dataset, some performing skills may change the timbre largely in different sentences. Additionally, the average prosody style information can also be shared by different utterances.

Author response:
**[About the assumption in section 3.1] (Question 1)**  
Yes, we agree that: 1) some performing skills may change the timbre largely in different sentences; 2) the the global style information can also be shared by different utterances. For case 1), we select $\tilde{y}$ randomly from regions near $y_{t}$ as much as possible, ensuring that the timbre information of $\tilde{y}$ is close to that of $y_{t}$. We have made descriptions of it in Appendix I in the new version of the paper (marked in red). For case 2), under this circumstance, the timbre encoder may capture **global style information**. However, the **local prosodic patterns** vary dramatically across different utterances by the same speaker. The P-LLM in our paper mainly aims at capturing the **fine-grained (local) prosodic patterns**. To enhance the clarity of our paper, we have provided additional clarification for assumption (1) and explained that we only disentangle the fine-grained prosody and timbre information in the new version of the paper (marked in red). Furthermore, our experiments demonstrate the capability of Mega-TTS to independently prompt prosody and timbre in a zero-shot setting. Even in some emotional datasets (the ESD dataset) and in some in-the-wild corpus (as shown in the demo page), our method is also able to clone non-audiobook style voices effectively.

Reviewer comment (Experiments):
Is it possible to show the baseline methods with 300s context?

Author response:
**[About the baseline methods with 300s context]**  
There are two types of baseline methods in our work: 1) VALL-E; 2)  a baseline that incorporates the GAN used in our Mega-TTS to the FastSpeech 2
backbone [1]. 

1. For VALL-E, it is impossible to show the results with 300s context. Due to the low compression rate of the Encodec model, the maximum context is limited (around 30 seconds) during training. When the length exceeds this limit during inference, the generation quality will be very poor. As shown in "long-form zero-shot TTS experiments" on our demo page, VALL-E fails to generate intelligible results with long context.
2. For the baseline 2), it is fine-tuned on the corresponding context. Therefore, for Baseline-300s, it has actually encountered and learned the speech information from each speaker's 300-second context, which is more powerful than directly feeding the 300s context into the reference encoder and encoding it to a time-averaged global vector.

Reviewer comment (Experiments):
In the case of datasets like LibriVox, which comprises audiobooks, it is commonly observed that there is not a significant variation in speaking style across different utterances by the same speaker. In this context, it becomes difficult to uphold assumption (1). As a result, the timbre encoder may capture prosody information as well, and the experiment does not conclusively demonstrate the complete separation of these two elements.

Author response:
**[About the assumption (1) ]**  
Yes, we agree that since we use the audiobook-style training corpus, the **global** speaking style across different utterances by the same speaker remains unchanged. Under this circumstance, the timbre encoder can capture **global** style information. On the other hand, the **local prosodic patterns** vary dramatically across different utterances by the same speaker. The P-LLM in our paper mainly aims at capturing the **fine-grained (local) prosodic patterns**.  To enhance the clarity of our paper, we have provided additional clarification for assumption (1) and explained that we only disentangle the fine-grained prosody and timbre information in the new version of the paper (marked in red). Furthermore, our experiments demonstrate the capability of Mega-TTS to separately control prosody and timbre in a zero-shot setting. Even in some emotional datasets (the ESD dataset) and in some in-the-wild corpus (as shown in the demo page), our method is also able to clone non-audiobook style voices effectively.

Reviewer comment (Reproducibility):
The description of the information bottleneck, a crucial component of the proposed method, is lacking in detail. Specifically, there is an absence of clear guidelines and processes for setting variables such as $r$, $d$, and the hidden dimensions to ensure an appropriate information bottleneck.

Author response:
**[About the detailed descriptions of the information bottleneck]**  
Thank you for raising this concern. Intuitively, there are four crucial variables for ensuring an appropriate information bottleneck: 1) the number of codebook embedding; 2) the codebook channel size; 3) the compression rate $r$ of the VQ Encoder; 4) and the downsampling rate $d$ of the MRTE. However, the search space of these hyperparameters is too large. Since the settings of r and d also influence the reconstruction quality, the burden of P-LLM, and the computational requirements, we first consider $r$ and $d$, fix them, and find the best setting for the hyperparameters of the codebook. We have introduced the detailed guidelines for setting the information bottleneck and designed experiments for evaluating the disentanglement performance in Appendix F in the new version of the paper (marked in red). Moreover, we also include the results with different sizes of training data to demonstrate that our system performs well with different data volumes consistently.

Reviewer comment (Reproducibility):
From the perspective of a practitioner, the prerequisite of using Montreal Forced Aligner (MFA) to extract alignments beforehand could be seen as a cumbersome step.

Author response:
**[About using Montreal Forced Aligner (MFA)]**  
Yes, Mega-TTS needs an external aligner (Montreal Forced Aligner) to obtain the alignment beforehand. We agree that the data preprocessing pipeline is more complicated than those with end-to-end aligners. However, in our experiments, we find that the external aligner based methods are more robust in large-scale and multi-speaker scenarios. Here we conduct WER evaluations for models based on the external aligner and models based on end-to-end aligners (Like YourTTS [4] and VALL-E [1]). All systems are trained on the large-scale LibriLight dataset and tested on the LibriSpeech test-clean set, which is a zero-shot test setting. For fair comparisons, we keep the number of parameters consistent among all systems. The results are shown in the following table. We can see that the WER of the models based on the external aligner is significantly lower than those based on the end-to-end aligner, demonstrating their effectiveness in practical scenarios.

|    Model     |                    Description                    |   Aligner Type   |    WER    |
| :----------: | :-----------------------------------------------: | :--------------: | :-------: |
|   YourTTS    |                Official Checkpoint                |    End-to-End    |   7.10%   |
|   YourTTS    | Trained on LibriLight (1k hours), 400M parameters |    End-to-End    |   6.83%   |
|    VALL-E    |         Trained on LibriLight (1k hours)          |    End-to-End    |  15.73%   |
| FastSpeech 2 |         Trained on LibriLight (1k hours)          | External Aligner | **4.81%** |
|   Mega-TTS   |         Trained on LibriLight (1k hours)          | External Aligner | **4.69%** |

Reviewer comment (Experiments):
What would be the performance outcome if we generate the prosody latent for the target speaker using only the prosody information from another speaker, instead of using interpolation?

Author response:
**[About the performance of using only the prosody information from another speaker]**     
Here we compare the use of prosody information solely from another speaker with the use of the interpolation technique. The results are presented in the table below. It is evident that relying only on prosody information from another speaker significantly increases the word error rate (WER). This phenomenon may be due to the fact that speakers A and B may not have identical vocal ranges. Directly transferring B's prosody to A may cause speech intelligibility issues, especially for some special prosody patterns. Our proposed prosody interpolation technique resolves this issue by transferring the prosody patterns in a soft and fine-grained manner during the autoregressive generation process.

| Setting                             |    WER    |   SIM-AB    |    DE    |     QMOS      |  SMOS-AB  |
| ----------------------------------- | :-------: | :---------: | :------: | :-----------: | :-------: |
| Interpolation ($\gamma=0.8$)        | **4.82%** | 0.920/0.513 |   32.8   | 3.92$\pm$0.08 | 3.87/2.64 |
| Another Speaker Only ($\gamma=1.0$) |   5.29%   | 0.918/0.510 | **32.2** | 3.84$\pm$0.12 | 3.88/2.60 |

Reviewer comment (Writing):
It is unclear in which dimension the concatenation is performed in the Multi-Reference Timbre Encoder. While Figure 1 seems to suggest that concatenation occurs along the hidden dimension axis, the description in Section 3.2 of the timbre-to-content attention module implies that it happens along the time axis.

Author response:
**[About the dimension in which the concatenation is performed in the Multi-Reference Timbre Encoder]**   
The concatenation is performed in the **time dimension** in the Multi-Reference Timbre Encoder. We are sorry that the corresponding part in Figure 1 is misleading. We have added more descriptions to the title of Figure 1in the rebuttal version of the paper (marked in red).

Reviewer comment (Evaluation):
In the comparison experiments with VALL-E, why was a different model chosen to measure speaker similarity (SIM) instead of using WavLM the one utilized [1]?

Author response:
**[About using a different model to measure SIM instead of using WavLM the one utilized by VALL-E [1]]**  
Yes, we agree that replicating the experimental conditions described in the baseline papers in the evaluation is important. Therefore, we use the WavLM utilized by VALL-E [1] to measure SIM and the results are shown in the following table. It can be seen that the trend and conclusion of the new experimental results are consistent with the results of the speaker verification model used in our paper.   
We acknowledge the significance of replicating the experimental conditions outlined in the baseline papers during the evaluation. As a result, we employ the WavLM, which was utilized by VALL-E [1], to assess SIM. The corresponding results are presented in the table below. Moreover, it is evident that the recent experimental findings align with the outcomes of the speaker verification model employed in our study.

| Model         |    SIM    |
| ------------- | :-------: |
| GT            |   0.683   |
| Baseline-10s  |   0.544   |
| Baseline-60s  |   0.577   |
| Baseline-300s | **0.591** |
| VALL-E-3s     |   0.517   |
| VALL-E-10s    |   0.539   |
| VALL-E-20s    |   0.364   |
| Ours-3s       | **0.551** |
| Ours-10s      | **0.556** |
| Ours-60s      | **0.582** |
| Ours-300s     | **0.587** |

Reviewer comment (Experiments):
Unclear Performance Across Languages: The experiments presented in the paper only use English datasets. Therefore, it's unclear how well the system performs with different languages or dialects. This limits the generalizability of the findings and may hinder the application of the system in diverse linguistic contexts. While the authors acknowledge some limitations of their approach, a more extensive exploration and testing of these constraints could have provided a more comprehensive understanding of the model. This additional analysis could guide future research addressing these limitations and further refining the Mega-TTS model.

Author response:
**[About the performance with different languages] (Question 1)**  
Here, we train our Mega-TTS model on the multilingual *WenetSpeech dataset (Chinese) + LibriLight dataset (English)* and use the *AISHELL-3 dataset (Chinese)* as the test set. We split the dataset and make evaluations according to Section 4.2 in our paper. To evaluate the WER for the Chinese dataset, we fine-tune the wav2vec2-large model. The results are shown in the following table. It can be seen that our Mega-TTS demonstrates superior performance with Chinese datasets. Due to the limited time constraints, we only conduct evaluations on the Chinese test set. Chinese audio examples and cross-lingual examples can be found in the "Additional Examples for Rebuttal" Section of our demo page: [https://boostprompt.github.io/boostprompt/](https://boostprompt.github.io/boostprompt/).

| Dataset Usage |    WER    |    SIM    |    DTW    |       QMOS        |       SMOS        |
| ------------- | :-------: | :-------: | :-------: | :---------------: | :---------------: |
| GT            |   2.26%   |     -     |     -     |   4.02$\pm$0.07   |   4.23$\pm$0.09   |
| Baseline-10s  |   5.76%   |   0.906   |   21.43   |   3.87$\pm$0.09   |   4.01$\pm$0.08   |
| VALL-E-10s    |   7.01%   |   0.901   |   19.52   |   3.80$\pm$0.13   |   3.98$\pm$0.08   |
| Ours-10s      | **3.13%** | **0.914** | **18.40** | **3.92$\pm$0.10** | **4.05$\pm$0.08** |

Reviewer comment (Evaluation):
The authors have not provided any information about the inference times of Mega-TTS compared to other models. This omission makes it difficult to evaluate the model's performance in real-world scenarios where speed may be as important as accuracy.

Author response:
**[About the inference times and computational requirements of Mega-TTS compared to other models] (Question 2)**  
We agree that the computational requirements are crucial for understanding the trade-offs involved in using Mega-TTS. In Table 1, we have provided the real-time factor (RTF),  the time (in seconds) required for the system to synthesize one-second waveform, in the inference stage of Mega-TTS compared to other models. Besides, we have also compared the total number of parameters in Table 1 and provided the computational requirements for training Mega-TTS in Section 4.1.

Reviewer comment (Experiments):
How does Mega-TTS handle non-English languages or different dialects? Exploring this could help assess the generalizability of the model across various linguistic contexts.

Author response:
**[About the performance with different languages] (Question 1)**  
Here, we train our Mega-TTS model on the multilingual *WenetSpeech dataset (Chinese) + LibriLight dataset (English)* and use the *AISHELL-3 dataset (Chinese)* as the test set. We split the dataset and make evaluations according to Section 4.2 in our paper. To evaluate the WER for the Chinese dataset, we fine-tune the wav2vec2-large model. The results are shown in the following table. It can be seen that our Mega-TTS demonstrates superior performance with Chinese datasets. Due to the limited time constraints, we only conduct evaluations on the Chinese test set. Chinese audio examples and cross-lingual examples can be found in the "Additional Examples for Rebuttal" Section of our demo page: [https://boostprompt.github.io/boostprompt/](https://boostprompt.github.io/boostprompt/).

| Dataset Usage |    WER    |    SIM    |    DTW    |       QMOS        |       SMOS        |
| ------------- | :-------: | :-------: | :-------: | :---------------: | :---------------: |
| GT            |   2.26%   |     -     |     -     |   4.02$\pm$0.07   |   4.23$\pm$0.09   |
| Baseline-10s  |   5.76%   |   0.906   |   21.43   |   3.87$\pm$0.09   |   4.01$\pm$0.08   |
| VALL-E-10s    |   7.01%   |   0.901   |   19.52   |   3.80$\pm$0.13   |   3.98$\pm$0.08   |
| Ours-10s      | **3.13%** | **0.914** | **18.40** | **3.92$\pm$0.10** | **4.05$\pm$0.08** |

Reviewer comment (Evaluation):
What are the specific computational requirements of Mega-TTS compared to other models? This information is crucial for understanding the trade-offs involved in using Mega-TTS.

Author response:
**[About the inference times and computational requirements of Mega-TTS compared to other models] (Question 2)**  
We agree that the computational requirements are crucial for understanding the trade-offs involved in using Mega-TTS. In Table 1, we have provided the real-time factor (RTF),  the time (in seconds) required for the system to synthesize one-second waveform, in the inference stage of Mega-TTS compared to other models. Besides, we have also compared the total number of parameters in Table 1 and provided the computational requirements for training Mega-TTS in Section 4.1.

Reviewer comment (Experiments):
How does the performance of Mega-TTS scale with the size of the training data? Understanding this can provide insights into how well the model might perform in scenarios with varying amounts of available data.

Author response:
**[About the performance of Mega-TTS scale with different sizes of the training data]**  
Here we evaluated the performance of our Mega-TTS scale with varying amounts of available data. In this experiment, all of the systems use 3 seconds of speech prompts. The results are shown in the following table. We can see that Mega-TTS performs well with different sizes of the training data, while VALL-E fails to obtain satisfying results when the data is insufficient. We also scale our Mega-TTS with 200K hours of speeches and the results can be found in Appendix B in the original version of the paper.

| Model    |     Dataset Usage      |     WER     |     SIM     |     DTW     |        QMOS         |        SMOS         |
| -------- | :--------------------: | :---------: | :---------: | :---------: | :-----------------: | :-----------------: |
| VALL-E   | LibriLight (1k hours)  |   15.73%    |    0.782    |    54.05    |    3.63$\pm$0.13    |    3.52$\pm$0.10    |
| VALL-E   | LibriLight (10k hours) |    8.29%    |    0.870    |    38.93    |    3.84$\pm$0.12    |    3.65$\pm$0.12    |
| VALL-E   | LibriLight (60k hours) |    5.83%    |    0.885    |    36.59    |    3.90$\pm$0.12    |    3.71$\pm$0.10    |
| Mega-TTS | LibriLight (1k hours)  |    4.69%    |    0.861    |    43.07    |    3.88$\pm$0.11    |    3.61$\pm$0.10    |
| Mega-TTS | LibriLight (10k hours) |    2.91%    |    0.882    |    35.76    |    3.95$\pm$0.11    |    3.70$\pm$0.12    |
| Mega-TTS | LibriLight (60k hours) | ***2.46%*** | ***0.898*** | ***34.39*** | ***3.98$\pm$0.09*** | ***3.77$\pm$0.08*** |

Reviewer comment (Experiments):
How does Mega-TTS handle non-standard speech patterns such as shouting, laughing, or other forms of emotional expression? This question could illuminate the model's ability to accurately capture and reproduce a wider range of human speech nuances.

Author response:
**[About handling non-standard speech patterns]**  
We have generated some speech samples by directly utilizing non-standard speech voice prompts, such as laughing and other forms of emotional expression. The generated speech samples can be accessed in the "Additional Examples for Rebuttal" Section of our demo page: [https://boostprompt.github.io/boostprompt/](https://boostprompt.github.io/boostprompt/). From the demo page, we can see that Mega-TTS can accurately capture timbre information even from a short clip of a non-standard speech pattern. This demonstrates its effectiveness in "capturing and reproducing a wider range of human speech nuances". Again, we greatly appreciate your valuable suggestions and assistance.