Paper: DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation (ICLR-2024)

Reviewer comment (Experiments):
Compared with FastSpeech2 and WaveGrad2, the proposed method consumes even more GPU memory in normal-length sentences.

Author response:
One of the benefits of the diffusive method and working directly on the waveform is the inherent stochasticity of the signal. Therefore, if we partition the signal into independent segments and then concatenate them, there's a possibility that one segment may be spoken in a certain prosody, while another segment may be spoken differently. In such cases, even if the concatenation is of excellent quality, the delivery of words may not be continuous in terms of prosody. Hence, the significance of autoregressive processes and the transfer of information over time. For models that lack stochasticity, the suggested method of concatenation may be suitable, but it comes with the drawback of determinism.

While very long signals like 512 words are indeed uncommon in everyday language usage (except in cases of reading an article or a book), it is noticeable that for an E2E model like Wavegrad2, the transition occurred even for texts of reasonable length (32 words). One possible strategy to decrease GPU consumption is to **simplify or substitute the denoiser architecture.** As an illustration, we executed a model with a lower number of residual channels (C=128). This resulted in a reduction in GPU consumption, although not to a sufficient extent. (A figure for the analysis is available on our Git repository) We haven't had the opportunity to conduct a comprehensive comparison regarding quality results, but initial observations suggest promising and good-sounding outcomes. Reducing the frame length could be considered as another method for decreasing GPU consumption, but we will address this in the response to question 3. 

We recognize the significance of optimizing the synthesis process, and it presents a direction for further research. Another potential research direction involves applying the autoregressive method to other time-dependent domains and assessing its effectiveness. For instance, in video synthesis, it's likely that GPU consumption over time may increase more significantly. Alternatively, the synthesis of DNA chains serves as an example for very long signal lengths. Exploring the application in these contexts presents interesting avenues for further research.

Reviewer comment (Reproducibility):
More details about the linguistic representation need to be clarified. In the conditional setting, is the linguistic representation ${y}^l$ aware of the whole sentence text information or just aware of the text information of the corresponding frame? Is there a length regulator to align the text representation to the sample points?

Author response:
Our model generates time and energy predictions by incorporating information from the entire text. One of the advantages of this approach is its ability to facilitate more stable conditioning frameworks. If the prediction were local, the entire speech would likely be less smooth or stable.
Regarding the length regulator, we consistently utilize $\alpha=1$ (normal speed) and extend the signal based on the predictions of the duration predictor to ensure alignment with the actual samples. We will provide further refinement and will change the model architecture figure accordingly.

Reviewer comment (Experiments):
What is the practical scenario of synthesizing extremely long sentences (>512 words)? Such long sentences are rare in natural language. Compared with FastSpeech2 and WaveGrad2, the proposed method consumes even more GPU memory in normal-length sentences, which is below 512 or 32 words. There are obvious baseline methods to generate long sentences, such as splitting sentences or linguistic representation into chunks, synthesizing them separately, then concatenating the waveform. Under this setting, both of FastSpeech2 and WaveGrad2 can generate unlimited waveform durations while preserving the computational resources, and with even lower computation. Could the proposed method outperform these baselines by scaling down the computation cost to match these models?

Author response:
One of the benefits of the diffusive method and working directly on the waveform is the inherent stochasticity of the signal. Therefore, if we partition the signal into independent segments and then concatenate them, there's a possibility that one segment may be spoken in a certain prosody, while another segment may be spoken differently. In such cases, even if the concatenation is of excellent quality, the delivery of words may not be continuous in terms of prosody. Hence, the significance of autoregressive processes and the transfer of information over time. For models that lack stochasticity, the suggested method of concatenation may be suitable, but it comes with the drawback of determinism.

While very long signals like 512 words are indeed uncommon in everyday language usage (except in cases of reading an article or a book), it is noticeable that for an E2E model like Wavegrad2, the transition occurred even for texts of reasonable length (32 words). One possible strategy to decrease GPU consumption is to **simplify or substitute the denoiser architecture.** As an illustration, we executed a model with a lower number of residual channels (C=128). This resulted in a reduction in GPU consumption, although not to a sufficient extent. (A figure for the analysis is available on our Git repository) We haven't had the opportunity to conduct a comprehensive comparison regarding quality results, but initial observations suggest promising and good-sounding outcomes. Reducing the frame length could be considered as another method for decreasing GPU consumption, but we will address this in the response to question 3. 

We recognize the significance of optimizing the synthesis process, and it presents a direction for further research. Another potential research direction involves applying the autoregressive method to other time-dependent domains and assessing its effectiveness. For instance, in video synthesis, it's likely that GPU consumption over time may increase more significantly. Alternatively, the synthesis of DNA chains serves as an example for very long signal lengths. Exploring the application in these contexts presents interesting avenues for further research.

Reviewer comment (Experiments):
Although there is a justification of the choice of a long frame length in the unconditional setting, the author did not explain why they also used such a long frame in the conditional setting. There is also no ablation study on this. Note that these two settings are different in aspect of the global information. The global information provided by the long frame length in the unconditional setting can be provided by linguistic representation ${y}^l$ in the conditional setting. If the frame length can be shortened without performance drop in the conditional setting, the GPU memory could be reduced further, but the proposed method would be like a vocoder rather than a TTS system.

Author response:
Indeed, for the unconditional case, choosing a long frame is more comprehensible since generating a continuous signal requires obtaining all the necessary information directly from the waveform. In the conditional model, choosing the frame length involved several considerations:
- Balancing between memory consumption and synthesis time: If the frame lengths were very small, the GPU consumption at any given time would indeed be lower. However, simultaneously, more serial rounds of diffusion would be required to generate a given text. This would result in a longer synthesis time.
- While the representation $y^l$ from the past is valuable, from our perspective, it cannot substitute conditioning on past waveform values. This limitation arises from the stochasticity inherent in diffusion models and the waveform - given a certain representation $y^l$, it can be heard in many prosodies. Hence, to achieve natural-sounding continuity, referencing the past waveform is essential.
- In certain cases, it might be acceptable to rely on a relatively recent history. However, for long phonemes or silent phonemes (which can extend up to 200 ms or more), ensuring continuity requires information from a more distant history of the waveform signal.
- During the model's development, we acknowledged the significance of processing complete phonemes. Consequently, when conditioning synthesis on phoneme segments (leading to shorter conditioning), the model may struggle to discern the relative portion of the resulting phoneme. Our observations indicate that working with full phonemes produces more robust and "healthier" results.

Reviewer comment (Experiments):
Weak baselines and limited reference: The author should acknowledge and compare their proposed model against other works in a relevant research direction (i.e., diffusion-based TTS), e.g., FastDiff, Diff-TTS, Grad-TTS, PriorGrad, ProDiff, etc. It is understandable that comparing with all the conventional methods is computationally prohibitive, yet the authors should at least discuss the pros and cons of these methods in comparison to DiffAR. The latest diffusion-based TTS model that the authors referred to was WaveGrad 2 (released in 2021), which is not up-to-date and not the SOTA in this research direction. I am skeptical whether this baseline is strong enough, and I suggest the authors to use a more recent TTS model as for a stronger baseline.

Author response:
Thank you for the comments. You brought up interesting points, and we would like to respond to them comprehensively. 

**TL;DR:**  The models **FastDIFF, DiffTTS, Grad-TTS, PriorGrad, and ProDiff**  utilize diffusion-based approaches, functioning either as acoustic models generating a mel-spectrogram (thus requiring a vocoder for waveform synthesis) or as diffusion-based vocoders. A common feature among most of these models is the aim to accelerate the diffusion process, often at the expense of audio quality, the stochasticity of synthesis, or the simplicity of the model.

**Q1:**

In hindsight, we acknowledge that our comparison to existing models may not have been sufficiently comprehensive. Considering your feedback,  we added a comparison and explanation here and in the paper for the models you mentioned and others.

Our goal was to develop an E2E model that, in addition to being constructed as a single unit, conducts the synthesis process directly on the waveform. We utilized qualitative metrics for comparison, such as MOS and MUSHRA, but also delved into examining the stochasticity in the model (as a "one-to-many" model) and the controllability of stylistic features. In our responses to question 2, we will also address the synthesis time comparison.

**DiffTTS** is among the pioneering models that integrate the Text-to-Speech (TTS) task with diffusion. It utilizes the DDIM sampling equation to enable faster inference speed for Mel-spectrogram generation. **To the best of our knowledge, there is no publicly available implementation of DiffTTS, but based on official samples in the Git repository,** it seems to produce good-sounding results. However, the use of a vocoder occasionally introduces metallic or noisy sounds.

**FastDIFF** incorporates noise scheduling algorithms and predicts faster sampling noise scheduling than the one used in training. The model is mainly employed as a vocoder component, but we recognize the potential for **FastDIFF-TTS** as an end-to-end model. **Given its official comparison in the paper and the Git repository, we decided to perform a comparison with ProDiff, a model written by joint authors and published later.**

**PriorGrad** serves both as a vocoder and as an acoustic model. It formulates the sampling and training procedures to utilize an adaptive prior derived from data statistics. As an acoustic model, PriorGrad achieves slightly inferior results compared to GradTTS with T=10 diffusion steps. **Consequently, we chose to compare our model's results to the Grad-TTS with T=1000 diffusion steps.**

**Grad-TTS** explicitly manages the trade-off between sound quality and inference speed. A notable modification involves initiating the diffusion process from $N(\mu, \sigma)$ rather than white noise.
**ProDiff** adopts the generator-based method and also incorporates knowledge distillation and the DDIM method to enable synthesis with only 2 diffusion steps.

To compare the quality of **Grad-TTS** and **ProDiff** , we conducted a MUSHRA test following the methodology described in the paper. **The models evaluated were Grad-TTS with T=100 diffusion steps and with ProDiff T=4.**

| Model      | MUSHRA |
| ----------- | ----------- |
| GT     | 70.0 &plusmn; 2.1     |
| DiffAR 200   | 66.6 &plusmn; 2.4      |
| DiffAR 1000   | **67.5 &plusmn; 2.3**     |
| ProDiff  | 64.4 &plusmn; 2.4      |


| Model      | MUSHRA |
| ----------- | ----------- |
| GT     | 73.7 &plusmn; 2.4       |
| DiffAR 200   | **69.4 &plusmn; 2.5**        |
| DiffAR 1000   | 67.7 &plusmn; 2.6      |
| Grad-TTS   | 68.5 &plusmn; 2.5      |


(Each table is based on a different set of listeners, hence GT is not the same)

Despite the advantage of fast synthesis, changing and shortening the diffusion process in both models led to a **decrease in their stochasticity and creativity as one-to-many models.** Regarding ProDiff, different syntheses for the same text were nearly identical in terms of pitch and energy values.
In the case of GradTTS, initiating the diffusion process not with white noise and significantly shortening the diffusion process diminishes the stochasticity of the model. In the example provided on the website, we can see that **using T=1000 steps produces almost identical draws. Therefore, even with numerous diffusion steps, the guidance is highly explicit, leaving minimal room for stochasticity.** Furthermore, it is not possible to directly control characteristics throughout the synthesis; instead, changes can only be made indirectly by adjusting phoneme duration.

**Grad-TTS mentions the option of an E2E model, but the results are not of high quality for meaningful comparison.**

Reviewer comment (Evaluation):
Implicit description on synthesis time: In section 5.4, while the section is titled "COMPUTATIONAL LIMITATIONS AND SYNTHESIS TIME", I expect to see exact numbers or a graph reporting the inference speed of DiffAR. However, the authors only show the memory usage of DiffAR and implicitly describe the limitation of DiffAR in synthesis time, which makes it unclear how much we need to sacrifice in efficiency to exchange for an improved quality. It would be unfair if we focus on quality and compare DiffAR with FastSpeech 2, since FS2 is clearly much more efficient in time. And, conceivably, as an autoregressive model similar to WaveNet, DiffAR should be quite sensitive to the length of the waveform to be generated.

Author response:
**Q2:**

In retrospect, we acknowledge that more explanation could have been given Regarding the synthesis time. In response to your question, we conducted the **RTF values** for the models we compare so far.


| Syntax      | RTF |
| ----------- | ----------- |
| Fastspeech2     | 2.79     |
| ProDiff  | 0.25     |
| DiffGAN  | 0.06      |
| GradTTS  | 3.24      |
| VITS  | 0.09     |
| Wavegrad2   | 5.14       |
| DiffAR  | 31.06      |


As mentioned earlier, there is a significant tradeoff, particularly in diffusion models, between synthesis time, synthesis quality, and the level of creativity or stochasticity. The primary focus of our paper was to introduce a novel approach which utilizes the information in the waveform (phase and amplitude). We aimed to develop a model capable of generating natural speech phenomena and facilitates stochasticity as a one-to-many model.
Following your comment, we acknowledge a potential research direction, alongside exploring architectural changes, involves maintaining synthesis quality and stochasticity while reducing synthesis time.

Reviewer comment (Experiments):
The paper has a somewhat limited experimental evaluation, particularly with respect to dataset size and variety, and lacks references or comparisons to recent relevant literature.

Author response:
As mentioned earlier, we acknowledge the possibility of having conducted a more comprehensive comparison with existing TTS models and regret not doing so sooner. We hope that the attached explanation provides a broader perspective on how our work is positioned within the current TTS landscape. It's important to emphasize that the choice of wavegrad2 was not arbitrary; it is an end-to-end model utilizing diffusion, which was published around the same time as VITS. However, due to its parallel signal generation, it has inherent limitations, resulting in significantly lower audio quality compared to our model. Recognizing the importance of an extensive comparison with more advanced models than fastspeech2 and wavegrad2, we have included additional results.

Reviewer comment (Experiments):
The paper fails to adequately analyse closely related variants which are cited but not evaluated against, specifically DiffTTS (Jeong et al., 2021) and GradTTS (Popov et al., 2021). Given their methodological similarity to DiffAR, it is important to differentiate the proposed method, but the only justification provided is possibly better representation compared to using MelSpectrograms. Without more comprehensive experiments (only comparisons to FastSpeech2 + HiFi GAN and Wavegrad 2 are presented), this is insufficient to establish superiority over acoustic feature-based variants.

Author response:
Thank you for the detailed comments. You raised interesting points, and we would like to respond to them comprehensively. In hindsight, we acknowledge that our comparison to existing models may not have been sufficiently comprehensive. Considering your feedback, we added a comparison and explanation here and in the paper for the models you mentioned and others. Furthermore, we answer here and in the paper the other concerns you raised.

**TL;DR:** The models DiffTTS, GradTTS, and DiffGAN-TTS generate a spectrogram, and subsequently, a vocoder is employed to produce the waveform, while our model is a single end-to-end model that directly generates a waveform. 
Below (and soon in the paper), we compare our model to DiffGAN, GradTTS, and VITS. The models DiffTTS, VALL-E, Voicebox, and EATS have no implementation, while YourTTS architecture is very similar to VITS and focuses on slightly different tasks than ours, namely Zero-shot-multi-speaker. Details below.

**Major:**

As mentioned, our objective was to introduce an E2E model that, beyond being constructed as a single unit, conducts the synthesis process **directly on the waveform ​**​(without replying on an intermediate representations such MFCC, Mel-spectrograms), hence can captures prosody (such as vocal fry). In the comparison, we referred to qualitative metrics such as MOS and MUSHRA, but **it was also important for us to examine the stochasticity/creativity** (as a "one-to-many" model) as well as the controllability over stylistic features.

The models DiffTTS, GradTTS, and DiffGAN-TTS all fall into the category of diffusion-based acoustic models. These models, given text input, generate a spectrogram, and subsequently, a vocoder is employed to produce the waveform. A common characteristic among most of these models is the desire to accelerate the diffusion process, often impacting audio quality, the stochasticity of synthesis, or the model's complexity.

**DiffTTS** is among the pioneering models that integrate the Text-to-Speech (TTS) task with diffusion. It utilizes the DDIM sampling equation to enable faster inference speed for Mel-spectrogram generation. **To the best of our knowledge, there is no publicly available implementation of DiffTTS, but based on official samples in the Git repository, it seems to produce good-sounding results.** However, the use of a vocoder occasionally introduces metallic or noisy sounds.

**Grad-TTS** explicitly manages the trade-off between sound quality and inference speed. A notable modification involves initiating the diffusion process from $N(\mu, \sigma)$ rather than white noise.
**DiffGAN-TTS** achieves a significant reduction in synthesis time by employing adversarial training of a GAN, which can occasionally introduce instability in the training process.

To compare the quality of **GradTTS** and **DiffGAN-TTS,** we conducted a MUSHRA test following the methodology described in the paper. **The models evaluated were GradTTS with T=1000 diffusion steps and DiffGAN-TTS with T=4.**


| Model      | MUSHRA |
| ----------- | ----------- |
| GT     | 73.7 &plusmn; 2.4       |
| DiffAR 200   | **69.4 &plusmn; 2.5**        |
| DiffAR 1000   | 67.7 &plusmn; 2.6      |
| GRAD-TTS   | 68.5 &plusmn; 2.5      |


| Model      | MUSHRA |
| ----------- | ----------- |
| GT     | 71.2 &plusmn; 2      |
| DiffAR 200   | **69.5 &plusmn; 2.1**       |
| DiffAR 1000   | 68.4 &plusmn; 2.2      |
| DiffGAN-TTS   | 68.0 &plusmn; 2.2      |

(Each table is based on a different set of listeners, hence GT is not the same)

**We also investigated the stochasticity of Grad-TTS and DiffGAN-TTS** by generating five syntheses of the same text. The charts illustrating these comparisons are available on the website and in the article. **In both cases, the energy and pitch values were found to be the same or nearly identical for different samples.** This indicates that, despite being diffusion models, the stochasticity in these models is reduced or non-existent. While the small manifold of the spectrogram compared to that of the waveform might contribute to the reduced stochasticity, it may not be the sole factor influencing the phenomenon.

In the case of GradTTS, initiating the diffusion process not with white noise and significantly shortening the diffusion process diminishes the stochasticity of the model. In the example provided on the website, we can see that using **T=1000 steps produces almost identical draws. Therefore, even with numerous diffusion steps, the guidance is highly explicit, leaving minimal room for stochasticity.**

DiffGAN-TTS drastically reduces synthesis time by minimizing the number of diffusion steps, **resulting in an outcome that closely resembles a deterministic model.**

For the models DiffTTS, GradTTS and DiffGAN-TTS, explicit control over signal properties. While in some cases, phoneme duration can be indirectly adjusted, it remains the extent of controllability.

Reviewer comment (Novelty):
An entire class of models that have established state-of-the-art performance (VITS → YourTTS → VALL-E → Voicebox) are not mentioned. All of these outperform the baselines used in the paper, meaning the work fails to position itself and articulate tradeoffs in relation to recent literature.

Author response:
We turn now to the models - **EATS, VITS, YourTTS, VALL-E and Voicebox**, we would like to conduct a similar comparison.

EATS, although a pioneer in E2E models, exhibits performance below the baselines. Moreover, it has a relatively complex architecture and is challenging to reproduce. Due to the absence of a public implementation, we decided to focus on the other models.

**VITS** and **YourTTS** share a similar E2E architecture utilizing VAE, Normalizing Flow, MAS algorithm, and adversarial training for the TTS task. While VITS addresses the TTS scenario, YourTTS focuses on the Zero-One-shot task with multiple speakers and languages. Both models employ a stochastic time predictor to synthesize speech with diverse rhythms. However, for the same text the samples exhibit energy and pitch with very similar but shifted frames, as evidenced by examples attached to the paper and the Git repository. We should note that in the training of VITS and YourTTS, the latent variables z are based on linear spectrograms obtained by stft, so the synthesis doesn't directly work on the waveform. The reconstruction loss also involves the spectrogram, and the adversarial loss on the output may reduce stability during learning.

**In terms of qualitative metrics,** we compared VITS as it is more closely related to the examined task. We will elaborate on the extension to multiple speakers in our responses to the questions. 

| Syntax      | MUSHRA |
| ----------- | ----------- |
| GT     | 74.9 &plusmn; 2.2     |
| DiffAR 200   | 69.1 &plusmn; 2.2       |
| DiffAR 1000   | **71.5 &plusmn; 2.2**      |
| VITS  | 69.0 &plusmn; 2.3      |


**VALL-E** and **Voicebox** represent state-of-the-art models trained on large-scale datasets. Voicebox can be utilized as a zero-shot TTS model by infilling missing parts in the Mel-spectrogram and as an acoustic TTS model. VALL-E is an E2E model that uses a Codec code, working indirectly on the waveform, to generate speech and handle the zero-one-shot task.

While these models excel on large-scale datasets, replicating their success on LJspeech and VCTK proved challenging. Voicebox lacks an available implementation, and unofficial samples suggest that VALL-E faces difficulties in generalizing well when trained only on Lj-speech. **We want to emphasize that our primary focus was introducing a novel approach and exploring the benefits of working directly on the waveform.** We acknowledge that with more resources, it could be feasible to model additional phenomena like background noise, potentially leading to improved results.

Reviewer comment (Experiments):
The model is trained on a single speaker dataset with limited samples, in contrast to larger multi-speaker datasets such as LibriTTS (Kahn et al., 2020), LibriSpeech, VCTK (Veaux et al., 2016), and LibriTTS (Zen et al., 2019).

Author response:
- As previously highlighted, our article primarily addresses single-speaker synthesis. Nevertheless, we recognize the potential for extending the model to accommodate multiple speakers and varied prosodies. One approach involves leveraging the model's versatility by integrating the speaker's embedding into the synthesis process. **We have explored this option using Titanet embedding and datasets like VCTK/LibriTTS** with detailed insights provided on GIT regarding results and architecture structure. Additionally, a combination of classifier guidance or free classifier guidance presents an alternative method for introducing features such as emotion.

Reviewer comment (Experiments):
Quantitative/qualitative comparisons are only made against models such as FastSpeech2 + HiFi GAN and Wavegrad 2, which were not SOTA even as of January 2023. Important references like VITS, YourTTS, and VALL-E are missing. (The reviewer acknowledges the difficulty of obtaining some implementations, as noted in the footnote about FastSpeech 2s and Wave-Tacotron.)

Author response:
As mentioned earlier, we acknowledge the possibility of having conducted a more comprehensive comparison with existing TTS models and regret not doing so sooner. We hope that the attached explanation provides a broader perspective on how our work is positioned within the current TTS landscape. It's important to emphasize that the choice of wavegrad2 was not arbitrary; it is an end-to-end model utilizing diffusion, which was published around the same time as VITS. However, due to its parallel signal generation, it has inherent limitations, resulting in significantly lower audio quality compared to our model. Recognizing the importance of an extensive comparison with more advanced models than fastspeech2 and wavegrad2, we have included additional results.

Reviewer comment (Experiments):
The authors claim to benchmark against “the most high-performing and relevant models” without defining how relevance is established. Based on alternative methodologies and/or performance, models such as DiffTTS, GradTTS, DiffGAN-TTS, VITS, and YourTTS are also extremely viable candidates. Even a qualitative written explanation of the critical tradeoffs between this work and these models would significantly clarify the contribution.

Author response:
Thank you for the detailed comments. You raised interesting points, and we would like to respond to them comprehensively. In hindsight, we acknowledge that our comparison to existing models may not have been sufficiently comprehensive. Considering your feedback, we added a comparison and explanation here and in the paper for the models you mentioned and others. Furthermore, we answer here and in the paper the other concerns you raised.

**TL;DR:** The models DiffTTS, GradTTS, and DiffGAN-TTS generate a spectrogram, and subsequently, a vocoder is employed to produce the waveform, while our model is a single end-to-end model that directly generates a waveform. 
Below (and soon in the paper), we compare our model to DiffGAN, GradTTS, and VITS. The models DiffTTS, VALL-E, Voicebox, and EATS have no implementation, while YourTTS architecture is very similar to VITS and focuses on slightly different tasks than ours, namely Zero-shot-multi-speaker. Details below.

**Major:**

As mentioned, our objective was to introduce an E2E model that, beyond being constructed as a single unit, conducts the synthesis process **directly on the waveform ​**​(without replying on an intermediate representations such MFCC, Mel-spectrograms), hence can captures prosody (such as vocal fry). In the comparison, we referred to qualitative metrics such as MOS and MUSHRA, but **it was also important for us to examine the stochasticity/creativity** (as a "one-to-many" model) as well as the controllability over stylistic features.

The models DiffTTS, GradTTS, and DiffGAN-TTS all fall into the category of diffusion-based acoustic models. These models, given text input, generate a spectrogram, and subsequently, a vocoder is employed to produce the waveform. A common characteristic among most of these models is the desire to accelerate the diffusion process, often impacting audio quality, the stochasticity of synthesis, or the model's complexity.

**DiffTTS** is among the pioneering models that integrate the Text-to-Speech (TTS) task with diffusion. It utilizes the DDIM sampling equation to enable faster inference speed for Mel-spectrogram generation. **To the best of our knowledge, there is no publicly available implementation of DiffTTS, but based on official samples in the Git repository, it seems to produce good-sounding results.** However, the use of a vocoder occasionally introduces metallic or noisy sounds.

**Grad-TTS** explicitly manages the trade-off between sound quality and inference speed. A notable modification involves initiating the diffusion process from $N(\mu, \sigma)$ rather than white noise.
**DiffGAN-TTS** achieves a significant reduction in synthesis time by employing adversarial training of a GAN, which can occasionally introduce instability in the training process.

To compare the quality of **GradTTS** and **DiffGAN-TTS,** we conducted a MUSHRA test following the methodology described in the paper. **The models evaluated were GradTTS with T=1000 diffusion steps and DiffGAN-TTS with T=4.**


| Model      | MUSHRA |
| ----------- | ----------- |
| GT     | 73.7 &plusmn; 2.4       |
| DiffAR 200   | **69.4 &plusmn; 2.5**        |
| DiffAR 1000   | 67.7 &plusmn; 2.6      |
| GRAD-TTS   | 68.5 &plusmn; 2.5      |


| Model      | MUSHRA |
| ----------- | ----------- |
| GT     | 71.2 &plusmn; 2      |
| DiffAR 200   | **69.5 &plusmn; 2.1**       |
| DiffAR 1000   | 68.4 &plusmn; 2.2      |
| DiffGAN-TTS   | 68.0 &plusmn; 2.2      |

(Each table is based on a different set of listeners, hence GT is not the same)

**We also investigated the stochasticity of Grad-TTS and DiffGAN-TTS** by generating five syntheses of the same text. The charts illustrating these comparisons are available on the website and in the article. **In both cases, the energy and pitch values were found to be the same or nearly identical for different samples.** This indicates that, despite being diffusion models, the stochasticity in these models is reduced or non-existent. While the small manifold of the spectrogram compared to that of the waveform might contribute to the reduced stochasticity, it may not be the sole factor influencing the phenomenon.

In the case of GradTTS, initiating the diffusion process not with white noise and significantly shortening the diffusion process diminishes the stochasticity of the model. In the example provided on the website, we can see that using **T=1000 steps produces almost identical draws. Therefore, even with numerous diffusion steps, the guidance is highly explicit, leaving minimal room for stochasticity.**

DiffGAN-TTS drastically reduces synthesis time by minimizing the number of diffusion steps, **resulting in an outcome that closely resembles a deterministic model.**

For the models DiffTTS, GradTTS and DiffGAN-TTS, explicit control over signal properties. While in some cases, phoneme duration can be indirectly adjusted, it remains the extent of controllability.

Reviewer comment (Evaluation):
There are no benchmarks on synthesis speed (Real Time Factor), though given the use of DDPM + AR, the model will likely be quite slow.

Author response:
- We have attached **RTF values** compared to the other models. As mentioned, employing DDPM+AR and working directly on the waveform comes with a cost in terms of synthesis time. Recognizing this tradeoff, we acknowledge the potential for follow-up work to explore methods for reducing processing time. One approach could involve modifying the denoiser architecture to expedite the process. Alternatively, considering more conventional methods, such as DDIM, presents its own set of advantages and disadvantages.


| Syntax      | RTF |
| ----------- | ----------- |
| Fastspeech2     | 2.79     |
| ProDiff  | 0.25     |
| DiffGAN  | 0.06      |
| GradTTS  | 3.24      |
| VITS  | 0.09     |
| Wavegrad2   | 5.14       |
| DiffAR  | 31.06      |

Reviewer comment (Experiments):
There is no discussion on extending the model to multi-speaker scenarios and other prosodies; this is only briefly mentioned in the conclusion as future work.

Author response:
- As previously highlighted, our article primarily addresses single-speaker synthesis. Nevertheless, we recognize the potential for extending the model to accommodate multiple speakers and varied prosodies. One approach involves leveraging the model's versatility by integrating the speaker's embedding into the synthesis process. **We have explored this option using Titanet embedding and datasets like VCTK/LibriTTS** with detailed insights provided on GIT regarding results and architecture structure. Additionally, a combination of classifier guidance or free classifier guidance presents an alternative method for introducing features such as emotion.