Paper: CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech (ICLR-2024)

Reviewer comment (Experiments):
It's mentioned by the authors that different codeword rates can be compared. Including a detailed analysis of the tradeoff between RTF and performance in relation to varying codeword rates would greatly enhance the paper. Such an analysis would provide a clearer understanding of how changes in codeword rate impact both the efficiency and effectiveness of the model, offering valuable insights for optimizing model performance in practical applications.

Author response:
We sincerely appreciate your positive review and constructive feedback.

> `Regarding the tradeoff between efficiency and effectiveness in varying codeword rate`

As per your great suggestion, we conducted an ablation study to analyze the impact of varying codeword rates in the proposed system and addressed it in the first section of “General Response to AC and Reviewer”. The experimental results demonstrate a trade-off: *reducing the code emitting frequency degrades audio quality while increasing it diminishes language modeling performance*. We also reported on the inference speed of the entire system in relation to varying code frequencies. This finding illustrates that our main model operates at a balance point within this trade-off, efficiently managing both aspects.

Reviewer comment (Experiments):
It is not very clear to me why the probabilistic RVQ can lead to much compressed token sequence. For example, audiolm uses the 50Hz semantic token, while this work use a 10Hz codeword rate. It would be great if the author can explain or provide experimental results showing the effect of codeword emitting frequency.

Author response:
Thank you for pointing out the parts in our paper that were not explained clearly. The semantic tokens in AudioLM are a sequence of 50Hz, while our tokens consist of 32 sequences of 10Hz due to the use of 32-depth residual vector quantization. This means that when representing audio, we can achieve a higher "length" compression rate compared to AudioLM by compressing the audio into multiple sequences using RVQ. We have conducted experiments and evaluations regarding the codeword emitting frequency, which we addressed in the first section of “General Response to AC and Reviewer”. These results demonstrate that our main model operates in the sweet spot for the trade-off. We sincerely request you to review them for additional context.

Reviewer comment (Writing):
What's y and \hat{y} in Eq(7)? It looks like it is never defined?

Author response:
For the definitions of the variables, please refer to the second paragraph in Sec. 3.2.1. $y$ and $\hat{y}$ denote a mel-spectrogram and its reconstruction from the decoder, respectively.

Reviewer comment (Novelty):
Novelty. The VAE or latent diffusion models are not something new, and thus the overall novelty could be questioned.

Author response:
We would like to thank the reviewer for providing constructive feedback and spotting unclear points.

> `Contribution of this paper`

To address the reviewer's concerns and clarify our methodology, we emphasize the distinctions between our approach and similar existing generative methodologies, VQGAN and RQ-Transformer, among generative models such as VAE and latent diffusion.

1. **Introduction and utilization of probabilistic residual vector quantization**: This feature not only enhances codebook usage to improve reconstruction quality but also offers a principled approach to generate residual codes from latent language modeling within a variational framework.
2. **Employing a single autoregressive transformer**:  Our method proposes a process where a single autoregressive transformer can generate $N$ tokens at each decoding step by utilizing the probabilistic modeling of the residual vector quantizer, without the need for depth-transformers or non-autoregressive (NAR) transformer decoders. Our method shares similarities with VQGAN in that both involve 1) autoencoding of input data using VQVAE and 2) generation of the quantized latent representation using an autoregressive transformer. However, the vector quantizer in VQGAN employs plain vector quantization, producing a sequence of tokens, thereby enabling the transformer to model the discrete sequence with a softmax output. In contrast, RQ-Transformer and neural audio codec language models, which use $N$-depth residual vector quantization, produce $N$ sequences of tokens. This results in the need for not only an autoregressive transformer to decode along the time axis but also an additional depth-transformer to model $N$ tokens along the depth axis or a NAR transformer decoder to generate a token sequence at each decoding step.

Reviewer comment (Experiments):
Experimental comparison. The authors mostly compare their work with the outdated model YourTTS in English Librispeech, or without comparison in multilingual TTS. Thus, some competing models are better to include. The demos are too limited, it is recommended to include demo audios from other work for comparison.

Author response:
Moreover, as per your great suggestion, we have included samples from other competitive models (VALL-E, NaturalSpeech 2, and Mega-TTS) on our demo page for comparison. These samples were taken from each model's demo pages. We note that the small number of these additional samples is due to the lack of official codes or pre-trained models. We acknowledge that the performance analysis using this limited number of additional samples may not have statistical significance. Nevertheless, we kindly request the reviewer to consider these additional samples in their evaluation.

demo page: https://clam-tts-mos.s3.us-east-2.amazonaws.com/demo/index.html

Reviewer comment (Evaluation):
Results. In most tasks, the model cannot achieve state-of-the-art results but claims that in the paper, and thus the performance can be questioned.

Author response:
Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review. 

We acknowledge the observed performance gap of our method compared to Voicebox, particularly in cross-sentence tasks where models are required to generate outputs from discontinuous texts and partially truncated audio prompts. To address this issue, we have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) and durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. In response to the concerns you raised, we have provided detailed explanations in the second section of our “General Response to AC and Reviewer”.

Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

Reviewer comment (Novelty):
voicebox significantly outperforms this work. while this not necessarily detracts from the contributions of this work, the fact that it includes duration predictions and works on phones is not a great excuse as many other tts systems use this recipe

Author response:
Thank you for your constructive criticism, which highlights the weaknesses in our research. We are truly grateful for your thorough review. We acknowledge the observed performance gap of our method compared to Voicebox.

In response to the concerns, we would like to clarify that our model can easily incorporate phoneme or duration to improve the performance. We have empirically analyzed the performance improvements when incorporating phonemes as inputs (as done in VALL-E, Voicebox, and SPEAR-TTS) or durations (as in Voicebox and YourTTS) into our model. We also discuss methods that directly train on cross-sentence tasks, such as those employed by SPEAR-TTS. These investigations are critical in understanding the reasons behind our model's performance drop in cross-sentence tasks compared to other models. We provide the empirical results and detailed discussion in the second section of “General Response to AC and Reviewer”.

Nonetheless, it is important to emphasize why our approach uniquely relies on plain text input, unlike other approaches. This is due to our model's ability to seamlessly integrate a pre-trained large language model, which is trained on a diverse array of texts and tasks. This integration allows us to achieve performance comparable to other baselines that employ more complex modeling.

Reviewer comment (Writing):
the paper is quite dense and difficult to follow. claims such as parallel generation of multiple tokens are seen in abstract and conclusion but finding how this is actually done is not straight forward and the description of "parallel predictors" is obtuse.

Author response:
Thank you for pointing out the parts of our paper where the explanation was insufficient. We are grateful for your detailed feedback. In our work, the latent language model produces three distinct outputs, representing 1) the mixture weights, 2) the means of the mixture of Gaussian distribution, and 3) the probability of ending the generation. Initially, we described these three components collectively as “parallel predictors”. However, in light of your feedback, we have replaced this term in the text with the aforementioned description. Furthermore, we elucidate the decoding step in which multiple tokens are generated from the output mixture of Gaussians (MoG) distribution, consisting of the mixture weights and the means. During each decoding step, a latent vector sampled from this distribution is transformed into latent codes through the residual vector quantizer of Mel-VAE. The autoregressive decoding step concludes when the probability of ending the generation reaches or exceeds 0.5.

Reviewer comment (Novelty):
Missing references: one of the first speech discretization and speech vector quantization techniques was proposed in Vq-wav2vec (https://arxiv.org/abs/1910.05453)

Author response:
In response to your feedback, we have added the missing reference and corrected typos, as well as other areas requiring revision, in the updated version of the paper.

Reviewer comment (Writing):
section 5.2: ablation studty -> ablation study

Author response:
In response to your feedback, we have added the missing reference and corrected typos, as well as other areas requiring revision, in the updated version of the paper.

Reviewer comment (Experiments):
One of the central claims of this paper is that it manages to describe speech with discrete tokens at much lower frequency than many other approaches. While there is discussion about how lower frequency may lead to worse performance, it would be nice to see empirical ablations study of how frequency affects accuracy and performance of the system

Author response:
To address the critical concern you raised, we conducted an ablation study to analyze the impact of varying codeword rates in the proposed system and addressed it in the “General Response to AC and Reviewer”. The experimental results demonstrate a trade-off: reducing the code emitting frequency degrades audio quality while increasing it diminishes language modeling performance. This finding illustrates that our main model operates at a balance point in this trade-off. We sincerely request you to review them for additional context.

Reviewer comment (Experiments):
why are mel spectrograms necessary to obtain the desired compression? can you not progressively downsample raw wave forms to acheive the same effect?

Author response:
Our choice to use mel-spectrograms instead of raw waveforms as the input for the VAE is driven by a practical consideration. It requires substantially more memory and trainable parameters to train VAE with raw waveforms due to the extended input length. In contrast, using mel-spectrograms allows us to train VAE with considerably fewer resources as well as to use an off-the-shelf vocoder to convert mel-spectrograms to raw waveforms. Nonetheless, as the reviewer pointed out, pursuing higher reconstructed audio quality with the desired compression in an end-to-end manner with raw waveforms can be a promising direction for future work.