Paper: Vec-Tok Speech: Speech Vectorization and Tokenization for Neural Speech Generation (ICLR-2024)

Reviewer comment (Experiments):
The paper may need more experimental results to improve its soundness. The state-of-the-art (SOTA) model for cross-lingual zero-shot TTS is not VALL-E X, but Mega-TTS[1]. Including such a comparison would make the paper sounder.

Author response:
Mega-TTS [2] is a powerful zero-shot TTS model. However, the MRTE and PLM modules in Mega-TTS require speaker ID during the training phase, which is not available in our large-scale crawled dataset, and labelling the entire dataset with a speaker diarization model and extracting their phoneme-level alignments surpasses our limited resources' capacity. 

Moreover, according to the [review guideline](https://iclr.cc/Conferences/2024/ReviewerGuide) of ICLR,
> if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper.

And due to limited resources, we have not invested our resources in the replication of Mega-TTS.

Reviewer comment (Experiments):
The paper may need more experimental results to improve its soundness. The reconstruction quality of the proposed codec is not assessed thoroughly.

Author response:
We are sorry for insufficient training in vocoder, which leads to low reconstruction quality. We have supplemented the speech reconstruction demo and qualitative results of reconstruction quality, which proves our codec is effective. We test the PESQ and STOI metrics in 100 speech clips from our test set. The results are shown at next comment. 

The results indicate that our codec is capable of reconstructing speech with a low bitrate and minor loss of sound quality.

Reviewer comment (Evaluation):
The quality of the provided samples is not satisfactory when compared with the samples from [1] (demo link: https://mega-tts.github.io/demo-page/).

Author response:
Mega-TTS [2] is a powerful zero-shot TTS model. However, the MRTE and PLM modules in Mega-TTS require speaker ID during the training phase, which is not available in our large-scale crawled dataset, and labelling the entire dataset with a speaker diarization model and extracting their phoneme-level alignments surpasses our limited resources' capacity. 

Moreover, according to the [review guideline](https://iclr.cc/Conferences/2024/ReviewerGuide) of ICLR,
> if a paper was published (i.e., at a peer-reviewed venue) on or after May 28, 2023, authors are not required to compare their own work to that paper.

And due to limited resources, we have not invested our resources in the replication of Mega-TTS.

Reviewer comment (Novelty):
The abstract contains some overstatements. Indeed, this work is not the first work to unify multiple speech generation tasks ([1-4]).

Author response:
We clarify that a unified framework overriding more speech generation tasks is attracting attention [4,5]. However, these speech-generative models obtain good speech quality while the number of speech-generation tasks is limited.

Reviewer comment (Writing):
The abstract contains the statement: `"In contrast, the current speech generative models are still struggling regarding speech quality and task generalization."` which may be an overstatement.

Author response:
We clarify that a unified framework overriding more speech generation tasks is attracting attention [4,5]. However, these speech-generative models obtain good speech quality while the number of speech-generation tasks is limited.

Reviewer comment (Novelty):
The use of BPE for semantic tokens was first proposed in [5]. Including the reference and a discussion on this topic may be necessary.

Author response:
The article [3] became available on Arxiv on 12th October 2023. Unfortunately, we are unable to provide the reference or a discussion due to the ICLR submission deadline of 28th September 2023. So we will discuss this article here.

In the article [3], they use the deduplicated final layer representation of HuBERT base model,  in which the operation of deduplication leads to the loss of rhythmic information[1], and the final layer representation leads to worse pitch and energy reconstruction accoding to [6]. Therefore, this token will be difficult to reconstruct, and thus this representation is not suitable for speech generation tasks.

Reviewer comment (Evaluation):
Does the SCS metric measure the similarity between the generated audio and the original recording or the reconstructed samples?

Author response:
The SCS metric measures the similarity between the generated audio and the original recording.

Reviewer comment (Experiments):
Do consecutive repeating tokens impact the BPE efficiency? Could you provide the average token sequence length after deduplicating the tokens?

Author response:
Consecutive repeated tokens can affect the efficiency of the BPE algorithm. Nevertheless, these repeated tokens carry speech prosody such as rhythm [1].  Moreover, when we remove the repeated tokens, the language model would not be able to model the rhythms of speech, and we must use a duration module to model the rhythm. This poses a challenge to the Non-Autoregressive inverse K-means module. As a result, we have not provided the average length of the token sequence after deduplication.

Reviewer comment (Novelty):
The proposed model seems to be the combination of the existing technique: vectorization, tokenization, and BPE, without any novel architecture.

Author response:
The techniques already exist. We explore and introduce the pre-trained self-supervised model and the idea of inverse k-means into speech codec. We decouple speech into linguistic, para-linguistic, and non-linguistic features, trying to simplify the speech tokens for better performance of speech LMs. Thus, we can stimulate the potential of speech LM and build a speech generation framework easily applied to multiple speech generation tasks.

Reviewer comment (Novelty):
I am not sure it is novel enough to adapt Byte-Pair Encoding technique to compress the length of semantic tokens. Many existing works have leveraged the BPE technique in audio processing [1,2], so it is better to distinguish the proposed model from the ones that have utilized the BPE.

Author response:
Thank you for your suggestion. BPE has been investigated in the pre-training speech model [1] where BPE, deduplication, DP smoothing and various techniques for a less redundant representation. We explore the utilization of BPE in speech-generation tasks. Specifically, directly applying BPE to acoustic tokens leads to different sequence lengths for different layers of acoustic tokens. Applying BPE to semantic tokens benefits the speech LM from lower exposure bias and longer context coverage, which reduces the accumulative error due to the nature of the autoregressive language model.

Reviewer comment (Experiments):
I think even though the authors have presented various amount of downstream task, the performances of previous works that they are comparing with seem to be limited overall. For example in zero-shot tts, why not comparing with Voicebox?

Author response:
The large-scale speech generation model is developing rapidly, but there is very little open-source work or benchmark. Besides, the dataset used in each work is very different. This brings us a great challenge to reproduce. We successfully reproduced VALLE-X and other models with open-source implementations but failed in Voicebox.

Reviewer comment (Evaluation):
Moreover, there is no quantitative performance on speech denoising. Although the results are shown in the demo page, it is better to report the qualitative results with metrics like MCD, STOI, PESQ for performance reporting.

Author response:
Thank you for your suggestion. We have evaluated our framework in the speech denoising task. We use the synthetic test set with reverberation of DNS challenge-Interspeech 2021 as our test set, and we follow the evaluation metric of DNS challenge 2021 [2]. Additionally, we compare our result with some models trained explicitly with denoising objectives [3-6]. The results are shown as follows.

| Model | DNSMOS| | |
| --- | --- | --- | --- |
|  | SIG | BAK | OVL |
| Noisy [2] | 1.760 | 1.497 | 1.392 |
| Conv-TasNet [3] | 2.415 | 2.710 | 2.010 |
| Demucs [4] | 2.510 | 2.641 | 2.215 |
| Inter-SubNet [5] | 2.651 | 2.581 | 2.362 |
| CDiffuSE [6] | 2.541 | 2.300 | 2.190 |
| Vec-Tok Speech | 2.681 | 2.912 | 2.235 |


[1] Elkahky, Ali, et al. "Do Coarser Units Benefit Cluster Prediction-Based Speech Pre-Training?." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.

[2] Chandan K.A. Reddy et al., “INTERSPEECH 2021 Deep Noise Suppression Challenge,” in Proc. Interspeech, 2021, pp. 2796–2800.

[3] Yi Luo and Nima Mesgarani, “Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation,” IEEE ACM Trans. Audio Speech Lang. Process., vol. 27, no. 8, pp. 1256–1266, 2019.

[4] Alexandre D ́efossez, Gabriel Synnaeve, and Yossi Adi, “Real Time Speech Enhancement in the Waveform Domain,” in Proc. Interspeech, 2020, pp. 3291–3295.

[5] Jun Chen et al., “Inter-subnet: Speech enhancement with subband interaction,” in Proc. ICASSP. IEEE, 2023, pp. 1–5.

[6] Simon Welker, Julius Richter, and Timo Gerkmann, “Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain,” in Proc. Interspeech, 2022, pp. 2928–2932.

Reviewer comment (Writing):
The paper is poorly written. Section 3.2 is very hard to understand. I would suggest anything in the figure should be clearly defined in the method section. For example, "codec decoder". I also highly suggest the author describe tortose (Betker, 2023) in the related work. It seems some component is borrowed from here, but it's unclear which part is being used.

Author response:
We have revised section 3.2 and described TorToise-TTS (Betker, 2023) in the related work. We borrowed the contrastive Language-Voice pre-trained (CLVP) module from Tortoise-TTS. Thank you for your suggestion. We have carefully polished the writings and the figures in the updated manuscript.

Reviewer comment (Experiments):
One key contribution of the paper is to replace acoustic tokens in audioLM. However, these claims are very vague; are there any empirical or theoretical analyses to justify this claim?

Author response:
There are many articles that claim the acoustic tokens are too complex to predict for Speech LMs [3,4,5]. Therefore, Mel or continuous vectors are utilized to replace acoustic tokens in current works[3,7]. In addition, kNN-VC[1] demonstrated that speech waveforms can be reconstructed from features of WavLM. We follow these works using WavLM features to replace acoustic tokens.

Reviewer comment (Writing):
The citation regarding "a neural vocoder is used to reconstruct speech waveforms..." is unclear. I highly recommend the author fix citations like this—what is being cited here? The claim, the decoder architecture, or something else?

Author response:
The citation here intends to confirm that the waveforms generated by vocoders are highly generalizable outside of the training set. Thank you for your suggestion. We have carefully polished the writings and the figures in the revised manuscript to avoid misunderstandings. We have updated our manuscript and fixed these citations.

Reviewer comment (Novelty):
Quite complex and depends on previous pre-trained models (e.g. WavLM)

Author response:
Yes, we ablate WavLM during the review period. We choose the 6-layer feature of the XLSR model, which is a wav2vec 2.0 model pre-trained on a multilingual dataset. We find the performance of XLSR is similar to that of WavLM. We speculate that the shallow-layer feature has high generalization ability for various speech inputs [6].

Reviewer comment (Experiments):
No comparison with diffusion based models like NaturalSpeech2

Author response:
Indeed, it makes more sense to compare the diffusion model. However, since there is no open source code, we do not get a reasonable performance when we reproduce diffusion models  during experiments such as naturalspeech2 or voicebox. Therefore, we did not compare diffusion models.

Reviewer comment (Experiments):
Missing ablations make it difficult to understand which parts contribute to the performance of the model. While certain parts of the architecture are ablated for certain tasks, it would be nice to ablation results for each task to understand how modeling choices (BPE, inverse K-means) contribute to the performance. It would also be good to know how much WavLM contributes to the overall performance or if it is replaceable with any other model. Choices on the LM side (how necessary it is to produce 256 candidates and then rank them? etc) are not ablated at all.

Author response:
Thank you for your suggestion. Since we propose a multi-task framework, we validated the proposed module in several specific tasks, such as verifying the validity of inverse k-means in zero-shot VC and BPE in zero-shot TTS in Table 1, 2, and 3.  We supplement the ablation of the CLVP module and the WavLM model. The results are shown in next comment. 

In the process of our ablation experiments, we found that CLVP can reduce the frequency of extremely poor samples, which in turn increases the stability of the generation, and after removing the CLVP, we found that the extremely poor samples will affect the evaluation indexes to a large extent, so we reduced the number of samples selected by CLVP for ablation. As for the pre-training representation selection, we found that different representation selections have little effect on the model performance.

Reviewer comment (Evaluation):
Speech to speech translation results compare to one other prior work that outperforms the proposed architecture on BLEU and is missing all other metrics making comparison difficult.

Author response:
We try to compare our model with other S2ST models. However, the models either involve internal datasets [1,2] or have complex pre-training and fine-tuning pipelines [3,4]. The model trained on the same open-source  dataset as ours is GigaST [5]. However, it is a speech-to-text translation (S2TT) model; therefore only BLUE is available and comparable. As our model is a text-free S2ST model and transfers source speaking style and speaker timbre when translating, it is reasonable to obtain a lower BLEU score than an S2TT model (GigaST).