Paper: SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models (ICLR-2024)

Reviewer comment (Experiments):
My first concern is about the necessity for a unified speech tokenizer. In the last several sentence of page 1, the authors argue that "the multi-stage modeling approach is more complex, leading to several drawbacks such as error accumulation and slower processing speed". Is there any evidence to support the claim? In addition, I may somewhat disagree with how the paper describe the characteristic of "Semantic LM and Acoustic LM" in Table 1. In my view, Semantic LM does not generate Accurate Content but it is speed is much faster. And Acoustic LM does generate Accurate Content.

Author response:
### Q1: Necessity for a unified speech tokenizer
AudioLM [1] is a representative of Hierarchical LMs which is composed of three cascaded autoregressive models: Semantic modeling, Coarse acoustic modeling, and Fine acoustic modeling. The complex tri-stage autoregressive cascade approach is more prone to error accumulation and introduces a high time complexity. Recently, SoundStorm [2] replaces the last two stages of AudioLM with one NAR model, and the experimental results show they alleviating the issues of slow processing speed, and achieves better audio generation quality. However, the redundancy of information between semantic tokens and acoustic tokens adds unnecessary complexity and difficulty to the NAR modeling. So we proposed SpeechTokenizer, by sequentially decoupling semantic and acoustic information within the RVQ1 and RVQ2:8 frameworks, hoping to reduce the modeling difficulty of NAR. This has been proven in our Hierarchical LM comparative experiments.

 In terms of the advantage of Semantic LM over Acoustic LM in generating accurate content, the AudioLM [1] paper states, 'Using the model trained only on the acoustic tokens, we sample speech continuations from a prompt of 4 seconds. While both the recording conditions and the speaker identity from the prompt are preserved, the linguistic content is inconsistent, and often akin to babbling.' This demonstrates the lack of superiority of Acoustic LM in generating accurate content. Furthermore, our paper's Section 2.3 analysis reveals that in SLMTokenBench, semantic tokens have better text alignment than acoustic tokens, suggesting a more effective mapping relationship with the content for semantic tokens.

Reviewer comment (Experiments):
The second concern, which is also my major concern, is about whether the complicated training pipeline really leads to better performance in zero-shot TTS than Hierarchical LM. If using VALL-E with the semantic token for HuBERT and RVQ1:7 from EnCodec (It has the same bit rate as USLM), what is performance?

Author response:
### Q2: Comparsion with Hierarchical LM
We implemented a Hierarchical LM that combines Semantic token modeling with SoundStorm, and compared its performance with USLM's in zero-shot TTS, under the same data and experimental settings. We use HuBERT L9 units as semantic tokens and codes generated by EnCodec as acoustic tokens here. Given that the open-source EnCodec and HuBERT have different sampling rate, and aslo in order to ensure a fairer comparison, we used our own version of EnCodec trained on LibriSpeech, maintaining consistency with SpeechTokenizer. At the same time, we adjusted USLM to the same architecture, where we used SpeechTokenizer RVQ-1 for Semantic modeling, and SoundStorm then generates the remaining RVQ2-8 tokens conditioned on the input RVQ1 tokens. We followed the original paper in implementing SoundStorm’s model structure, mask scheme, and decoding strategy. We conduct evaluation on the VCTK dataset and the results are summarized in the Table below.
|  | AR (Transformer Decoder) |  | NAR（SoundStorm） |  | Evaluation |  |
| --- | --- | --- | --- | --- | --- | --- |
| Exp | Input | Output | Input | Output | WER↓  | Speaker Similarity ↑ |
| 1 USLM | text | SpeechTokenizer RVQ-1 | SpeechTokenizer RVQ-1 | SpeechTokenizer RVQ-2:8 | 4.49 | 0.88 |
| 2 Hierarchical LM | text | HuBERT units | HuBERT units | Encodec RVQ-1:8 | 7.18 | 0.79 |
| 3 Hierarchical LM | text | HuBERT units | HuBERT units | Encodec RVQ-1:7 | 7.5 | 0.56 |
| 4 Hierarchical LM | text | HuBERT units | HuBERT units | SpeechTokenizer RVQ-1:8 | 5.94 | 0.81 |

As shown in the Table, USLM (Exp 1) significantly outperforms Hierarchical LM (Exp 2 3 4) in both WER and Speaker Similarity. We attribute this improvement to the decoupling of information by SpeechTokenizer. In Hierarchical LM, the modeling process from HuBERT units to acoustic tokens still requires modeling all the information of speech in the acoustic tokens, which increases the modeling difficulty. However, USLM’s NAR model receives RVQ-1 tokens as input and needs to output RVQ-2:8 tokens that contain only paralinguistic information, significantly reducing the modeling complexity. 
We also observed that in Hierarchical LMs, generating only EnCodec's RVQ1-7 (Exp 3) leads to a decline in model performance, with an increase in WER and a decrease in Speaker Similarity.

[1] Borsos, Zalán, et al. "Audiolm: a language modeling approach to audio generation." IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).

[2] Borsos, Zalán, et al. "SoundStorm: Efficient Parallel Audio Generation." arXiv preprint arXiv:2305.09636 (2023).

Reviewer comment (Experiments):
At last, I am doubtful whether the SpeechTokenizer only works for AR + NAR model like VALL-E. As VALL-E only uses the first speech token in the AR model, putting more information on the first token may be beneficial to its performance so that the improvement of USLM in zero-shot TTS is much larger than the improvements of SpeechTokenzier in speech reconstruction. I am not sure such large improvements will still exist in Hierarchical model like AudioLM.

Author response:
### Q3: Applying SpeechTokenizer to Hierarchical LM
As shown in the Table above, Hierarchical LM with SpeechTokenizer (Exp 4) yields better results compared to with Encodec (Exp 2). We think this is partly because the task of HuBERT unit -> Speechtokenizer RVQ1 is simpler than HuBERT unit -> Encodec. Moreover, due to NAR's layered modeling of RVQ, the process of NAR modeling different layers of RVQ in Speechtokenizer is implicitly decoupled into two relatively independent processes: semantic modeling (HuBERT units -> SpeechTokenizer RVQ-1) and paralinguistic modeling (HuBERT units + SpeechTokenizer RVQ-1:(i - 1) -> RVQ-i). This reduces the complexity of modeling, enabling Hierarchical LM to achieve better results with Speechtokenizer than with Encodec.

Reviewer comment (Evaluation):
Can the author provide some explainations why the improvement of USLM in zero-shot TTS is much larger than the improvements of SpeechTokenzier in speech reconstruction (Table 4 v.s. Table 2)?

Author response:
### Q4: Explanation of larger improvement of USLM in zero-shot TTS
We attribute the performance improvement of USLM in zero-shot TTS to the information decoupling feature in SpeechTokenizer, rather than just improvements in speech reconstruction quality. Concretely, the information disentanglement strategy in SpeechTokenizer significantly simplifies both autoregressive (AR) and non-autoregressive (NAR) modeling in zero-shot TTS.

Firstly, for AR models, the RVQ-1 tokens in SpeechTokenizer align more easily with text compared to RVQ-1 tokens in EnCodec. In the codebook analysis section of our paper's Appendix F, we visualized the alignment between the phones and codes of SpeechTokenizer's and EnCodec's RVQ-1, and computed pnmi. It is clearly evident that the alignment of SpeechTokenizer's codes with phones is significantly higher than that of EnCodec. Therefore, theoretically, it is simpler for the model to learn phoneme to SpeechTokenizer RVQ-1 mapping than phoneme to EnCodec RVQ-1. 

Secondly, for non-autoregressive (NAR) models, the RVQ-2:8 tokens in SpeechTokenizer only contain acoustic details of the speech. This contrasts with EnCodec's RVQ-2:8 tokens, where various types of information are intermingled, making the tokens more difficult to predict. So the NAR model in USLM performs an easier task.

During the AR and NARtraining process, we recorded the model's top10 accuracy in predicting tokens on the validation set, as shown in the following table. USLM achieved a higher top10 accuracy than VALL-E in both AR and NAR models. This result validates our intuition.
|  | AR | NAR |
| --- | --- | --- |
| VALL-E | 0.77 | 0.57 |
| USLM | 0.99 | 0.7 |

Reviewer comment (Experiments):
The paper misses the comparison with Hierarchical speech language models as listed in the paper. if the comparison could be added, it would be more convincing.

Author response:
### Q1: Comparision with Hierarchical speech language model
We implemented a Hierarchical LM that combines Semantic token modeling with SoundStorm [1], and compared its performance with USLM's in zero-shot TTS, using the same data and experimental settings. We use HuBERT L9 units as semantic tokens and codes generated by EnCodec as acoustic tokens here. Given that the open-source EnCodec and HuBERT have different sampling rate, and aslo in order to ensure a fair comparison, we used our own version of EnCodec trained on LibriSpeech, maintaining consistency with SpeechTokenizer. At the same time, we adjusted USLM to the same architecture, where we used SpeechTokenizer RVQ-1 for Semantic modeling, and SoundStorm then generates the remaining RVQ2-8 tokens conditioned on the input RVQ1 tokens. We followed the original paper in implementing SoundStorm’s model structure, mask scheme, and decoding strategy. We conduct evaluation on the VCTK dataset and the results are summarized in the Table below.
|  | AR (Transformer Decoder) |  | NAR（SoundStorm） |  | Evaluation |  |
| --- | --- | --- | --- | --- | --- | --- |
| Exp | Input | Output | Input | Output | WER↓  | Speaker Similarity↑ |
| USLM | text | SpeechTokenizer RVQ-1 | SpeechTokenizer RVQ-1 | SpeechTokenizer RVQ-2:8 | 4.49 | 0.88 |
| Hierarchical LM | text | HuBERT units | HuBERT units | Encodec RVQ-1:8 | 7.18 | 0.79 |

As shown in the Table, USLM significantly outperforms Hierarchical LM in both WER and Speaker Similarity. We attribute this improvement to the decoupling of information by SpeechTokenizer. In Hierarchical LM, the modeling process from HuBERT units to acoustic tokens still requires modeling all the information of speech in the acoustic tokens, which increases the modeling difficulty. However, USLM’s NAR model receives RVQ-1 tokens as input and needs to output RVQ-2:8 tokens that contain only paralinguistic information, significantly reducing the modeling complexity.

Reviewer comment (Experiments):
My primary concern lies with the experimental section of the paper. Firstly, the authors trained SpeechTokenizer on the LibriSpeech set, while the EnCodec was trained on a mixed dataset of 10,000 hours. As a result, EnCodec should possess better generalization capabilities for speech reconstruction, particularly in the presence of noise. However, the authors only evaluated the performance on LibriSpeech, which is an in-domain test set for their method but falls outside the domain of EnCodec. This unfair experiment fails to demonstrate whether their method outperforms EnCodec. A fair experiment should involve testing the model on various test sets, especially those unseen during training.

Author response:
Thank you for your valuable suggestions on our experimental setting. We would like to clarify that the purpose of proposing SpeechTokenizer is not to achieve a Codec model that surpasses EnCodec in terms of speech reconstruction quality or robustness. Instead, our goal is to provide a more suitable speech discretization tool for building speech language models by unifying semantic tokens and acoustic tokens. Therefore, we train it on LibriSpeech but not a large and diverse speech dataset to develop a high-performance codec. Our experimental evaluation focuses on the disentanglement effects between different RVQ layers of SpeechTokenizer and its performance in speech language modeling. Due to space limitations, we did not design extensive and comprehensive evaluation experiments for codec-related tasks.
### Q1: comparision with Encodec: 
We evaluate SpeechTokenizer on the VCTK dataset, which is beyond our training set. As indicated in the table below, the experimental results show that SpeechTokenizer outperforms EnCodec on VCTK. Moreover, we noticed a recent paper [1] mentioning that incorporating semantic information into EnCodec enhances sound quality. This finding aligns with our experimental observations, further validating the effectiveness of our approach.
|  | VCTK |  |
| --- | --- | --- |
|  | WER↓  | VISQOL↑ |
| EnCodec(official) | 3.6 | 4.14 |
| SpeechTokenizer | 3.7 | 4.30 |

Reviewer comment (Evaluation):
How did you arrive at the WER number of 7.9 in Table 4? I recall that the VALL-E paper reported a WER of around 3.X on LibriSpeech. Additionally, other studies on LibriSpeech have achieved comparable or even lower WER values.

Author response:
### Q2: Explanation of WER number of 7.9 of VALL-E:
As VALL-E's model is not open-sourced, to ensure a fair comparison with USLM, we retrained VALL-E under the same experimental setup and data conditions. We believe there are two main reasons for the discrepancy between our self-trained version and the WER reported in the original VALL-E paper:
- Differences in the test set and prompt selection method: The original VALL-E [2] paper distinguished between VALL-E-continual and VALL-E based on whether the prompts and texts are semantically continuous. VALL-E-continual achieved a WER of 3.8 on the LibriSpeech dataset, compared to 5.9 for VALL-E. In our experiments, we did not use the continual prompt method, thus the appropriate WER to compare with is 5.9, not 3.8. Additionally, our test set was the VCTK dataset, for which VALL-E's paper did not report zero-shot TTS WER. However, they noted in the limitations section that 'The worse result on VCTK than LibriSpeech also implies insufficient coverage of accent speakers.' This suggests that VALL-E's performance on VCTK might be inferior to that on LibriSpeech. Therefore, it is reasonable that our self-trained VALL-E achieved a WER of 7.9 on VCTK, which is slightly weaker than its 5.9 performance on LibriSpeech.
- Differences in training data: The original VALL-E paper used a 60,000-hour LibriLight dataset for training, while we used a 40,000-hour multilingual LibriSpeech dataset. This difference in the amount of training data could lead to variations in performance.

Reviewer comment (Experiments):
In the context of the speech tokenizer, it is necessary to take into account additional baselines like SoundStream and the multi-band diffusion-based method, rather than solely comparing it with Encode.

Author response:
### Q3: Additional baselines like SoundStream
Since SoundStream and the multi-band diffusion-based codec do not have official open-source versions available, we are unable to directly compare our approach with these baselines.

Reviewer comment (Experiments):
Why the authors not directly cite the number in VALL-E paper for evaluation? I don't think it is a good idea to re-train VALL-E on a smaller training set.

Author response:
### Q4: not directly citing the number in VALL-E paper for evaluation
Since the VALL-E paper only report WER results on the LibriSpeech dataset, to cite numbers from the VALL-E paper and ensure a fair comparison, we evaluate the zero-shot TTS performance of USLM on the LibriSpeech dataset. Our evaluation experiment setting was completely identical to that in the VALL-E paper, using non-continual 3-second audio clips as prompts. The results are shown in the following table.
|  | WER↓  | SIM↑  |
| --- | --- | --- |
| VALL-E report | 5.9 | 0.580 |
| USLM | 5.4 | 0.675 |

 As can be seen, USLM outperforms VALL-E in both WER and SIM. Furthermore, we have provided audio demos on our demo page, where the superiority of our approach over VALL-E can be clearly heard.

Reviewer comment (Experiments):
It's not quite clearly explained why the authors chose the first RVQ to inject semantic information. Why not the later ones or even for all RVQs

Author response:
We chose to inject semantic information into the first RVQ because our goal was to unify the semantic token (hubert token) and the acoustic token (encoded token) within a single framework. Since the HuBERT token is a one-dimensional sequence, we decided to inject semantic information into only one RVQ layer to align it with the semantic token. Additionally, restricting semantic information to only RVQ-1 means that in the USLM's AR model, only the RVQ-1 token needs to be predicted, avoiding the issue of predicting long sequences.

Reviewer comment (Experiments):
It could be better if the authors can show some experiments using the new tokenizers for speech understanding tasks like recognition/speech translation with relatively large scale training data, and compare to Hubert/soundstream/encodec tokens.

Author response:
In our subsequent work, we plan to utilize the tokenizer with a substantial amount of data to develop a model capable of both speech understanding and speech generation.