Paper: RepCodec: A Speech Representation Codec for Speech Tokenization (ICLR-2024)

Reviewer comment (Writing):
In the introduction, the term "semantic quality" is mentioned. For clarity and the benefit of readers, could you provide a definition on what "semantic quality" encompasses?

Author response:
We believe we do not directly mention  "semantic quality". Instead, we mention the “quality of semantic tokens”, which refers to suitability of the semantic tokens for downstream tasks like ASR or TTS. When integrating LLMs with semantic tokens, we need to discretize speech into tokens to perform different downstream tasks. Some discrete tokens may not be suitable for downstream tasks and leads to inferior downstream task performance.

Reviewer comment (Experiments):
In previous works such as VALLE, AuidoLM, and PolySpeech mentioned in this paper, they all use the k-means clustering method to obtain semantic tokens. For the speech generation task (TTS, VC), the semantic information is important in discrete tokens, but the other encoded information such as speaker timbre is harmful to the task. Given the importance of speaker timbre information for downstream speech generation tasks, it would be of great value if a verification for speaker information of the encoded features is incorporated.

Author response:
We agree with the reviewer that the semantic units should carry as much semantic information as possible and as less speaker information as possible. Otherwise they may affect the speaker effect brought by other components, e.g. speaker embedding, acoustic tokens. We verify the speaker information carried by our units and the baseline k-means units through the voice conversion task:
    
    | Methods                   | Speaker Similarity ↑| MOS↑ |
    |:---|:---:|:---:|
    |k-means(baseline)     |0.771                    |2.82±0.36|
    |RepCodec                 |0.764                    |3.48±0.74|

    In our voice conversion experiment, the speaker characteristics of the generated speech is controlled by the speaker embedding. However, the speaker information carried by the semantic units (if any) may also affect the result. In the above table, we measure the similarity of the voice converted speech and the original speech from the same speaker. A higher similarity score implies less speaker information carried by the units. It can be seen that the similarity scores are very close but RepCodec units have a much higher MOS score due to the improved semantic part. This means that RepCodec does not carry extra speaker information which is harmful to the task.

Reviewer comment (Writing):
Section II.Speech Tokenization. "However, the discretization step of k-means discards plenty of information of the speech“. Can you give some examples so that readers can better understand the limitations of k-means?

Author response:
The limitation of k-means is reflected in the degraded performance of the ASR modeling. Even if we train the seq-to-seq model with large amounts of data, using k-means tokens as input always results in much worse WER. The explanation to this empirical observation is that k-means discards some information of speech and makes the speech recognition of some similar words impossible so that the WER is inferior to using raw speech as input.

Reviewer comment (Writing):
The first paragraph in Sectiion III. "In AudioLM (Borsos et al., 2023), the WER is dramatically increased from 2.5% to 6.0% by using the discrete tokens of k-means from w2v-BERT XL". What does this number 2.5% refer to?

Author response:
We refer to Table II of the AudioLM paper, where WER of the original speech is 2.5%. However, the reconstructed speech using k-means has a much worse WER of 6.0%.

Reviewer comment (Writing):
In Equation 5, some symbols are not defined.

Author response:
Thanks for your advice. We will add explanations to symbols $\tilde{n}_k$ and $e_k$ in our revision. They represent the moving average of EMA. $\tilde{n}_k$ represents the moving average of the number of clusters $k$ and $e_k$ represents the moving average of the code book of cluster $k$.

Reviewer comment (Evaluation):
One question not related to the proposed method. Why does the performance gap change among different speech encoders after VQ and K-means quantization compared with the original speech representations, especially whisper. In other words, what kind of representations are suitable for clustering?

Author response:
From our experiments, we believe that encoders trained with SSL objective function may be more suitable for clustering. Whisper is trained by end-to-end encoder-decoder architecture so that its representation from the encoder may require complicated transformation to be decoded as text. Therefore, its representation may be less suitable for clustering.

Reviewer comment (Evaluation):
Can you provide some samples of the speech resynthesis?

Author response:
We have uploaded some resynthesized speech to supplementary material.

Reviewer comment (Novelty):
Overall, this paper lacks novelty, as compared to SouldStream, it simply replaces the input from raw waveform with SSL representations.

Author response:
**“Novelty of the paper”**
  
    We believe our paper proposes completely new methods to generate semantic units and provides new understanding of the quality of the semantic units. Before RepCodec, semantic units can only be produced by traditional k-means methods following HuBERT [1], which was proposed about two years ago. Our paper, despite technical similarity to SoundStream, provides a novel method that largely outperforms long existing k-means clustering. 

    In addition, we also provide a new understanding about the semantic units. In previous papers such as HuBERT [1] and AudioLM [2], similarity to phonemes is used to measure the quality of the semantic units. Our paper, however, shows that PNMI is not the only metric for evaluating the semantic tokens. Instead, the reconstruction loss provides more meaningful indication for the performance of downstream tasks. It brings new insight for future study of semantic tokens.

Reviewer comment (Writing):
Equation (5) lacks sufficient explanation. I am unsure of its correctness as neither ${\overset{\sim}{n_k}}$ nor $\mathbf{e}_{i}$ is adequately defined or explained.

Author response:
**“Confusing details of the paper: Equation (5)”**

    Thanks for your advice. We will add explanations to symbols $\tilde{n}_k$ and $e_k$ in our revision. They represent the moving average of EMA. $\tilde{n}_k$ represents the moving average of the number of clusters $k$ and $e_k$ represents the moving average of the code book of cluster $k$.

Reviewer comment (Theory):
Shouldn't equation (7) be  
$ F^* = \arg\max_F p(\mathbf{y}|\mathbf{s}) = \arg\max_F \prod_{i=1}^{m} p(y_i|y_{<i}, \mathbf{s}) $?

Author response:
**“Confusing details of the paper: Equation (7)”**
 
    Thanks for pointing out our typo. We will fix it in our revision.

Reviewer comment (Experiments):
It would be better to include a more in-depth analysis of the weights ($\lambda_{r}$, $\lambda_{q}$) of reconstruction loss and quantization loss.

Author response:
**“Ablation Study of $(\lambda_q, \lambda_r)$”**

    As shown in the following Table, we change the weights of reconstruction loss $\lambda_r$ to train the encoder of RepCodec and report its WER on the downstream task of ASR modeling with representation from the 18th layer of data2vec large model. The results show that RepCodec is robust to the changes of the weights of $(\lambda_q, \lambda_r)$.
    | $\lambda_q$ | $\lambda_r$ | PNMI |Reconstruction Loss| WER of ASR Modeling |
    |:---|:---:|:---:|:---:|:---:|
    |1.0|30.0| 0.3689 | 0.2091 | 2.91 |
    |1.0|45.0| 0.3714 | 0.2100 | 2.87 |
    |1.0|60.0| 0.3697 | 0.2090 | 2.83 |

Reviewer comment (Evaluation):
Is WER a common metric for speech resynthesis?

Author response:
**“Using WER to evaluate speech resynthesis”**
 
    We follow previous papers and use WER to evaluate the quality of semantic units such as mHubert [3] and AudioLM [2]. As the role of semantic units is to provide the semantic information of the speech, the best way to evaluate the semantic information is to measure the content, i.e. ASR-WER, of the output speech.

Reviewer comment (Evaluation):
Insufficient evaluation metrics. The research predominantly relies on WER as the principal evaluation metric for the performance of semantic speech tokens. To make a compelling case for the proposed method's superiority, it's essential to include other evaluation metrics such as speaker similarity, F0 error, or mean-opinion score in the speech resynthesis experiments.

Author response:
First, our evaluation metrics follow previous papers on evaluating the quality of semantic units such as mHubert [1] and AudioLM [2]. They all use word error rate (WER) to measure the quality of semantic units. As the role of semantic units is to provide the semantic information of the speech, the best way to evaluate the semantic information is to measure the content, i.e. ASR-WER, of the output speech.  Other metrics such as speaker similarity or mean-opinion score (MOS) measure the acoustic quality of the speech, so they do not reflect the quality of the semantic units.

    While we do not think speaker similarity and MOS can measure the quality of RepCodec units, we still perform the experiments to calculate the speaker similarity, F0 error, and mean-opinion score on speech resynthesis task using LJSpeech. As shown in the following Table, RepCodec largely outperforms k-means on all these metrics, proving the superiority of the acoustic quality of RepCodec units in the speech generation tasks.

    | Methods|F0 VDE↓| F0 FFE ↓| Speaker Similarity ↑| MOS↑ |
    |:---|:---:|:---:|:---:|:---:|
    |k-means|0.199|0.248|0.771|2.82±0.36|
    |RepCodec|0.174|0.185|0.764|3.48±0.74|
    |Original Speech| - | - | - |4.32±0.92|

Reviewer comment (Evaluation):
I have concerns regarding the lack of evaluation results, as mentioned in the above weaknesses.

Author response:
First, our evaluation metrics follow previous papers on evaluating the quality of semantic units such as mHubert [1] and AudioLM [2]. They all use word error rate (WER) to measure the quality of semantic units. As the role of semantic units is to provide the semantic information of the speech, the best way to evaluate the semantic information is to measure the content, i.e. ASR-WER, of the output speech.  Other metrics such as speaker similarity or mean-opinion score (MOS) measure the acoustic quality of the speech, so they do not reflect the quality of the semantic units.

    While we do not think speaker similarity and MOS can measure the quality of RepCodec units, we still perform the experiments to calculate the speaker similarity, F0 error, and mean-opinion score on speech resynthesis task using LJSpeech. As shown in the following Table, RepCodec largely outperforms k-means on all these metrics, proving the superiority of the acoustic quality of RepCodec units in the speech generation tasks.

    | Methods|F0 VDE↓| F0 FFE ↓| Speaker Similarity ↑| MOS↑ |
    |:---|:---:|:---:|:---:|:---:|
    |k-means|0.199|0.248|0.771|2.82±0.36|
    |RepCodec|0.174|0.185|0.764|3.48±0.74|
    |Original Speech| - | - | - |4.32±0.92|