Paper: Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech (ICLR-2024)

Reviewer comment (Theory):
Very little theoretical analysis is found in the paper. When would the proposed method work and it would fail? Can anything be said about it?

Author response:
**Q1: Very little theoretical analysis is found in the paper. When would the proposed method work and it would fail? Can anything be said about it?**

- Our preliminary experiment results show that the VQScore is only comparable to SpeechLMScore in the VoiceMOS 2022 challenge test set (a challenge to predict MOS for synthesized speech from TTS and voice conversion). One possible reason is that VQScore trained on LibriSpeech clean-460 hours only uses <10% training data of SpeechLMScore. Another possible reason is that if we look at each frame generated by a TTS system, it may look like a clean frame. However, in the TTS evaluation, the evaluation may focus more on global conditions such as **NATURALNESS**, etc. In other words, it cares more about the relation of each frame with the other. On the other hand, VQScore pays more attention to the degradation of each frame.

Reviewer comment (Experiments):
Key parameters are not empirically, nor theorerically assessed. Especially codebook size appears to be extremely critical parameter.

Author response:
**Q2: Key parameters are not empirically, nor theorerically assessed. Especially codebook size appears to be extremely critical parameter**

- Thank you for pointing this out, we have added more description in the paper (section I in the Appendix) to describe how we decide the hyper-parameters. We decided the hyper-parameters (e.g., \beta, and codebook size, etc.) based on the performance of DNSMOS (OVRL) on validation set. For quality estimation, it is the linear correlation coefficient (LCC) with VQScore. For the speech enhancement, it is the score itself.

- LCC between DNSMOS (OVRL) and VQScores with different codebook sizes are shown in Table r1. From this table, we can observe that except for very small codebook dimensions (i.e., 16), the performance is quite robust to codebook number and dimension. The case for speech enhancement is similar, except for very small codebook dimensions and numbers, the performance is robust to the codebook setting. These results have been added to the Appendix of the paper.

- Table r1: LCC between DNSMOS (OVRL) and VQScores with different codebook size.
| Codebook size (number, dim)     | LCC | 
| :----:        |    :----:   |   
| (1024, 32)| 0.8332| 
| (2048, 16)| 0.7668 | 
| **(2048, 32)**|	**0.8386** | 
| (2048, 64)|	0.8317 | 
| (4096, 32)|	0.8297 |

Reviewer comment (Evaluation):
Significance testing should be reported for each computed correlation.

Author response:
**Q3: Significance testing should be reported for each computed correlation.**

- Thank you for the suggestion. In the following tables, we report the T-test between Proposed + AT and different baselines on the DNS1 test set to show the statistical significance. **!!Please check Section K in the Appendix for the colored version.!!** In the tables, results shown in red, and blue represent Proposed + AT is statistically significant (p-value<0.05) better and worse than the baseline, respectively (results with black color represent no statistically significant). From the tables, we can observe that Proposed + AT is usually statistically significant better on the DNSMOS (BAK) and DNSMOS (OVR) which implies better noise removal ability. This improvement mainly comes from VQ and AT. 

- Table r2: P-value of DNSMOS (SIG) between Proposed + AT and baselines 
on the DNS1 test set.
|| Noisy     | Wiener | Demucs| CNN-Transformer|
| :----:  |    :----:   |   :---  |    :----:   |   :----:   | 
| Real  |   0.683   |   0.128 |   0.0191   |   0.0025  | 			
| Noreverb	|   0.743   |   2.29e-14 |   0.00126   |   0.265  |				
| Reverb	|   4.35e-24   |   3.46e-09 |   1.55e-11   |   0.0019  |				
			

- Table r3: P-value of DNSMOS (BAK) between Proposed + AT and baselines 
on the DNS1 test set.
|| Noisy     | Wiener | Demucs| CNN-Transformer|
| :----:  |    :----:   |   :---  |    :----:   |   :----:   | 
| Real  |  3.77e-104	 | 3.90e-83	 | 3.53e-14	 | 3.01e-10  | 			
| Noreverb	|   3.79e-61| 4.69e-49| 2.97e-08| 0.0018  |				
| Reverb	|   4.74e-91| 1.92e-46| 0.114| 0.422 |	


- Table r4: P-value of DNSMOS (OVR) between Proposed + AT and baselines 
on the DNS1 test set.
|| Noisy     | Wiener | Demucs| CNN-Transformer|
| :----:  |    :----:   |   :---  |    :----:   |   :----:   | 
| Real  |  1.35e-36  | 	4.84e-31 | 8.97e-07	 | 0.0001  | 			
| Noreverb	|   7.84e-43| 1.18e-47| 0.0042| 0.141  |				
| Reverb	|   9.20e-54| 8.70e-34| 3.21e-08| 0.205 |

Reviewer comment (Experiments):
How was the commitment weight \beta = 3 decided?

Author response:
**Q4: How was the commitment weight \beta = 3 decided?**

- As pointed out in the original paper of VQ-VAE [r1]: “We found the resulting algorithm to be quite robust to β, as the results did not vary for values of β ranging from…”. Our ablation study shown in the following table also appears a similar trend. Although, the performance is not sensitive to β, we still select the β based on the performance on the validation set. Specifically, the DNSMOS (OVRL) of the validation set.

- Table r2: DNSMOS (OVRL) of enhanced speech from models trained with different β.
| β      | DNSMOS (OVRL) | 
| :----:        |    :----:   |   
| 1      | 2.865       | 
| 2	  | 2.872 | 
| **3**	  | **2.876**      | 


[r1] A. Van Den Oord, et al., “Neural discrete representation learning,” Nips, 2017.

Reviewer comment (Evaluation):
Basically quantization error is the measure that you are using and it for sure does make sense. It would be interesting to see whether some distrubutional arguments can be made about the quantization errors. Note that those errors are scalar quantities and thus could be easily plotted and visually inspected.

Author response:
**Q5: Basically quantization error is the measure that you are using and it for sure does make sense. It would be interesting to see whether some distrubutional arguments can be made about the quantization errors. Note that those errors are scalar quantities and thus could be easily plotted and visually inspected.**

- Thank you for this suggestion, it is a very interesting direction to observe the distribution!! We have added the distribution of the VQScore on the IUB test set in the updated appendix (section H). To study the distribution of the VQScore, we first divide the test set into 3 subsets with equal size based on the sorted MOS. The first and the third subset corresponds to the group of the lowest and highest MOS, respectively. Figure 9 shows the histogram of the VQScore on the IUB test set, where blue and orange represent the first and third sets, respectively. Although there is some overlap in between for IUB_cosine, speech with higher MOS usually also has higher VQScore.

Reviewer comment (Novelty):
The novelty of the paper is limited, as quality metrics comparing embedding have already been proposed in multimodal or generative contexts including speech, such as the Fréchet Audio Distance.

Author response:
**Q1: The novelty of the paper is limited. Quality metrics comparing embedding has already been proposed in multimodal or generative contexts including speech such as the Fréchet Audio Distance.**

- Thank you for the comment. One main difference between our VQscore and Fréchet Audio Distance (FAD) is that FAD cannot evaluate the quality of each **INDIVIDUAL** audio clip as VQscore. As pointed out in the paper of FAD [r1] (section 3.1 Definition): “Unlike existing audio evaluation metrics, FAD does not look at individual audio clips, but instead compares embedding statistics generated on the whole evaluation set with embedding statistics generated on a large set of clean music (e.g. the training set).” In other words, FAD compares the distance between two sets (distributions), and the number of elements in the set has to be large enough to make FAD reliable. **In summary, FAD can only be used to evaluate the performance of a generative model, not a single audio clip.**

- As far as we know, we are the **FIRST** one to propose using quantization error to measure speech quality without the need for any quality label during training. **Note that being able to calculate the quantization error is a unique property of the VQ-VAE that other autoencoders do not have.**

[r1] Kilgour, K., Zuluaga, M., Roblek, D., & Sharifi, M. (2018). Fr\'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms. arXiv preprint arXiv:1812.08466.

Reviewer comment (Evaluation):
The authors found a low correlation between the proposed score and the quality benchmarks when the speech quality is poor, limiting the proposed measure's reliability.

Author response:
**Q2: Moreover, the authors found a low correlation between the proposed score and the quality benchmarks when the speech quality is poor, limiting the proposed measure's reliability. The results of the proposed approach for speech enhancement are still behind those of supervised models.**

- It is difficult to ask an objective quality metric to always align with subjective scores perfectly in **EVERY** condition. **For example, in Fig. 2 of [r2], the widely-used quality metrics, PESQ and POLQA also perform less reliably when the speech quality is poor.** From our experimental results shown in Table 2, our VQScore is comparable to supervised SOTA (DNSMOS), which is trained on large-scale paired data while our VQscore doesn’t need any label for model training.

- Our proposed SE model is only behind supervised models in the **matched** condition (Table 3), where the training and testing sets were from the **SAME** source. However, in practical applications, where the training and testing sets were from **DIFFERENT** sources (Tables 4 and 5), our self-supervised SE model shows better generalization capabilities than the supervised Demucs model (from Meta) and is comparable to CNN-Transformer, which has a similar model architecture as our self-supervised SE. 

- We have added an ASR experimental result in Table r1. We apply Whisper [r3] as the ASR model and compute the WER of speech generated by different SE models [r4] on the VoiceBank-DEMAND noisy test set. The results show that all the SE can improve the WER performance, and our proposed method can achieve the **lowest** WER.

- Table r1: WER of speech generated by different SE models on the 
VoiceBank-DEMAND noisy test set.
|| Noisy|	Wiener|	Demcus|	CNN-Transformer|	Proposed | 
| :----:        |    :----:   |    :----:   |  :----:        |    :----:   |    :----:   | 
|Whisper ASR|	14.25|	12.60|	13.75|	11.84|	**11.65**|


[r2] Reddy, C. K., Gopal, V., & Cutler, R. (2021, June). DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.

[r3] Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023, July). Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (pp. 28492-28518). PMLR.

[r4] Defossez, A., Synnaeve, G., & Adi, Y. (2020). Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847.

Reviewer comment (Evaluation):
The results of the proposed approach for speech enhancement are still behind those of supervised models.

Author response:
**Q2: Moreover, the authors found a low correlation between the proposed score and the quality benchmarks when the speech quality is poor, limiting the proposed measure's reliability. The results of the proposed approach for speech enhancement are still behind those of supervised models.**

- It is difficult to ask an objective quality metric to always align with subjective scores perfectly in **EVERY** condition. **For example, in Fig. 2 of [r2], the widely-used quality metrics, PESQ and POLQA also perform less reliably when the speech quality is poor.** From our experimental results shown in Table 2, our VQScore is comparable to supervised SOTA (DNSMOS), which is trained on large-scale paired data while our VQscore doesn’t need any label for model training.

- Our proposed SE model is only behind supervised models in the **matched** condition (Table 3), where the training and testing sets were from the **SAME** source. However, in practical applications, where the training and testing sets were from **DIFFERENT** sources (Tables 4 and 5), our self-supervised SE model shows better generalization capabilities than the supervised Demucs model (from Meta) and is comparable to CNN-Transformer, which has a similar model architecture as our self-supervised SE. 

- We have added an ASR experimental result in Table r1. We apply Whisper [r3] as the ASR model and compute the WER of speech generated by different SE models [r4] on the VoiceBank-DEMAND noisy test set. The results show that all the SE can improve the WER performance, and our proposed method can achieve the **lowest** WER.

- Table r1: WER of speech generated by different SE models on the 
VoiceBank-DEMAND noisy test set.
|| Noisy|	Wiener|	Demcus|	CNN-Transformer|	Proposed | 
| :----:        |    :----:   |    :----:   |  :----:        |    :----:   |    :----:   | 
|Whisper ASR|	14.25|	12.60|	13.75|	11.84|	**11.65**|


[r2] Reddy, C. K., Gopal, V., & Cutler, R. (2021, June). DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.

[r3] Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023, July). Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (pp. 28492-28518). PMLR.

[r4] Defossez, A., Synnaeve, G., & Adi, Y. (2020). Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847.

Reviewer comment (Writing):
Notation of equations 5 and 6 is inconsistent. According to Eq. 5, Lce is a function of two arguments, but Eq. 6 does not develop it correctly. Notation, in general, should be reviewed.

Author response:
**Q3: Notation of equations 5 and 6 is inconsistent. According to Eq. 5, Lce is a function of two arguments, but Eq. 6 does not develop it correctly. Notation in general, should be reviewed.**

- Sorry for the confusion. The additional argument Cv in Eq. 6 is the **FIXED** dictionary which can be treated as constant during AT. We have modified Eq.5 in the paper to include Cv as a function input to avoid confusion.

Reviewer comment (Experiments):
The authors should show evidence of the training stability during the proposed adversarial training. Is there any risk that during training, the model collapses to select the same token?

Author response:
**Q4 :The authors should show evidence of the training stability during the proposed adversarial training. ¿Is there any risk that during training, the model collapses to select the same token?**

- Different from GAN, which makes two models (generator and discriminator) compete with each other, our proposed AT **doesn’t contain a discriminator** (please see eqs (5)-(7)), therefore, the AT is pretty stable. 

- We have added learning curves of AT on the validation set in the Appendix (Section G) of the paper. The starting point (iteration 0) is the result after Step 1 in Section 2.5 (VQ-VAE trained on clean speech has converged). From the curves, we can observe that the process of AT is very stable. The scores of evaluation metrics (except for SIG) first gradually increase and then converge to a better optimum compared to the initial, which shows the effectiveness and stability of AT.	

- The model collapse usually happens in the case of a **UNCONDITIONAL GAN**, where the input is **pure noise**, and the model only generates certain classes of data (e.g., dog, and cat, etc.). In the conditional case (noisy speech as input in our case), because of the strong input prior, model collapse seldom happens.

Reviewer comment (Evaluation):
Other adversarial training strategies suffer from the high cost of generating adversarial samples, and the proposed approach does not seem to do differently. The authors should include analyses regarding computational load and scalability.

Author response:
**Q5: Other adversarial training strategies suffer from the high cost of generating adversarial samples, and the proposed approach does not seem to do differently. The authors should include analyses regarding computational load and scalability.**

- Unlike the iterative-fast gradient sign method (I-FGSM) [r5] which requires an iterative procedure to generate adversarial samples, the computational cost of our method is similar to FGSM [r6] which only needs a single pass to generate adversarial samples.
Compared to normal training (such as eq (7)), our adversarial training only needs another forward and backward pass of the computational graph (eq (5)), therefore, the computation cost is roughly **twice** of normal training. However, as illustrated in the learning curves on the validation set in the updated Appendix (Fig. 8), the AT training is efficient and can converge within **5000 iterations** (within **2 hours** using a common V100 GPU). Therefore, scalability should not be a concern. 

[r5] Kurakin, A., Goodfellow, I., & Bengio, S. (2016). Adversarial examples in the physical world.

[r6] Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.

Reviewer comment (Experiments):
The paper should also include experiments to support the claims that the proposed approach can exhibit better generalization capabilities to new domains than supervised models. Telephony speech or artificially generated speech should also be included.

Author response:
**Q6: The paper should also include experiments to support the claims that the proposed approach can exhibit better generalization capabilities to new domains than supervised models. Telephony speech or artificially generated speech should also be included.**

- **About speech quality estimation:**

- The test sets used for speech quality estimation actually come from many new domains as pointed out in the paper. For the VOiCES dataset, acoustic conditions such as foreground speech (reference), low-pass filtered reference (anchor), and reverberants were included. For the COSINE dataset, close-talking mic (reference) and chest or shoulder mic (noisy) data were provided. We even include the Tencent corpus, which covers many acoustic conditions encountered in an **online meeting in Chinese speech**.

- **About speech enhancement:**

- In fact, we have presented several experiment results (Tables 4, 5, and 8) to support our self-supervised SE model exhibits better generalization capabilities to **NEW** domains. Table 4 is based on the DNS1 test set, and contains three acoustic conditions, 1) without reverberation (noreverb), 2) noisy reverberant speech (reverb), and 3) noisy real recordings (real). Table 8 is based on the DNS3 test set, and it can be divided into subcategories based on realness (real or synthetic) and language (English or non-English). Therefore, we have verified the generalization capabilities in different conditions (**acoustic and language**, etc.) 
Under these new domains, our self-supervised SE model shows better generalization capabilities than the supervised Demucs model.

Reviewer comment (Reproducibility):
How to determine the values for several hyper-parameters, e.g. \beta in equation (3), codebook size etc.

Author response:
**Q 1:	How to determine the values for several hyper-parameters, e.g. \beta in equation (3), codebook size etc.**

- Thank you for pointing this out, we have added more description in the paper (section I in the Appendix) to describe how we decide the hyper-parameters. We decided the hyper-parameters (e.g., \beta, and codebook size, etc.) based on the performance of DNSMOS (OVRL) on validation set. For quality estimation, it is the linear correlation coefficient (LCC) with VQScore. For the speech enhancement, it is the score itself.

- As pointed out in the original paper of VQ-VAE [r1]: “We found the resulting algorithm to be quite robust to β, as the results did not vary for values of β ranging from…”. Our ablation study shown in the following table shows a similar trend. Although the performance is not sensitive to β, we still select β based on the performance on the validation set. Specifically, the DNSMOS (OVRL) of the validation set.

- Table r1: DNSMOS (OVRL) of enhanced speech from models trained with different β.
| β      | DNSMOS (OVRL) | 
| :----:        |    :----:   |   
| 1      | 2.865       | 
| 2	  | 2.872 | 
| **3**	  | **2.876**      | 
		

- LCC between DNSMOS (OVRL) and VQScores with different codebook sizes are shown in Table r2. From this table, we can observe that except for very small codebook dimensions (i.e., 16), the performance is quite robust to codebook number and dimension. The case for speech enhancement is similar, except for very small codebook dimensions and numbers, the performance is robust to codebook setting. These results have been added to the Appendix of the paper.

- Table r2: LCC between DNSMOS (OVRL) and VQScores under different codebook size.
| Codebook size (number, dim)     | LCC | 
| :----:        |    :----:   |   
| (1024, 32)| 0.8332| 
| (2048, 16)| 0.7668 | 
| **(2048, 32)**|	**0.8386** | 
| (2048, 64)|	0.8317 | 
| (4096, 32)|	0.8297 |

Reviewer comment (Evaluation):
For the results tables, could authors include std to show if difference is statistically significant?

Author response:
**Q2:	For the results tables, could authors include std to show if difference is statistically significant?**

- Thank you for the suggestion. In the following tables, we report the T-test between Proposed + AT and different baselines on the DNS1 test set to show the statistical significance. **!!Please check Section K in the Appendix for the colored version!!** In the tables, results shown in red, and blue represent Proposed + AT is statistically significant (p-value<0.05) better and worse than the baseline, respectively (results with black color represent no statistically significant). From the tables, we can observe that Proposed + AT is usually statistically significant better on the DNSMOS (BAK) and DNSMOS (OVR) which implies better noise removal ability. This improvement mainly comes from VQ and AT. 

- Table r3: P-value of DNSMOS (SIG) between Proposed + AT and baselines 
on the DNS1 test set.
|| Noisy     | Wiener | Demucs| CNN-Transformer|
| :----:  |    :----:   |   :---  |    :----:   |   :----:   | 
| Real  |   0.683   |   0.128 |   0.0191   |   0.0025  | 			
| Noreverb	|   0.743   |   2.29e-14 |   0.00126   |   0.265  |				
| Reverb	|   4.35e-24   |   3.46e-09 |   1.55e-11   |   0.0019  |				
			

- Table r4: P-value of DNSMOS (BAK) between Proposed + AT and baselines 
on the DNS1 test set.
|| Noisy     | Wiener | Demucs| CNN-Transformer|
| :----:  |    :----:   |   :---  |    :----:   |   :----:   | 
| Real  |  3.77e-104	 | 3.90e-83	 | 3.53e-14	 | 3.01e-10  | 			
| Noreverb	|   3.79e-61| 4.69e-49| 2.97e-08| 0.0018  |				
| Reverb	|   4.74e-91| 1.92e-46| 0.114| 0.422 |	


- Table r5: P-value of DNSMOS (OVR) between Proposed + AT and baselines 
on the DNS1 test set.
|| Noisy     | Wiener | Demucs| CNN-Transformer|
| :----:  |    :----:   |   :---  |    :----:   |   :----:   | 
| Real  |  1.35e-36  | 	4.84e-31 | 8.97e-07	 | 0.0001  | 			
| Noreverb	|   7.84e-43| 1.18e-47| 0.0042| 0.141  |				
| Reverb	|   9.20e-54| 8.70e-34| 3.21e-08| 0.205 |

Reviewer comment (Evaluation):
For Table 3, could authors add a short summary about comparing model complexity for the proposed approach vs. baselines?

Author response:
**Q 3: For Table 3, could authors add a short summary about comparing model complexity for the proposed approach vs. baselines?**

- We compare model complexity based on the number of parameters and the number of computational operations as shown in Table r6. MACs stand for multiply–accumulate operation and are calculated based on 1 sec of audio input. Because NyTT doesn’t release the model, it is difficult to accurately estimate its model complexity. However, its model structure is based on CNN-BLSTM, so we can expect it to have higher model complexity compared to MetricGAN-U, which is based on simple BLSTM. CNN-Transformer is the supervised version (and removing VQ) of our proposed model and hence has a similar model complexity. Demucs is a CNN encoder and decoder framework with BLSTM in between to model temporal relationships. Because directly models the waveform, its model complexity is significantly higher than others.

- Table r6: Model complexity for the proposed approach vs. baselines.
|| Params (M) | MACs (G) | 
| :----:        |    :----:   |    :----:   |  
| CNN-Transformer	| 2.51| 0.32 | 
| Demucs| 60.81| 75.56 | 
| MetricGAN-U| 1.90| 0.24 | 
| NyTT| -| - | 
| Proposed| 2.51| 0.32 |

Reviewer comment (Experiments):
No downstream evaluation (diarization, speaker recognition, ASR, etc.) provided which I would expect for ICLR.

Author response:
**Q1: No downstream evaluation (diarization, speaker recognition, ASR, etc.) provided which I would expect for ICLR.**

**Q8: Why downstream evaluation is not done? To publish a new enhancement solution in ICLR, in my personal opinion, it becomes critical.**

- Thank you for the suggestion, we have added an ASR experimental result in Table r1 and the Appendix of the paper. We apply Whisper [r1] as the ASR model and compute the WER of speech generated by different SE models [r2] on the VoiceBank-DEMAND noisy test set. The results show that all the SE can improve the WER performance, and our proposed method can achieve the lowest WER.

- Table r1: WER of speech generated by different SE models on the 
VoiceBank-DEMAND noisy test set.
||Noisy|Wiener|Demcus|CNN-Transformer|Proposed|
|  :----: |  :----: |  :----: |  :----: |  :----: |  :----: |
|Whisper ASR |14.25 |12.60 |13.75 |11.84 |**11.65**| 

 [r1] Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023, July). Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (pp. 28492-28518). PMLR.

[r2] Defossez, A., Synnaeve, G., & Adi, Y. (2020). Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847.

Reviewer comment (Evaluation):
I am not able to determine if high linear correlation of proposed metric is enough to say this enhancement will work on real noisy datasets. Remember the goal of enhancement is to remove noise (and other unwanted information) such that it can be used on a plethora of end applications. It is not just about perceptually making it better. STOI-like metrics are ignored in this work which quantifies intelligibility. Note that it is also possible to produce good sounding audio which is not very intelligible.

Author response:
**Q 2: I am not able to determine if high linear correlation of proposed metric is enough to say this enhancement will work on real noisy datasets. Remember the goal of enhancement is to remove noise (and other unwanted information) such that it can used on a plethora end applications. It is not just about perceptually making it better. STOI-like metrics are ignored in this work which quantifies intelligibility. Note that it is also possible to produce good sounding audio which is not very intelligible.**

- **For speech quality estimation:**
We have added new results of linear correlation between STOI and our VQScore on the VoiceBank-DEMAND noisy test set in Table r2 (also added the results in Table 1 in the paper). We can observe that, for STOI, the VQScore calculated in the code space (z) still has a much larger correlation than that calculated in the signal space (x) and the self-supervised baseline, SpeechLMScore.

- For real noisy datasets, our test sets, ‘Tencent_wR’ contains speech recorded in a realistic reverberant room, and ‘IUB_cosine’ dataset includes a close-talking mic (reference) and chest/shoulder mic (noisy). Therefore, real noisy datasets have already been considered in our test sets.

- Table r2: LCC between STOI and our VQScore on the VoiceBank-DEMAND noisy test set.
||SpeechLMScore|VQScore(cos,x)|VQScore(cos,z)|
|  :----: |  :----: |  :----: |  :----: | 
|STOI| 0.6023|0.5012|**0.7490**|


- **For speech enhancement:**
The reason we downplayed the role of intrusive metrics (clean reference is needed) such as PESQ, and STOI for SE evaluation is detailly explained in the response of Q7. In summary, the goal of generative models is to model the **distribution** of clean speech **rather than minimize the difference with its clean counterpart in the signal space**. Therefore, intrusive metrics may not be able to appropriately evaluate its performance.

- Please refer to the response of Q7 for the discussion and related citations, etc. On the other hand, the results shown in Table r1 show that the proposed method can effectively decrease WER, which quantifies the intelligibility of an ASR.

- For real noisy datasets, both DNS 1 (Table 4) and DNS3 (Table 8) test sets contain REAL noisy subsets, where our self-supervised SE model shows better performance than the supervised model, Demucs.

Reviewer comment (Experiments):
Lack of ablation or other analysis on proposed method. Since the proposed method is the main technical contribution, I would expect it to be evaluated more robustly.

Author response:
**Q3: Lack of ablation or other analysis on proposed method. Since the proposed method is the main technical contribution, I would expect it to be evaluated more robustly.**

- **We have added several new experiments and analyses in the Appendix, including 1) Computational load of adversarial training, 2) Learning curves of adversarial training, 3) Distribution of VQScore, 4) Sensitivity to hyper-parameters, 5) Model complexity, 6) Statistical significance, and 7) ASR results of enhanced speech.**

- An ablation study about four combinations of distance metrics and targets for calculating VQScore has already been presented in section B “COMPARISON OF DIFFERENT VQSCORES” of the Appendix. Here we present another ablation study about the codebook size. LCC between DNSMOS (OVRL) and VQScores with different codebook sizes are shown in Table r3. From this table, we can observe that except for very small codebook dimensions (i.e., 16), the performance is quite robust to codebook number and dimension. The case for speech enhancement is similar, except for very small codebook dimensions and numbers, the performance is also robust to the codebook setting. These results have been added to the Appendix of the paper.

- Table r3: LCC between DNSMOS (OVRL) and VQScores with different codebook size.
|Codebook size (number, dim)|LCC|
|  :----: |  :----: | 
|(1024, 32)|0.8332|
|(2048, 16)|0.7668|
|**(2048, 32)**|**0.8386**|
|(2048, 64)|0.8317|
|(4096, 32)|0.8297|

Reviewer comment (Experiments):
Noise corpora is not used which is readily available. It would be interesting to see how using external noises can improve model performance. AT noise is not the only noise that is readily available. In fact AT is slow.

Author response:
**Q 4: Noise corpora is not used which is readily available. It would be interesting to see how using external noises can improve model performance. AT noise is not the only noise that is readily available. In fact AT is slow.**

- Thank you for pointing this out, this is a very interesting direction to compare real noise and adversarial noise. We tried to add real noise from the DEMAND dataset just as those used in the VoiceBank-DEMAND training set. In the **dataset-matched** conditions (the training and testing sets were from the same source), adding real noise can provide **further improvement** compared to adversarial noise as shown in Table r4. On the other hand, in the **dataset-mismatched** conditions, the performance of adding real noise is **similar** to that of adding adversarial noise as shown in Table r5.

- **About AT is slow**: Unlike the iterative-fast gradient sign method (I-FGSM) [r3] which requires an iterative procedure to generate adversarial samples, the computational cost of our AT is similar to FGSM [r4] which only needs a single pass to generate adversarial samples. Compared to normal training (such as eq (7)), our AT only needs another forward and backward pass of the computational graph (eq (5)). Therefore, the computation cost is roughly **twice** of the normal training. However, as illustrated in the learning curves on the validation set (Fig. 8) in the updated Appendix, the AT training is efficient and can converge within **5000 iterations** (within **2 hours** using a common V100 GPU). Therefore, scalability should not be a concern.


- Table r4 Comparison of adding different noises for SE model training on the VoiceBank-DEMAND noisy test set.
||Training data|SIG|BAK|OVR|	
|  :----: |  :----: | :----: |:----: |:----: |
|Proposed|clean speech|3.329|3.646|2.876|
|Proposed + AT|clean speech|3.300|3.838|2.941|
|Proposed + real noise|Clean speech+noise|**3.326**|**3.960**|**3.026**|		
	

- Table r5 Comparison of adding different noises for SE model training on the DNS1 test set.
		
|Real|Training data|SIG|BAK|OVR|
|  :----: |  :----: | :----: |:----: |:----: |
|  Proposed |  clean speech | 3.095 |3.365 |2.589 |
|Proposed + Gaussian|	clean speech	|3.152	|3.458	|2.673|
|Proposed + AT| clean speech| **3.156**| **3.640**| **2.750**|
|Proposed + real noise| Clean speech+noise| 3.142| 3.604| 2.728|

|Noreverb|Training data|SIG|BAK|OVR|
|  :----: |  :----: | :----: |:----: |:----: |
|  Proposed |  clean speech | 3.463| 3.764| 3.066 |
|Proposed + Gaussian|	clean speech	|3.484 | 3.830 | 3.115|
|Proposed + AT| clean speech| 3.481 |  **3.960** |  **3.162** |
|Proposed + real noise| Clean speech+noise|  **3.485** |3.881 |3.139|
			
|Reverb|Training data|SIG|BAK|OVR|
|  :----: |  :----: |  :----: |:----: |:----: |
|  Proposed |  clean speech | 2.911| 3.097| 2.325 |
|Proposed + Gaussian|	clean speech	|2.930 | 3.222 | 2.394|
|Proposed + AT| clean speech| **2.949** |  3.361 |  2.456 |
|Proposed + real noise| Clean speech+noise|  2.859 |**3.534** |**2.465**|			

[r3] Kurakin, A., Goodfellow, I., & Bengio, S. (2016). Adversarial examples in the physical world.

[r4] Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.

Reviewer comment (Experiments):
If TorchaudioSquim has mismatch issues (as authors point out), it can be retrained to make it more appropriate for comparison with proposed method.

Author response:
**Q5:	If TorchaudioSquim has mismatch issues (as authors point out), it can be retrained to make it more appropriate for comparison with proposed method.**

- Thank you for pointing this out. Because TorchaudioSquim is a **supervised** training framework (quality labels are needed), our training data (no quality labels) cannot be used to train this model. This also shows the value and flexibility of our self-supervised VQScore.

Reviewer comment (Evaluation):
Table 4,5 is missing PESQ, STOI numbers. (CHECK: https://paperswithcode.com/sota/speech-enhancement-on-demand). I don't understand how Weiner is best in SIG (real subset, Table 4). I am not sure dereverberation should be investigated in this paper. DNS1 details are also not mentioned.

Author response:
**Q6:	Table 4,5 is missing PESQ, STOI numbers. (CHECK: https://paperswithcode.com/sota/speech-enhancement-on-demand). I dont understand how Weiner is best in SIG (real subset, Table 4). I am not sure dereverberation should be investigated in this paper. DNS1 details are also not mentioned.**

- Sorry for the confusion. The DNS1 dataset is the test set of the 1st Deep Noise Suppression Challenge held by Microsoft. Because the **corresponding clean speech is not released** (in the ‘real’ subset, the corresponding clean speech may even not exist), we cannot calculate PESQ/STOI scores where the **corresponding clean speech is needed** for evaluation (from which we can see the limitation of intrusive metric and why non-intrusive metrics such as VQScore is more practical). 


- For the traditional signal processing method such as Wiener, because of limited model ability, the result usually has less speech distortion (high SIG) but more noise remains (low BAK).

Reviewer comment (Evaluation):
Why role of PESQ is downplayed? Authors say it is something to do with generative models but they did not expand or give citations to support this idea.

Author response:
**Q7: Why role of PESQ is downplayed? Authors say it is something to do with generative models but they did not expand or give citations to support this idea.**

- The discussion about the problem of using PESQ to evaluate generative models is presented in **footnote 1 of page 7**: “Note that several recent papers have shown that PESQ can not reflect the true speech quality, especially for speech generated by a **generative model** (such as **GAN** Kumar et al. (2020); Liu et al. (2022), **vocoder** Du et al. (2020); Li & Yamagishi (2020); Maiti & Mandel (2020), **diffusion model** Serrà et al. (2022), etc.) We also observe that the PESQ scores of enhanced speech generated by models with VQ usually **CANNOT** reflect the true speech quality. This is mainly because the discrete tokens in VQ are shared by similar sounds, which makes the generated speech have less fidelity. However, as pointed out by DNSMOS and the following subjective listening test, this does not imply its generated speech has lower quality.”

- The reason is that PESQ is an intrusive metric that is based on the comparison between the enhanced speech with its clean counterpart. On the other hand, generative models (such as GAN, VQ-VAE, diffusion model, etc.) try to model the **distribution** of real data, and hence the generated image/speech may look/sound real but the **signal distance to their clean counterpart may be large**. In summary, intrusive metrics such as PESQ, and STOI, etc. can only provide some information about signal fidelity for generative models. In the following, we list two representative examples in detail:

- 1) Microsoft recently found that PESQ may not perfectly reflect the perceptual quality (please refer to Table 2 and Fig 2 in [r5]. From which they show the correlation between PESQ and Mean Opinion Score (MOS) is only **0.78** while DNSMOS can reach to **0.93**). 

- 2) In Meta’s ReVISE method [r6], although their method has **much better** intelligibility (WER), and quality (MOS), the low-level detail intrusive metrics (ESTOI and MCD) are **significantly worse** than the baseline (see section B.3. in their paper). 

[r5] Reddy, C. K., Gopal, V., & Cutler, R. (2021, June). DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.

[r6] Hsu, W. N., Remez, T., Shi, B., Donley, J., & Adi, Y. “Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech enhancement.” CVPR, 2023.

Reviewer comment (Experiments):
Why downstream evaluation is not done? To publish a new enhancement solution in ICLR, in my personal opinion, it becomes critical.

Author response:
**Q1: No downstream evaluation (diarization, speaker recognition, ASR, etc.) provided which I would expect for ICLR.**

**Q8: Why downstream evaluation is not done? To publish a new enhancement solution in ICLR, in my personal opinion, it becomes critical.**

- Thank you for the suggestion, we have added an ASR experimental result in Table r1 and the Appendix of the paper. We apply Whisper [r1] as the ASR model and compute the WER of speech generated by different SE models [r2] on the VoiceBank-DEMAND noisy test set. The results show that all the SE can improve the WER performance, and our proposed method can achieve the lowest WER.

- Table r1: WER of speech generated by different SE models on the 
VoiceBank-DEMAND noisy test set.
||Noisy|Wiener|Demcus|CNN-Transformer|Proposed|
|  :----: |  :----: |  :----: |  :----: |  :----: |  :----: |
|Whisper ASR |14.25 |12.60 |13.75 |11.84 |**11.65**| 

 [r1] Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023, July). Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (pp. 28492-28518). PMLR.

[r2] Defossez, A., Synnaeve, G., & Adi, Y. (2020). Real time speech enhancement in the waveform domain. arXiv preprint arXiv:2006.12847.