Paper: Large Language Models are Efficient Learners of Noise-Robust Speech Recognition (ICLR-2024)

Reviewer comment (Experiments):
The way to extract the audio noise embedding is from the ASR encoder (i.e., Whisper Large-V2). This may only make sense for the Whisper ASR, as a recent paper [r1] pointed out that the noise-robustness of Whisper does not come from noise-invariant, but recognizes speech conditioned on the noise type. In summary, the Whisper encoder is a suitable model to extract noise information. On the other hand, other ASR models may not have such ability and they achieve noise-robustness through the noise-invariant encoder. If this is the case, those ASR encoders may not be suitable to extract audio noise embedding. A discussion and simple experiment about this would be great.

Author response:
Thanks for your comment. We agree with you that Whisper encoder is better than other encoders for audio noise embedding extraction. We have added an experiment on CHiME-4 in Table 17 (Appendix), which compares Whisper encoder with WavLM encoder and Wav2vec2 encoder. Results on audio-space denoising shows that Whisper encoder performs better than WavLM and Wav2vec2, which verifies the claim above. However, since audio denoising itself does not perform well in our experiments, we may need more studies to compare these ASR encoders for noise information extraction, which we would like to leave for future work. Thanks again for bringing up this topic.

Reviewer comment (Theory):
As pointed out on page 5, the noise embedding is calculated by their diversity “similar to variance”, however in eq (4) and (6), the sentence embedding differences are simply summed, so I guess an abs or square operation is needed?

Author response:
Thanks for pointing out this. We apologize for making you confused, and we have removed the term “similar to variance”. 

  As for Eq.(4) and (6), we also make a mistake (sorry again), please kindly refer to our updated manuscript, it should be the concatenation of all sentence embedding differences, instead of summing them up. From N-best hypotheses, we can collect $N\cdot(N-1)/2$ sentence-level embedding differences from each pair of hypotheses in the N-best list (i.e., $1 + 2 + \cdots + (N-1) = N\cdot(N-1)/2$), where each of them has tensor shape $ \mathbb{R}^{1\times D_\text{sbert}}$. Samely, we can also obtain $N\cdot(N-1)/2$ token-level embedding differences. Therefore, we concatenate all of these $N\cdot(N-1)$ embedding differences along the first dimension to get a language-space noise embedding $E_\text{LN} \in \mathbb{R}^{N\cdot(N-1)\times D_\text{sbert}}$.

  We have made these clear in the updated manuscript.

Reviewer comment (Theory):
In figure 2, and eq (3), why is the language-space noise embedding ‘subtracted’ from the prompt? I found another related equation in eq (13) of the Appendix and the authors only mention “the subtraction operation denotes ‘denoise’”, more explanation is needed.

Author response:
Thanks for pointing out this, and we apologize for making you confused. After calculating language-space noise embedding $E_\text{LN}$, we add a minus sign before it to build a **denoise** embedding, which is then sent into LLM to teach it to perform language-space denoising (i.e., **reduce** the noise from LLM finetuning). This is an intuitive design to implement the idea of language-space **denoising**. We have made this clear in Section 4.1 and modified the “subtract” in Fig.2 to “denoise”.

Thank you for comment. The design of subtraction operation is **mainly** to align the concept of denoise, but also considering the performance. If we remove the minus sign, the proposed approach still outperforms the baselines but are worse than our current result with this minus sign. We speculate the reason could be, the LLaMA-Adapter itself can learn this minus sign to "denoise" instead of "add noise" during finetuning, but if we explicitly define the minus sign, it might be easier for LLaMA-Adapter to conduct denoising.

Thank you again for bringing up this question, we believe it could become a promising interpretability topic for future work.

Reviewer comment (Writing):
In eq(8), IΘ(X; Y) should be IΘ(X; Z).

Author response:
Thanks for pointing out this, we have corrected it in the updated manuscript.

Reviewer comment (Reproducibility):
There lacks the practical discussion on additional workload, especially on resources.

Author response:
Thanks for pointing out this. In our experiments, we use 1 NVIDIA A40 GPU (48GB) for model training, which takes 1.5 hours for CHiME-4 dataset, 2.0 hours for VB-DEMAND, 1.6 hours for NOIZEUS, 4.5 hours for LS-FreeSound, and 3.8 hours for RATS, respectively. Actually we have introduced them in Appendix C.2 (first paragraph).

Reviewer comment (Presentation):
Section 5.4 - What is Table 14?

Author response:
We apologize for making you confused. Table 14 indicates the ablation study of the audio distillation approach in terms of different KD techniques (i.e. teacher-student learning, contrastive learning, and our utilized MINE), where the analysis is provided in Appendix D.4. We mention it in Section 5.4 to highlight the effectiveness of proposed audio distillation approach.

Reviewer comment (Experiments):
I wonder the motivation of using Robust Hyporadise dataset for noisy ASR condition. What kind of noise it exactly contains? Is it replacible with other noisy datasets that are more commonly known to the ASR community, such as Switchboard and VoxCeleb (just two examples, they may not be good fit)?

Author response:
Thanks for your comment. Our Robust HyPoradise dataset covers many common noisy ASR scenarios in real world, in order to evaluate the generality of our proposed approach. The details of their noise conditions are introduced in Appendix A.2, which include:
  1) **CHiME-4:** bus, cafe, pedestrian area, street junction;
  2) **VB-DEMAND:** training data contains 10 noise types (babble, cafeteria, car, kitchen, meeting, metro, restaurant, speech-shaped noise, station, traffic), test data contains 3 new noise types (baby-cry, helicopter, and crowd-party) – simulate an unseen noise testing condition that is common in real-world scenarios;
  3) **NOIZEUS:** 8 noise types from Aurora-2 database (suburban train noise, babble, car, exhibition hall, restaurant, street, airport and train-station noise);
  4) **LibriSpeech-FreeSound:** 6 noise types from FreeSound database (metro, car, traffic, cafe, babble and ac/vacuum);
  5) **RATS:** radio communication speech in ultra high frequency data category that is extremely noisy and challenging for ASR.

  Yes, our RobustGER can be extended to any noisy ASR datasets, and we are exactly planning to make such extensions for future work, which would then be open sourced to benefit the community. 

  The two datasets you mentioned are both good fit for Robust Hyporadise, we did not select them before due to their less diverse noise categories than our utilized five datasets. We will add them for future extensions, thanks again for your suggestion.

Reviewer comment (Experiments):
Do you think your model will be sensitive to sampling frequency? I mentioned Switchboard in the last question, which is an 8KHz dataset.

Author response:
Thanks for your comment. We think our model is not sensitive to sampling frequency. The whole pipeline of our system consists of two stages: 
  1) use Whisper for ASR decoding to generate N-best hypotheses; 
  2) employ LLM-based RobustGER to generate ground-truth transcription from N-best hypotheses. 

  The sampling frequency of source speech is directly related to stage 1), where Whisper is a general ASR model that can handle input speech of different sampling rates.

  We also list the sampling rates of our selected ASR datasets here for your reference: 
  1) **CHiME-4:** 16 kHz; 
  2) **VB-DEMAND:** 48 kHz; 
  3) **NOIZEUS:** 8 kHz; 
  4) **LibriSpeech-FreeSound:** 16 kHz; 
  5) **RATS:** 8 kHz. 

  Our approach has achieved improvements on all of them.

Reviewer comment (Writing):
The notation seems to be mixed up. $\mathcal{P}$ may be used in the sense of probability density function in equation (2), but this is also used to mean "prompt". In the definition of KL, the probability distribution is denoted as $\mathbb{P}$.

Author response:
Thanks for pointing out this. We have modified them as: 

  1) use $\mathcal{P}$ to denote the probability density function in Eq. (2); 
  2) modify the prompt to $\mathcal{G}$; 
  3) use $\mathbb{P}$ to denote the probability distribution in definition of KL. 

  Considering 1) and 3) are both probability-related variables, we prefer to both use $P$ but in different font styles. We are willing to discuss more if you have any further suggestions.

Reviewer comment (Writing):
I don't feel the need to use too much fancy notation like tensor product $\mathbb{P}_X \otimes \mathbb{P}_Z$ and Radon-Nikodym derivative $\log \frac{d\mathbb{P}}{d\mathbb{Q}}$.

Author response:
Thanks for your suggestion. We have removed them to make a plain style.

Reviewer comment (Writing):
It may be just because I am conservative but $1e^{-2}$ looks like $1/\exp(2)$.

Author response:
Sorry for making you confused. We have modified it to $10^{-2}$.

Reviewer comment (Writing):
$\textit{i.e.}$ should not be italicized in standard writing convention.

Author response:
Thanks for your suggestion. We have corrected all of them in the paper.