Paper: Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction (ICLR-2024)

Reviewer comment (Writing):
The writing/presentation of the paper could be improved.

Author response:
Dear Reviewer nPyR,

Thank you for your supportive feedback on our work. We are pleased to see the alignment in our perspectives on the potential future trajectories of our research. In terms of writing and presentation, we are diligently integrating the collective advice from all reviewers, including notation linkage between figure and content, rectifying formulation errors, fixing missed descriptions, and elucidating experimental setups. Should you have any particular recommendations for further improvement, we would be eager to hear them.

Reviewer comment (Experiments):
Have you or are you considering to evaluate your method on the ASR accuracy on the real conversational data, e.g., AMI, ICSI, Ali-meeting, etc.? If you have already done with these evaluations, how is the performance compared with the original HuBERT method?

Author response:
- We have not yet evaluated our system with real conversational data as it was beyond the scope of our initial focus. Indeed, compared to WavLM, which benefits from additional and noisy data augmentation, our system might not currently perform as well. We recognize the significance of this direction and intend to explore it in subsequent research.

Reviewer comment (Experiments):
In the paper, it seems two types of resolution are adopted in the method. What the performance will be if you increase the resolution types to 3 or more?

Author response:
- Concerning your question about multiple resolutions, we direct you to Appendix B.2. Here we assessed the impact of incorporating three resolutions. The breadth of the investigation necessary to thoroughly examine this, including the consideration of more aggressive utterance-level embeddings and finer 10ms resolutions, would be vast. Given this, we have earmarked the exploration of diverse resolutions as a future direction, as noted in Appendix F.

Reviewer comment (Novelty):
Somewhat lower in ML novelty, as a more straightforward evolution of HuBERT.

Author response:
We concur that the core concept behind our approach is straightforward. Nevertheless, we are thrilled about the introduction of MR-HuBERT to the research community. We believe that it can unlock numerous novel pathways in the field of speech representation learning, as thoroughly discussed in Appendix F of our paper.

Reviewer comment (Experiments):
As acknowledged (limitations, appendix E), MR-HuBERT was not trained on augmented data like WavLM, leaving this as future work, and so performance lags behind WavLM.

Author response:
With regards to the performance comparison with WavLM, we acknowledge that since we have not replicated the WavLM setups precisely, including data augmentation and the use of additional datasets such as Gigaspeech&Voxpopuli, a direct comparison is not feasible at this stage. We aim to explore these enhancements in our future research.

Reviewer comment (Writing):
The important section on MR-HuBERT's architecture (3.2) could be improved. The processing steps are adequately described, and Figure 1 is for the most part clear enough, but with several functions f and outputs H, the relation between figure 1 and the description could be better---is equation 2 correct? Also, the operators in (eq. 2) and (eq. 3) should be introduced, as should $\phi$. Are the high resolution encoders in figure 1 the same?

Author response:
We appreciate your attention to detail concerning the notation in our formulation. We will make the necessary corrections to our figures to improve clarity, including the reversal of $f_1^q$ and $f_2^q$. We also apologize for the oversight of not including the definition of $\phi$ in the manuscript. This will be corrected in our revised edition. Regarding the high-resolution encoders depicted in Figure 1, we will clarify that different $f_1^q$ and $f_3^q$ functions represent them, which should resolve any confusion.

Reviewer comment (Writing):
minor: speech Self-Supervised Learning (SSL) models -> Self-Supervised Learning (SSL) speech models

Author response:
Finally, we are grateful for your suggestions regarding writing improvements. These will be addressed in our forthcoming submission.

Reviewer comment (Writing):
minor: we evaluate MR-HuBERT at two resolutions -> we evaluate a two resolution variant of MR-HuBERT

Author response:
Finally, we are grateful for your suggestions regarding writing improvements. These will be addressed in our forthcoming submission.

Reviewer comment (Evaluation):
According to the SuperB evaluation, HuBERT large is still better (though not by much) compared to the proposed method.

Author response:
In the SUPERB benchmark, we have noticed the ASR performance is not as robust compared to the original HuBERT models. We have elaborated on potential reasons for this in Appendix C.3 (\textbf{ASR} bullet), and we regard this as an avenue for future improvements to MR-HuBERT. The comprehensive explanation provided in the appendix should serve to clarify this point further.

Reviewer comment (Experiments):
One major invention, the sampling module that employs a blend of upsampling and downsampling to achieve a flexible ratio between any two resolutions, does not show clear benefits when compared to a simple sampling module.

Author response:
We are thankful for your scrutiny of our detailed ablation analysis. While there is some divergence in model performance across the base setting in Librispeech experiments, the flexible up/downsampling module demonstrates more consistent results in large-scale experiments and the SUPERB benchmark. We encourage you to refer to Appendix B.8 (\textbf{Simplified sampling is not recommended} bullet) and the results depicted in Table 18 for models \texttt{(B.8)-i} and \texttt{(B.8)-j}. Based on these observations, we maintain that our proposed flexible up/downsampling module is the more reliable choice over its simplified counterpart.

Reviewer comment (Evaluation):
According to Table 10, the proposed sampling strategy only becomes more powerful when using 100 hours of labeled speech, and the benefit is not significant. When using only 1 and 10 hours of labeled speech, the simple sampling strategy actually performs better in terms of ASR.

Author response:
We are thankful for your scrutiny of our detailed ablation analysis. While there is some divergence in model performance across the base setting in Librispeech experiments, the flexible up/downsampling module demonstrates more consistent results in large-scale experiments and the SUPERB benchmark. We encourage you to refer to Appendix B.8 (\textbf{Simplified sampling is not recommended} bullet) and the results depicted in Table 18 for models \texttt{(B.8)-i} and \texttt{(B.8)-j}. Based on these observations, we maintain that our proposed flexible up/downsampling module is the more reliable choice over its simplified counterpart.

Reviewer comment (Evaluation):
According to Table 18, different design choices and configurations clearly affect downstream performance, and there is no clear winner.

Author response:
We are thankful for your scrutiny of our detailed ablation analysis. While there is some divergence in model performance across the base setting in Librispeech experiments, the flexible up/downsampling module demonstrates more consistent results in large-scale experiments and the SUPERB benchmark. We encourage you to refer to Appendix B.8 (\textbf{Simplified sampling is not recommended} bullet) and the results depicted in Table 18 for models \texttt{(B.8)-i} and \texttt{(B.8)-j}. Based on these observations, we maintain that our proposed flexible up/downsampling module is the more reliable choice over its simplified counterpart.

Reviewer comment (Theory):
The positions of $f_1^q$ and $f_2^q$ are reversed in equation 2. According to your description, the $\tilde{H}_0$ is first processed by $f_1^q$.

Author response:
We acknowledge the reversal error between $f_1^q$ and $f_2^q$ that you have identified. We will fix this mistake in our forthcoming revised manuscript.

Reviewer comment (Reproducibility):
In HuBERT, there's only one output sequence, so it is sent to a CTC layer when fine-tuning. However, in MR-HuBERT, there're two output sequences. How are the two sequences of different lengths combined and sent to CTC? I didn't find the details in the paper.

Author response:
Your second point has brought to our attention the need for greater clarity in describing the use cases of pre-trained MR-HuBERT. To address this:
  - For the fine-tuning setting, we indeed utilize the hidden states from the last layer for downstream models, followed with a shallow linear CTC decoder. It is important to note that the lower-resolution layers are not directly engaged in this process. The intention is to demonstrate the effectiveness of the pre-trained MR-HuBERT as a comprehensive encoder. We will enhance the manuscript by including an additional sentence to elucidate this for the benefit of our readers.
  - Regarding the frozen setting (SUPERB and ML-SUPERB), we employ a weighted summation of hidden representations from all layers. To synchronize the varying resolutions, we repeat-upsample the time axis of the lower-resolution representations, ensuring alignment across all layers. This approach is in line with established practices found in prior work, which can be reviewed at https://arxiv.org/abs/2306.01084.