Paper: Resource Efficient Self-Supervised Learning for Speech Embeddings (ICLR-2024)

Reviewer comment (Experiments):
Despite the title of this paper mentioning “speech embeddings”, experiments only evaluate the learned representations in the context of the ASR task. A broader evaluation including common SSL benchmarks such as SUPERB[1] would provide a more comprehensive understanding.

Author response:
We have added some results and are planning to add more if time permits.

| Model | SLU | KS |
| -------- | ------- | ------- |
| Data2vec-2 | 73.3% | 98.5% |
| Wav2vec-2 | 73.1% | 98.0% |
| Ours | 73.4% | 98.0% |

Reviewer comment (Novelty):
Related works on SSL do not include recent SSL papers such as WavLM[2] or BEST-RQ[3].

Author response:
Thanks for the suggestion. We will extend the related work and add a proper discussion about these works in the related work.

Reviewer comment (Experiments):
I am not quite sure that speech SSL methods are either contrastive or non-contrastive. For example, HuBERT is more like a BERT-style Masked Language Modeling (MLM) approach rather than a contrastive one. This line of work includes w2v-BERT, WavLM, and BEST-RQ. As HuBERT / WavLM are gaining popularity, I think a comparison with these methods would be beneficial.

Author response:
We agree with the reviewer that comparison with other masking based SSL approaches would benefit the paper. We did not add the other baselines since Data2Vec-2 showed SOTA compared to all others. However, we will add the suggested SSL approaches to the paper and compare them with our approach.

Reviewer comment (Experiments):
The authors discuss static and dynamic scaling techniques for balancing the loss; however, there are no corresponding experimental results.

Author response:
We were not able to perform ablation on those two loss scalings since we couldn’t converge the model with static scaling. We will clarify this in the paper.

Reviewer comment (Presentation):
(minor) The reference style is inconsistent across different sections of the paper, which makes it difficult to read. Please consider unifying the style throughout the paper.

Author response:
Thanks for pointing this out. We will fix this issue.

Reviewer comment (Evaluation):
The performance gap of ‘non-contrastive’ vs. ‘sequentially combined’ is not small for low-data scenarios. This raises the question of whether the non-contrastive approach sufficiently provides information for speech recognition. Is the performance gap caused by contrastive learning’s ability, or, is it a by-product of longer training? I’d like to hear your thoughts on this.

Author response:
We observed that during the pre-training the models do not converge further after around 350k steps. Therefore, we believe that the performance gain comes with the combination of contrastive and non-contrastive learning.

Reviewer comment (Presentation):
Regarding the numbers in Table 3, for sequentially combined cases, where do 3040 (2376+664) and 380 (297+83) come from? It does not seem that there are clear explanations for these numbers.

Author response:
3040 = 2376 (i.e., GPU hours for Wav2Vec-2) + 664 (i.e., GPU hours with our approach). 
380 = 297 (wall clock time for Wav2Vec-2) + 83 (wall clock time for our approach)
We will clarify this in the paper.

Reviewer comment (Presentation):
The authors mention the Appendix in Table 5, but I cannot find the Appendix.

Author response:
That’s our fault. We will fix this in the revised version of the paper.

Reviewer comment (Experiments):
It's not clear which of the two terms is contributing more to the performance of the model, in the proposed loss function L_U and L_M.

Author response:
We were not able to perform ablation on those two loss scalings since we couldn’t converge the model with static scaling. Therefore, we could not even show the individual contribution of the two losses. We will clarify this in the paper.

Reviewer comment (Evaluation):
It seems that the proposed method could save some GPU time, but not too significantly (1.3X less).

Author response:
We agree with the reviewer that mentioning this absolute improvement in that way may look less exciting or impactful. However, we also wish to highlight one important thing: training SSL models is notoriously expensive, and a 1.3X improvement in speed can very quickly translate into days of training and cost savings, especially from a multi-GPU training perspective. We also know well that deployed or usable SSL models are not the outcome of a single training run, meaning that these needed resources must be multiplied by the number of trials. This is a 1.3X reduction in time, energy consumption and cost. From this regard, any improvement is something positive for the community. The refinement of today’s models comes from the progressive incremental improvements of training efficiency [1] – and we believe that a 1.3X improvement already is a significant step for anyone interested in training a SSL model with a limited GPU budget. 

[1] Vyas, A., Hsu, W. N., Auli, M., & Baevski, A. (2022). On-demand compute reduction with stochastic wav2vec 2.0. arXiv preprint arXiv:2204.11934.

Reviewer comment (Writing):
The presentation of the loss function is somewhat abrupt, lacking explanation.

Author response:
Thanks for the suggestion. We will address these comments in the revised version of the paper.

Reviewer comment (Presentation):
Literature review could be more organized.

Author response:
Thanks for the suggestion. We will address these comments in the revised version of the paper.

Reviewer comment (Evaluation):
One of the main claims is that the proposed method provides a new SOTA result on the low-resource labeled data, which is good. However, it's not clear why the proposed method cannot compete with the Data2Vec2 method in the high-resource labeled data experiments.

Author response:
We believe that Data2Vec-2 was well optimised to achieve SOTA WER results, which require a very thorough hyper-parameter optimization (HPO). However, we do not perform rigorous HPO due to limited computational resources. Still we show that our approach marginally higher WER for LibriSpeech splits, except for dev-clean where all methods achieve the same WER (i.e., 2%).

Reviewer comment (Experiments):
Despite the similarities to Wav2Vec2 and Data2Vec2, it would be nice to include more non-contrastive comparisons (especially since the application focus is on speech), these would ideally include at least one of HuBERT and WavLM. Since the takeaway here seems to be about reducing resource requirements while maintaining high-quality performance, comparison with these popular approaches both in terms of training resources required and inclusion in the WERs table would be helpful.

Author response:
We agree with the reviewer that comparison with HuBERT and WavLM as baselines would benefit the paper. We did not add the other baselines since Data2Vec-2 showed SOTA compared to all others. However, we will add the suggested SSL approaches to the paper and compare them with our approach in table 1, 2, 3, and 4.

Reviewer comment (Experiments):
In the abstract, SSLs are mentioned as being great for a variety of tasks. Seeing the performance of this non-contrastive approach on not only ASR but other speech tasks as well could help to distinguish it from Data2Vec2 (since it often needs to be sequentially combined with wav2vec2 to match Data2Vec2 performance on the "other" partition of dev or test). These SSLs are useful in a variety of cases, so an idea of the general performance hit (in service of resource savings) on these other tasks would be helpful.

Author response:
Following the reviewer's advice, we added two new tasks from the SUPERB benchmark to compare our model to wav2vec2 and data2vec2: Spoken Language Understanding with SLURP and Keyword Spotting with Google Speech command. Following the open-source recipe of these tasks on SpeechBrain we fine-tuned three different models to obtain the results reported on the following Table. SLURP results are reported with the introduced SLU-F1 score while Keyword Spotting is a simple accuracy. In both cases, higher is better. Finally, we would like to highlight that other accepted articles on SSL, such as BEST-RQ, are also only evaluated on ASR. 

| Model | SLU | KS |
| -------- | ------- | ------- |
| Data2vec-2 | 73.3% | 98.5% |
| Wav2vec-2 | 73.1% | 98.0% |
| Ours | 73.4% | 98.0% |

Reviewer comment (Experiments):
Resource and performance comparisons with HuBERT and WavLM would be helpful (for more non-contrastive comparisons).

Author response:
We agree with the reviewer that comparison with HuBERT and WavLM as baselines would benefit the paper. We did not add the other baselines since Data2Vec-2 showed SOTA compared to all others. However, we will add the suggested SSL approaches to the paper and compare them with our approach in table 1, 2, 3, and 4.