Paper: MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training (ICLR-2024)

Reviewer comment (Novelty):
The presented approach is primarily minor modifications of existing SSL models and the significance of the MERT training approach itself is limited. It’s not fully established that the modifications over Hubert are really adding substantive improvements in performance.

Author response:
**Novelty of Our Method Compared to HuBERT**: We understand your concern regarding the apparent similarity of our mask language modelling approach to HuBERT. However, we believe MERT significantly contributes to the field as it is the first model to apply masked language modelling specifically for music understanding. Our approach incorporates RVQ-VAE discrete labels for general semantic information and CQT reconstruction loss for pitch learning, tailored to the unique requirements of music modelling. The exploration of different combinations of acoustic features and cluster numbers for K-Means (as detailed in Table 3) is particularly noteworthy. While there are similarities with HuBERT in MLM tasks, MERT's distinct training logic and additional features, such as CQT reconstruction targets and transformer architecture optimizations for training stability (Section 4.3, paragraph 2), substantiate its novelty.

Reviewer comment (Evaluation):
“Computationally affordable”, “cost-effective”, “lightweight sizes” etc. are frequently used for the proposed MERT but it is not really clear how all of these are attained for MERT. How is MERT more computationally efficient or lightweight than say Hubert-Base? Aren’t the models similar? What efficiencies are we expecting here? Is it just about K-Means vs uses of codebook from Encodec? This aspect has been highlighted several times in the paper so it would be good to properly establish (quantitatively ??) how MERT is better than others.

Author response:
**Claim of Being Lightweight**: We apologise for any ambiguity in our earlier statement. Our comparison was intended against the performance of probing Jukebox5B (Castellon et al., 2021, Dhariwal et al., 2020), not HuBERT. We highlight MERT's efficiency by noting its significantly fewer parameters (1.9% and 6.6% compared to Jukebox5B) in Section 5.1, Paragraph 3.

Reviewer comment (Experiments):
Is it necessary to use all 8 layers of codebooks? Perhaps some additional experiments to better show how results vary with codebooks from different layers would be good.

Author response:
**Necessity of Using All Codebooks**: Your question regarding the use of all eight layers of codecs in RVQ-VAE is well-founded. Specifically, we have tested:

        a. Predicting the most informative codebook at layer 0=> score 60.5

        b. Predicting the most informative codebook at layer 7 => score 57.6

        c. Predicting one random layer of the codebook each time => score 65.8

        d. Predicting all layers of codebook in parallel => score 66.9

    Our investigation (referred in Table 3) shows that utilising all layers concurrently is optimal. However, we speculate that selecting a random layer for prediction and extending training could potentially yield similar results. Detailed analysis is presented in the last paragraph of Section 5.2.

Reviewer comment (Experiments):
Related to the previous point, how about using Encodec itself as SSL representation. Codec is used in MERT as a teacher – can codec itself be used as a SSL model for music? Isn’t that a baseline one can have?

Author response:
**Effectiveness of EnCodec Representation**: Thank you for this valuable suggestion. We have conducted additional experiments to address this, which are now included in the revised paper. As Reviewer-7KYT also raised this question, we invite you to refer to point 2 in our [reply to Reviewer-7KYT](https://openreview.net/forum?id=w3YZ9MSlBu&noteId=PhDl8AtHDl) to avoid redundancy in our rebuttal.

Reviewer comment (Evaluation):
The “AVG” column in Table 2 — what is it the average of? Of all the other columns? I am not sure looking at average of different types of metrics over different datasets is a good way to look at overall results.

Author response:
**Averaging Metrics**: While averaging may not always be the optimal approach, it is a common practice in evaluating pre-trained models across established benchmarks (JukeMIR, GLUE, OpenCompass). We agree that more robust aggregation metrics could be developed for a more nuanced evaluation of pre-trained models.

Reviewer comment (Experiments):
For all of the downstream experiments, is the full training set for each dataset used in the experiments? I think some experiments on “limited training data” would be useful. Otherwise, these models are not really outperforming the supervised baselines – which does not fully justify the SSL pre-training.

Author response:
**Downstream Training Set Usage for Evaluation**: The diverse training set sizes (from hundreds to 300K) across the 14 downstream tasks demonstrate the robustness of our evaluation method and generalisability of the model. We are also conducting few-shot setting evaluations, to further address your concerns. The results are anticipated in the next 2-3 days.

Reviewer comment (Evaluation):
In Sec 5.1, the paper describes that the model is doing better on local-level music tasks compared to global-level tasks. Some more discussion and perhaps illustration of why this is happening would be super helpful.

Author response:
**Performance Gap Between Local and Global Level Tasks**: As discussed in Section 5.1, MERT is trained with 5-second contexts to balance performance and computational resources. Longer contexts might benefit certain tasks, but this involves a trade-off with batch size. Our decision aligns with recent findings in the field (Ma et al. 2023, On the Effectiveness of Speech Self-supervised Learning for Music, ISMIR), which is described at *Training Settings* paragraph in section 4.3.

Reviewer comment (Experiments):
Works on short excerpts. The authors argue that this limitation could be overcome in future work.

Author response:
**Regarding Short Training Contexts**: We acknowledge your point about the potential benefits of longer segments in pre-training. Our current use of a 5-second context is a starting point, and we agree that extending this to capture more global information in music audio would be advantageous. Recent advancements in efficient transformer architectures offer promising avenues for achieving this with improved memory efficiency. Additionally, we are considering alternatives to the Conv1D encoder, which currently poses challenges in terms of stability and memory consumption, to facilitate longer context windows.

Reviewer comment (Experiments):
We mined 160K hours of music recordings from the Internet ... What are the typical sources for mining? Youtube, streaming services, Freesound, or something else? What is the typical audio quality? Are they copyrighted, or not? Do you keep the audio or only the relevant features (MFCC, CQT?)

Author response:
**Audio Collection and Quality**: We sourced audio from open-access streaming services and open-source datasets. The audio quality varies, with sampling rates of 16KHz, 24KHz, and 48KHz, necessitating upsampling or downsampling for different training settings. We ensured quality through random sampling and conducting hearing tests from the dataset. Related ethical considerations are added in Appendix D.

Reviewer comment (Presentation):
Some references are not well formatted and/or they miss key information (in particular the conference). Examples:  
Alonso-Jimenez, P., Serra, X., and Bogdanov, D. (2022).  
Bogdanov, D., Won, M., Tovstogan, P., Porter, A., and Serra, X. (2019)  
Chen, W., Keast, J., Moody, J., Moriarty, C., Villalobos, F., Winter, V., Zhang, X., Lyu, X., Freeman, E., Wang, J., et al.

Author response:
**Reference Formatting**: Thank you for pointing out the issue with reference formatting. We have added the missing conference information and corrected the numbering in the bibliography.

Reviewer comment (Writing):
Although they should be known in general, I would suggest the authors to mention the full name of all the metrics such as R2 or ROC used in the experiments.

Author response:
**Terminology Clarifications**: We have reintroduced the full names of ROC and R2 in the experimental section for clarity.

Reviewer comment (Evaluation):
Some of the "previous SOTA" (e.g. 26, 36) are still the best. Wouldn't it mean that they are still the state-of-the-art?

Author response:
**Describing “Previous SOTAs”**: We appreciate your observation on the use of "Previous SOTAs". To avoid potential confusion, we have revised Table 2 to read “(Previous) SOTA”.

Reviewer comment (Writing):
The writing switches between British and American spelling, e.g. "masked language modeling" vs. "masked language modelling."

Author response:
**General Formatting and Language Consistency**: Based on your suggestions, we have:

- Unified the language to British English.
- Modified the phrase on page 4 from “speech and language dataset” to “sequential dataset”.
- Changed “advanced SOTA” to “current SOTA”.
- Corrected punctuation and added missing commas.
- Improved figure accessibility by adopting markers and a new colour palette suitable for colour-blind users.

We are thankful for your detailed formatting suggestions, which have been incorporated into the latest version of our paper. We hope that the challenges and solutions we have shared will be valuable for future research in music audio modelling.

Reviewer comment (Writing):
Nitpick: Page 4 "data sample in a speech or language dataset..." -> the dataset doesn't have to be speech or language, e.g. it can contain instrumental music.

Author response:
**General Formatting and Language Consistency**: Based on your suggestions, we have:

- Unified the language to British English.
- Modified the phrase on page 4 from “speech and language dataset” to “sequential dataset”.
- Changed “advanced SOTA” to “current SOTA”.
- Corrected punctuation and added missing commas.
- Improved figure accessibility by adopting markers and a new colour palette suitable for colour-blind users.

We are thankful for your detailed formatting suggestions, which have been incorporated into the latest version of our paper. We hope that the challenges and solutions we have shared will be valuable for future research in music audio modelling.

Reviewer comment (Writing):
"Additionally, we present the advanced SOTA for each task including" -> This phrase could be read as the proposed model advances the state of the art for all tasks, which is not necessarily the case. If I understand correctly, "the current SOTA" is a better wording.

Author response:
**General Formatting and Language Consistency**: Based on your suggestions, we have:

- Unified the language to British English.
- Modified the phrase on page 4 from “speech and language dataset” to “sequential dataset”.
- Changed “advanced SOTA” to “current SOTA”.
- Corrected punctuation and added missing commas.
- Improved figure accessibility by adopting markers and a new colour palette suitable for colour-blind users.

We are thankful for your detailed formatting suggestions, which have been incorporated into the latest version of our paper. We hope that the challenges and solutions we have shared will be valuable for future research in music audio modelling.

Reviewer comment (Writing):
"... longer contexts if required" -> longer contexts, if required (missing comma)

Author response:
**General Formatting and Language Consistency**: Based on your suggestions, we have:

- Unified the language to British English.
- Modified the phrase on page 4 from “speech and language dataset” to “sequential dataset”.
- Changed “advanced SOTA” to “current SOTA”.
- Corrected punctuation and added missing commas.
- Improved figure accessibility by adopting markers and a new colour palette suitable for colour-blind users.

We are thankful for your detailed formatting suggestions, which have been incorporated into the latest version of our paper. We hope that the challenges and solutions we have shared will be valuable for future research in music audio modelling.

Reviewer comment (Presentation):
Figures 2 - 6 are very useful, however, they are not suitable for color-blind readers. I would suggest to change the line/marker styles for each element in the legends.

Author response:
**General Formatting and Language Consistency**: Based on your suggestions, we have:

- Unified the language to British English.
- Modified the phrase on page 4 from “speech and language dataset” to “sequential dataset”.
- Changed “advanced SOTA” to “current SOTA”.
- Corrected punctuation and added missing commas.
- Improved figure accessibility by adopting markers and a new colour palette suitable for colour-blind users.

We are thankful for your detailed formatting suggestions, which have been incorporated into the latest version of our paper. We hope that the challenges and solutions we have shared will be valuable for future research in music audio modelling.

Reviewer comment (Experiments):
Since RVQ-VAE is pre-trained on a larger dataset, comparing MERT with CQT/k-means and MERT with RVQ-VAE is somewhat unfair.

Author response:
1. **On Potentially Unfair Comparison between Pre-Training Targets**: We acknowledge your observation regarding the comparison between RVQ-VAE and K-Means targets. While it may appear unfair at first glance due to the variance in training-data size, it's important to note that K-Means models benefit from handcrafted acoustic features, providing a distinct inductive bias. This aspect makes a direct comparison with self-supervised RVQ-VAE not entirely straightforward. However, we concur with your insight that RVQ-VAE models, given their unsupervised nature, potentially have higher capacity limits, especially in learning from large-scale datasets.

Reviewer comment (Experiments):
The experiment should verify how RVQ-VAE code performs on downstream tasks to prove that the proposed MLM training phase with RVQ-VAE code as the target is required. Otherwise, one can directly utilize codes and codebook vectors from RVQ-VAE as upstream representations instead of MERT.

Author response:
2. **Performance of Vanilla RVQ-VAE Representation**: We have conducted additional experiments to evaluate the audio embeddings from the RVQ-VAE encoder, which are the continuous vectors used for quantization to discrete codecs. These supplementary results, now included in Table R1, indicate that the codec continuous representations alone are insufficient for a robust music understanding baseline. We are grateful for your suggestion of this complementary experiment, which substantiates the effectiveness of the self-supervised learning phase. Please note, these findings and discussions have also been added to Appendix A.2 in the revised manuscript for the benefit of future readers.

**Table R1: Evaluating the EnCodec RVQ-VAE Embeddings**

| Model               | MTT_{ROC} | MTT_{AP} | GS_{ACC} | GTZAN-Genre_{ACC} | EMO_{R2V} | EMO_{R2A} | Vocal-Tech_{ACC} | Vocal-Singer_{ACC} |
| ------------------- | --------- | -------- | -------- | ----------------- | --------- | --------- | ---------------- | ------------------ |
| RVQ-VAE  Embedding  | 83.4      | 26.2     | 12.1     | 36.5              | 10.3      | 47.4      | 46.3             | 69.4               |
| MERT-95M^{K-Means}  | 90.6      | 38.4     | 65.0     | 78.6              | 52.9      | 69.9      | 74.6             | 77.2               |
| MERT-95M^{RVQ-VAE}  | 91.0      | 39.3     | 63.5     | 78.6              | 60.0      | 76.4      | 74.2             | 83.7               |

Reviewer comment (Evaluation):
How do you calculate "Previous SOTA average score" in Table 2? The number did not match any baselines listed in the table, is it referenced from another work?

Author response:
3. **Clarifications on Table 2 Presentation**: Regarding the average score calculation in Table 2, we chose this approach as all metrics positively correlate with task performance. We apologise for the missing bibliography numbers; this was an oversight, and we intended to use numbered hyperlinks for easier reading. This error has been rectified in the revised manuscript.