Paper: LLark: A Multimodal Foundation Model for Music (ICLR-2024)

Reviewer comment (Novelty):
The assertion that LLark serves as a foundational music model is overstated. Upon closer examination, LLark emerges as a language-based audio understanding model, enriched with music instruction-based data. It lacks significant technical novelty for the following reasons: (1) The audio encoder (Jukebox) and text encoder (Llama 2) employed in LLark were introduced in previous works, and their training primarily relied on large-scale data pre-training. Consequently, the paper doesn't place substantial emphasis on advancing the early-stage representation learning of both audio and language models. (2) The model architecture itself, present in both encoders, exhibits no novel adaptations to better suit the domains of music understanding or language comprehension. (3) The instruction-based data generation process, while employed effectively, has been previously explored in various machine learning domains, including audio, as demonstrated by works such as LAION-CLAP [1] and WavCaps [2]. The method used for prompt creation in this paper does not introduce particularly novel design elements, potentially falling short of the requisite technical novelty standards for ICLR.

Author response:
* **Lack of novelty (numbered according to reviewers’ original numbering):** (1, 2) The reviewer is correct that LLark is built upon existing modality-specific models, and uses instruction-following techniques that have worked well for vision and language. As we discuss in our response to reviewer gHco above, this is both a benefit to our approach (not requiring expensive retraining of language or audio models) and in line with existing multimodal contributions in other domains [1-6]. The use of a straightforward architecture with “no novel adaptations to better suit the domains'' is also in line with recent multimodal models such as [1, 6, 7] which use simple projection layers as multimodal adapters. (3) Instruction tuning is neither used in LAION-CLAP (which uses keyword-to-caption augmentation to train a contrastive model) nor in WavCaps (which uses an LLM for pseudolabeling captioning data). However, we do not claim to be the first to use instruction-tuning, even for music. Instead, our contribution is in line with data-centric machine learning, where our results are distinguished by their scale (1.2M musical query-response pairs), diversity (over 160k tracks), and quality (using high-quality GPT models) to achieve results not demonstrated by any existing multimodal model for music. We believe that this is in line with similar works in the audio space appearing in ICLR; for example, MT3 [8, ICLR’22], also used preexisting, unmodified architecture and datasets.

Reviewer comment (Novelty):
Apart from LLark's exploration of LLM's potential applications in the context of music as a sub-category of the audio modality, both the audio and text encoders, as well as the data creation approach, lack substantial innovation. It appears to be more of a fusion of pre-existing works rather than a meticulously designed contribution to the field of music understanding.

Author response:
[continued from previous point]
  * **Additional considerations beyond architectural novelty:** We believe that our contribution extends beyond purely proposing a novel model (since other audio-language models do exist, as the reviewer and our paper both note, and as discussed in our point "Lack of novelty" in the original Reviewer p6wE response (1/4)). This is why we have emphasized our contribution as one of data-centric machine learning and encompassing data, models, and evaluation. In addition to a novel model (which is *not* what we consider our standalone contribution), we believe that the training data generation strategy, strong empirical results, and the quality and rigor of evaluation are important to our work. To our knowledge, LLark contains the most thorough set of musical evaluations of any audio-text model, where we evaluate MIR tasks (tempo estimation, key estimation, genre classification, instrument identification), conduct high-quality human evaluations for both music captioning and reasoning tasks, and also provide a set of objective (non-human evaluated) metrics across these tasks. Our ablation studies and extensive supplementary results provide further insights related to scaling and model components, as we discuss above and in the paper. We believe these evaluations and empirical insights are also critical contributions of our work, as rigorous evaluation is also critical to reliable progress in the field of machine learning [1]. As we noted in our previous response, other music-related works have also appeared in the ICLR conference without proposing novel model architectures, such as MT3 (ICLR 2021), which uses "an off-the-shelf T5 architecture" (see p.2 of [2]).

Once again, we thank the reviewer for their thoughtful engagement. Please let us know if we are able to provide any additional clarifications or further improve the paper to reflect the reviewers' concerns during the author-reviewer discussion window.

References:

[1] Liao, Thomas, et al. "Are we learning yet? A meta review of evaluation failures across machine learning." NeurIPS 2021.

[2] Gardner, Josh, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. "MT3: Multi-task multitrack music transcription." ICLR 2022.

Reviewer comment (Novelty):
LLark's design, which is more of a combination of existing components than a novel creation, results in a limitation. It primarily excels at simple classification-based and language-generation tasks, thus falling short of being a comprehensive music foundation model capable of addressing diverse aspects of music content, such as music source separation and music generation. Consequently, labeling LLark as a music foundation model appears somewhat overambitious, as a language-output-only model does not meet the criteria for such a designation.

Author response:
* **Status as foundation model:** The term “foundation model” is nebulous; we apologize to the reviewer for any confusion and think they raise a fair objection. We are using the term in the sense of the Stanford Center for Research on Foundation Models (CRFM), which states on its website (https://crfm.stanford.edu/) that “In recent years, a new successful paradigm for building AI systems has emerged: Train one model on a huge amount of data and adapt it to many applications. We call such a model a foundation model.” While we believe LLark meets this definition, we have removed the term “foundation model” from the title and have updated the paper text to avoid characterizing LLark as a “foundation model” (we are not currently able to change the OpenReview submission title, but have changed the title in the PDF).

Reviewer comment (Experiments):
LLark does not exhibit superior performance despite leveraging two large models, Jukebox-5b and Llama2-7B. State-of-the-art models often have parameter sizes of less than 100 million or even 1-5 million, and smaller models like MERT achieve equal or superior performance with far fewer parameters. Furthermore, models like MERT, CLAP, and others can extend their capabilities to broader applications, such as music separation and text-to-music generation. Introducing a language model with billions of parameters that delivers fewer tasks and less performance seems counterintuitive. Comparisons with language-audio (LU-AST) or language-image (IB-LLM) models are not comprehensive, as they are trained on different datasets and do not primarily focus on music data. Additionally, these models are also far from achieving state-of-the-art performance to demonstrate practical usability.

Author response:
* **Comparison to representation learning models (MERT and CLAP):** The reviewer states that “Furthermore, models like MERT, CLAP, and others can extend their capabilities to encompass broader applications, such as music separation and text-to-music generation tasks.” Both MERT and CLAP are music representation-learning models. These models do not directly make predictions for any downstream task; they are either used in a zero-shot fashion by comparing embedding distances between an audio input and a text label (as in CLAP [10], see Fig. 1, and LAION CLAP [11], Fig. 1 middle) or via linear probing (as in MERT [9], sec. B.2, and LIAON CLAP [11], Fig. 1 bottom). LLark is capable of directly outputting text responses for any query or task, and this is because it addresses a fundamentally different task: generating language from music + instructions. However, just as the reviewer suggests that the representations learned by MERT and CLAP could be used for source separation or music generation, the same task could be performed with LLark. For example, LLark’s hidden states could be used as general-purpose embeddings, and LLark could be extended to output discrete acoustic tokens to generate music. In this sense, the model we propose is capable of the same hypothetical extensions as those described for MERT and CLAP, but unlike MERT and CLAP, it is also capable of performing classification, regression, and text generation tasks out of the box without zero-shot adaptation or probing.

Reviewer comment (Experiments):
LLark's inference setting may not be as rigorously defined as proposed. The instruction-based data generation method provides the model with pre-structured slots for various music elements, such as genre, tempo, instrument, and chord. In this setup, it is expected that LLark can generate outputs corresponding to these slots when prompted with questions. Consequently, it may not truly qualify as a zero-shot setting.

Author response:
* **Inference setting:** We agree with the reviewer that our description of “zero-shot datasets” is confusing; we use the term “zero-shot dataset” to refer to datasets unseen at training time. We have updated the text for clarity.

Reviewer comment (Novelty):
Regarding music captioning, while LLark's performance demonstrates the power of language models, it is arguable whether this achievement is solely attributed to LLark or whether it mainly leverages the adaptation learned within Llama 2. Furthermore, when it comes to evaluation metrics such as BLEU scores, LLark does not consistently outperform previous state-of-the-art approaches.

Author response:
* **Music captioning:** The reviewer states “it is arguable whether this achievement is solely attributed to LLark or whether it mainly leverages the adaptation learned within Llama 2” -- it is a combination of both. Certainly, Llama 2 already contains very strong text generation capabilities. However, Llama has no multimodal capabilities on its own. The power of a multimodal model is in aligning the LM’s generation to the musical inputs so that the generated text corresponds to the musical inputs, not just the instruction text; we believe our captioning results demonstrate that this has largely been achieved. While we provide the linguistic metrics (BLEU,  ROUGE, etc.) in the supplement, that these are unreliable indicators of quality as they only measure adherence to a reference caption, while many captions are possible for a given piece of audio. As a result, we believe that human evaluation of captioning is the best indicator of model quality; as we note, this is known as the “gold standard” of chatbot evaluation in part for this reason [14].

Reviewer comment (Experiments):
The motivation behind incorporating a language model within a music foundation model remains somewhat unclear. While LLM represents a remarkable advance in AI, the necessity of using a language model to answer questions about music attributes like tempo, key signature, chord estimation, or genre is debatable. Such an interface may be more welcome when provided with labels or temporal markers, rather than a language-based query. Although music captioning demonstrates the potential, it is not fully explored in this paper. To harness the full potential of LLM in music understanding, it is crucial to either prove that language can serve as an effective tool or instruction for new music tasks (e.g., text-to-music generation, text-guided separation) or show that LLark can learn more from limited instruction-based data. However, this aspect remains unaddressed in the paper, as LLark's performance appears to rely heavily on specific instruction texts, and it may struggle to comprehend music without those explicit cues.

Author response:
* **Motivation for use of a language model for music:** We share the reviewer’s thoughtful consideration of whether a language model (LM) is the “right” approach for some music tasks. We believe the reviewer would agree that a LM is certainly appropriate for some tasks, such as captioning and certain reasoning tasks. Their main concern seems to be for certain MIR and audio tasks. Here, we feel that the main value of a LM is the useful fact that many tasks can be formulated in natural language (as we demonstrate with e.g. key, tempo). This flexibility is what makes LMs so useful as general models, and is our motivation for using an instruction-following framework. This flexibility relieves us of having to adapt the model to individual tasks (as is required for a probing step for CLAP or MERT) unless further fine-tuning is desired.

  The reviewer states that “To harness the full potential of LLM in music understanding,  it is crucial to either prove that language can serve as an effective tool or instruction for new music tasks (e.g., text-to-music generation, text-guided separation) or show that LLark can learn more from limited instruction-based data”. We believe that the audio tasks mentioned, especially text-to-music generation, have already been demonstrated widely with LLMs [15-18]. In the second case (limited-data learning), our results in Figure 8 may provide some relevant data: in particular, we show that, indeed, LLark can be trained on smaller datasets, with only limited performance changes on the MIR tasks even when using 50% of the original data. This provides at least some evidence that, indeed, LLark can learn more from limited instruction-based data, and is in line with existing works [19] which show that quality and diversity are far more important than scale for instruction-tuning.

Reviewer comment (Novelty):
In summary, LLark appears to lack the necessary level of technical novelty in designing new architectures for a music foundation model, does not provide comprehensive evidence to establish itself as a complete music foundation model, and fails to demonstrate superior performance or conduct exhaustive comparisons against established Music Information Retrieval (MIR) state-of-the-art models or other music understanding approaches. As such, it may not meet the standards expected for acceptance at the ICLR conference, and more substantial evidence is required to demonstrate its functionality and efficiency.

Author response:
[continued from previous point]
  * **Additional considerations beyond architectural novelty:** We believe that our contribution extends beyond purely proposing a novel model (since other audio-language models do exist, as the reviewer and our paper both note, and as discussed in our point "Lack of novelty" in the original Reviewer p6wE response (1/4)). This is why we have emphasized our contribution as one of data-centric machine learning and encompassing data, models, and evaluation. In addition to a novel model (which is *not* what we consider our standalone contribution), we believe that the training data generation strategy, strong empirical results, and the quality and rigor of evaluation are important to our work. To our knowledge, LLark contains the most thorough set of musical evaluations of any audio-text model, where we evaluate MIR tasks (tempo estimation, key estimation, genre classification, instrument identification), conduct high-quality human evaluations for both music captioning and reasoning tasks, and also provide a set of objective (non-human evaluated) metrics across these tasks. Our ablation studies and extensive supplementary results provide further insights related to scaling and model components, as we discuss above and in the paper. We believe these evaluations and empirical insights are also critical contributions of our work, as rigorous evaluation is also critical to reliable progress in the field of machine learning [1]. As we noted in our previous response, other music-related works have also appeared in the ICLR conference without proposing novel model architectures, such as MT3 (ICLR 2021), which uses "an off-the-shelf T5 architecture" (see p.2 of [2]).

Once again, we thank the reviewer for their thoughtful engagement. Please let us know if we are able to provide any additional clarifications or further improve the paper to reflect the reviewers' concerns during the author-reviewer discussion window.

References:

[1] Liao, Thomas, et al. "Are we learning yet? A meta review of evaluation failures across machine learning." NeurIPS 2021.

[2] Gardner, Josh, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. "MT3: Multi-task multitrack music transcription." ICLR 2022.

Reviewer comment (Evaluation):
Some of the most impressive results in the paper are regarding the long-form text outputs, but this is also where the non-expert evaluations are limited and where the negative side effects of LLMs come into play. Lumping these together under the term “model hallucinations” loses important nuance about how this will play out when LLark gets deployed into products. Looking at the outputs in the online supplement, the text appears really impressive with many musical terms related to the track, but more focused evaluations with experts would likely reveal that much of this captioning output is nonsense. This creates a problem of nicely packaged nonsense that appears correct to non-experts, a sense of authority and confusion that is less problematic in pre-LLM baseline models. While this kind of evaluation may be out of scope for the paper and could be explored in future work once the model is open-source, there are immediate examples of this issue in the online supplement that are worth highlighting in the paper.

Author response:
Thank you to the reviewer for their thorough analysis of the paper and even the supplementary material. We are pleased that the reviewer acknowledged the modeling improvements in the work, found the data and technical details “comprehensive” and “clearly communicated,” and felt the paper “presents a clear and compelling path for future MIR work in this direction”. We discuss the reviewers’ main concerns below, but would like to highlight up front that we have made several changes and additions to the paper in order to reflect the reviewers’ concerns, and we believe they have improved the paper considerably. These include: a new supplementary section on “Failure Cases” (Section N), a Model Card (Section M) to highlight limitations and concerns about potential downstream use, and updates to the Ethics statement (Section A). We are adding content to the online supplement to highlight the behaviors noted by the reviewer and discussed in these sections as well. 

We interpret one of the reviewers’ main concerns to be a need for more clearly highlighting and understanding the ways in which our model’s outputs can be misleading or incorrect. As the reviewer notes, LLMs have a tendency to generate “impressive text” with a “sense of authority” and this can be confusing or downright detrimental in some use cases when the model is incorrect. The persuasiveness of this kind of text only grows as the models’ language modeling capabilities improve. And while we made careful efforts to evaluate the music understanding capabilities of the model (e.g. Section 6.1), even long-form text responses are only helpful insofar as they are also correct, whether users know it or not. We share the reviewers’ concern, and both assessing and handling the (un)reliability of outputs is widely acknowledged to be an open research question [1] and a  major challenge for any language mode. This is made more challenging by the fact that there is no current methodology for language models to convey their uncertainty in the way that a human expert would; instead, autoregressive text generation models simply output the sequence of tokens which maximize the estimated probability conditional on the inputs -- regardless of whether the models’ estimates of these probabilities is reliable, or even whether the predictions themselves are confident. 

We share this concern, and have made several revisions to both clearly highlight potential issues with LLark’s outputs, make clear potential (mis)uses of the model, and clearly highlight risks and limitations.

In particular, we have made the following changes:
* The reviewer identifies a subset of problematic examples from the online supplement. In order to better highlight and categorize such behaviors across all tasks and datasets, we have introduced a new “Failure Cases” section to the paper (Section N), motivated by similar efforts in recent multimodal model releases (e.g. [2]). We have explicitly analyze some of the examples the reviewer mentions, among many others, in this new section, and we identify and discuss five important failure modes of the LLark model.
* We have added a Model Card (Section M; we discuss this in detail below).
* We have added a clear statement regarding the risks and limitations of using language models in this context to the ethics statement of the paper in (Section A).

**Response to question:** The reviewer asks *“Given the time to do it, how would you design an evaluation to address the kinds of issues raised under "weaknesses" above? How much of that can be communicated within this paper rather than left to future work?”*

The reviewer raises an important question. We offer several thoughts:

* (1) One approach we believe is clearly motivated here: to clearly identify and highlight failure cases of the model. This has been used in other multimodal works, for example [2], and is in line with a broader movement toward investigating and publicizing the undesirable outputs of models (“red teaming” [3] is a version of this for safety-critical applications, where the model is explicitly prompted to perform dangerous tasks). We have added this to the paper in a new “Failure Cases” section (Section N), and we discuss five prominent failure cases (N.1-N.5), including examples, potential causes, and offer potential solutions to address each.
* (2) For each failure mode, we propose specific mitigation strategies in Section N. We believe that each of these strategies, if implemented, could significantly reduce these failure modes, and each could be verified with a small set of targeted evaluations specific to the failure mode.

Reviewer comment (Writing):
For example, under “How could a music producer recreate the sounds in this track? —> they would likely use a variety of synthesizers, both digital and analog” — this is typical LLM word-salad that may seem harmless but is misleading. You definitely do not need to buy a bunch of analog synthesizers to produce that kind of track.

Author response:
* (3) The community has existing tools to highlight the relevant attributes of models and their intended uses in order to mitigate the potential for adverse impact on users. One key tool is Model Cards [4]. We have added a Model Card to the paper (Section M), which we believe will also help to avoid the kinds of issues raised under the reviewers’ “weaknesses” concerns, and provide as a single guide to where critical information can be found throughout the paper.
* (4) “Fact checking” of the free-text responses to captioning and reasoning tasks could be a useful evaluation approach. This would entail identifying musical “facts” stated by the model (i.e. references to a specific key, tempo, or instrument) in its response, and determining whether these are correct, given the audio. This strategy could provide quantifiable evidence to either confirm or disconfirm a hypothesis of whether these errors are prevalent in free-text outputs. However, properly conducting this procedure is a subtle and complex undertaking and beyond the scope of the reviewer response window.
* (5) An extreme, but plausible, option could be to rely solely on expert reviewers. However, this is expensive, not scalable, and -- we believe -- not necessary to address some of the most significant concerns the reviewer raises. (This is also an option that would be left to future work, due to the time and cost of designing, recruiting for, and conducting a study with expert musicians.)

Again, we are grateful to the reviewer for their thorough assessment and valuable feedback which have improved the paper. Please notify us if there are further concerns the reviewer has that we may address during the author response window, and we are grateful for the reviewers’ thorough efforts!

References
* [1] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).
* [2] Han, Jiaming, et al. "Imagebind-llm: Multi-modality instruction tuning." arXiv preprint arXiv:2309.03905 (2023).
* [3] Ganguli, Deep, et al. "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned." arXiv preprint arXiv:2209.07858 (2022).
* [4] Mitchell, Margaret, et al. "Model cards for model reporting." Proceedings of the conference on fairness, accountability, and transparency. 2019.

Reviewer comment (Evaluation):
Another example is “What are some characteristics that potentially differentiate the song from other similar songs —> the use of the synthesizer as the main instrument…Additionally, the song's tempo of 120 BPM and the use of the E minor key… gives the song its own unique identity within the electronic genre.” This is nonsensical because almost every song in the electronic genre features a synthesiser, 120 BPM is the least distinctive tempo possible, and E minor is a very typical key.

Author response:
* (3) The community has existing tools to highlight the relevant attributes of models and their intended uses in order to mitigate the potential for adverse impact on users. One key tool is Model Cards [4]. We have added a Model Card to the paper (Section M), which we believe will also help to avoid the kinds of issues raised under the reviewers’ “weaknesses” concerns, and provide as a single guide to where critical information can be found throughout the paper.
* (4) “Fact checking” of the free-text responses to captioning and reasoning tasks could be a useful evaluation approach. This would entail identifying musical “facts” stated by the model (i.e. references to a specific key, tempo, or instrument) in its response, and determining whether these are correct, given the audio. This strategy could provide quantifiable evidence to either confirm or disconfirm a hypothesis of whether these errors are prevalent in free-text outputs. However, properly conducting this procedure is a subtle and complex undertaking and beyond the scope of the reviewer response window.
* (5) An extreme, but plausible, option could be to rely solely on expert reviewers. However, this is expensive, not scalable, and -- we believe -- not necessary to address some of the most significant concerns the reviewer raises. (This is also an option that would be left to future work, due to the time and cost of designing, recruiting for, and conducting a study with expert musicians.)

Again, we are grateful to the reviewer for their thorough assessment and valuable feedback which have improved the paper. Please notify us if there are further concerns the reviewer has that we may address during the author response window, and we are grateful for the reviewers’ thorough efforts!

References
* [1] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).
* [2] Han, Jiaming, et al. "Imagebind-llm: Multi-modality instruction tuning." arXiv preprint arXiv:2309.03905 (2023).
* [3] Ganguli, Deep, et al. "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned." arXiv preprint arXiv:2209.07858 (2022).
* [4] Mitchell, Margaret, et al. "Model cards for model reporting." Proceedings of the conference on fairness, accountability, and transparency. 2019.

Reviewer comment (Experiments):
Limiting the audio encoder to 25-second clips (mentioned in Sections 4.1 and 7) is a notable drawback, as it lacks the understanding of important contextual information in real-world scenarios (most music audio is longer than 25 seconds), leading to potential misinterpretations and missed insights in audio content analysis.

Author response:
We acknowledge that this is a limitation of the work as presented (as discussed in Section 7). However, this is not a fundamental limitation of the approach. The audio encoding size could be extended indefinitely (up to the language model’s context window size; 4096 tokens in the case of Llama 2 7B). This could be achieved by passing chunked audio into Jukebox, and could be further compressed to accommodate longer audio by decreasing the audio frame rate (currently 10fps, or 250 tokens per 25-second clip). We adopt the 25-second limitation due to convenience in this work, and to limit the computational expense of using Jukebox (since audio clips in these datasets can be an hour or more). We leave analysis of longer audio encodings to future work - but note that our code can easily support such experiments, as it can run on arbitrarily-sized audio encodings out of the box.

Reviewer comment (Experiments):
Section 4.2.1 should investigate how LLark's performance is affected by the features extracted from traditional MIR models [5], especially regarding temporal-related music properties like timestamped chords and beat grids. Additional ablation studies should be included to confirm the performance gain of metadata augmentation.

Author response:
We agree that this would be a useful analysis, and would have liked to conduct this absent computational or budgetary constraints. However, an ablation study which removed the metadata augmentation would require completely rerunning our entire dataset creation pipeline, withholding the augmented metadata features when the query-response pairs are created (the dataset creation, not the training, was the most expensive component of our work, so this was not tractable for us). Additionally, we felt the results of this ablation would be clear: we would expect a performance degradation on the “music understanding” tasks in particular, due to the lack of any supervision on these tasks during training, and a corresponding decrease in musical detail in free-text tasks. Our open-source code release would enable others to conduct this analysis in the future -- or, perhaps more likely, to *extend* the metadata augmentation to further improve the model by incorporating more and better pseudolabeling models, as MIR research progresses and new models become available.

Reviewer comment (Evaluation):
In Sections 6.3 and 6.4, it’s ok to rely on human evaluation as we don't have holistic objective metrics to assess captioning and reasoning tasks. But leaning entirely on non-experts for this might not be the best call. It might be better to balance this out with some expert raters rather than going all-in on non-experts.

Author response:
The reviewer is highlighting a crucial and unresolved challenge in the evaluation of text generation models in general (discussed in e.g. [1]), and for music in particular. We agree that non-experts are limited in what they can assess. Overall, our approach to evaluation was to let existing works in the MIR and language modeling space be our guides, to maximize our ability to compare results to previous works. To our knowledge, prior work conducting human evaluation of audio-language models has exclusively used non-experts for both musical and non-musical applications [e.g. 2, 3] -- but human evaluations tend to be rare due to the high cost of conducting these studies. We believe there are three separate considerations raised by the reviewers’ point (and others in the review):
    * **Objective vs. subjective metrics:** Any human evaluation is necessarily subjective. Objective metrics can help to offset the human element from evaluations, helping us to avoid “leaning entirely on non-experts” while measuring different behaviors. While the reviewer is correct that “we don’t have holistic objective metrics to assess captioning and reasoning tasks”, we provide several objective metrics assessing our models’ outputs on the tasks discussed in 6.3 and 6.4, in the supplementary material. For example: (i) Figures 11 and 12, which show that LLark provides captions that are competitive with existing models in terms of both diversity and length, for a fixed prompt (keeping in mind that the higher diversity of tokens in LLaMA-Adapter are largely due to detailed hallucinations, as our user study shows; we can provide examples if the reviewer would find this useful); (ii) Figure 13, which shows that LLark can follow instructions; (iii) Tables 3 and 5, which show (using GPT-4 as judge) that LLark consistently contains more musical detail in its responses; and (iv) Table 6, which shows that, with minimal prompt tuning (i.e. asking for a “short” caption vs. the standard captioning prompt), LLark is competitive with or outperforms captioning models even in its ability to match reference captions in various language metrics (BLEU, BLEU-4, METEOR, ROUGE, CIDER). We caution against overreliance on the language metrics, however, as these only measure adherence to a reference caption, which can reflect the biases of the original captions [4] and penalizes even high-quality captions which deviate from the original caption.
    * **Experts vs. non-experts:** We acknowledge the reviewer’s concerns about non-experts being used in the subjective evaluations, and attempted to highlight this limitation in Section 7. Our goal is to construct a model which is useful to both experts and non-experts, which motivated our use of non-experts for selected tasks. Our study design (including the use of non-experts) was matched to recent language modeling evaluations, for example [1, 2, 3] which use non-experts for language, music, and audio models. As the reviewer is undoubtedly aware, the cost of assembling musical experts to evaluate our model was also prohibitively high. However, since our intention was to construct a model that is useful for a broad user base, we believe non-expert evaluations are still useful for this. Furthermore, there is not only a binary choice between expert vs. non-expert evaluators -- objective metrics (discussed above) are also a useful tool in assessing model quality without relying exclusively on non-experts.
    * **Correctness of musical aspects of text responses:** One possible concern with overreliance on non-expert human evaluation is that non-experts cannot easily assess the correctness of musical information (tempo, key, etc.). We share this concern. We point to our formal evaluations on music understanding tasks (Section 6.1), which evaluate the model’s outputs of musical properties that may be otherwise assessed by experts (key, tempo, and instrument estimates) and show that the model’s musical responses regarding these attributes are of high correctness. At the same time, we acknowledge that incorrect, but convincing-sounding outputs may affect the model. As a result, we have (1) updated the ethics section of the paper to reflect these issues with truthfulness of outputs; (2) added a Model Card [11] to the paper (Section M); (3) added a “Failure Modes” section to the paper (Section N) to highlight and categorize cases where these behaviors may emerge with LLark to ensure that users and researchers alike are appraised of the risks unique to music-language models. We also propose strategies to address each failure mode.

Reviewer comment (Evaluation):
In Figure 4, the matching rates are quite low across all models (around the random guessing level), leading to questions about whether LLark's performance on certain tasks is only marginally better than random guessing. Section 6.4 attributes this result to “limitations in the musical expertise of the (non-expert) raters in our study," further emphasizing my previous concerns about relying solely on non-experts for human evaluation.

Author response:
We acknowledge the reviewers’ concern here. Open-ended reasoning tasks are, by far, the most challenging task in our study, as they require combining music understanding, language modeling, and external “world knowledge” to answer correctly. While our model outperforms existing instruction-following audio-text models, there is room for future improvement. We believe that the rigorous formal reasoning evaluation in our study, which has not been attempted for similar models in either music-, audio-, or vision-language models to our knowledge, provides a useful benchmark for future work in this direction. We provide two additional points of context:
    * Our above discussion of objective vs. subjective metrics is relevant here. In particular, the subjective metrics can only tell “part of the story” with respect to reasoning evaluation. We hesitate to rely only on audio-text matching (see next point), and believe that the objective metrics for this task also provide useful context. Specifically, this includes the results in Table 3 and Figure 12.
    * In practice, “random guessing” is not the lower bound for audio-text matching performance for models in our study, and “perfect matching” (100%) may be an impractically high bar. Recall that our design presents raters with one piece of audio + a text query, and three responses to that query (from the same model, in random order).

      **Lower bound:** we observe that many models produce “convincing hallucinations” that attracted reviewers at a high rate whenever they appeared, regardless of whether they were correct or not. This was a major contributor to the poor performance of ImageBind-LLM (see Figure 12, which shows that ImageBind-LLM’s responses were both extremely long, and had a high number of unique tokens; these often described fictional artists and elaborate visual scenes not associated with the input but were selected by raters at high rates). In contrast, for LTU-AS, the descriptions tended to be short and generic (as shown in Figure 12), which led reviewers to select the more detailed responses at a high rate. Both behaviors can lead to lower-than-chance performance. (Consider the extreme case, not possible with this specific design, where one answer is always chosen regardless of the correct response; in that design the matching rate would plummet to 1 / size of response pool, effectively zero.)
    
      **Upper bound:** note that our audio-text matching setup is a very challenging task even under the best of circumstances: it requires that a response addresses a question correctly, and that the response matches the audio in sufficient detail such that a rater can link the textual response to the audio in the presence of other potentially-similar responses from other tracks. We hypothesize that even in the presence of expert-human-provided responses, matching rates would likely not achieve 100%. This is particularly true for non-expert raters, as the reviewer notes, since they may not be able to match attributes such as key or tempo to the audio, and again motivates a more holistic approach to our evaluations (i.e. considering the level of musical detail in Table 3, and the music understanding results in Table 2). As a result, while we acknowledge that performance even higher than LLark’s is achievable, we do not expect that 100% matching rate would be possible.
We have also added further analysis summarizing these points to Section G.3.

Reviewer comment (Experiments):
With no clear indication of randomness in option order (or maybe I missed?), such as whether LLark is consistently presented as the first option, this raises concerns about potential biases [6] in the evaluation results of GPT-4 in Sections 6.3 and 6.4.

Author response:
The ordering of all models and answer choices in all of our human and GPT-4 studies are always randomized. This is mentioned in the “interface” sections in L.1 (captioning) and L.2 (reasoning); we have also added it to the main text in Sections 6.3 and 6.4.

Reviewer comment (Writing):
Wrong citation in Appendix F: the ablation study used the CLAP developed by LAION [7], but wrongly cited the Microsoft one [8]. This made me confused for a while. Please correct the citation.

Author response:
We have corrected this mistake; thank you for noting it.

Reviewer comment (Experiments):
In Appendix F, ablation studies may not offer a fair comparison due to the significant model size differences between CLAP [7] and Jukebox [9]. To assess temporal information's significance, please consider an additional ablation study where Jukebox's weights are still frozen but average pooling is applied to its output embeddings without preserving temporal information.

Author response:
We acknowledge the reviewers’ concern that this ablation does not control for the temporal dimension of the embedded audio, and have added more detail on this point in the ablation study (Section F.1). We explored this approach (average pooling over the entire time dimension in Jukebox embeddings) in the early stages of the project, and found that adding the temporal information improved the model substantially, but did not conduct a complete ablation study. We commit to preparing such a study for the camera-ready deadline and believe it will make an interesting comparison -- but also note that it is an explanatory ablation study and ultimately will not improve the model’s results, only our understanding of them.

Reviewer comment (Novelty):
It's largely an empirical study / engineering type of project.

Author response:
We are grateful to the reviewer for the thoughtful response, and glad that the reviewer assessed that “[t]he results are promising,” that the work is “well thought out” and the paper is easy to read. As we discuss below, the reviewers’ questions also helped us to improve the clarity of the paper - thank you!

The reviewer’s generally positive feedback seems somewhat at odds with their rating. It appears that the reviewers’ main reservation about our study is the nature of the contribution: in particular, the reviewer appears to feel that the empirical nature of the work, and a belief that it “merely rehashes existing datasets and models,” mean the work is not significant. We would respectfully like to reframe our contribution to the reviewer, and put it into context with existing works. Our paper falls in line with existing highly impactful contributions in the multimodal modeling space, and with work on data-centric machine learning, where the emphasis is on empirically understanding how to build high-performing models and on solving important applied problems, regardless of the architecture used.

We provide a few specific responses below on how our work relates to others in the data-centric multimodal learning space:

  * (a) *Using existing models is a feature, not a bug:* We believe the claim that we use “existing datasets and models” is a benefit, not a drawback, of our approach. Our ablation studies (Figures 6, 7) show the impact of using these specific models, and demonstrates that starting from the highest-quality pretrained (language, audio) models is critical to the quality of the resulting output. Utilizing existing resource-intensive foundation models is also an effective use of training resources. Furthermore, neither our study nor the work of others suggests that new (audio, language) models are necessary for multimodal music understanding (see next point).

  * (b) *Many impactful multimodal models utilize pretrained modality-specific encoders:* It is standard practice, and in line with the contributions in other multimodal models, to use “existing datasets and models” as the building blocks of a multimodal model. This includes recent SOTA multimodal models in the vision-language space, such as LLaVA [1] (Llama, CLIP), Otter [2] and OpenFlamingo [3] (Llama, CLIP), LLaMA-Adapter [4] (Llama, CLIP), ImageBind-LLM [5] (Llama, ImageBind) Mini-GPT4 [6] (Vicuna, ViT). 

  * (c) *Open-source data is a benefit in the music domain:* The reviewer also voices a concern about using existing datasets. To this, we offer  two considerations:
    * (1) All of the works discussed in the previous paragraph [1-6] start by building a multimodal training set from existing open-source datasets. We follow this widely-used approach, albeit in the music domain. Our contribution is in line with these widely-cited and used works. Using open-source data is a benefit for music researchers in particular, as it allows others to reliably reproduce and extend our results (due to the strict regulations surrounding recorded music, this can be challenging for closed-source datasets!). Reviewer zBmL also highlighted this as a **strength** of the paper: “The paper leverages publicly accessible, open-source, and permissively licensed music datasets for training, which ensures data accessibility and reproducibility.” 
    * (2) We do not simply use unmodified off-the-shelf data to train the model. Indeed, our data preprocessing (via instruction-tuning across 3 task families) and metadata augmentation (which uses music-specific feature extractors to add critical musical information) is a key contribution of our work. Again, this is in line with existing instruction-tuning works in the language and vision domains [1-6]. LLark is the most comprehensive such effort in the music domain.

* (d) *Simple multimodal projection architecture is in line with existing work:* The simplicity of the multimodal adapter is both a benefit of our approach, and in line with existing work. Recent state-of-the-art models across the multimodal space have shown that simple adapter layers are highly effective: e.g. LLaVA [1], Mini-GPT4 [6], FUYU [7], among others. Given no clear reason to add further complexity to the architecture (and the emerging likelihood that this is the most effective technique for multimodal alignment), we felt this approach was appropriate and is validated by our results.

Reviewer comment (Reproducibility):
The authors seem to say they will give code and data but no checkpoint.

Author response:
* *“The authors seem to say they will give code and data but no checkpoint.”* 
  * We have provided the complete end-to-end code to recreate our datasets and train the model in the supplementary material with our initial submission, including Python scripts and Dockerfiles for executing the code. If the reviewer has any issues accessing this, please let us know!
  * All of the raw data used in our work is entirely open-source, and we provide links to the sources in the supplement (Section H). Due to the non-derivatives license of many of the audio files used in these datasets, we are legally prohibited from providing model checkpoints, but our training code enables retraining or extended the model.

Reviewer comment (Experiments):
Have you included the latest results on the benchmarking tasks?

Author response:
* *“Have you included the latest results on the benchmarking tasks?”* Our goal in this study is to compare to state-of-the-art *audio-language models* on the target tasks, which allows for an apples-to-apples comparison to LLark. Our evaluations compare against the open-source SOTA for audio-language modeling at the time of our submission. If the reviewer is asking about task-specific models (i.e., a tempo estimation model, or a key estimation model), we do not explicitly compare to such models, because (a) they are fundamentally far more limited than an audio-text model, and (b) they are often explicitly trained only on the target task and dataset (i.e. only on the training split of the target task). However, we provide some numbers from related literature (provided by reviewer p6wE) here (these references and results have also been added to the paper):
    * **Tempo estimation on Giant Steps:** 88.6% Acc2 score (the model of [8], as benchmarked in [11]). Note that the model of [8] was trained on MTG-tempo, from the same source (Beatport) and genre distribution as the evaluation dataset, whereas Llark was not.
    * **Key estimation on Giant Steps:** 74.3% accuracy (the model of [9], as cited in [10], Table 1). Note that the model of [9] was trained on MTG-Key, from the same source (Beatport) and genre distribution as the evaluation dataset, whereas Llark was not.
    * **Genre classification on GTZAN:** 83.5% (model of [11], as cited in [10], Table 2). This result is achieved by fine-tuning (via probing) directly on the GTZAN dataset. We discuss issues with the GTZAN dataset in our response to p6wE below.
    * We are not aware of prior published results performing genre recognition or instrument recognition on MedleyDB, but believe that our evaluations provide strong baselines via other audio-text models.
    * Please note that these are task-specific, not generalist models. We expect that the “best” model for a given task will often be a model architected and trained specifically for that task, which is why we focus on evaluating against other audio language models, not task-specific models, in Section 6.1.