Evaluation of Pretrained Language Models on Music Understanding (2024)

Abstract: Music-text multimodal systems have enabled new approaches to Music Information Research (MIR) applications. Despite the reported success, there has been little effort in evaluating the musical knowledge of Large Language Models (LLM). We demonstrate that LLMs suffer from prompt sensitivity, inability to model negation and sensitivity towards specific words. We quantified these properties as a triplet-based accuracy, evaluating the ability to model the relative similarity of labels in a hierarchical ontology. We leveraged Audioset ontology to generate triplets consisting of anchor, positive and negative label for genre/instruments sub-tree and use six general-purpose Transformer-based models. Triplets required filtering, as some were difficult to judge and therefore relatively uninformative for evaluation purposes. Despite the relatively high accuracy reported, inconsistencies are evident in all six models, suggesting that off-the-shelf LLMs need adaptation to music before use.

## introduction the capability of large language models (llm) to obtain informative context-dependent word embeddings with long-range inter-token dependencies showed that they can be used effectively to encode knowledge from several domains without manually curating datasets. during the last 5 years, the scientific community combined audio-based deep neural networks (dnn) with llms to form audio-text models, leading to improved performance on several music applications such as audio-to-text retrieval and textto-audio retrieval @xcite @xcite , music captioning @xcite and text-based song generation @xcite . llms are usually used pretrained and off-theshelf @xcite . while datasets for semantic similarity of general language @xcite are available, we are not aware of any such datasets for music. therefore, llms haven't been thoroughly evaluated on their musical knowledge and potential issues might be obscured. in this paper, we quantify musical knowledge in llms using triplets obtained through an ontology and report three shortcomings when used off-the-shelf. we leverage audioset, a hierarchical ontology, to extract the triplets of (anchor, positive, negative) format. the anchor label is chosen arbitrarily from the ontology, a similar label is selected as the positive, and a relatively less similar label as the negative term of the triplet. we quantify the relative similarity using the ontology-based distance between pairs of labels. thus, we evaluate llm's musical knowledge by comparing the relative similarity between anchor-positive and anchor-negative labels. we collected 13633 music genre and 37640 music instrument triplets. we evaluated the sensitivity of llms to 20 different musically informed prompts and their inability to model negation. finally, we report performance improvements when both labels and their definitions are used. both code snippets and sets of triplets used are made publicly available for reproducibility reasons foot_0 . ## evaluation of language models on musical knowledge as far as we are concerned, a linguistic evaluation dataset of musical knowledge doesn't exist apart from language-based artist similarity @xcite . information used for semantic similarity is usually scraped from websites and we argue that this information is not directly useable. generally, these websites highlight the history of the queried label without juxtaposing related concepts, audio attributes or providing slang labels and abbreviations. also, their massive size can hinder inspection and therefore, reduce their value as evaluation sets. we argue that an evaluation dataset needs to be cleaned and inspected thoroughly before increasing its size. this hasn't been done in captioning and tagging datasets, as most are weakly annotated and have highly noisy annotations @xcite . therefore, we chose to utilize an ontology with less than 200 musical labels which have a manageable size, can be manually inspected and filtered. however, we need to acknowledge that most existing ontologies are far from being exhaustive. we drew inspiration from the semantic textual similarity task @xcite @xcite that contains pairs of sentences and their degree of similarity but proposed a method of obtaining such sentences automatically leveraging a taxonomy. we evaluated 6 general-purpose transformerbased models @xcite for sentence similarity using musical terminology. in detail, a global average pooling layer is appended on top of the final layer and the sentence embedding is calculated as the mean of the respective token embeddings. the models used are mpnet, distil-roberta, minilm and albert trained on different corpora. more information about the models is provided in appendix section b and tables 1, 2. ## conclusions and future work in this paper, we quantified the musical knowledge of six transformer-based models based on triplet accuracy with musical labels for genres and instruments. we identified three shortcomings: prompt sensitivity, difficulty modeling negation and sensitivity to specific words. to overcome these shortcomings, we propose using augmentation during training and varying the prompt structures to avoid prompt sensitivity. this approach can utilize definitions to substitute labels with their definitions. to address negation modeling, we suggest multi-task learning that includes tagging negative labels in a caption and maximizing the distance between negative and positive versions of the tags in contrastive losses. we recommend using lexical databases (e.g. wordnet), which offer more elaborate music concept relationships, instead of using a tree to obtain triplets. we highlight that further filtering needs to be done to form meaningful triplets and produce good-quality evaluation datasets. lastly, despite reporting increments when definitions are used, further testing is required.