The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models (2024)

Abstract: Benchmark plays a pivotal role in assessing the advancements of large language models (LLMs). While numerous benchmarks have been proposed to evaluate LLMs’ capabilities, there is a notable absence of a dedicated benchmark for assessing their musical abilities. To address this gap, we present ZIQI-Eval, a comprehensive and large-scale music benchmark specifically designed to evaluate the music-related capabilities of LLMs.ZIQI-Eval encompasses a wide range of questions, covering 10 major categories and 56 subcategories, resulting in over 14,000 meticulously curated data entries. By leveraging ZIQI-Eval, we conduct a comprehensive evaluation over 16 LLMs to evaluate and analyze LLMs’ performance in the domain of music.Results indicate that all LLMs perform poorly on the ZIQI-Eval benchmark, suggesting significant room for improvement in their musical capabilities.With ZIQI-Eval, we aim to provide a standardized and robust evaluation framework that facilitates a comprehensive assessment of LLMs’ music-related abilities. The dataset is available at GitHub and HuggingFace.

## introduction in recent years, large language models (llms) have made significant advancements, revolutionizing various natural language processing tasks @xcite . these models have showcased their proficiency in tasks such as accessing and reasoning about world knowledge. benchmark evaluation has played a crucial role in assessing and quantifying the performance of llms across different domains. specific benchmarks tailored to particular tasks such as coding @xcite , reading comprehension @xcite , and mathematical reason-ing @xcite , in light of the advancements made by llms, are increasingly regarded as inadequate for assessing their comprehensive capabilities. consequently, there has been a surge in the emergence of more comprehensive benchmarks @xcite . however, both the specific and comprehensive benchmarks have failed to adequately address the musical capability of large language models. music is an essential part of human life and culture, and assessing llms' comprehension and generation of music presents a unique and challenging task. this oversight emphasizes the necessity for a comprehensive evaluation framework specifically designed to capture the nuances of the musical domain. therefore, we present ziqi-eval, an extensive and comprehensive music benchmark specifically crafted to assess the music-related abilities of llms. ziqi-eval comprises a diverse range of questions, systematically organized into 10 major categories and 56 subcategories. these categories cover various aspects of music, including music theory, composition, genres, instruments, and historical context. in addition, this music benchmark actively contributes to the recognition of female music composers. by incorporating valuable content from these composers, it rectifies the gender disparity prevalent in historical literature, fostering advancement and inclusivity within the realm of music scholarship. with over 14,000 carefully crafted data entries, ziqi-eval provides a rich and extensive resource for evaluating llms' comprehension and generation of music-related content. utilizing ziqi-eval, we conducted a comprehensive experiment over 16 llms, comprising apibased models and open-source models, to evaluate the performance of llms in the realm of music. specifically, we fed music knowledge or the first half of a musical score, along with four options, to the llms to assess their ability to select the correct option and provide meaningful explanations. with an average f1 of just 58.68, even the topperforming model, gpt-4, falls short in demonstrating comprehensive music understanding and generation capabilities. this observation not only exposes the overlooked aspect of music in llms but also emphasizes the significance of ziqi-eval in bridging this gap and tackling the inherent challenges associated with it. our contribution (1) we find that existing evaluations of the capabilities of llms have overlooked their musical abilities. therefore, we propose ziqi-eval benchmark, a manually curated, large-scale, and comprehensive benchmark for evaluating music-related capabilities. it consists of 10 major categories and 56 subcategories, encompassing over 14,000 data entries. (2) we conduct evaluations on the music comprehension and music generation capabilities of 16 llms and find that almost all of them struggle to understand music effectively, let alone generate it. (3) we explore the issue of bias in llms' music capabilities, focusing on gender bias, racial bias, and region bias. analysis indicates that over 35% of llms exhibit biases, with region bias being the most severe. later, with the development of llms, recent research has utilized the modeling capabilities of these models to further enhance the understanding of music @xcite . midibert @xcite and musicbert @xcite both utilize pre-trained bert to tackle symbolicdomain discriminative music understanding tasks. musicbert further designs octuplemidi encoding and bar-level masking strategy to enhance pretraining with symbolic music data. @xcite extracts music-related information from an open-source music dataset and uses instructiontuning to instruct their proposed model llark to do music understanding, music captioning, and music reasoning. ng-midiformer @xcite first processes music pieces into sequences, followed by leveraging n-gram encoder to understand symbolic music. clamp @xcite retrieves symbolic music information through learning cross-modal representations between natural language and symbolic music. music generation before the proliferation of llms, there are some other traditional methods proposed for music generation, mainly falling into three categories: neural networks, neural audio codecs, and diffusion models. engel et al. (2019), marafioti et al. (2020), greshler et al. (2021) employ neural network architectures such as cnns, rnns, or gans to achieve music generation. a neural audio codec typically contains an encoder and a decoder. @xcite follows the typical structure. some models such as jukebox @xcite , and musiclm @xcite further insert a vector quantizer between the encoder and the decoder to learn a discrete latent representation. a diffusion model iteratively adds gaussian noise and then learns to reverse the diffusion process to construct desired data samples from the noise. @xcite proposes diffwave, a non-autoregressive model that converts the white noise signal into structured waveform through a markov chain. @xcite utilizes latent diffusion approach to generate high-quality music. since the advent of llms, researchers gradually began to explore the application of llms in music domain @xcite @xcite @xcite . audiogen @xcite and musicgen @xcite both use an autoregressive transformer-based decoder @xcite that operates on the discrete audio tokens. macaw-llm @xcite incorporates visual, audio, and textual information by using an alignment module to unite multi-modal features to textual features for llm to generate response. some models @xcite exploit the potential of llm to bridge multi-modal music comprehension and generation. benchmark evaluations benchmark evaluation plays a crucial role in assessing the development of llms. previous specific benchmarking efforts @xcite @xcite focused on evaluating certain capabilities of models in individual tasks or single-task types. however, with the advancement of llms, these benchmarks have become insuffi- composition theory music theory harmony musical form world ethnic music east asian music southeast asian music south asian music european music black african music north american music west asia, north african music south american music oceanian music female music female composers female performers female conductors female vocalists popular music blues jazz rock music rhythm and blues country music pop music folk music hip-hop disco film music funk soul music chinese music history ancient chinese music history ancient and xia-shang music zhou-qin music two han music wei-jin-nanbei dynasties music sui-tang music song-yuan music ming-qing music chinese modern music history (20th century) western music history baroque music romantic music impressionist music 20th century music ancient greek and roman music medieval music renaissance music classical music chinese traditional music folk songs rap music traditional opera music vocal music instrumental music other music musical performance instrument and instrumental performance vocal singing foundations of music aesthetics theory chinese music aesthetics western music aesthetics music aesthetics music education figure 1: ziqi-eval task overview. ## analysis in addition to the overall evaluation of llms on the dataset, we are also interested in the models' ability for specific categories. ## human experiments we also conduct relevant user studies, and we choose the precision metric to compare the performance of sota llm (gpt-4) and humans. due to time and cost limitations, we randomly selected 2 questions for each subcategory (including music comprehension questions and music generation questions), totaling 114 questions. we distributed surveys and recruited 165 individuals with and without a musical background, of different educational qualifications, to answer the questions. among them, 32.1% are non-music majors without any musical background, 8.5% are non-music majors with some musical background, and 59.4% are music majors with a musical background. as for educational qualifications, 21.2% have a high school diploma, 31.5% have a bachelor's degree, 35.2% have a master's degree or are currently pursuing one, and 12.1% have a doctorate or are currently pursuing one. the results can be seen in ii. individuals with higher educational qualifications tend to score higher. specifically, the order of scores is as follows: individuals with a doctorate degree or currently pursuing a @xmath1. iii. llms have an advantage in terms of knowledge breadth, while music master's and doctoral students have an advantage in their specialized domain knowledge. gpt-4 demonstrates music abilities that are equivalent to, or even surpass, the average level of a music doctoral student (including those currently pursuing a doctorate). this clearly proves the exceptional music capabilities of llms. in this regard, the remarkable abilities of llms are highly commendable, and it is believed that they will continue to bring us more surprises in the future. however, on the other hand, in certain knowledge domains (such as chinese traditional music, chinese music history, music education, composition theory etc.), it is evident that the scores of music master's or doctoral students are higher. this may be due to their focused and in-depth research in a specific field during the master's and doctoral stages. ## conclusion and future work our research sheds light on the oversight of existing evaluations in recognizing the musical abilities of large models. to address this gap, we introduce ziqi-eval, a comprehensive benchmark that encompasses 10 major categories and 56 subcategories, comprising over 14,000 data entries. notably, this benchmark also actively contributes to the acknowledgment of female music composers, rectifying the gender disparity and promoting inclusivity. we conducted a comprehensive experiment involving 16 llms, including both api-based and open-source models, to assess their performance in the domain of music. the results indicate that there is significant scope for enhancing the musical capabilities of existing llms. we intend to create a multimodal benchmark to evaluate the musical expertise of llms in the future.