The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines? (2024)

Abstract: In this work, we explore the use and reliability of Large Language Models (LLMs) in musicology. From a discussion with experts and students, we assess the current acceptance and concerns regarding this, nowadays ubiquitous, technology. We aim to go one step further, proposing a semi-automatic method to create an initial benchmark using retrieval-augmented generation models and multiple-choice question generation, validated by human experts. Our evaluation on 400 human-validated questions shows that current vanilla LLMs are less reliable than retrieval augmented generation from music dictionaries. This paper suggests that the potential of LLMs in musicology requires musicology driven research that can specialized LLMs by including accurate and reliable domain knowledge.

## introduction in recent years, research on large language models (llms) has led to notable advancements within the text generation domain @xcite . this is the result of training large models on vast non-domainspecific data @xcite . well-known families of models include llama (ai@meta, 2024) or gpt @xcite , which can generate coherent and contextually relevant text, making them valuable tools in numerous applications and professions such as healthcare @xcite , journalism @xcite , customer support (kolasani, 2023) or education @xcite . despite their potential, llms' so-called hallucinations @xcite , i. e., the lack of confidence and accuracy in the text they generate, prevents the use of this technology in most arts and humanities research tasks @xcite figure 1 : fictitious interaction illustrating why llms' hallucinations might prevent musicologists' trust. in this work, we focus on musicology, a field where the impact of llms still needs to be explored. musicology, the scholarly study of music, spans from historical research to theoretical analysis @xcite . our research mainly focuses on the former, an area which might be greatly supported by llms, e. g., by breaking language barriers, enhancing information retrieval, or supporting teaching and learning. however, reliable sources, such as musicspecialized lexica, monographies, and research articles, are often, unlike in more technical disciplines, not open-access, which prevents llms to access high quality information. this knowledge deprivation further increases the risk of llms to hallucinate, which often leads to non-reliable text generation in musicology related topics. through a pilot-survey involving experts and students from the field of musicology, we gather initial insights into the acceptance and trustworthiness of llms in domain-related tasks, and its potential impact for music professionals. subsequently, we propose a methodology to measure to which extent such models posses domain expertise in the field of musicology, by this assessing their practical value for the discipline. we adopt a multiple-choice question generation @xcite approach to semi-automatically construct a benchmark leveraging recent advancements in retrieval-augmented generation models @xcite . to automatically generate high-quality questions, we provide the generation model with domain-knowledge from the new grove dictionary of music and musicians @xcite , an established and reliable source. the final benchmark, made up by 400 question-answer pairs validated by a human expert, is evaluated on several open-source models. this dual approach-survey and benchmarkprovides a comprehensive understanding of the challenges and potential solutions for meaningful integration of llms in musicology. ## pilot-survey: llms in musicology we conducted a survey targeting professionals related to musicology. the survey included questions to identify the respondent's domain of study (e.g., musicology, composition, music pedagogy, music performance), the highest level of music education completed or being pursued, and their familiarity with technologies known as llms such as chatgpt. additionally, the survey inquired about the frequency of interactions with llms, particularly in the context of musical topics like music theory and music history. participants were asked to rate the trustworthiness and usefulness of llms for these subjects, as well as to consider its revolutionary impact on the field of musicology. lastly, the survey explored the possible consequences of llms on music professionals, both presently and in the future. a total of 33 participants, having or pursuing a bachelor's degree in music, completed the survey: 20 students, 7 lecturers, 11 researchers, and 8 music educators (multiple areas can be selected). in terms of discipline, the respondents are distributed as the following: 22 musicology and related studies, 10 music performance, 3 music pedagogy, 2 composition, 1 conduction, and 1 music therapy. while only one participant (from the field of musicology) had not heard about llms before, in terms of the participants' frequency of use and trustworthiness, a noticeable gap between students and teachers foot_0 can be observed. figure 2 illustrates how often students and teachers interact with llms in general and about music topics. teachers frequently or not at all, while most students use it weekly or monthly. however, both groups tend to not use llms for music-related topics. participants' judgement of llms' trustworthiness is depicted in figure 3 and the trend of ratings is similar across both groups. additionally, confidence in llms is slightly higher for music history than for music theory, indicating a nuanced perception of their reliability in different musicology subfields. most of the participants (78%) agreed that llms might revolutionize the field of musicology (cf. figure 4 , left). while the anticipated potential consequences of llms for the field are varied, professional transformation seems to be the most prominent (20 votes), as illustrated in the histogram (cf. figure 4 , right). in conclusion, despite limited current usage and trust, experts anticipate a significant future impact of llms on musicology, motivating current research on the topic. ## musicology benchmark: trustmus this section outlines our strategy for evaluating how much llms hallucinate in musicology. it summarizes the creation of the human-validated multiple-choice benchmark trustmus, i. e., a collection of reliable questions related to various musical topics and concepts, and analyzes the models' performance on the benchmark. following previous works @xcite , multiple-choice questions are generated after extracting relevant information from a text source: here, the grove dictionary online @xcite ). 2 in order to identify the most relevant articles within the text source, we used a pagerank-like algorithm @xcite . to accelerate the creation of trustmus, we designed a workflow inspired by recent works @xcite @xcite , as shown in figure 5 . first, we generated five questions from each article, each with four possible answer options, using a fine-tuned llm for retrieval-augmented generation (rag) @xcite , resulting in 7 500 questions. second, we discard questions that did not have relation with musicology or a unique and unambiguous answer, by prompting the same llm to decide based on the article, eliminating 2 632 questions. next, we attempted to answer the remaining questions using a rag-like model that we term llama professor by giving the arti2 the grove dictionary is a copyrighted work. using its content for generating questions is under fair use for research purposes. the eu directive on copyright in the digital single market allows text and data mining for research purposes. cle as context to the llm. questions for which llama professor chooses the wrong answer option are considered ambiguous or unusable and are thus removed, resulting in 3 285 valid questions. all previous prompts used the chain of thought (cot) method to enhance the model's reasoning skills @xcite . before human intervention, we attempted to answer the questions with llama3-8b (ai@meta, 2024) without rag and in one shot, i. e., without the chain of thought (cf. the difficulty filter in figure 5 ), which lead an accuracy of 67.4%. thus, arguably simple questions are eliminated, resulting in 1 081 domain-ones. the resulting set of questions was automatically classified with a cot prompt into four classes, according to their topic: people (ppl); instruments and technology (i&t); genres, forms, and theory (thr); culture and history (c&h). an expert human annotator validated questions until 100 valid ones per class were identified (on average, 17% of those assessed were discarded). 3 4 results and discussion ## conclusions our paper shows that while current usage and trust in llms in musicology are low, there is a strong expectation of future impact. however, llms are not yet at the required level for the field and do not meet the minimum quality, ethical and likely legal standards currently being discussed. through the proposed semi-automatic benchmark, we present a first attempt to measure llms hallucinations on musicology-related tasks. this approach aims to facilitate the evaluation of future models, which promotes transparency and trustworthiness of the technology. despite the effort, this initial experiments are insufficient. besides a more thorough evaluation, there is the need to specialize current models for musicology-related tasks, while reducing their environmental footprint. further research should focus on ensuring llms reliability to avoid misinformation, protecting user privacy and data security, and mitigating training data biases to promote responsible use in musicology. collaboration between the technological, musicological, and content owner communities is essential for the proper development of this technology.