The Interpretation Gap in Text-to-Music Generation Models (2024)

Abstract: Large-scale text-to-music generation models have significantly enhanced music creation capabilities, offering unprecedented creative freedom. However, their ability to collaborate effectively with human musicians remains limited. In this paper, we propose a framework to describe the musical interaction process, which includes expression, interpretation, and execution of controls. Following this framework, we argue that the primary gap between existing text-to-music models and musicians lies in the interpretation stage, where models lack the ability to interpret controls from musicians. We also propose two strategies to address this gap and call on the music information retrieval community to tackle the interpretation challenge to improve human-AI musical collaboration.

## introduction in recent years, the field of human-ai music co-creation has experienced significant advancements @xcite @xcite . the advent of large-scale text-to-music generation models has played a crucial role in this progress, enabling generating music with good sonic quality and well-defined musical structures @xcite @xcite . a primary focus of recent research has been to enhance these models through the incorporation of control signals @xcite @xcite @xcite . this has led to significant success in manipulating dynamics, melody, and chord progressions in generated music contents. while precision in following these control signals can still be improved, these developments represent substantial progress. although extensive efforts are made to allow these models follow control signals precisely, misalignments between musicians' intent and model * both authors contributed equally. figure 1 : the comparison between human-human and human-ai interaction processes. we observe that the gap exists at both the interpretation stage and the execution stage, while the interpretation stage is often overlooked by current research. in this paper, our contribution is threefold: ## interpretation of controls to begin with, we propose a general framework that conceptualizes the musical interaction procedure in three stages: the expression, interpretation, and execution of controls, as shown in figure 2 . in an interaction between parties a and b: in this section, we examine the musical communication process following this framework. we observe that musical interactions often involve varying degrees of ambiguity in control expression, and skilled musicians can effectively interpret and execute these ambiguous instructions. in contrast, current text-to-music generation models struggle with this ambiguity, and can only understand highly semantical or highly precise instructions. ## potential solutions to improve interpretation of controls addressing the interpretation gap between musicians and models is challenging due to the complex, multi-modal nature of musician communication, which includes visual cues, textual prompts, vocalizations, and musical references. no existing data sources comprehensively capture all modalities of music interactions, and creating such a dataset would be resource-intensive. thus, we must approach the problem of learning interpretation under resource constraints. given these limitations, two potential solutions emerge: directly learning from many aspects of human interpretation data, or leveraging a strong prior understanding of human interpretation, such as that encapsulated in large language models (llms). in the following sections, we explore these two avenues for enhancing ai models' ability to interpret musical controls. ## conclusion we identify a critical gap in text-to-music generation models: their inability to effectively interpret musicians' controls. we propose a three-stage framework for musical interaction: expression, interpretation, and execution, and highlight how current ai models often struggle with the crucial interpretation stage. to address this gap, we suggest two potential solutions: directly learning from various sources of human interpretation data and leveraging large language models for musical interpretation. we call on the mir community to prioritize research in this area, as improving the interpretation capabilities is crucial for their integration into creative workflows and for realizing their full potential as collaborative tools for musicians.