Generative Music Models’ Alignment with Professional and Amateur Users’ Expectations (2025)

Abstract: Recent years have witnessed rapid advancements in text-to-music generation using large language models, yielding notable outputs. A critical challenge is understanding users with diverse musical expertise and generating music that meets their expectations, an area that remains underexplored.To address this gap, we introduce the novel task of Professional and Amateur Description-to-Song Generation. This task focuses on aligning generated content with human expressions from varying musical proficiency levels, aiming to produce songs that accurately meet auditory expectations and adhere to musical structural conventions. We utilized the MuChin dataset, which contains annotations from both professionals and amateurs for identical songs, as the source for these distinct description types. We also collected a pre-train dataset of over 1.5 million songs; lyrics were included for some, while for others, lyrics were generated using Automatic Speech Recognition (ASR) models.Furthermore, we propose MuDiT/MuSiT, a single-stage framework designed to enhance human-machine alignment in song generation. This framework employs Chinese MuLan (ChinMu) for cross-modal comprehension between natural language descriptions and auditory musical attributes, thereby aligning generated songs with user-defined outcomes. Concurrently, a DiT/SiT model facilitates end-to-end generation of complete songs audio, encompassing both vocals and instrumentation. We proposed metrics to evaluate semantic and auditory discrepancies between generated content and target music. Experimental results demonstrate that MuDiT/MuSiT outperforms baseline models and exhibits superior alignment with both professional and amateur song descriptions.

## introduction music is a universal language that transcends cultural barriers, yet the automated creation of music that aligns with human thoughts is a complex endeavor. in recent years, large language models for text-to-music generation have advanced rapidly, resulting in notable systems. a key challenge for these models is aligning generated music with the diverse expectations of users with varying musical expertise. however, this issue remains underexplored. to address this gap, we introduce the "professional and amateur description-to-song generation" task, which prioritizes the alignment of aigenerated music with both professional and amateur human expressions. we propose mudit/-musit, a single-stage framework based on two commonly used diffusion-transformer-like models @xcite for music generation. to address the limited coverage of chinese colloquial phrases in existing text-to-audio pre-trained models, we trained a chinese mulan (chinmu) cross-modal encoder based on clap @xcite and mulan @xcite architectures. the mudit/musit framework integrates cross-modal understanding via chinmu to generate music that aligns with user auditory expectations and musical conventions. during inference, chinmu converts user colloquial descriptions into vectors. these vectors, concatenated with random noise, serve as conditional inputs to dit/sit modules. operating in the vae (kingma and welling, 2013) latent space, dit and sit then generate songs corresponding to these descriptions. llm-generated lyrics can also be incorporated as conditions via a cross-attention mechanism. mudit/musit underwent a three-stage training process: module preparation, large-scale data pretraining, and annotated data fine-tuning. for finetuning, we utilized the muchin dataset @xcite . unlike datasets often limited to expertonly annotations or those with semantic gaps from automated music information retrieval (mir) tag-ging, muchin provides paired professional and amateur annotations for identical songs. this unique characteristic, previously validated for its richness with models like mert @xcite and jukebox @xcite , makes it highly suitable for training our model to align with these varied user expressions. models were evaluated using subjective and objective metrics, focusing particularly on their ability to understand meanings conveyed by users with varying musical expertise (amateur or professional) and to generate music meeting user expectations. in addition to fréchet audio distance (fad) for assessing generation quality, we introduced sst and asoa-novel metrics for semantic and auditory alignment with input descriptions. given that music generation lacks standard references, unlike text annotation, we supplemented with human evaluation as subjective metrics. for experimental comparison of alignment performance, we selected audioldm @xcite , stableaudio @xcite , and musicgen @xcite as baseline models. results indicate that mudit/musit surpasses baseline models in aligning with professional and amateur colloquial expressions, marking a significant step in human-ai collaborative music creation. this research, by introducing the "professional and amateur description-to-song generation" task, is the first to explicitly address the challenge of generating music that resonates with the everyday language of the general public. it tackles a critical gap in aligning ai-generated music with non-expert colloquial expressions and paves the way for more nuanced human-ai interaction in artistic expression. code implementation 1 . ## method this study adopts supervised learning techniques, complemented by the dual perspectives of professional and amateur annotators, and constructs the training process using an end-to-end single-stage approach. this ensures that the ai-generated songs align with human expectations, while also enabling the ai model to understand and incorporate the standard structure of human songs. ## conclusion our work represents a significant advancement in the field of human-ai collaborative music generation. the mudit/musit framework has demonstrated the potential to combine ai-generated music with professional and amateur colloquial expressions, paving the way for more inclusive and nuanced ai-generated artistic creations. future work will explore expanding datasets and applying our methods to other creative domains.