Mustango: Toward Controllable Text-to-Music Generation (2024)

Abstract: The quality of the text-to-music models has reached new heights due to recent advancements in diffusion models. The controllability of various musical aspects, however, has barely been explored. In this paper, we propose Mustango: a music-domain-knowledge-inspired text-to-music system based on diffusion. Mustango aims to control the generated music, not only with general text captions, but with more rich captions that can include specific instructions related to chords, beats, tempo, and key. At the core of Mustango is MuNet, a Music-Domain-Knowledge-Informed UNet guidance module that steers the generated music to include the music-specific conditions, which we predict from the text prompt, as well as the general text embedding, during the reverse diffusion process. To overcome the limited availability of open datasets of music with text captions, we propose a novel data augmentation method that includes altering the harmonic, rhythmic, and dynamic aspects of music audio and using state-of-the-art Music Information Retrieval methods to extract the music features which will then be appended to the existing descriptions in text format. We release the resulting MusicBench dataset which contains over 52K instances and includes music-theory-based descriptions in the caption text. Through extensive experiments, we show that the quality of the music generated by Mustango is state-of-the-art, and the controllability through music-specific text prompts greatly outperforms other models such as MusicGen and AudioLDM2.

## introduction recently, diffusion models @xcite have shown prowess in image (openai, 2023a), audio (liu et al., 2023a,b; ghosal et al., 2023; borsos * co-first authors. both authors contributed equally. † both authors contributed equally and led this project. @xcite and music @xcite generation tasks. generating music directly from a diffusion model poses some unique challenges. first, music adheres to specific rules related to, for instance, tempo, key, and chord progressions. evaluating whether or not the generated music follows these conditions remains challenging. for instance, musiclm @xcite , a text-to-music model, ensures that the generated music matches the text prompts in terms of instrumentation and music vibe. however, the musicality of the generated music (e.g., musically meaningful harmonies and steady tempo) remains only partially addressed. secondly, the availability of paired music and textual description datasets is limited @xcite . although the textual descriptions in the existing datasets include details like instrumentation or vibe, more representational descriptions that capture the structural, melodic, and harmonic aspects of music are missing. we thus argue that including this information during generation may improve the current text-to-music models in terms of musicality and controllability (e.g., following metrical structure and chord progressions). more information on related work can be found in appendix e. beyond existing text-to-music systems' capability (e.g., setting correct instrumentation), our proposed mustango model enables musicians, producers, and sound designers to create music clips with specific text-specified conditions like following a chord progression, setting tempo, and key selection. in this paper, we release the musicbench dataset which is derived from the musiccaps @xcite dataset and propose mustango to address these challenges. to create the musicbench dataset, we use two augmentation methods: description enrichment and music diversification. the aim of description enrichment is to augment the existing text descriptions with beats and down-beats location (inferred from tempo information in the text prompt), underlying chord progression, key, and tempo as control information. during inference, these additional descriptive texts could steer the music generation towards user-specified music quality. we use state-of-the-art music information retrieval (mir) methods @xcite @xcite to extract such control information from our training data. furthermore, to diversify the music samples in the training set, we augment this dataset with variants of the existing music, altered along three aspects-tempo, pitch, and volumethat essentially determine the rhythmic, harmonic, and interpretive aspects of music. the text descriptions are also altered accordingly. the resulting musicbench dataset is 11 times the size of the original musiccaps @xcite dataset. our proposed controllable text-to-music model mustango incorporates a novel munet (musicdomain-knowledge-informed unet) that can instill the input chords, beats, key, and tempo, along with the textual description, in the generated music during the reverse-diffusion process. the results in §4 indicate mustango creates more musically meaningful output and shows improved controllability (e.g., changing chords) over the existing text-to-music models. our musicbench dataset, mustango implementation, and comparative music samples are available through https: //github.com/amaai-lab/mustango. the overall contributions of this paper are: (i) we propose mustango, a text-to-music diffusion model with our novel munet module to explicitly guide the music generation towards input tempo, key, chords, and general textual description. (ii) we release the musicbench dataset with ∼53k pairs of music audio and description with information on musical attributes like chords, key, and beats. this is achieved by altering the music samples of musiccaps along the harmony, tempo, and volume dimensions and enriching the captions with the aforementioned musically-relevant attributes. (iii) we empirically verify that our mustango model is able to generate high quality music faithful to the input text descriptions, chords, and beats. ## dataset creation in this section, the methods of music feature extraction and data augmentation are introduced. then, the application of these methods and the details of our dataset are discussed. ## mustango mustango consists of two components: 1) latent diffusion model; 2) munet. ## experiments we conduct extensive objective and subjective evaluations to answer these research questions: i) how is the audio quality of the music generated by mustango? ii) does mustango generate music with better music quality compared to other baselines? iii) is mustango more controllable in terms of music-specific instructions? iv) is our data augmentation approach effective -can models trained on only this dataset compete with large-scale pretrained models? ## conclusion in conclusion, mustango presents a significant advancement in the field of controllable text-to-music generation. mustango is a controllable diffusionbased text-to-music system inspired by musicdomain knowledge which is able to generate music that follows certain music properties embedded within user-specified text prompts. the integration of the munet module within mustango enables greater music controllability over stateof-the-art text-to-music systems such as tango, audioldm2 and musicgen. we also made our dataset musicbench and model publicly available. musicbench contains 11 times more data than the original musiccaps dataset and includes text prompts that contain music-theory-based description and augmented music audio.