Moûsai: Efficient Text-to-Music Diffusion Models (2024)

Abstract: Recent years have seen the rapid development of large generative models for text; however, much less research has explored the connection between text and another “language” of communication – music. Music, much like text, can convey emotions, stories, and ideas, and has its own unique structure and syntax. In our work, we bridge text and music via a text-to-music generation model that is highly efficient, expressive, and can handle long-term structure. Specifically, we develop Moûsai, a cascading two-stage latent diffusion model that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions. Moreover, our model features high efficiency, which enables real-time inference on a single consumer GPU with a reasonable speed. Through experiments and property analyses, we show our model’s competence over a variety of criteria compared with existing music generation models. Lastly, to promote the open-source culture, we provide a collection of open-source libraries with the hope of facilitating future work in the field. We open-source the following: Codes: https://github.com/archinetai/audio-diffusion-pytorch. Music samples for this paper: http://bit.ly/44ozWDH. Music samples for all models: https://bit.ly/audio-diffusion.

## introduction in recent years, natural language processing (nlp) has made significant strides in understanding and generating human language, due to the advancements in deep learning and large-scale pre-trained models @xcite @xcite . while the majority of nlp research has focused on textual data, there exists another rich and expressive "language" of communication -music. music, much like text, can convey emotions @xcite , stories @xcite , and ideas @xcite , and has its own unique structure and syntax @xcite . ## moûsai: efficient long-context music generation from text our model moûsai contains a two-stage training process. in stage 1, we use diffusion magnitudeautoencoding (dmae), which compresses the audio waveform 64x using a diffusion autoencoder. in stage 2, we use a latent text-to-audio diffusion model, to generate a novel latent space by diffusion while conditioning on text embeddings obtained from a frozen transformer language model. ## conclusion in this work, we presented moûsai, a novel textto-music generation model using latent diffusion. we show that, in contrast to earlier approaches, our model can generate minutes of music in realtime on a consumer gpu, with good music quality and text-audio binding. in addition, we provide a collection of open-source libraries to facilitate future work in the field. the work helps pave the way towards higher-quality, longer-context text-tomusic generation for future applications.