A Retrieval Augmented Approach for Text-to-Music Generation (2024)

Abstract: Generative text-to-music models such as MusicGen are capable of generating high fidelity music conditioned on a text prompt. However, expressing the essential features of music with text is a challenging task. In this paper, we present a retrieval-augmented approach for text-to-music generation. We first pre-compute a dataset of text-music embeddings obtained from a contrastive language-audio pretrained encoder (CLAP). Then, given an input text prompt, we retrieve the top k most similar musical aspects and augment the original prompt. This approach consistently generates music of higher audio quality as measured by the Frechét Audio Distance. We analyze the internal representations of MusicGen and find that augmented prompts lead to greater diversity in token distributions and display high text adherence. Our findings show the potential for increased control in text-to-music generation.

## introduction modeling discrete representations of audio obtained from a neural audio codec has been an effective approach in tasks such as audio generation @xcite , speech synthesis @xcite and self-supervised learning @xcite . in particular, text-to-music generation @xcite @xcite @xcite has seen widespread adoption. despite their impressive capabilities, these models still suffer from distributional shifts, where underspecified user prompts lead to an audio quality degradation. furthermore, constructing text prompts that accurately capture the user's creative intent while also expressing the essential features of music remains a challenge. inspired by the success of retrieval augmented generation (rag) in natural language processing tasks, we present a retrieval augmented approach for text-to-music generation. while relatively simple, we show our approach consistently generates music of higher audio quality, while also displaying high text adherence. our findings show potential for increased control in text-to-music generation. ## results for the baseline generation methods, unconditional generation consistently generates music of poor audio quality. specifying a single musical aspect or first sentence of the caption improves quality, while the full text caption achieves the best scores. this suggests that the conditioning text encoder plays a key role in influencing the generation process. the clap score is highest when specifying a single musical aspect, likely because matching the audio to a single word presents a simpler task than aligning with a more complex sentence. for the rag methods, retrieving similar items based on the caption outperforms retrieving similar items based on the musical aspects. this is reasonable as the musical aspects aim to capture more qualitative features and as a result could diverge more from the intended caption description. retrieving five similar items based on the caption achieves the best fad scores, suggesting a tradeoff with prompt length. despite their relevance, too many aspect qualities may hinder the models performance instead of focusing on a select few. the best clap score is achieved by retrieving three similar items based on the caption, which aligns with the notion that matching fewer relevant words presents a simpler task. interestingly, the best kl score is achieved by retrieving ten similar items based on the musical aspect list. this could be due to our implementation of calculating kl, where we use a classifier trained for a multi-label task on audioset. by retrieving many diverse aspects, we increase the probability of matching with multiple labels. finally, retrieving random items generates music of worse or comparable audio quality to baseline generation methods. again, this suggests a tradeoff with the prompt length and relevance. this also demonstrates the ability of the pre-trained text encoder to transfer useful representations when generating diverse music. ## future work in this work, we explored the overall effect of an augmented prompt when generating music with musicgen. however, it would be more valuable to investigate specifically how the augmented tokens affect the internal representations. smitin @xcite trains classifier probes to identify selfattention heads that perform instrument recognition. then, they introduce an inference time intervention technique for steering the generated output towards the desired musical trait. extending this approach to other specific control methods for various musical features is a desirable goal. people value agency and control over creative collaborations with generative ai models. as such, we want to build systems that promote interactive, human-centered approaches. equipping musicgen with the ability to refine and build upon previously generated output is another valuable direction. ## conclusion in this paper, we presented a retrieval augmented approach for text-to-music generation. while relatively simple, we show our method consistently generates music of higher audio quality while displaying high text adherence. we compare the tradeoffs of various retrieval strategies and suggest extensions to this work. our findings show the potential for increased control in text-to-music generation. large scale generative models raises questions regarding ethics and societal consequences of their use. generative text-to-music models can represent an unfair competition for artists which is an open problem. another potential bias is the lack of diversity in the musiccaps dataset, which contains a larger proportion of western music. through open research, we hope that such generative models can become useful as a tool for amateur musicians and professionals.