FUTGA : Towards Fine-grained Music Understanding through Temporally-enhanced Generative Augmentation (2024)

Abstract: We propose FUTGA, a model equipped with fined-grained music understanding capabilities through learning from generative augmentation with temporal compositions. We leverage existing music caption datasets and large language models (LLMs) to synthesize fine-grained music captions with structural descriptions and time boundaries for full-length songs. Augmented by the proposed synthetic dataset, FUTGA is enabled to identify the music’s temporal changes at key transition points and their musical functions, as well as generate detailed descriptions for each music segment. We further introduce a full-length music caption dataset generated by FUTGA, as the augmentation of the MusicCaps and the Song Describer datasets. The experiments demonstrate the better quality of the generated captions, which capture the time boundaries of long-form music.

## introduction natural language music understanding, which extracts music information and generates detailed music captions, is a fundamental task within the mir community, beneficial for a series of applications including music generation @xcite @xcite , editing @xcite , question-answering @xcite , and retrieval @xcite @xcite . recent developments in music foundation models @xcite @xcite enable free-form music prompts and multitasking. these foundation models are developed based on pre-trained large language models (llms) and aligned with the music modality. although llm-powered music understanding models can leverage the abundant pre- the music begins with a slow, sustained note from the bansuri. the tabla joins in with a rhythmic pattern. the string instruments join in with a simple melody. the tempo increases slightly. the bansuri plays a more prominent role in the melody. the string instruments continue to play a simple melody. the chord progression remains the same. the music fades out with the sound of people clapping. there are no other instruments or voices in this segment. the tempo is very low. ## temporally-enhanced generative augmentation in this section, we introduce our proposed temporally-enhanced generative augmentation. due to the limitation of existing music caption datasets, music captioning and understanding models can only generate global music descriptions for short music clips @xcite @xcite . to address this limitation, we propose the augmentation of synthetic music and caption composition, which empowers music understanding models with capacities of time-aware music segmentation and fine-grained music description generation. for each sampled set of music clips c k , the corresponding music caption set t k and the clip length information l k are interleaved and composed by the template, we use a relative time-boundary representation approach to minimize training bias towards specific numbers of music lengths in our model. in addition, relative time-boundary representation enables the model's ability to comprehend music of varying lengths, thereby improving the model's generalizability. by generating these relative time boundaries, our generative music understanding model gains a better awareness of the music's overall progression, which further enhances the model's temporal understanding of music. to further augment rich mir features in music captions, we further propose to use a text-only large language model (llm) to augment the templatebased caption xk with natural language descriptions, in which additional information, such as global captions, musical changes, and music structures, can be automatically extracted from the llm. since llms are pre-trained with abundant domain knowledge including music analysis @xcite and music information retrieval @xcite , with enough context provided, such llms can accurately extract music information via languagebased summarization and reasoning @xcite . inspired by llms' capacities in language reasoning, we propose to paraphrase and augment additional music information with instructions as follows: context: music analysis { xk }. this is a music analysis of a song. note that the numbers indicate the time-boundaries of functional segments in this musiccaps: the low quality recording features footsteps, birds chirping, distant traffic and wind sound effects. at the very end, there is a suspenseful, low, widely spread pad chord. it sounds like it is a sound for some kind of video. song describer: a rock song with a french speaking male voice. it has the typical feeling of political/social rock song with a really simple guitar, bass and drum skeleton. ## dataset creation: futga based on the final version of our proposed music captioning model, we automatically generate music captions for whole songs between 2 minutes and 5 minutes in musiccaps @xcite and song describer @xcite . during inference time, we set the repetition penalty as 1.5 to prevent repetitive descriptions of the same music segments. in addition, we also set the beam search number to 10 to find the statistically best captions. we allow a maximum of 2048 tokens to be generated from futga. as demonstrated in the comparison example in in terms of music caption diversity, we first show that our captions have significantly larger numbers of tokens and vocabulary size, compared to existing music caption datasets. second, our dataset still maintains good diversity in terms of unique genre, instrument, and music mood vocabularies, which are comparable to human or gpt-3.5 annotations. thus, the futga dataset can serve to augment existing music captioning models with strong temporal reasoning abilities without harming the model's generalizability, which will be further evaluated in our evaluation section. ## experiments and results we obtain 5k synthetic training samples by prompting the gemma-7b model @xcite with the template-based caption xk and the designed instructions. then we adopt lora @xcite instruction finetuing of the salmonn-7b @xcite backbone model for 100 epochs and the learning rate of 1e-5, with 2 nvidia rtx a6000 gpus with 48gb each. we use the bfloat16 type for training with the batch size set to 4 and gradient accumulation steps to 8. we first evaluate the generated data samples' quality by comparing them to existing human annotation datasets, musiccaps @xcite and song describer @xcite . we follow the previous works @xcite and report the metrics, bleu (b), meteor (m), rouge (r), and bert-score (b-s), in based on the results in however, the complete music captions generated from futga show relatively inferior performance on musiccaps, which is mainly due to the different forms of music captions. since futga focuses on the temporal reasoning of a whole song, the time segment information and musical changes are completely new to both the original musiccaps captions. whereas, lp-musiccaps is directly augmented from musiccaps, which makes their captions formally more similar. such observations can motivate future works to explore more fine-grained and complex music caption forms in terms of evaluating the model's generation capacities. ## conclusion in this work, we propose a temporally-enhanced music caption augmentation method through generative large language models. by bootstrapping existing music captions with time boundary tags, mir features, and musical changes, we fine-tune the pretrained music understanding model salmonn-7b, where we observe emerging music segmentation capacities and enable instruction prompting to guide the generation with ground-truth time segments. we use the fine-tuned model to reannotate the existing musiccaps and song describer datasets with full-length songs. the generated captions are shown to be more fine-grained and beneficial for various downstream tasks. for future works, since our model is the first to enable end-to-end full-length song captioning with significantly longer context provided (10 times more than conventional music captions), we are motivated to further develop a long-context-based clap model, which can enable more complex and longer music retrieval tasks. in addition, with more fine-grained details provided by our captions, we propose to further use such captions for more complex music understanding tasks, including music question-answering and whole-song generation.