Cross-Modal Learning for Music-to-Music-Video Description Generation (2025)

Abstract: Music-to-music-video generation is a challenging task due to the intrinsic differences between the music and video modalities. The advent of powerful text-to-video diffusion models has opened a promising pathway for music-video (MV) generation by first addressing the music-to-MV description task and subsequently leveraging these models for video generation. In this study, we focus on the MV description generation task and propose a comprehensive pipeline encompassing training data construction and multimodal model fine-tuning. We fine-tune existing pre-trained multimodal models on our newly constructed music-to-MV description dataset based on the Music4All dataset, which integrates both musical and visual information. Our experimental results demonstrate that music representations can be effectively mapped to textual domains, enabling the generation of meaningful MV description directly from music inputs. We also identify key components in the dataset construction pipeline that critically impact the quality of MV description and highlight specific musical attributes that warrant greater focus for improved MV description generation.

## introduction generating a music-video (mv) to match a given piece of music is a challenging task due to the inherent differences between the music and video modalities. despite the challenges, mv generation holds significant potential for enhancing the music experience by providing a visual narrative that aligns with the music's tone, style, and mood, offering a more immersive and engaging way for audiences to connect with the music. compared to generating music or audio from a given video (tian et al., 2024; @xcite , the reverse task is more complex, as the video modality typically conveys richer spatial and temporal information than music. however, with the advent of text-to-video diffusion models @xcite @xcite , videos can now be generated from textual descriptions. this development enables mv generation to be divided into two subtasks: (1) music-to-mv description generation and (2) mv description-to-mv generation. as illustrated in fig. 1 , mv descriptions can be further refined using large language models (llms) like gpt (openai, 2023) to fit specific text-to-video models @xcite . in this study, we focus on the first task: generating mv descriptions from music. to this end, we propose a practical pipeline for data construction and model training to generate meaningful mv descriptions based on music inputs. additionally, we explore methods to enhance the alignment of the generated descriptions to the given music. specifically, we investigate the impact of various data sources-such as music, music genre tags, mv type tags, and lyrics understanding text-on the quality of the generated mv descriptions when fine-tuning multimodal llms like next-gpt (wu et al., 2024). as shown in fig. 1 , our approach first leverages existing music understanding models @xcite to extract lyrics understanding text. we then fine-tune a multimodal llm to process these diverse inputs and generate mv descriptions. the training data is constructed from gold-standard mvs, incorporating music-related information to enhance the connection between music and the generated descriptions. unlike prior studies on mv generation, such as vipe (shahmohammadi et al., 2023), which focused solely on lyrics as input, our work emphasizes leveraging multiple modalities and evaluates the effectiveness of various combinations of input data in connecting multimodal representations for mv description generation. to facilitate this study, we construct a musicto-mv description training and evaluation dataset using the music4all dataset (santana et al., 2020). ## proposed method in this section, we present the pipeline proposed for training a multimodal llm specifically tailored to the music-to-mv description generation task. for the first time, our pipeline incorporates a broader range of musical information beyond lyrics as inputs, aiming to enrich the holistic understanding of the music. additionally, we introduce strategies to ensure the generated mv descriptions are more closely aligned with the musical inputs. the curated dataset is then utilized to fine-tune a multimodal llm for performing the mv description generation task.