C o T - VTM : Visual-to-Music Generation with Chain-of-Thought Reasoning (2025)

Abstract: The application of visual-to-music generation (VTM) is rapidly growing. However, current VTM methods struggle with capturing the relationship between visuals and music in open-domain settings, mainly due to two challenges: the lack of large-scale, high-quality visual-music paired datasets and the absence of direct semantic correspondence between visuals and music. In this work, we propose CoT-VTM, a framework that distills Chain-of-Thought (CoT) reasoning to enable visual-to-music generation without paired data, while efficiently producing music aligned with visual content in open-domain settings. We first bridge the gap between visual, music, and text data using appropriate foundation models. Next, we identify key elements of the visual-music relationship and design a CoT prompt for visual-to-music mapping. To fully distill the reasoning of CoT, we incorporate latent information from intermediate reasoning steps as supervisory signals alongside visual and music supervision. Finally, we design a two-stage mapping distillation training process: the first stage uses discriminative MLP modules, while the second uses a generative embedding diffusion model (EDM). Our model achieves optimal performance on both image-to-music and video-to-music tasks. Project page: https://xxkkxxx.github.io/cot-vtm/

## introduction synesthesia, the phenomenon of associating music with specific visuals @xcite , has practical applications across various fields. in the film industry, appropriate background music enhances viewer immersion, a technique now widely used in advertising, animation, and social media content. however, manually selecting or composing music is costly and poses copyright challenges. thus, this study focuses on generating instrumental music from visual content within an open visual domain. ## conclusion in this work, we introduced cot-vtm, a framework that leverages cot reasoning to address open-domain vtm without relying on large-scale paired datasets. by integrating latent information from cot as supervisory signals, our two-stage approach-using mlp for visual-to-latent mapping and embeddiff for latent-to-music generation-efficiently produces high quality and semantically aligned music. experimental results show that cot-vtm outperforms existing models in both image-to-music and video-to-music tasks, providing a scalable and data-efficient solution for vtm. this work demonstrates the potential of cot reasoning in bridging visual and musical domains, advancing cross-modal generation tasks.