Can Impressions of Music be Extracted from Thumbnail Images? (2024)

Abstract: In recent years, there has been a notable increase in research on machine learning models for music retrieval and generation systems that are capable of taking natural language sentences as inputs. However, there is a scarcity of large-scale publicly available datasets, consisting of music data and their corresponding natural language descriptions known as music captions. In particular, non-musical information such as suitable situations for listening to a track and the emotions elicited upon listening is crucial for describing music. This type of information is underrepresented in existing music caption datasets due to the challenges associated with extracting it directly from music data. To address this issue, we propose a method for generating music caption data that incorporates non-musical aspects inferred from music thumbnail images, and validated the effectiveness of our approach through human evaluations.

## introduction the enjoyment of music is a highly personal experience, and the ways in which individuals find pleasure in it vary greatly. one method of enjoying music involves listening to tracks that best suit one's current mood or situations, such as during events like birthdays or in accordance with the season. furthermore, individuals often find pleasure in music through discovering songs that match their preferred genres or instruments, and even in creating new music by utilizing the characteristics of existing music. as our interaction with music becomes more sophisticated, the development of music retrieval and generation systems that are customized to individual tastes and preferences is ## related work to address the scarcity of music caption datasets, the lp-musiccaps (llm-based pseudo music captioning) initiatively utilized gpt-3.5 turbo (openai, 2022), a large language model (llm), for generating music captions across various tasks employing a comprehensive music tag dataset @xcite . the dataset produced, encompassing approximately 2.2 million captions and 500,000 audio clips paired with tags, has been made publicly available. it facilitates the automated generation of music captions with an enriched vocabulary while ensuring alignment with tags and grammatical precision. notably, these methods utilize llms to focus on generating music captions that incorporate musical elements derived from tags, in contrast to our approach of enriching captions with non-musical aspects. while our study focuses on the automatic generation of music captions enriched with non-musical aspects using thumbnail images, there exists related research that leverages image intermediary representations for generating lyrics @xcite . this research tackles the challenge of assisting users in conveying appropriate messages and words in lyric creation. contrastive language-image pre-training (clip) @xcite uses a dual-encoder architecture to embed images and text into a shared latent space. clip has been influential in its demonstration of extracting highly generic visual representations from natural language, and muscall @xcite also adopts this dual-encoder contrastive learning approach pioneered by clip. mulan @xcite represents a similar endeavor in this field. additionally, clap @xcite is a model that applies clip's methodology to the audio domain. a cross-modal retrieval method between music and images has been also investigated in @xcite . in this study, we train a music retrieval model using pairs of music captions and music data created through the proposed method, and assess its effectiveness in music retrieval tasks where input queries are written in natural language sentences. evaluation metric details of evaluation metric score positive assuming a music generation model that generates music from the provided text, this text contains and accurately expresses the corresponding non-musical aspects. 2 neutral assuming a music generation model that generates music from the provided text, this text contains the corresponding non-musical aspects, but their expression is not accurate. 1 negative this text does not contain the corresponding non-musical aspects. 0 ## proposed method this study is centered around thumbnail images linked to music clips on platforms like youtube. figure 1 provides an overview of our proposed methodology. in our approach, we first input the thumbnail image into a large vision-language model (lvlm), a large-scale language model capable of processing images. following this, the lvlm, adapted using a meticulously crafted prompt, generates a music caption. this procedure facilitates the automatic generation of music captions that incorporate non-musical elements. to ensure that the features of the thumbnail images are not included in the captions generated by the lvlm, we developed a strategic prompting method. specifically, we guide the model to initially describe the features of the image itself, such as the content depicted in the thumbnail or its overall mood, in the initial segment of the generated text. in the subsequent part, we prompt the model to articulate non-musical aspects that are evoked by these features or the overall mood. moreover, by directing the lvlm to distinctly express the features of the thumbnail image in the first portion of the generated text, we successfully separate the description of thumbnail image features from the music caption generation. this not only achieves effective compartmentalization but also leverages techniques like the chain of thought @xcite , resulting in the production of high-quality music captions enriched with diverse non-musical elements. ## conclusion in this study, we examined the importance of nonmusical aspects, such as suitable situations for listening, times of day or seasons, and emotions evoked by listening. specifically, we addressed the issue of the shortage of music caption data that includes non-musical aspect information, which is essential for constructing music retrieval and generation models. to address this issue, we proposed a method for generating music captions by leveraging thumbnail images. the effectiveness of the proposed method was validated through human evaluations. it has been reported that using the latest lvlms to generate descriptions related to artworks has been successful @xcite @xcite , and we plan to update our experiments in accordance with the further advancements of lvlms.