Harnessing High-Level Song Descriptors towards Natural Language-Based Music Recommendation (2024)

Abstract: Recommender systems relying on Language Models (LMs) have gained popularity in assisting users to navigate large catalogs. LMs often exploit item high-level descriptors, i.e. categories or consumption contexts, from training data or user preferences. This has been proven effective in domains like movies or products. In music though, understanding how effectively LMs utilize song descriptors for natural language-based music recommendation is relatively limited. In this paper, we assess LMs effectiveness in recommending songs based on user natural language requests and items with descriptors like genres, moods, and listening contexts. We formulate the recommendation as a dense retrieval problem and assess LMs as they become increasingly familiar with data pertinent to the task and domain. Our findings reveal improved performance as LMs are fine-tuned for general language similarity, information retrieval, and mapping longer descriptions to shorter, high-level descriptors in music.

## introduction music recommender systems are often used to assist users in navigating the vast catalogs offered by streaming platforms. recently, language models (lms) have demonstrated efficacy in recommending items such as songs, movies, or books @xcite . they achieve this by 1) matching concise user profiles derived from item preferences expressed in natural language against the available set of items; and 2) drawing upon their knowledge about specific items gained during pre-training or after fine-tuning for the specific task or domain. a common thread among these strategies is the reliance of lms on item high-level descriptors, such as genres (e.g. "romantic comedy" for a movie) or consumption contexts (e.g. "running" for music), within the pre-training corpus or during user preference elicitation phase. empirical evidence suggests that these descriptors play a crucial role in generating relevant recommendations with lms, in cases like near cold-start or exploration in movie or product domains @xcite @xcite . leveraging these descriptors has been proven particularly valuable and could surpass the importance of item examples alone such as in collaborative filtering @xcite . in music however, our understanding of how efficiently lms utilize item textual features and user preference description for music recommendation is comparatively limited. previous research has emphasized the significance of natural language features such as tags in improving music retrieval @xcite and captioning algorithms @xcite . these accessible descriptors help bridge the semantic gap between audio and more complex song descriptions provided by humans @xcite . yet, such a study is lacking in the field of music recommendation despite advancements in related areas like conversational recommendation systems @xcite @xcite . in this work, we study the efficacy of lms for song recommendation when users express their preferences in natural language and items are associated with high-level descriptors such as music genres, styles, moods, and listening contexts. as no large dataset linking natural language user preferences to music descriptors is available, we re-purpose an existing dataset originally created for music captioning @xcite . subsequently, we formulate the recommendation task as a dense retrieval problem @xcite and propose evaluating the lms as they become increasingly familiar with data pertinent to the target task (recommendation as retrieval) and domain (music). this contrasts with previous approaches to integrating lms, that typically involve encoding low-level item data (such as audio or embedded metadata) and user queries @xcite , and are likely less effective because of the inherent semantic gap in music. our results show that a) pre-trained lms with no further fine-tuning perform quite poorly on this task in music; b) the performance gets better when the lms are progressively fine-tuned for text similarity, for multi-domain query retrieval and for mapping longer descriptions on shorter, high-level descriptors in music. our approach renders the dataset and models suitable also for retrieving explanations based on song high-level descriptors or encoding any type of item textual information in multi-modal music systems. we release the code and fine-tuned models at https: //github.com/deezer/nlp4musa_melscribe . ## related work pre-trained lms are widespread and their usage spans various tasks, including generating synthetic data, producing system utterances, and providing recommendations. penha and hauff (2020) finetune bert on recommendation conversations on movies, books, or music extracted from reddit and assess the resulting model's capability to retrieve the most relevant utterance containing a suitable recommendation from existing conversations. similarly, chaganty et al. (2023) model the recommendation task as retrieval and fine-tune multiple checkpoints of a t5 encoder to create a common embedding space of conversations and song metadata such (i.e. title, artist and album). @xcite build a new dataset by using instruct-gpt to generate narrative-driven recommendations in the point-of-interest (poi) domain. then, they fine-tune dense retrieval models for this domain. other studies depend on lms for generating system inputs in natural language. hayati et al. (2020) build a dialogue model comprising two distinct language modules tailored to the recommendation seeker and the recommender system. @xcite fine-tune a t5 for learning to generate relevant indirect questions about the context of item consumption in order to help the user to elicit preferences when these are not clearly defined. similar to these works, we adopt a recommendation as retrieval approach and assess lms for our song recommendation task in scenarios where music preferences are given in natural language. how-ever, our recommendation module relies solely on high-level descriptors rather than low-level ones extracted from audio or embedded metadata (e.g. song titles or artists). our hypothesis is that as the model needs to bridge a narrower semantic gap between these two music information sources, it would yield improved outcomes. ## method the recommendation task we consider is the learning to rank setup: given r, a recommendation request in natural language, and s, a corpus of n songs where each song s has associated t s , |t s | ≥ 1, a set of high-level descriptors (words or short phrases describing music), the recommendation task is addressed with a ranking function ρ over the collection s. if we consider an encoder lm with an embedding function f , which takes as input text and outputs a vector, then the ranking could be computed based on the dot-product similarity between the embeddings of the request r and the concatenated, and alphabetically sorted, high-level descriptors of each song the problem revolves around having an efficient text encoder f in music for embedding high-level descriptors and recommendation requests. for this, we train a music text bi-encoder using the generative pseudo-labeling method (gpl) @xcite . bi-encoders @xcite rely on two siamese encoders, comprising a pre-trained lm such as bert @xcite followed by a pooling layer (e.g. mean). to adapt it to music, we fine-tune it using contrastive learning, which entails that similar input texts (song high-level descriptors on one side and natural language recommendation requests on the other side) are embedded close together in the vector space, while dissimilar ones are far apart. two steps are essential in this method: the automatic negative mining and the pseudo-labeling of training examples. we mine hard negatives (r, t - s ) using pre-trained bi-encoders as follows: for each query r we retrieve the top k most similar high-level descriptors, t k , k ̸ = s. however, sometimes negatives could be closer to an actual positive example or even a false negative. for example, there might be instances when a particular user request for recommendation matches other songs as well, but this was not explicitly labeled in the training data. as in gpl @xcite , we resort to soft-labeling the training data, instead of considering a negative as a true negative. for each tuple (r, t s , t - s ), we compute a margin score δ @xmath0, concat(t s )) -g(r, concat(t - s )) with a cross-encoder g as a teacher. compared to biencoders that separately encode input texts to a shared vector space, cross-encoders take as input the concatenated texts and produce as output a similarity score. cross-encoders are known to be more effective at the text similarity task than bi-encoders but do not scale well @xcite . implementation details we choose models pretrained on ms-marco, a dataset with search queries and documents from various domains @xcite , because of its similarity to our domain and task, that proved useful experimentally too. we fine-tune msmarco-bert-base-dot-v5 foot_0 on our music data for 1 epoch and 140k steps with a batch size of 4. this model is chosen as the backbone of our bi-encoder as it yields the best performance in our experiments. for each r, we use msmarco-distilbert-base-v3 and msmarco-minilm-l-6-v3 to mine 30 negative examples. to soft-label the training data, we fine-tune a domain-specific cross-encoder on mu-siccaps @xcite , a dataset with human-created song captions and descriptors. other baselines bert @xcite and mpnet @xcite ) are pre-trained on general language corpora using various objectives such as masked language modeling or permuted language. we apply mean as pooling function to all token embeddings to derive a fixed-size embedding for the given input. then, we consider the best-performing bi-encoders reported on sentence-transformers @xcite . other baselines we include are text encoders obtained from multimodal (audio-text) representation learning @xcite . their text encoding branch is initialized with bert or roberta weights. one difference between ttmr @xcite and clap (wu* et al., 2023) lies in the training dataset and the audio encoding branch. a tf-idf sparse representation is also considered in the experiments. although such a text encoding does not generalise to new vocabulary, we expect it to work well when recommendation requests and music descriptors have high exact term overlap. ## datasets and evaluation details since there is no large dataset linking natural language user preferences (r) with high-level song descriptors (t s ) in music recommendation, we repurpose a dataset originally created for music captioning, lp-musiccaps @xcite , to finetune our music text bi-encoder. lp-musiccaps was created from three pre-existing datasets of audio tracks annotated with tags (the ecals subset of the million song dataset (msd) @xcite , musiccaps (mc) @xcite , and magnatagtune (mtt) @xcite ) by using an instruction-based llm to generate captions from the given high-level descriptors. our goal is to extract training pairs (r, t s ) from lp-musiccaps by ensuring the compatibility with the desired use case of the text encoder, for conversational music recommendation. compared to narrative-driven recommendations, user requests are unlikely long in synchronous conversations @xcite . as the length of each caption in lp-musiccaps ranges from a sentence to a paragraph, we first split paragraphs in sentences. then, as in gpl where each query is seen with multiple documents during training, we ensure multiple sets of high-level descriptors for each sentence. we sample up to 3 variations of high-level descriptors from the original t s : first from overlapping high-level descriptors (i.e. tags or phrases from t s found in the sentence) and then from the non-overlapping ones (i.e. tags or phrases from t s not found in the sentence). like this we simulate cases where there is a varying number of high-level descriptors per song, and some may be irrelevant to the description. with llama3 2 (more details in appendix a). we use only the msd and mtt train splits of lp-musiccaps for training and keep mc completely unseen. each song s is associated with a concatenated set of high-level descriptors t s . the mapping from the description r to the most likely t k , @xmath2. then, we compute recall@10 at the level of descriptors (t s ), and not at the song level (s) as multiple songs could have the same set of high-level descriptors t s and t s is the only song information that we consider in this work. for each dataset, we produce 3 test sets by sampling a different set of high-level descriptors per song s and description r. we then report mean and standard deviation (std). ## results baselines' scores on the rephrased mc test set are lower compared to those on the original mc dataset. manual checks revealed that rephrasing descriptions as requests sometimes omitted initial descriptors, thus making it difficult to distinguish between the effects of rephrasing and information loss (a couple of examples are shown in ## conclusion conversational recommender systems or interfaces based on natural language have emerged as a practical alternative to dynamically elicit preferences from the users in cold-start or exploration cases. lms have emerged as central to these systems. in this work, we analysed the efficacy of lms towards song recommendation when users express their preferences in natural language and items have high-level descriptors. we showed that a biencoder fine-tuned in multiple phases first for the task and then for the domain is quite competitive. future works aims at improving the model on outof-distribution data, integrating more specialized music knowledge and personalisation during its fine-tuning, and joining the proposed encoder with other modalities for song recommendation. the fine-tuned models, which will be released, target only english-language content and have been exposed primarily to music descriptions and descriptors that mostly refer to western-centered music, with a limited number of music descriptions and descriptor set pairs. additionally, we are aware that music descriptors such as mood or genre can be specific to individuals, groups, or cultures. however, the embeddings we obtain with the fine-tuned models are deterministic and do not take into account any form of localization or personalization, which is a limitation with ethical implications. yusong wu*, ke chen*, tianyu zhang*, yuchen hui*, taylor berg-kirkpatrick, and shlomo dubnov. 2023. large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. in ieee international conference on acoustics, speech and signal processing, icassp.