Data-Efficient Playlist Captioning With Musical and Linguistic Knowledge (2022)

Abstract: Music streaming services feature billions of playlists created by users, professional editors or algorithms. In this content overload scenario, it is crucial to characterise playlists, so that music can be effectively organised and accessed. Playlist titles and descriptions are proposed in natural language either manually by music editors and users or automatically from pre-defined templates. However, the former is time-consuming while the latter is limited by the vocabulary and covered music themes. In this work, we propose PlayNTell, a data-efficient multi-modal encoder-decoder model for automatic playlist captioning. Compared to existing music captioning algorithms, PlayNTell leverages also linguistic and musical knowledge to generate correct and thematic captions. We benchmark PlayNTell on a new editorial playlists dataset collected from two major music streaming services.PlayNTell yields 2x-3x higher BLEU@4 and CIDEr than state of the art captioning algorithms.

## introduction playlists are a popular feature of music streaming services. users consume playlists for the 31% of their listening time @xcite . and, 55% of users create their own playlists @xcite . playlists are also created for users by professional editors or algorithms. for instance, the popular music streaming service, spotify, was hosting more than four billion playlists in 2021 @xcite . in this content overload scenario, it is crucial to characterise playlists, so that music can be effectively organised and accessed @xcite . a common approach is to rely on automatic playlist tagging with tags such as music genres or decades. however, tagging solutions are limited because they often resort to a pre-defined set and fall short of filling the semantic gap between audio and human-like descriptions @xcite . in contrast, natural language could be used to char-acterise playlists with less ambiguity and using a richer vocabulary. in fact, curators often provide a title and/or a description of their created playlist in natural language. however, describing playlists, especially the ones created by recommendation algorithms, is very labor-intensive and time-consuming when done manually @xcite . when automatic captions are proposed, these often rely on pre-defined templates thus cannot cover all kinds of cases, similar to tags @xcite . to address the above limitations, @xcite introduce the task of playlist captioning, that is automatically describing a playlist using natural language. playlist captioning can enable several useful applications, such as assisting curators in the process of finding an appropriate caption for a playlist; enabling search and discovery of playlists through human-like queries @xcite ; assigning captions to algorithm-generated playlists, that could be also used as explanations for automatic playlist recommendations @xcite . even so, playlist captioning is still an under-researched topic. as of today, we are aware of only two contributions in the field: @xcite and @xcite . although promising, these are afflicted by two main limitations. the first limitation is poor data quality. public datasets for playlist captioning rely on playlists created by users for personal use @xcite . instead, editorial playlists are created by professional editors for a public audience. @xcite find that user playlists may not have a strictly defined theme, while the editorial ones usually do. therefore, user playlists may not be an optimal source to learn representative captions, especially considering that the theme is regarded as a common playlist descriptor @xcite . the second limitation is the semantic gap. algorithms for playlist captioning consider as input embeddings of tracks, and strive to generate a cor-rect caption as output. for example, @xcite represent tracks as audio embeddings; and @xcite as random embeddings indexed by track id and updated by back-propagation. in both cases, the low-level input embeddings and the high-level output caption are separated by a "semantic gap" @xcite that algorithms are tasked to close. closing the semantic gap is challenging, especially because playists may be built around themes not easily deducible from embeddings alone. for example, an ireland and a uk thematic playlist could be confused given the cultural similarity these countries share. artist thematic playlists, e.g. "100% the beatles", are even more difficult to caption due to data sparsity. in the whole dataset, an artist can be absent or be mentioned only in some playlists, making it difficult for existing models to learn tracks-to-artists mappings that could be exploited to produce relevant captions @xcite . in this work, we propose playntell, a new multi-modal, data-efficient @xcite playlist captioning model that overcomes the above limitations by leveraging linguistic and musical knowledge, to generate english-language thematic captions. first, playntell narrows the semantic gap with musical knowledge. in particular, it leverages tags, e.g. "ireland", "rock" and "90s", that provide information at the same high semantic level as the expected captions. we also introduce an ad-hoc strategy to deal with artist thematic playlists, by masking artist mentions, and informing playn-tell with an artist distribution vector at encoding. second, we train playntell on a new highquality dataset of editorial playlists assembled from two major music streaming services, which we release together with the code 1 . however, as editorial playlists have the drawback of sparsity, i.e. our dataset is composed by only few thousand samples, training playntell from scratch to generate correct natural language captions is challenging @xcite . inspired by existing work in computer vision @xcite , we address this limitation by warm-starting the decoder with a pre-trained gpt-2 @xcite . we validate playntell with extensive quantitative experiments: playntell outperforms existing playlist captioning algorithms, achieving 2x higher bleu@4 and 3x higher cider. also, we observe via qualitative evaluation that it can gen-1 https://github.com/deezer/playntell caption playntell "80s smash hits the best tracks of the decade" ground truth "all out 80s the biggest songs of the 1980s" 3. an extensive evaluation of playntell reporting 2x-3x higher bleu@4 and cider than existing playlist captioning algorithms. we also provide a qualitative analyses of playntell, as well as an ablation study, sensitivity analysis to validate the contribution of different modalities and model components, and a user study. ## related work automated captioning is an active research field that has attracted much attention in the recent years. we can find attempts to caption images @xcite , videos @xcite , audio @xcite and music @xcite . here we review works in audio and music captioning, which are closest to our contribution. audio captioning focuses on identifying the human-perceived information in a general audio signal and expressing it through text, using natural language @xcite . for example, an audio caption is: "a door creaks as it slowly revolves back and forth". @xcite propose a transformer architecture for audio captioning that is similar to the original transformer for machine translation @xcite . the input audio is embedded with a pre-trained convolutional neural network (cnn) @xcite . the embeddings are input to a transformer encoder. then, a transformer decoder is tasked to generate the target caption. while audio captioning is concerned with general audio signals, music captioning deals with music audio signals. @xcite propose a recurrent neural network (rnn) encoder-decoder to tackle single song captioning. the multi-modal encoder takes as input the song audio embedding and the caption embedding up to token t. the song embedding is obtained with a pre-trained cnn @xcite . the caption embedding is obtained with a pre-trained word2vec (w2v)-like model. the decoder is tasked to generate the token t + 1. @xcite propose a rnn encoderdecoder to tackle playlist captioning. however, their attempt is not successful, mainly due to overfitting on a small training set. @xcite frames playlist captioning as a machine translation task. the source playlist is treated as a sequence of song ids. the target caption is treated as a sequence of token ids. the authors apply the seq2seq @xcite and transformer @xcite machine translation models to translate a playlist to a caption. they train the models on user playlists from the million playlist dataset @xcite . as detailed in section 1, current work on playlist captioning suffer from two main limitations: 1) poor data quality of existing datasets; 2) the semantic gap between the low-level input playlist embeddings and the high-level output caption. here, we tackle the two limitations by designing a model that takes advantage of high-level musical and linguistic knowledge, apart from audio, and by training it on a newly collected dataset of high-quality editorial playlists, better suited to capture themes. ## dataset public playlist captioning datasets are limited to user playlists, known to be noisier and sometimes not focused on a theme @xcite . we address the above limitation by introducing a new dataset of thematic editorial playlists. we collected public editorial playlists from spotify and deezer, which are two major music streaming services @xcite . each playlist consists of a sequence of track ids, a title and a description. both tracks, title and description are curated by a professional editor. each track is associated to at least one artist. some playlist are extremely short, e.g. just two tracks. we get rid of these outliers by filtering out playlists which number of tracks is below the fifth percentile. some other playlists are extremely long, e.g. more than 200 tracks, or have long captions, e.g. more than 50 words. in practice, such outliers have the effect of increasing, respectively, the memory requirements and inference time of algorithms. for these reasons, we filter out playlists which number of tracks is above the 95 th percentile, or which caption length in words is above the 95 th percentile. we end up with 5467 deezer playlists and 1104 spotify playlists. we observe high data quality, so no further pre-processing is needed. we release both raw and pre-processed datasets with the code. we present dataset statistics in appendix a. we consider both title and description to be part of a caption. in particular, a caption is defined by the template: <@xmath0. an example is: <@xmath1. other captions are the ground truths in the captions we consider account for a variety of playlist themes, such as musical genres, moods, activity, events and artist, and were annotated by a number of professional editors coming from different cultural backgrounds. also, the dataset statistics we report in we partition the deezer playlists randomly in training, validation and test, accounting respectively for 60%, 20% and 20% of the total. validation and test splits allow internal evaluation, i.e. on in-distribution samples. we use the spotify playlists as an additional test set for an external evaluation, i.e. on out-of-distribution samples. playntell, the model we propose, is designed to bridge the semantic gap between a playlist and the relative caption by leveraging different sources of musical knowledge, among which tracks audio and tags. as for audio, we retrieve 30-seconds audio previews of playlist tracks. audio previews are convenient because they can be freely accessed from music streaming public api. as for tags, we resort to the crowdsourced database discogs, which offers tags at album level. we propagate album tags to every track in the album. the tags cover a range of aspects: genres, countries, years and moods (e.g. "pop", "italy", "2020" and "happy"). we release the tracks tags with the captioned playlists. tracks audio and tags differ in availability. while the existence of audio is implied by the existence of a track, the availability of tags depends on the crowdsourcing databases, such as discogs. thus, tags may not be available for a new release, just because no crowdsourcer have tagged it yet. the model we propose leverages tags if available, and, if not, can rely on audio only to generate sensible captions, as we show in section 5.3. ## method editorial data are available in small quantities, as their existence depends on the expensive work of professional editors. in our case, we can rely on a few thousand editorial playlists, as we detail in section 3. in this setting, it may be difficult to learn a model from scratch. we introduce playntell, a new data-efficient playlist captioning model that leverages linguistic and musical knowledge to generate editorial-like captions. playntell is composed of an encoder and decoder, shown in the left and right hand-sides of figure 1 . ## conclusion and future work with the contribution at hand, we add to the literature on playlist captioning, as we present a new dataset and a new model that sets a new state-ofthe-art. playntell leverages general linguistic knowledge from a pre-trained language model to generate coherent captions, and musical knowledge to make the captions consistent with the playlist content. planned future work include the extension to non-english captions, and to apply playn-tell in the field of music recommender systems, where captions could explain automatically generated playlists @xcite .