Leveraging User-Generated Metadata of Online Videos for Cover Song Identification (2024)

Abstract: YouTube is a rich source of cover songs. Since the platform itself is organized in terms of videos rather than songs, the retrieval of covers is not trivial. The field of cover song identification addresses this problem and provides approaches that usually rely on audio content. However, including the user-generated video metadata available on YouTube promises improved identification results. In this paper, we propose a multi-modal approach for cover song identification on online video platforms. We combine the entity resolution models with audio-based approaches using a ranking model. Our findings implicate that leveraging user-generated metadata can stabilize cover song identification performance on YouTube.

## introduction music is a popular content category on youtube @xcite . uploaders share music in a variety of contexts, ranging from amateur covers to mashups @xcite . since youtube is not organized in terms of songs but rather in terms of online videos, foot_0 finding cover versions of songs is a non-trivial retrieval task. driven by applications such as copyright infringement detection, cover song identification (csi) deals with the retrieval of covers. the key challenge of csi is to compare songs based on properties which can indicate their association (e.g., melody, lyrics) while discarding irrelevant information (e.g., timbre). consequently, current research efforts are mainly audio-based @xcite @xcite with limited consideration of non-audio features such as lyrics @xcite . however, the utilization of user-generated metadata of online videos like in similar tasks @xcite , has not yet been considered in csi. we propose to model csi on online video platforms (ocsi) as a multi-modal problem, based on the hypothesis that uploaders tend to describe their videos using attributes of songs (e.g., song title, performer name) to make them easily findable. in this work, we propose multi-modal ensembles combining entity resolution models with audio-based csi models in a late-fusion fashion using the ranking model lambdamart @xcite . we compare the performance of the proposed ensembles with the performance of the csi models. further, we study the robustness of er approaches in difficult cases such as song title variations or hard negatives on youtube and provide our code and results. foot_1 ## multi-modal online cover song identification the items in our task are online videos. the goal of ocsi is the retrieval of items associated with the same musical work as a query item. in traditional csi, only representations concerning musical content (e.g., audio and lyrics) are considered. we additionally leverage user-generated metadata. in figure 1 we show an example of a query-candidate pair. each item is represented by attributes derived from its audio data and attributes from its usergenerated metadata (video title, channel name, video description, and a set of keywords). for simplicity, we assume that each item contains only one song. a song has the attributes song title, performer name, and a work identifier. songs which are associated with the same musical work are considered relevant in the retrieval scenario when one song is used as a query to retrieve the other song. we model the task of ocsi as a multi-modal problem involving metadata and audio representations. our overall system will consist of modules to compute the pairwise similarities for each of both modalities. then, a ranking model combines both outputs to compute an overall rank. in the following, we explain our entity-resolution (er) methods used to model similarities in the metadata domain. ## experimental setup our experiments aim to evaluate a) whether ensembles of er and csi models outperform csi models and b) whether er models are robust against hard negatives and song title variations (e.g., translations of song titles or parodies). 3 we report two evaluation metrics suggested by mirex: 4 map and mean rank of the first relevant item (mr1). we use subsets of two popular csi datasets: shs100k @xcite and datacos (yesiler 3 an example for a parody title is "bye, bye johnny" by the rattles covering "johnny b. ## results in the highest results in map are achieved with s-bert and ditto with s-bert as blocker and roberta as underlying lm. considering the computational overhead for using ditto, even with blocking and @xmath0, makes its use for the task questionable. however, ## limitations and conclusion in this paper, we implemented er approaches as means to support the task of cover song identification in online videos on the example of the online video platform youtube. we showed that simple fuzzy matching can partly help to increase model performances. better results were achieved by using s-bert. additionally, using ditto appears to lastly, we outline limitations of this study. our selected text input structure for s-bert and fuzzy matching can only detect the song title in the video titles. while this might be sufficient for our utilized datasets, other videos which contain the title only in the description or keywords are not uncovered. secondly, song titles that are completely different than the reference (e.g., parodies or medleys) cannot be detected by er models. hence, we see these models as supporting tool in ocsi rather than independent approaches. in future research, we aim to leverage more recent large lms which are starting to get used for er @xcite .