Modeling the Music Genre Perception across Language-Bound Cultures (2020)

Abstract: The music genre perception expressed through human annotations of artists or albums varies significantly across language-bound cultures. These variations cannot be modeled as mere translations since we also need to account for cultural differences in the music genre perception. In this work, we study the feasibility of obtaining relevant cross-lingual, culture-specific music genre annotations based only on language-specific semantic representations, namely distributed concept embeddings and ontologies. Our study, focused on six languages, shows that unsupervised cross-lingual music genre annotation is feasible with high accuracy, especially when combining both types of representations. This approach of studying music genres is the most extensive to date and has many implications in musicology and music information retrieval. Besides, we introduce a new, domain-dependent cross-lingual corpus to benchmark state of the art multilingual pre-trained embedding models.

## introduction a prevalent approach to culturally study music genres starts with a common set of music items, e.g. artists, albums, tracks, and assumes that the same music genres would be associated with the items in all cultures @xcite . however, music genres are subjective. cultures themselves and individual musicological backgrounds influence the music genre perception, which can differ among individuals @xcite . for instance, a westerner may relate funk to soul and jazz, while a brazilian to baile funk that is a type of rap @xcite . thus, accounting for cultural differences in music genres' perception could give a more grounded basis for such cultural studies. however, ensuring both a common set of music items and culture-sensitive annotations with broad coverage of music genres is strenuous @xcite . to address this challenge, we study the feasibility of cross-culturally annotating music items with music genres, without relying on a parallel corpus. in this work, culture is related to a community speaking the same language @xcite . the specific research question we build upon is: assuming consistent patterns of music genres association with music items within cultures, can a mapping between these patterns be learned by relying on language-specific semantic representations? it is worth noting that, since music genres fall within the class of culture-specific items @xcite , cross-lingual annotation, in this case, cannot be framed as standard translation, as one also needs to model the dissimilar perception of music genres across cultures. our work focuses on four language families, germanic (english-en and dutch-nl), romance (spanish-es and french-fr), japonic (japanese-ja), slavic (czech-cs), and on two types of languagespecific semantic representations, ontologies and multi-word expression embeddings. first, ontologies are often used to represent music genres, showing how they relate conceptually @xcite . we identify wikipedia foot_0 , the online multilingual encyclopedia, to be particularly relevant to our study. it extensively documents worldwide music genres relating them through a coherent set of relation types across languages (e.g. derivative genre, sub-genre). though the relations types are the same per language, the actual music genres and the way they are related can differ. indeed, pfeil et al. @xcite have shown that wikipedia contributions expose cultural differences aligned with the ones in the physical world. second, music genres can be represented from a distributional semantics perspective. word vector spaces are generated from large corpora following the distributional hypothesis, i.e. words with similar contexts have akin meanings. as languages are passed on culturally, we assume that the languagespecific corpora used to create these spaces are sufficient to convey concepts' cultural specificity into their vector representations. in our study, we focus on multiple recent multilingual pre-trained models to generate word or sentence foot_1 embeddings @xcite @xcite @xcite , to account for variances in the used corpora or model designs. lastly, we combine the semantic representations by retrofitting distributed music genre embeddings to music genre ontologies. retrofitting @xcite modifies each concept embedding such that the representation is still close to the distributed one, but also encodes ontology information. initially, we retrofit music genres per language, using monolingual ontologies. then, by partially aligning these ontologies, we apply retrofitting to learn multilingual embeddings from scratch. the results show that we can model the crosslingual music genre annotation with high accuracy by combining both types of language-specific semantic representations. when comparing the representations derived from multilingual pre-trained models, the smooth inverse frequency averaging @xcite of aligned word embeddings outperforms the state of the art approaches. to our knowledge, this simple method has been rarely used to embed multilingual sentences @xcite , and we hypothesize its potential as a strong baseline on other cross-lingual datasets and tasks too. finally, embedding learning based on retrofitting leads to better multilingual music genre representations than when inferred with pretrained embedding models. this opens the possibility to learn embeddings for rare music genres or languages when aligned music genre ontologies are available. summing up, our contributions are: 1) a study on how effective language-specific semantic representations of music genres are for modeling crosslingual annotation, without relying on a parallel music item corpus; 2) an extensive evaluation of multilingual pre-trained embedding models to derive representations for multi-word concepts in the music domain. our study can enable complete musicological research, but also localized music information retrieval. this latter application is crucial for online music streaming platforms that leverage music genre annotations to provide worldwide, user-personalized music recommendations. our domain-specific study complements other works benchmarking general-language sentence representations @xcite . finally, we provide an in-depth formal analysis of the retrofitting part of our method. we prove the strict convexity of retrofitting and show that the ontology concepts' final embeddings converge to the same values despite the order in which concepts are iteratively updated, on condition that we know a single initial node embedding in each connected component of the ontology. ## related work music genres are conceptual representations encompassing a set of conventions between the music industry, artists, and listeners about individual music styles @xcite . from a cultural perspective, it has been shown that there are differences in how people listen to music genres. @xcite . average listening habits in some countries span across many music genres and are less diverse in other countries @xcite . also, cultural dimensions proved strong predictors for the popularity of specific music genres @xcite . despite the apparent agreement on the music style for which the music genres stand, conveyed in the earlier definition and implied in the related works too, music genres are subjective concepts @xcite . to address this subjectivity, @xcite proposed a dataset of music items annotated with english music genres by different sources. in this line of work, we address the divergent perception of music genres. still, we focus on multilingual, unsupervised music genre annotation without relying on content features, i.e. audio or lyrics. we also complement similar studies in other domains (art: @xcite with another research method. then, in the literature, there are other works that benchmark pre-trained word and sentence embedding models. @xcite compares multilingual contextual language models for named-entity recognition and part-of-speech tagging. @xcite use multiple static and contextual word embeddings to represent multi-word expressions and assess their capacity to capture meaning shift and implicit meaning in compositionality. @xcite formulate a new task to evaluate cross-lingual sentence representation centered on natural language inference. compared to these works, our benchmark is aimed at cross-lingual annotation; we target a specific domain, music, for which we try concept embedding adaptation with retrofitting; and we also test a multilingual sentence representation obtained with smooth inverse frequency averaging of multilingual word embeddings. as discussed in a recent survey on cross-lingual word embedding models @xcite , there is a need to unlock domain-specific data to assess if general-language sentence representations are also accurate across domains. our work builds towards this goal. ## cross-lingual music genre annotation further, we formalize the cross-lingual annotation task and the strategy to evaluate it in section 3.1. we describe the test corpus used in this work, together with its collection procedure in section 3.2. ## language-specific semantic representations for music genres this work aims to assess the possibility of obtaining relevant cross-lingual music genre annotations, able to capture cultural differences too, by relying on language-specific semantic representations. two types of semantic representations are investigated given their popularity: ontologies to represent music genre relations (presented in section 4.1) and distributed embeddings to represent multiword expressions in general (presented in section 4.2). in contrast to this unsupervised approach, mapping patterns of associating music genres with music items across cultures could have also been enabled with a parallel corpus. however, gathering a corpus that includes all music genres for each pair of languages is challenging. ## experiments cross-lingual music genre annotation, as formalized in section 3, is a typical multi-label prediction task. for evaluation, we use the area under the receiver operating characteristic curve (auc, @xcite , macro-averaged. we report the mean and standard deviations of the macro auc scores using 3-fold cross-validation. for each language, we apply an iterative split @xcite of the test corpus that balances the number of samples and the tag distributions across the folds. we pre-process the music genres by either replacing special characters with space ( -/,) or removing them (()':.!$ ). for japanese, we introduce spaces between tokens obtained with mecab foot_4 . embeddings are then computed from pre-processed tags. we test two translation baselines, one based on google translate foot_5 (gtrans) and one on the dbpedia sameas relation (dbpsameas). in this case, a source music genre is mapped on a single or no target music genre, its embedding being in the form {0,1} |t | . for xlm ctxt , we compute the sentence embedding by averaging the token embeddings obtained with mean pooling across all layers. for mbert ctxt , we apply the same strategy, but by max pooling the token embeddings instead. we chose these representations as they showed the best performance experimentally compared to the other strategies described in section 4.2. when retrofitting language-specific music genre embeddings, we use the corresponding monolingual ontology (rfit uω ). when we learn multilingual embeddings from scratch with retrofitting, by knowing only music genre embeddings in one language (la), we use the partially aligned dbpedia ontologies which contain the sameas relations (rfit la aω ). for this case, we also propose a baseline representing a source concept embedding as a vector of geodesic distances in the partially aligned ontologies to each target concept (dbp aω nndist). results. the last column of discussion. the results show that using translation to produce cross-lingual annotations is limited as it does not consider the culturally divergent perception of music genres. instead, monolingual semantic representations can model this phenomenon rather well. for instance, from milton cardona's music genres in es, salsa and jazz, it correctly predicts the japanese equivalent of fusion (フュー ジョン) in ja. yet, while a thorough qualitative analysis requires more work, preliminary exploration suggests that larger gaps in perception might still be inadequately modeled. for instance, for santana's album welcome tagged with jazz in es, it does not predict pop in fr. when comparing the distributed embeddings, a simple method that relies on a weighted average of multilingual aligned word vectors significantly outperforms the others. although rarely used before, we question if we can notice such high performance with other multilingual data-sets. the cross-lingual annotations are further improved by retrofitting the distributed embeddings to monolingual ontologies. interestingly, the vector alignment does not appear degraded by retrofitting to disjoint graphs. or, the negative impact is limited and exceeded by introducing domain knowledge in representations. further, as shown in regarding the scores per language, we obtained the lowest ones for ja as the source. we could explain this by either a more challenging test corpus or still incompatible embeddings in ja, possibly because of the quality of the individual embedding models for this language and the completeness of the japanese music genre ontology. also, we did not notice any particular improvement for pairs of languages from the same language family, e.g. fr and es. however, we would need a sufficiently sizeable parallel corpus exhaustively annotated in all languages to reliably compare the performance for pairs of languages from the same language family or different ones. finally, by closely analysing the results in one possible explanation is that the prediction from languages with fewer music genre tags such as l 2 towards languages with more music genre tags such as l 1 is more challenging because the target language contains more specific or rare annotations. for instance, when checking the results per tag from cs to en we observed that among the tags with the lowest scores, we found moombahton, zeuhl, or candombe. however, other common music genres, such as latin music or hard rock, were also poorly predicted, showing that other causes exist too. is the unbalanced number of music genres used in annotations a cultural consequence? related work @xcite seems to support this hypothesis. then could we design a better mapping function that leverages the unbalanced numbers of music genres in cross-cultural annotations? we will dedicate a thorough investigation of these questions as future work. ## conclusion we have presented an extensive investigation on cross-lingual modeling of music genre annotation, focused on six languages, and two common approaches to semantically represent concepts: ontologies and distributed embeddings foot_6 . our work provides a methodological framework to study the annotation behavior across languagebound cultures in other domains too. hence, the effectiveness of language-specific concept representations to model the culturally diverse perception could be further probed. then, we combined the semantic representations only with retrofitting. however, inspired by paraphrastic sentence embedding learning, one can also consider the music genre relations as paraphrasing forms with different strengths @xcite . finally, the models to generate cross-lingual annotations should be thoroughly evaluated in downstream music retrieval and recommendation tasks.