A framework for evaluating multimodal music mood classification

This research proposes a framework of music mood classification that utilizes multiple and complementary information sources, namely, music audio, lyric text and social tags associated to music pieces. This article presents the framework and a thorough evaluation on each of its components. Experiment results on a large dataset of 18 mood categories show that combining lyrics and audio significantly outperformed systems using audio-only features. Automatic feature selection techniques were further proved to have reduced feature space. In addition, the examination of learning curves shows that the hybrid systems using lyrics and audio needed fewer training samples and shorter audio clips to achieve the same or better classification accuracies than systems using lyrics or audio singularly. Last but not least, performance comparisons reveal relative importance of audio and lyric features across mood categories. Introduction Music is an essential information type in people’s everyday life. Nowadays, there have been a large number of music collections, repositories, and websites that strive to provide convenient access to music for various users, from musicians to the general public. These repositories and users often use different types of metadata to describe music such as genre, artist, country of source and music mood 1 . (Vignoli, 2004; Hu & Downie, 2007). Many of these music repositories have been relying on manual supply of music metadata, but the increasing amount of music data calls for tools that can automatically classify music pieces. Music mood classification has thus been attracting researchers’ attention in the last decade, but many existing classification systems are solely based on information extracted from the audio recordings of music and have achieved suboptimal performances or reached a “glass ceiling” of performance (Lu, Liu, & Zhang, 2006; Trohidis, Tsoumakas, Kalliris, & Vlahavas, 2008; Hu, Downie, Laurier, Bay, & Ehmann, 2008, Yang & Chen, 2012, Barthet, Fazekas, & Sandler, 2013). At roughly the same time, studies have reported that lyrics and social tags associated with music have important values in Music Information Retrieval (MIR) research. For example, Cunningham, Downie, and Bainbridge (2005) reported lyrics as the most mentioned feature by respondents in answering why they hated a song. Geleijnse, Schedl, and Knees (2007) proposed an effective method of measuring artists similarity using social tags associated with the artists. As lyrics often bear with semantics of human language, they have been exploited in music classification as well (e.g., He et al, 2008; Hu, Chen, & Yang, 2009; Van Zaanen & Kanters, 2010; Dakshina & Sridhar, 2014). Furthermore, based on the hypothesis that lyrics and music audio 2 are different enough and thus may complement each other, researchers have started to combine lyrics and audio for improved classification performances (Laurier, Grivolla, & Herrera, 2008; Yang et al., 2008; Björn, Johannes, & Gerhard, 2010; Brilis et al., 2012). Such approach of combining multiple information sources in solving classification problems is commonly called multimodal classification (Kim, et al., 2010; Yang & Chen, 2012; Barthet et al., 2013). Multimodal classification approaches in general are reported to have improved classification performances over those based on a single source. However, there are many options and decisions involved in a multimodal classification approach, and to date, there has not been any general guidance on how to make these decisions in order to achieve more effective classifications. This study proposes a framework of multimodal music mood classification where research questions on each specific stage or component of the classification process could be answered. This is one of the first studies presenting a comprehensive experiment on a multimodal dataset of 5,296 unique songs, which exemplifies every stage of the framework and evidences how the performances of music mood classification can be improved by a multimodal approach. Specifically, novelty and contributions of this study can be summarized as follows: 1. Conceptualize a framework for the entire process of automatic music mood classification using multiple information sources. The framework is flexible in that each component can be easily extended by adding new methods, algorithms and tools. Under the framework, this study systematically answers questions often involved in multimodal classification: feature extraction, feature selection, ensemble methods, etc. 2. Following a previous study evaluating a wide range of lyric features and their combinations (Hu & Downie, 2010), this study further explores feature selection and effect of dimension reduction of feature spaces. Thus, it pushes forward the state-of-the-art on sentiment analysis for music lyrics; 3. Examine the reduction of training data brought by the multimodal approach. This aspect of improvement has rarely been addressed by previous studies on multimodal music classification. Both the effect on the number of training examples and that on the length of audio clips are evaluated in this study. 4. Compare relative advantages of lyrics and audio across different mood categories. To date, there is little evidence on which information source works better for which mood category(ies). Gaining insight on this question can contribute to deeper understanding of sources and components of music mood. 5. Build a large ground truth dataset for the task of multimodal music mood classification. The dataset contains 5,296 unique songs in 18 mood categories. This is one of the largest experimental datasets in music mood classification with both audio and lyrics available (Kim, et al., 2010; Yang & Chen, 2012; Barthet et al., 2013). Results from a large dataset with realistic and representative mood categories are more generalizable and of higher practical values. The rest of the paper is organized as follows. Related work is reviewed and research questions are stated. After that, a framework for multimodal music mood classification is proposed. We then report an experiment with ternary information and conclude by discussing the findings and pointing out future work on enriching the proposed framework.