Multitask Learning based Deep Learning Model for Music Artist and Language Recognition (2021)

Abstract: Artist and music language recognitions of music recordings are crucial tasks in the music information retrieval domain. These tasks have many industrial applications and become much important with the advent of music streaming platforms. This work proposed a multitask learning-based deep learning model that leverages the shared latent representation between these two related tasks. Experimentally, we observe that applying multitask learning over a simple few blocks of a convolutional neural network-based model pays off with improvement in the performance. We conduct experiments on a regional music dataset curated for this task and released for others. Results show improvement up to 8.7 percent in AUC-PR, similar improvements observed in AUC-ROC.

## introduction music is a universal language that we innately understand. it can influence or induce new emotions in the listeners. artists project their emotions and feeling onto their music that is felt and observed in the music. artist recognition of a music recording is an active area of research in music information retrieval (mir) @xcite @xcite and has various applications. artist recognition is crucial in the areas of music index, retrieval, and recommendation. the digitization of the music industry and music streaming platforms have created large volumes of digital music that need to be processed and stored on a large scale. this has reignited the research in the music domain. recognition of the artist of a song is crucial for these music streaming platforms. we also have our favorite artists, whom we search for on these streaming platforms as music listeners, and it shows how vital artist recognition is. machine learning-based approaches treat this problem as a multi-label classification problem. there have been many recent deep learning-based techniques that perform very well for this task. the approach in these techniques is to use variants of spectrogram and train the deep neural network model on that visual representation @xcite @xcite . some techniques have used raw waveforms to train sequence-based models. these techniques have revealed that related tasks specific noise filtering can boost the overall generalization of deep learning-based models for music-related tasks. surprisingly, given techniques by researchers do not leverage the shared representation learned by multitask learning of related tasks. in this paper, we propose a multitask learning-based model for artist recognition that leverages the shared representation learned from the related task of music language recognition. the results show improvement over single-task learning. we have used multitask learning with convolutional neural networks (cnn) for artist recognition, and we observed improved performance. multitask learning is a machine learning paradigm in which related tasks are trained together using the same model which shares bottom layers (in neural networks) among the related tasks. the training signals (gradients) from different related tasks force the model to learn more generalized data representation by filtering out noise for each related task @xcite . alternatively, we can say the knowledge learned for a task helps in the performance over another related task. in our case, we use two related tasks of artist recognition and music language recognition, where artist recognition is the primary task for leveraging shared representation from multitask learning. music recordings spectrograms are used as an input for the model, and corresponding artists and language are predicted as an output by the model. this paper is organized as follows: section 2 presents the proposed approach. the dataset, experimental setup is summarized in section 3. in section 4, we present the results and discussions of our findings. finally, we conclude in section 5. ## proposed method our proposed approach is a model based on convolutional neural networks (cnn) similar to vgg architecture. cnns are very popular in the computer vision domain. here we have chosen cnn because the spectrogram representation of music samples can be treated as an image, and cnns can extract relevant knowledge from them. this spectrogram approach is not new and has been used in the past with traditional machine learning. however, advances in computer vision have given new avenues in the spectrogram representation for music tasks. the architecture of our model is shown in figure 1 . a mel-spectrogram taken as a 2d-tensor with (187x96) is taken as an input to the network. the mel-spectrogram consists of 187 frames which correspond to 3 seconds of music data and 96 melbands. the extraction of the spectrogram from the music recordings is discussed in section 3.2.1. a batch of spectrograms with size 64 is passed to five blocks of cnn layers. each block consists of a cnn layer along with batch normalization, max-pooling, and dropout. the kernel size of cnn layers is fixed to (3x3), and the numbers of channels are 32, 64, 128, 64, and 32, respectively, with stride 1 of cnn layers in five blocks. the nonlinearity in the cnn is set relu. after each cnn layer, batch normalization (bn) is applied to normalize the weights during training. following bn, max-pooling is applied with a pool size of (2x) and stride of (2x2). lastly, a dropout layer with a 0.25 drop rate is applied. after these cnn layer blocks, a flatten layer is used to compress the output shape from cnn blocks to the 1d tensor. dense layers then use this tensor corresponding to each related task (two in our case, namely artist recognition, and music language recognition). each dense layer has the number of units equal to the number of labels in the corresponding task (64, 17 in our tasks). the softmax activation function is used in these dense layers' outputs to get the probability distribution of labels relating to the tasks. these are 193k trainable parameters in the top five blocks of our proposed model. the parameters in the output layers are dependent on the number of parameters in the previous layer and the number of labels in the corresponding task. the model shares the five blocks of the model between two related tasks. the representation learning during the training of these two tasks forces the model to generalize better than it would have been training on a single task. the loss used for training the model is a weighted sum of losses from individual tasks. categorical cross-entropy is used as a loss for both tasks. the weighted sum for overall loss function: here, l artist and l lang denote the corresponding loss for artist and music language recognition tasks, respectively. α represents the hyper-parameter for the weightage given to the loss of music language recognition task in the overall loss. setting it to zero disables the multitask learning and is done when we are done with training the network. the choice of α is discussed in section 3.2.4. ## results we performed multiple experiments over the described dataset in section 3.1. baseline and multitask are two models selected and trained for two tasks. both models are the same architecture except for the final dense layer. the baseline model has a single dense layer having units equal to the labels in the given task. while in the multitask model, there are two dense layers side by side for each task, having units equal to two task labels. comparing the result of baseline and multitask models allow us to observe the impact of multitask learning on the given task with the help of additional related task. the comparative analysis is presented in we report that the multitask model shows improvement over the baseline model for both tasks and can be observed in both evaluation metrics. on the curated dataset, we observe an increase of 4.43 percent for artist recognition in the auc-roc metric, while the auc-pr metric recorded a 7.48 improvement. for the music language recognition task, improvements are even further. 5.55 improvement is observed in the auc-roc and 8.69 for the auc-pr metrics. we observe from these experiments that the performance can be improved by adding more related tasks and increasing the number of data samples. ## conclusions in this paper, we present the importance of artist recognition from the perspective of music streaming platforms for storage, indexing and music information retrieval tasks. it is crucial to building a more generalized system by these platforms. towards building a more generalized system, we observe that multitask learning can help achieve a more generalized system by leveraging the model's representation across different related tasks. we propose a multitask learning model for artist and music language recognition tasks. experiments depict that the multitask learning approach improves the performance of the single-task baseline model.