ALCAP : Alignment-Augmented Music Captioner (2023)

Abstract: Music captioning has gained significant attention in the wake of the rising prominence of streaming media platforms. Traditional approaches often prioritize either the audio or lyrics aspect of the music, inadvertently ignoring the intricate interplay between the two. However, a comprehensive understanding of music necessitates the integration of both these elements. In this study, we delve into this overlooked realm by introducing a method to systematically learn multimodal alignment between audio and lyrics through contrastive learning. This not only recognizes and emphasizes the synergy between audio and lyrics but also paves the way for models to achieve deeper cross-modal coherence, thereby producing high-quality captions. We provide both theoretical and empirical results demonstrating the advantage of the proposed method, which achieves new state-of-the-art on two music captioning datasets.

## introduction learning to interpret music based on audio and lyrics has become an increasingly attractive research area for researchers in the filed music understanding and natural language processing @xcite . the insights gained from this research into multimodal representation learning enable a wide range of applications such as streaming media discovery @xcite and music recommendation with detailed and human-like descriptions @xcite , making the dynamics of search and recommendation engines more explainable. however, captioning music is a challenging task. the duality of music, with its often nebulous and repetitious lyrics intertwined with intricate audio compositions harboring multiple layers of information, * work done when zihao he interned at tiktok inc. previous works on music captioning have primarily focused on refining singular facets of the encoderdecoder paradigm, from the enhancement of the music encoder to the incorporation of sophisticated attention mechanisms and beam search strategies. however, little effort has been directed towards leveraging the correspondence between audio and lyrics, which could potentially provide useful information for generating high-quality captions. @xcite leverage the multimodal information from both lyrics and music through a crossmodal attention module, but the two modalities are not aligned before fusion. in reality, audio and lyrics are loosely aligned, as it is common for composers and lyricists to work separately in the music industry, resulting in different lyrics fitting the same melody. additionally, the same words with different song patterns and styles can express diametrically opposite emotions. therefore, the loose alignment between music and lyrics make them imperfect sources of data for existing multimodal learning methods that are not equipped with alignment mechanisms @xcite . in this regard, accurate and comprehensive music interpretation should leverage the subtle connections between music and lyrics. in addressing the complexities of music understanding, we propose to align audio and lyrics pairs with contrastive learning before modality fusion and caption generation. intuitively, paired audio and lyrics should be brought close together in the latent space, while non-paired ones should be pulled apart. by adding a contrastive loss, the multimodal input pairs are forced to be more aligned, which in turn guides the model to achieve stronger crossmodal consistency for a more meaningful fused latent space, thus generating better music captions. to this end, we propose alignment augmented music captioner (alcap), which is an extension of bart-fusion @xcite for music captioning with a contrastive learning based alignment augmentation module. we provide a theoretical explanation of why the proposed alignment module results in improved generalization from an information bottleneck perspective. extensive experiments on the song interpretation dataset @xcite and the netease cloud music review dataset demonstrate alcap's superiority with marked improvements in key metrics (rouge-l and meteor) over previous benchmarks. we also observe performance gain of al-cap in cross-modal text-music retrieval, which is a common application in industry, providing an indirect perspective to evaluate the caption quality. lastly, we explore the effect of contrastive loss weights on the model performance via grid search and conclude our ablation study by showing that our proposed multimodal alignment module leads to more concentrated attention on language tokens through visualization analysis. our contributions are summarized as follows: ## methodology in this section, we introduce the architecture of al-cap, which is based on bart-fusion @xcite . we first state the problem definition, then go through each module of the architecture. the overall framework of alcap is shown in figure 1 . ## an information theoretical perspective in this section, we explain the performance improvement of our alignment module based on contrastive learning from a mutual information perspective. given an input pair x i := {m i , t i }, information bottleneck (ib) @xcite encourages the model to find minimal but sufficient information about the input x i with respect to the target caption words y i . in other words, the objective of the training process in ib can be formulated as proposition 4.1. the mutual information of (x, z) in our setting is upper bounded by i(x; z) ≤ r(z) -i(m; t), where r(z) ≜ e p((m,t)|z) log e p(z) [p((m,t)|z)] p(m)p(t) in light of the fact that contrastive learning tends to maximize mutual information between (m, t) pairs, the above lower bound suggests that it can be considered as an approximate implementation of information bottleneck. furthermore, if the musiclyrics pairs used in contrastive learning are not well aligned, one can actually prove that the learning will fail. proposition 4.2. if the music-lyrics pairing in the learning process is random such that the music and lyrics are sampled independently, then the mutual information between the input x and the representation z will be zero, and thus the encoder cannot learn anything useful. the proof is provided in appendix a. to sum up, based on the infonce loss @xcite , the proposed alignment module can be interpreted as maximizing the mutual information lower bound between the music m and corresponding text t, which translates to minimizing the mutual information between the input x and the latent code z, and consequently improving the generalization ability of the model. ## experiments 6.1 experimental setup we resample each song at 16khz and take a 15s excerpt. the maximum caption length is 512. the model is implemented in pytorch @xcite . we use the bart implementation facebook/bart-base from huggingface @xcite . we use a batch size of 26 and a learning rate of 5e -5. the weight of contrastive learning α loss is set to 0.02. for better computation efficiency we freeze the parameters in the music encoder and precompute the music representations. we train the model for 20 epochs and report the results on the test split using the checkpoint with the best evaluation performance. all hyperparameter tuning is based on grid search. all models are trained on a tesla a100 gpu with 40gb memory. the training time for si-full, si w/@xmath0, si w/@xmath1, and ncm review are 28h, 28h, 5h, and 3h respectively. we use rouge-1,2,l (rouge, 2004) and me-teor @xcite as evaluation metrics. rouge measures the overlap of n-grams between the referenced text and the generated text. on top of rouge, meteor complementarily measures the semantic similarity between the two pieces of text by taking into account synonyms through wordnet. for both metrics, we use the implementation with default parameters from huggingface datasets library. we use three random seeds and report the average performance on the test set. ## conclusions in this paper, we introduce the alignment augmented music captioner (alcap), a pioneering model designed to enhance music captioning by incorporating an alignment augmentation module using cross-modal contrastive learning. our model's distinctiveness, particularly in the under-researched music domain, stems from its ability to successfully bridge the gap between music and its linguistic interpretation. we provide a theoretical analysis of the improved generalization of our model from an information bottleneck perspective. experiments on two music captioning datasets demonstrate the effectiveness of alcap, and we achieve the new state-of-the-art on both of them. our next steps will focus on collecting extensive music song data from the web and pretrain the music-lyrics alignment module, after which the model is further finetuned on small-scale music captioning. this model would be tailored to cater to large-scale song interpretation generation on various genres and styles of music. in addition, groundtruth interpretations, as drawn from user-generated content, inherently bring the risk of biases. we will be committed to developing strategies to effectively mitigate biases from these interpretations, ensuring that our model's outputs are as neutral and accurate as possible.