M usi L ingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response (2024)

Abstract: Large Language Models (LLMs) have shown immense potential in multimodal applications, yet the convergence of textual and musical domains remains not well-explored. To address this gap, we present MusiLingo, a novel system for music caption generation and music-related query responses. MusiLingo employs a single projection layer to align music representations from the pre-trained frozen music audio model MERT (CITATION) with a frozen LLM, bridging the gap between music audio and textual contexts. We train it on an extensive music caption dataset and fine-tune it with instructional data. Due to the scarcity of high-quality music Q&A datasets, we created the MusicInstruct (MI) dataset from captions in the MusicCaps datasets, tailored for open-ended music inquiries. Empirical evaluations demonstrate its competitive performance in generating music captions and composing music-related Q&A pairs. Our introduced dataset enables notable advancements beyond previous ones.

## introduction in the realm of music information retrieval (mir), prevailing methodologies for contemporary musical descriptions typically lean on discriminative learning. an illustrative instance is music tagging @xcite @xcite , where descriptors encompassing genres, composers, instruments, emotions, and tempos are ascribed to each music clip. in this case, the model output is confined to a pre-determined set of categorical labels, thereby constraining its applicability in contexts like music exploration and recommendation, where the ability to handle and generate handle natural language descriptions instead of individual tags such as music captions or answers to music instructions would boast a diverse array of practical applications. these include generating textual descriptions for items found within extensive music catalogues, annotating copious user-generated content; automatically providing descriptions for evocative music featured in videos, catering to the needs of the hearing-impaired; and furnishing explanations for automated music recommendations. furthermore, this advancement facilitates enhanced search and discovery of musical material for composers, all through user-friendly queries, while also serving as an inspiration for text-based music generation algorithms. given the potential alignment between musical and textual representations, there is some research to bridge the gap between acoustic music and natural language modalities, though still in its relatively nascent stages. a prospective avenue for better performance, in light of the recent triumphs of large language models (llms), entails integrating the conversation and generalisation proficiencies offered by llms into musical tasks. considering these insights, we introduce a novel music language model designed for music captioning, question answering, and query responses. our approach involves a single projection layer configuration with temporal compression applied to music embeddings. in contrast to the multilayer perceptron (mlp) approach for the llamaadapter @xcite in a contemporary work mu-llama @xcite , which projects music embeddings to the upper layers of llama, our method employs a straightforward projection to convey the embeddings to the initial layer of llama. the llama adapter is designed for finetuning the vanilla language model llama into an instruction-following model and enhances its applicability to visual-language tasks but vicuna @xcite , the llm we use as a language decoder, is already capable of instruction-following. our simple approach offers the advantage of accommodating larger batch sizes which possess an nlp backbone capable of instruction-following tasks. this has demonstrated considerable efficacy in visual-language contexts, as evidenced by its successful implementation in models like llava @xcite and mini-gpt4 @xcite . we also incorporate a pre-training phase to align music information with textual representations, utilising a large amount of music captioning data, and fine-tuning the model using our developed musicinstruct dataset derived from @xcite . this equips our model with the capability to understand different aspects of musical compositions and enables it to provide accurate and natural responses to user queries. in summary, our work features the following core contributions: section 2 details our methodology for the mi dataset creation and the music question-answering tasks. section 3 outlines the musilingo model structure and training procedure. section 4 presents experiments and evaluations of our model and baselines. our code is available on github 1 . ## related work several prior studies have explored the alignment of acoustic audio modalities with nlp. @xcite employed a cnn encoder based on a spectrum alongside an rnn with an attention-based natural language decoder to predict tag lists. however, the evaluation metrics utilized were not comprehensive, and the resulting tag lists exhibited noise, diminishing the robustness of the study. doh et al. 1 github repository (2023b) utilized clove or bert along with transformer architectures for music encoding to facilitate text-to-music retrieval. muscall @xcite leverages contrastive learning for textto-audio and audio-to-text retrieval tasks. @xcite curated a large-scale audio caption dataset comprising 630k samples and trained models utilizing various pre-trained audio encoders and natural language decoders, also equipped with contrastive learning capabilities. their approach yielded competitive results in text-to-audio retrieval tasks. in addition to raw audio processing, @xcite utilized transformer and gru models to map song ids and metadata from each playlist to the corresponding playlist titles. @xcite incorporated artist ids as part of the input for their investigation. work has additionally been carried out on music and audio captioning. muscaps @xcite leverages convolutional networks for music understanding and recurrent neural networks for captioning. mulan @xcite uses contrastive learning to align the text embedding to audio representations for music tagging and retrieval of music with text query. but the work is not open-sourced. lp-musiccaps @xcite and audio captioning transformer (act) @xcite utilise a cross-modality transformer-based encoder-decoder architecture for music/audio captioning. @xcite utilized an rnn to map cnn embeddings of playlists to word2vec embeddings of playlist captions. however, the performance of models in this earlier study is far from perfect. additionally, playntell @xcite employs tags, artist distributions, and audio as input for playlist captioning, thereby achieving state-of-the-art performance. although these studies have shown notable advancements in tackling music captioning, they are not designed for music instruction-following and their effectiveness in functioning within a genuine conversational context for question-answering remains restricted or not evaluated. several works have applied llms to multimodal tasks. unival @xcite offers a versatile model for image, video, audio, and language modalities, while ltu @xcite excels in audio quizzing. however, none of these models are designed for instruction-following on general audio, making them unsuitable for musicrelated question-answering and dialogue especially polyphonic music-related topics such as key and chord. to enable the bridge of two modalities on limited resources, we are inspired by the success of vision-language pre-training. in visionlanguage pre-training, the prevailing approach is to follow a new paradigm, connecting pre-trained unimodal encoders with llms via a learnable interface. this approach keeps encoders and language models fixed, using query tokens or adapter layers @xcite to transfer information between modalities. the interface can be a set of query tokens that extract information from the modality, as blip-2 @xcite and flamingo @xcite , or an adapter layer that projects embeddings from one modality to another. mini-gpt4 @xcite and video-chatgpt @xcite use simple linear adapters to project the visual embeddings onto text embedding space. video-llama @xcite adopts the q-former design from blip-2 for the adapter and incorporates 2 projections from the image and audio data in the video. llama-adapter @xcite ) employs a parameter-efficient approach with small adapter modules within transformer blocks. a contemporary work, mu-llama @xcite , extends the llama-adapter concept to music language tasks. these models, which utilise pre-trained frozen encoders and learnable interfaces, offer a promising approach to connecting any modality with language models, providing efficient training and maximal preservation of the model's original knowledge. ## experiment and results in this section, we introduce the experimental setup as well as present an evaluation of our model's performance on the question-answering of music on the musicqa and mi datasets. besides, we evaluate the performance of music captioning on the musiccaps dataset. we compare our results to state-of-the-art models and discuss the unique challenges posed by this dataset. last, we carry out an ablation study on training on different parts of the mi dataset. ## conclusion in summary, our submission introduces musilingo, a pioneering large language model that effectively bridges the gap between music and text domains. with the aid of a single projection layer, musilingo aligns music representations with textual contexts, delivering competitive performance in music captioning and question-answering tasks. the introduction of our innovative musicinstruct dataset further enhances its capabilities. we envision that our work lays the foundation for a new era of multimodal applications in the field of music, offering exciting possibilities for both music enthusiasts and researchers, promising to revolutionise the way we engage with and comprehend music.