C hat M usician: Understanding and Generating Music Intrinsically with LLM (2024)

Abstract: While LLMs demonstrate impressive capabilities in musical knowledge, we find that music reasoning is still an unsolved task.We introduce ChatMusician, an open-source large language model (LLM) that integrates intrinsic musical abilities. It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.ChatMusician can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. Interestingly, endowing musical abilities does not harm language abilities, even achieving a slightly higher MMLU score.ChatMusician is capable of composing well-structured, full-length music, condition on texts, chords, melodies, motifs, musical forms, etc.On our meticulously curated college-level music understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and GPT-3.5 by a noticeable margin. We show that ChatMusician preserves or even surpasses the original LLaMA2 7B’s language abilities by evaluating on MMLU benchmark.Our work reveals that LLMs can be an excellent compressor for music, which can be seen as humanity’s creative language, but there remains significant territory to be conquered.We release our 5B token music-language corpora MusicPiles, the collected MusicTheoryBench, code, model and demo.

## introduction the fusion of artificial intelligence and the arts, particularly music, has emerged as a pivotal area of research, for its profound implications on the essence of human creativity @xcite . music holds a unique position due to its inherent structure and complexity, and @xcite ; @xcite suggest that language and music may have evolved from the same source. large language models (llms) have recently revolutionized various domains with their remark-able capacity for generating long sequences. researchers have been exploring language modeling techniques for music generation @xcite @xcite @xcite @xcite @xcite . although it seems that symbolic music can be treated in a similar way to natural language, research has shown that many distinct challenges are encountered in the realm of music. for example, even state-of-the-art models such as gpt-4 perform marginally better than random in music reasoning foot_1 . we argue that the main reason is that the intricacies of musical composition remain inadequately represented in current llms, including the long-term, contrapuntal context dependency and the complex connections between music notes and text descriptions. attempting to find solutions to these challenges, we propose chatmusician, an open-source llm that integrates intrinsic musical abilities, with a pipeline as shown in figure 1 . our endeavors have focused on leveraging llms for symbolic music generation and understanding. our contributions: a) we introduce chatmusician, a text-based llm that unifies multiple symbolic music understanding and generation tasks, enriching their repertoire while maintaining or potentially enhancing their foundational general abilities. b) empirical evaluations demonstrate our model's superior musical composition capabilities, surpassing gpt-4 and established baselines in various music generation tasks, showcasing its prowess in generating coherent and structured musical pieces across diverse styles. c) we introduce the inaugural college-level symbolic music understanding a chord, in music, is any harmonic set of pitches consisting of multiple notes (also called "tones") that are sounded simultaneously ... which of the following chord progressions best describes the above example? ## conclusions in conclusion, our study introduces chatmusician, an innovative open-source llm capable of advanced music reasoning and composition. by leveraging a text-compatible music representation and achieving notable performance on both music and language benchmarks, chatmusician represents a significant step forward in integrating musical creativity within language models. our findings underscore the potential of llms as powerful tools for music understanding and creativity, highlighting the untapped possibilities in the fusion of music and artificial intelligence. the release of musicpiles, musictheorybench, and chatmusician provides a valuable resource for further research in this exciting domain.