Syllable-level lyrics generation from melody exploiting character-level language model (2024)

Abstract: The generation of lyrics tightly connected to accompanying melodies involves establishing a mapping between musical notes and syllables of lyrics. This process requires a deep understanding of music constraints and semantic patterns at syllable-level, word-level, and sentence-level semantic meanings. However, pre-trained language models specifically designed at the syllable level are publicly unavailable. To solve these challenging issues, we propose to exploit fine-tuning character-level language models for syllable-level lyrics generation from symbolic melody. In particular, our method aims to fine-tune a character-level pre-trained language model, allowing to incorporation of linguistic knowledge of the language model into the beam search process of a syllable-level Transformer generator network. Besides, by exploring ChatGPT-based evaluation of generated lyrics in addition to human subjective evaluation, we prove that our approach improves the coherence and correctness of generated lyrics, without the need to train expensive new language models.

## introduction generating lyrics from a given melody is a subjective and creativity-driven process that does not have a definitive correct answer. recognizing the importance of subjective and creativity-driven generation processes is essential for advancing the development of ai. by embracing and enabling such processes, we can pave the way for more nuanced and expressive ai-generated lyrics. accordingly, evaluating the quality of subjectively and creativity-driven generated lyrics has become a fascinating topic. our system focuses on generating lyrics from symbolic melodies and could serve as a valuable creative aid, collaborating with artists throughout the entire songwriting process. the use of symbolic melodies allows for effortless and frequent modifications, facilitating iterative creative exploration. in this work, we explore the generation of lyrics from simplified symbolic melodies consisting of 20 notes. our aim is to maintain the alignment between the syllables of the lyrics and the corresponding melody notes during the inference stage. to achieve this, we propose a melody-encodersyllable-decoder transformer architecture, which generates syllables sequentially in accordance with the melody. however, due to the scarcity of paired lyrics-melody data available for training, this approach could lead to producing lyrics that are not coherent and grammatically not correct, such as "you gotta o in what the you used to life". the dataset we are using is described in @xcite , and it only contains approximately 10,000 paired lyrics-melody sequences. each lyrics sequence in the dataset contains 20 syllables in length, and there may be samples where syllables are occasionally missing due to misalignment, or lack of corresponding notes. these problems significantly hinder the training of a model to comprehend and generate coherent language. on the other hand, due to the constraint of syllable-level generation, it is difficult to directly apply pre-trained language models that already have an understanding of linguistic knowledge, due to the scarcity of syllable-level language models. the utilization of the widely popular word-piece encoding is not feasible in our task because one word consists of different numbers of syllables. this would potentially affect the probabilities of generating multi-syllable words. a possible alternative approach to train a custom language model at the syllable level is using a large, clean text corpus that has been segmented into syllable-level texts, which can then be fine-tuned specifically for the task of generating lyrics, but it is also difficult to construct such kind of dataset. another solution is to fine-tune a character-level language model, refining it to generate syllable sequences. in this work, we focus on the latter approach, which aims to fine-tune a character-level language model for re-ranking the candidates generated by a melody-encoder-syllabledecoder transformer @xcite . we take inspiration from the usage of language models in re-ranking speech recognition token candidates @xcite . considering the sentence "last x was windy", and the speech recognition system candidates knight and night. due to the pronunciation similarities, the word knight could be given a higher probability when recognizing speech. however, a language model would easily fix the mistake, assigning a higher probability to the word night instead. another inspiring work by @xcite focused on video comment generation tasks, in this work, the probability of previous text token, the probability of future text token, and the mutual dependency between comment texts and video are modeled by three separately trained neural networks. the probabilities from all three models are then combined and the best candidate from the main comment generation transformer model is selected, improving coherence and relation between comments and video. in our study, using a real example from our models, given the sentence "you gotta", the lyrics generation model could predict possible next tokens as o rather than treat because of the limited training data it learned from, but it is neither grammatically correct nor semantically meaningful. in this case, a powerful language model would know that the latter is more likely to form a coherent sentence. using a fine-tuned language model to refine the semantic meanings within generated syllable-level lyrics, we are able to improve the generated sequence from "you gotta o in what the you used to life" to "you gotta treat me to maybe understand you". as one phrase of lyrics, the revised sequence is much more coherent and interesting than the original version. the main contributions of this work can be summarized as follows: ## proposed methods by exploiting fine-tuning a pre-trained language model, we have successfully designed syllablelevel lyrics generation architecture from symbolic melody exploiting character-level language model depicted in figure 1 . in this section, we will introduce the details of the proposed methods. ## conclusion in this work, we proposed a method to enhance the predictions of a syllable-level melody-conditioned lyrics generation transformer, which utilizes pretrained character-level language models fine-tuned on lyrics data. we propose a method for creating a dataset tailored to fine-tune the character-level language model for refining syllable-level semantic meanings. moreover, we present an algorithm for re-ranking candidate tokens during the beam search procedure. we prove that our syllable-level refinement leads to improved naturality, correctness, and coherence of lyrics, while maintaining them tightly related to the conditioning melodies via the use of the encoder-decoder architecture. in future work, we plan to work on pre-training a syllable-level language model on a large data corpus, and then fine-tuning it, as well as exploring fine-tuning character-level language models for the task of lyrics-conditioned melody generation.