Unsupervised Melody-to-Lyrics Generation (2023)

Abstract: Automatic melody-to-lyric generation is a task in which song lyrics are generated to go with a given melody. It is of significant practical interest and more challenging than unconstrained lyric generation as the music imposes additional constraints onto the lyrics. The training data is limited as most songs are copyrighted, resulting in models that underfit the complicated cross-modal relationship between melody and lyrics. In this work, we propose a method for generating high-quality lyrics without training on any aligned melody-lyric data. Specifically, we design a hierarchical lyric generation framework that first generates a song outline and second the complete lyrics. The framework enables disentanglement of training (based purely on text) from inference (melody-guided text generation) to circumvent the shortage of parallel data. We leverage the segmentation and rhythm alignment between melody and lyrics to compile the given melody into decoding constraints as guidance during inference. The two-step hierarchical design also enables content control via the lyric outline, a much-desired feature for democratizing collaborative song creation. Experimental results show that our model can generate high-quality lyrics that are more on-topic, singable, intelligible, and coherent than strong baselines, for example SongMASS, a SOTA model trained on a parallel dataset, with a 24% relative overall quality improvement based on human ratings. Our code is available at https://github.com/amazon-science/unsupervised-melody-to-lyrics-generation .

## introduction music is ubiquitous and an indispensable part of humanity @xcite . self-serve songwriting has thus become an emerging task and has received interest by the ai community @xcite @xcite . however, the task of melody-to-lyric (m2l) generation, in which lyrics are generated based on a given melody, is underdeveloped due to two major challenges. first, there is a limited amount of melody-lyric aligned data. the process of collecting and annotating paired data is not only labor-intensive but also requires strong domain expertise and careful consideration of copyrighted source material. in previous work, either a small amount (usually a thousand) of melody-lyrics pairs is manually collected @xcite , or @xcite use the recently publicized data @xcite in which the lyrics are pre-tokenized at the syllable level leading to less sensical subwords in the outputs. another challenge lies in melody-to-lyric modeling. compared to unimodal sequence-to-sequence tasks such as machine translation, the latent correlation between lyrics and melody is difficult to learn. for example, @xcite ; @xcite ; @xcite ; @xcite apply rnns, lstms, seqgans, or transformers with melody embeddings and cross attention @xcite , hoping to capture the melody-lyrics mapping. however, as shown in figure 1 , these methods may generate less singable lyrics when they violate too often a superficial yet crucial alignment: one word in a lyric tends to match one music note in the melody @xcite . in addition, their outputs are not fluent enough because they are neural models trained from scratch without leveraging large pre-trained language models (ptlms). in this paper, we propose lyra, an unsupervised, hierarchical melody-conditioned lyrics generator that can generate high-quality lyrics with content control without training on melody-lyric data. to circumvent the shortage of aligned data, lyra leverages ptlms and disentangles training (pure text-based lyric generation) from inference (melody-guided lyric generation). this is motivated by the fact that plain text lyrics under open licenses are much more accessible @xcite @xcite , and prior music theories pointed out that the knowledge about music notes can be compiled into constraints to guide lyric generation. specifically, @xcite argue that it is the durations of music notes, not the pitch values, that plays a significant role in melody-lyric correlation. as shown in figure 1 , the segmentation of lyrics should match the segmentation of music phrases for breathability. @xcite ; @xcite also find that long (short) note durations tend to associate with (un)stressed syllables. however, existing lyric generators, even when equipped with state-of-the-art neural architectures and trained on melody-lyrics aligned data, still fail to capture these simple yet fundamental rules. in contrast, we show that through an inference-time decoding algorithm that considers two melody constraints (segment and rhythm) without training on melody-lyrics aligned data, lyra achieves better singability than the best data-driven baseline. without losing flexibility, we also introduce a factor to control the strength of the constraints. in addition, lyra adopts the hierarchical text generation framework (i.e., plan-and-write @xcite ) that both helps with the coherence of the generation and improves the controllability of the model to accommodate userspecified topics or keywords. during training, the input-to-plan model learns to generate a plan of lyrics based on the input title and salient words, then the plan-to-lyrics model generates the complete lyrics. to fit in the characteristics of lyrics and melody, we also equip the plan-to-lyrics model with the ability to generate sentences with a predefined count of syllables through multi-task learning. our contributions are summarized as follows: • we design lyra, the first melody-constrained neural lyrics generator without training on parallel data. specifically, we propose a novel hierarchical framework that disentangles training from inference-time decoding, which is supported by music theories. our method works with most ptlms, including those black-box large language models (llms) when finetuning is replaced by in-context learning. ## background and problem setup representation of melody melody is a succession of pitches in rhythm consisting of a sequence of music phrases, which can be further decomposed into timed music notes. each music note is defined by two independent pivots: pitch values and durations. pitch represents the highness/lowness of a musical tone; duration is the note's length of time. namely, melody m can be denoted by @xmath0, p 2 , ...p m }, where each p i (@xmath1, 2, ..., m ) is a music phrase. the music phrase can be further decomposed into timed music notes (p @xmath2, n i2 , ...n in i }), where each music note n ij (@xmath3, 2, ..., n i }) comes with a duration and is associated with or without a pitch value. when a music note comes without a pitch value, it is a rest that indicates the absence of a sound and usually aligns with no lyrics. translation (mt) tasks (lample et al.; artetxe et al., 2019) which achieve melody (inference-time only) output title: in a little spanish town; genre: jazz; salient words: town, kiss, heart. decoding constraints: 1) music phrase segmentation, 2) rhythm alignment. keywords: line 1: [spanish, town], line 2: [love, kiss], line 3: [heart soul], … input plan plan-to-lyrics (aware of phonetics) inputto-plan figure 2: an overview of our approach that disentangles training from inference. blue represents components used during both training and inference, while brown means inference only. during training, our input-to-plan model learns to predict sentence-level plans (i.e., keywords) given the title, genre, and salient words as input. then, the planto-lyrics model generates the lyrics while being aware of word phonetic information and syllable counts. at inference time, we compile the given melody into 1) music phrase segments and 2) rhythm constraints to guide the generation. task formulation we aim to generate lyrics that comply with both the provided topic and melody. the input topic is further decomposed into an intended title t and a few salient words s to be included in the generated lyrics (see figure 2 for an example input). following the settings of previous work @xcite , we assume that the input melody m is predefined and consists of m music phrases (@xmath4, p 2 , ...p m }), and each music phrase contains n i music notes (p @xmath5, n i2 , ...n in i }). the output is a piece of lyrics l that aligns with the music notes: @xmath6, w 12 , ..., w m n }. here, for @xmath7, 2, ..., n i }, w ij is a word or a syllable of a word that aligns with the music note n ij . ## lyric generation model we draw inspirations from recent generation models with content planning. these models are shown to achieve increased coherence and relevance over end-to-end generation frameworks in tasks such as story generation @xcite @xcite . our lyrics generation model is similarly hierarchical as is shown in figure 2 . specifically, we finetune two modules in our purely text-based pipeline: 1) an input-to-plan generator that generates a keyword-based intermediate plan, and 2) a plan-to-lyrics generator which is aware of word phonetics and syllable counts. ## melody-guided inference in this section, we discuss the procedure to compile a given melody into constraints to guide the decoding at inference time. we start with the most straightforward constraints introduced before: 1) segmentation alignment and 2) rhythm alignment. note that both melody constraints can be updated without needing to retrain the model. ## experimental setup in this section, we describe the train and test data, baseline models, and evaluation setup. the evaluation results are reported in section 6. ## conclusion and future work our work explores the potential of lyrics generation without training on lyrics-melody aligned data. to this end, we design a hierarchical plan-and-write framework that disentangles training from inference. at inference time, we compile the given melody into music phrase segments and rhythm constraints. evaluation results show that our model can generate high-quality lyrics that significantly outperform the baselines. future directions include investigating more ways to compile melody into constraints such as the beat, tone or pitch variations, and generating longer sequences of lyrics with song structures such as verse, chorus, and bridge. future works may also take into account different factors in relation to the melody such as mood and theme.