U ni LG : A Unified Structure-aware Framework for Lyrics Generation (2023)

Abstract: As a special task of natural language generation, conditional lyrics generation needs to consider the structure of generated lyrics and the relationship between lyrics and music. Due to various forms of conditions, a lyrics generation system is expected to generate lyrics conditioned on different signals, such as music scores, music audio, or partially-finished lyrics, etc. However, most of the previous works have ignored the musical attributes hidden behind the lyrics and the structure of the lyrics. Additionally, most works only handle limited lyrics generation conditions, such as lyrics generation based on music score or partial lyrics, they can not be easily extended to other generation conditions with the same framework. In this paper, we propose a unified structure-aware lyrics generation framework named UniLG. Specifically, we design compound templates that incorporate textual and musical information to improve structure modeling and unify the different lyrics generation conditions. Extensive experiments demonstrate the effectiveness of our framework. Both objective and subjective evaluations show significant improvements in generating structural lyrics.

## introduction great progress has been made in natural language generation (nlg) with pre-trained language models in recent years @xcite @xcite . lyrics generation is also a special task of nlg @xcite . different from general natural language, lyrics eventually need to be presented with music after the composition. this requires the lyrics to follow song-writing rules (i.e., the structure of lyrics), such as clear paragraphs with chorus and verse concepts. however, most previous works ignore the musical concepts behind the lyrics and do not consider the structure of lyrics @xcite . to explicitly model the structure of lyrics, some works introduce additional structural labels (e.g., sentence-level chorus and verse label), which inevitably require much effort for additional human annotation @xcite . to avoid the huge annotation cost, other works either adopt predefined formats (e.g., the number of syllables in each sentence) or linguistic tags (e.g., pos, part-of-speech) to inject structural information @xcite . nevertheless, given that those methods cannot directly indicate the structure of lyrics, the generated lyrics are still difficult to realize the musical concepts (e.g., chorus and verse). moreover, most works only focus on certain lyrics generation conditions, such as generating lyrics given music scores or partially-finished lyrics etc., which hinders the application of a lyrics generation model in various scenarios. to mitigate the issues in previous works, we propose a unified structure-aware lyrics generation framework named unilg. as illustrated in figure 1 , the chorus parts of the songs share the same melody, so that the corresponding lyrics follow a similar pattern. such a phenomenon inspires us that shared musical signals indicate the structure of lyrics and can be used to infer the relation across lyrics. therefore, we design a compound template (i.e., a sequence of tuples) that incorporates both textual and musical information to model the structure of lyrics. the template is designed with rhythmic concepts in mind, and it can be extracted from different sources (e.g., audio, music score, etc.). as shown in figure 2 , the general interface in the template enables unilg to generate lyrics based on various conditional signals without re-training the model. additionally, we propose a cycle-consistency loss to enforce the reconstruction of the musical information from the generated lyrics to further improves the performance. to verify our proposed framework, we collect a test dataset named song8k with chorus and verse labels for each sentence. both objective and subjective evaluations on the test dataset demonstrate the effectiveness of our framework. in summary, the main contributions of this work are as follows: ## method our proposed unified framework for lyrics generation (unilg) contains two highlights: 1) it considers the structure of lyrics in the generation; 2) it can handle different lyrics generation conditions with different control signals, such as the music score, or music audio, or partial lyrics, etc. for the structure of lyrics, as illustrated in figure 1 , the melody implies the structure of lyrics, which can be leveraged for lyrics structure modeling. however, large-scale (melody, lyrics) parallel data is generally difficult to obtain. we, therefore, propose using rhythm patterns 6 that preserves the inter-correlation of lyrics as musical information to explicitly represent the structure of lyrics. as explored in previous works @xcite , the defined rhythm patterns can be efficiently extracted from lyrics and different rhythmic sources (e.g., music score, music audio, etc.) without extra human annotation. for handling various control signals of different 6 the rhythm patterns usually indicate when a note happens and how long it lasts. in this paper, the rhythm patterns only indicate when a note happens. lyrics generation conditions, the model should have the capacity to process different types of inputs, such as music score, music audio, rhythm patterns, partially-finished lyrics, etc. therefore, we design a compound template (i.e., a sequence of tuples) that can incorporate both textual and musical information. so any type of input can be converted into a compound template, and then lyrics can be generated based on the compound template. figure 2 illustrates the overview of our proposed lyrics generation framework unilg. we propose an intermediate compound template as a bridge between the rhythmic sources (e.g., audio, music score, etc.) and lyrics in unilg. specifically, the lyrics generation is decomposed into a two-stage pipeline consisting of an input-to-template stage and a template-to-lyric stage. in this section, we first describe the compound template in detail. we then present the two stages during training respectively. finally, we discuss the inference procedure of unilg to illustrate how to handle various control signals in different lyrics generation conditions with our unified framework. ## experiments results in this section, we report and analyze both objective and subjective experimental results. we also show some cases in appendix h to verify the ability of unilg to handle different generation conditions. ## conclusion in this paper, we propose unilg, a unified structure-aware lyric generation framework. with our designed compound template to indicate the structure of lyrics with textual and musical information, which acts as a bridge between the rhythmic sources and lyrics, unilg can handle different lyrics generation conditions. we also introduce a cycle-consistency loss to enhance the impact of musical information to improve performance. extensive experiments demonstrate the effectiveness of our framework, achieving significant improvement in both objective and subjective evaluations. we will explore topic-driven lyrics generation in our future work.