S ong R ewriter: A C hinese Song Rewriting System with Controllable Content and Rhyme Scheme (2023)

Abstract: Although lyrics generation has achieved significant progress in recent years, it has limited practical applications because the generated lyrics cannot be performed without composing compatible melodies. In this work, we bridge this practical gap by proposing a song rewriting system which rewrites the lyrics of an existing song such that the generated lyrics are compatible with the rhythm of the existing melody and thus singable. In particular, we propose SongRewriter, a controllable Chinese lyric generation and editing system which assists users without prior knowledge of melody composition. The system is trained by a randomized multi-level masking strategy which produces a unified model for generating entirely new lyrics or editing a few fragments. To improve the controllabiliy of the generation process, we further incorporate a keyword prompt to control the lexical choices of the content and propose novel decoding constraints and a vowel modeling task to enable flexible end and internal rhyme schemes. While prior rhyming metrics are mainly for rap lyrics, we propose three novel rhyming evaluation metrics for song lyrics. Both automatic and human evaluations show that the proposed model performs better than the state-of-the-art models in both contents and rhyming quality.

## introduction with the rapid development in natural language processing @xcite @xcite , lyrics generation has achieved significant advancement in recent years @xcite @xcite . prior works mainly focus on two research topics: figure 1 : an overview of the proposed chinese song rewriting system. given an existing song, the users first mask the part(s) of the lyrics that they want to rewrite. then, songrewriter generates new lyrics corresponding to the masked fragments. last, the rewritten lyrics are combined with the original melody to form a new song. during the generation, user can require the content to include specific keywords or control the rhyme scheme by setting the vowels of the output characters at specific positions. current methods of generating singable lyrics is typically conditioned on a given melody. accordingly, they treat the generation as a sequenceto-sequence translation task. however, there are two main challenges: 1) the parallel lyrics-melody dataset for training the model is limited, which mainly consists of 7,998 songs proposed by @xcite ; 2) melody notes and lyric syllables are loosely correlated and thus the alignments between them are hard to learn from the limited data. therefore, prior works simplify the task by assuming a one-to-one mapping between the melody notes and lyric syllables @xcite . however, such an assumption and restrictions may lead to a sub-optimal performance, as the mapping relationship is usually many-to-one in real life. another important research question on lyrics generation is how to control the generated content and rhyme. in prior works, the controllable content is usually achieved by conditioning the generation on given hint words or sentences @xcite . however, they usually ignore the requirements of generating lyrics from a draft where the model needs to edit some sentences or words. in addition, prior works on rhyme control mainly focus on rhymes at the end of sentences (end rhyme) @xcite @xcite . to the best of our knowledge, no work has been done on both internal rhyme and end rhyme schemes. in this work, we develop a user-assist ai system songrewriter which can generate singable lyrics by rewriting parts of or the whole lyrics of a given song (i.e., partial rewriting and full rewriting, respectively) with controllable contents and rhyme schemes, as illustrated in figure 1 . to address the difficulty of learning the correlation between the melody and lyrics from a limited amount of parallel dataset, we propose to generate lyrics which have the same number of syllables as the original lyrics of the song. therefore, the generated lyrics can well align with the melody. in addition, this method can directly learn the generation from text data so as to bypass the demand of parallel datasets. specifically, we adopt a transformer-based sequence-to-sequence auto-regressive model @xcite as our model backbone. the model is trained by masking a few random fragments from the encoder's inputs and predicting the masked fragments by the decoder. the generation process can be controlled in terms of three aspects. 1) to enable rewriting arbitrary parts of the lyrics, we train the model by randomly performing one of the three masking (i.e., token-level, sentence-level and song-level) strategies, corresponding to different levels of rewriting tasks. 2) to control the lexical choice of contents, we allow the generation to condition on given keyword prompts. this is achieved by training on extracted keywords from masked positions as additional encoder inputs. we evaluate our model on both generation controllability and quality in terms of keyword recall, vowel accuracy, lexical diversity, coherence, perplexity and rhyme quality. since prior evaluation metrics on rhyme quality are mainly for rap lyrics and ignore the problem of identical rhyming words, we propose three new rhyme metrics to measure the local rhyme between adjacent sentences, global rhyme and diversity of rhyming words, respectively. experimental results show that our model outperforms baseline models and state-of-the-art models on both full and partial rewriting tasks. our contributions are summarized as follows: ## related work lyrics generation can be mainly divided into two categories, rap lyrics generation @xcite @xcite and song lyrics generation @xcite @xcite @xcite @xcite . while rap lyrics generation mainly focuses on improving the rhyming performance @xcite , song lyrics generation concentrates on the alignment between melody and lyrics @xcite @xcite . rhyming is an essential element for lyrics and poetry. to model rhyme, current approaches can be mainly divided into three types. the first type is to encourage the model to learn the rhyme structure implicitly during the training by adding additional rhyme signals. @xcite append ⟨endline⟩ token to each sentence. li et al. (2020) employ an additional format embedding as inputs to emphasize the rhyming tokens. lau et al. (2018) incorporate an extra model to encode the ending tokens of the sentences. while zhang et al. (2020) and xue et al. (2021) both generate the rhyming word before the rest of the sentence, zhang et al. (2020) move the last word of the sentence to the front, and xue et al. ( 2021 ) generate sentence from right to left by reversing the word order. the second type is to apply explicit rhyme constraints during the training to force the model to generate rhyming sentences. @xcite apply a discriminator on the sentence-ending words to learn the rhyme pattern adversarially. the last type is through post-editing, where the model first generates the lyrics, then another model edits the ending words to fulfil the rhyming constraints @xcite . most prior works focus on the rhyme at the end of the sentences @xcite . in this work, we extend the rhyme control to arbitrary rhyme schemes for both internal rhyme and end rhyme. to the best of our knowledge, this is the first work enabling arbitrary rhyme schemes. ## method the proposed model is a transformer-based autoregressive sequence-to-sequence model @xcite . as shown in figure 2 , given an input of masked lyrics and keyword prompt to the encoder, the decoder generates output tokens corresponding to the masked tokens of the input, which contain the keywords in the prompt and satisfy the vowel constraints. such generation also improves decoding efficiency and forces the model to rely more on the source input. in a basic setting, given the lyrics of a song, the tokens of the input lyrics are of the following form, where x ij denotes the jth tokens in the ith sentence of the lyrics, [s] is the inter-sentence delimiter, [b] is placed at the beginning of the lyrics with [e] at the end, and the tokens to be rewritten are replaced by [m ] which will be predicted by the decoder. to enable the controllability of the generation process, we further incorporate the keyword prompt into the model to control the lexical choices, propose the multi-level masking strategy to enable rewriting arbitrary parts of the input in a single model and the rhyme control strategy to inject predefined rhyme schemes and improve rhyming word diversity. ## conclusion in this work, we propose to overcome the difficulties of modelling the melody-lyrics correlation from limited parallel datasets by directly rewriting the lyrics of songs. we propose a unified model for full and partial song rewriting by training with a multi-level randomized masking scheme. the proposed model allows rewriting arbitrary parts of the inputs according to the bidirectional context. besides, we introduce a partial vowel masking strategy into training to enable lyrics generation on any rhyme schemes. a novel decoding strategy is designed to improve the end rhyme consistency and rhyming word diversity. novel metrics are pro-posed for rhyme evaluation. both automatic and human evaluation shows our proposed model outperforms baseline and state-of-the-art models.