Sudowoodo: A C hinese Lyric Imitation System with Source Lyrics (2023)

Abstract: Lyrics generation is a well-known application in natural language generation research, with several previous studies focusing on generating accurate lyrics using precise control such as keywords, rhymes, etc. However, lyrics imitation, which involves writing new lyrics by imitating the style and content of the source lyrics, remains a challenging task due to the lack of a parallel corpus. In this paper, we introduce Sudowoodo, a Chinese lyrics imitation system that can generate new lyrics based on the text of source lyrics. To address the issue of lacking a parallel training corpus for lyrics imitation, we propose a novel framework to construct a parallel corpus based on a keyword-based lyrics model from source lyrics. Then the pairs (new lyrics, source lyrics) are used to train the lyrics imitation model. During the inference process, we utilize a post-processing module to filter and rank the generated lyrics, selecting the highest-quality ones. We incorporated audio information and aligned the lyrics with the audio to form the songs as a bonus. The human evaluation results show that our framework can perform better lyric imitation. Meanwhile, the Sudowoodo system and demo video of the system is available at Sudowoodo and https://youtu.be/u5BBT\_j1L5M

## introduction ai creative assistants are artificial intelligence systems that can learn from large amounts of text data to understand human language and culture and use this knowledge to create content such as story generation @xcite , poetry writing @xcite @xcite , grammar and spelling checking @xcite , etc. in addition, ai creative assistants can also assist in songwriting @xcite @xcite by learning from numerous songs, understanding human emotional expression, and creating * corresponding author music in a similar writing style to humans. previous research @xcite @xcite @xcite @xcite has focused on generating lyrics based on specified keywords (e.g., snow), lyrics styles, themes, or user input passages, which generate new lyrics with limited control over the content. however, in actual music production, users sometimes adapt excellent songs by adding their own creativity while remaining the original lyrical structure, resulting in new lyrics. this requires stronger control over the source lyrics such as text content, emotion, and fine-grained writing styles. to address this issue, this paper demonstrates sudowoodo foot_0 (a pokémon with the ability to imitate) a chinese lyrics imitation generation system based on source lyrics. sudowoodo is typically based on the encoder-decoder framework, where the encoder encodes the text and attributes of the source lyrics, and the decoder generates the imitated lyrics. however, since we only have the source lyrics and not the target ones, the parallel corpus is lacking to train the imitation model. to solve the problem, we also propose a method for constructing aligned training samples, which generated the target lyrics from the extracted keywords of source lyrics using a keywords-based lyrics generation model. specifically, we first collect the source lyrics corpus d k from the internet foot_1 and utilize the keyword extraction method described in section 2.1 to extract keywords from source lyrics. and we train a keywords-based model, named model k2l , which can generate lyrics from given keywords. then, we generate the target lyrics d k ′ using the model k2l . finally, we train a lyrics imitation model with the aligned lyrics corpus (d k ′ , d k ) based on the encoder-decoder framework. in addition, to improve the quality of generated lyrics lyrics: 这世界有那么多人 (there are so many people in the world) 人群里 敞着一扇门 (there is an open door among the crowd) source lyrics keywords: 世界(world)、人群(crowd) style: 流行 (pop) emotion: 积极 (positive) attributes generated lyrics: 这世界不声不响 (this world is quietly and silently) 好多人 也走得匆忙 (many people also leave in a hurry.) generated lyrics keywords to lyrics model lyrics to lyrics imitation model attributes extraction lyrics: 这世界有那么多人 (there are so many people in the world) 人群里 敞着一扇门 (there is an open door among the crowd) source lyrics generate lyrics model k2l transformer encoder transformer decoder model l2l transformer decoder transformer encoder lyrics quality scoring relevance scoring composing and singing post-processing figure 1: the framework of sudowoodo system proposes in this paper. model k2l denotes a model for generating lyrics based on keywords, while model l2l represents the generation model from source lyrics to imitation lyrics.encoder refers to the encoding portion of the encoder-decoder architecture, while decoder represents the decoding portion. post-processing is mainly aimed at the imitation lyrics generated based on the model l2l . and better showcase the results, we also employ post-processing modules including lyrics quality scoring and relevance scoring. meanwhile, to provide a more intuitive understanding of the generated lyrics through imitation, we incorporate audio information (the vocals and melody of the source song) and align the lyrics with the audio to produce a complete song. the main contributions of the sudowoodo system are summarized as follows: ## framework the sudowoodo system consists of two models and a post-processing module, as illustrated in figure 1 : model k2l , model l2l , and post-processing. these modules will be described in greater detail below. ## results of the experiment we conduct an ablation study to evaluate the framework proposed in this paper. metrics: we evaluate the generated lyrics from four perspectives: (1) thematic: the relevance of the imitation lyrics to the theme of the source lyrics, including love, friendship, family inspiration, etc. ( 2 ) fluency: it refers to the smoothness theme (avg.) flu (avg.) logic (avg.) overall (avg.) best (%) model k2l 4.168 4.103 3.480 4.078 32.75 model l2l 4.250 4.160 3.460 4.153 34.25 w/o ws 4.275 4.108 3.415 4.148 33 like waking up from a dream, empty-handed. empty-hearted. in the end, i chose to let go. learning to give oneself freedom. we are accustomed to using tenderness to soothe wounds. bowing our heads. don't want to play the role of a half-row audience anymore. don't want to let my impulses fall through anymore. every sincere heart can be understood. guessed correctly. i admit that i would rather be an audience. a self-witness is enough. love keeps spinning in life. and is very precious. and naturalness of the language used in the lyrics. in evaluating the fluency of a song's lyrics, we consider factors such as the fluency of the words and the rhythmic structure of the sentences. (3) logic: it refers to the coherence and smoothness of scene transitions in the lyrics. to evaluate the logic of a song's lyrics, we consider whether consecutive sentences describe a single scene. if m consecutive sentences describe a scene, we argue that those sentences are reasonable within logic. if n consecutive groups of m sentences are found to exist within n different scenes, the lyrics are considered to have a high degree of smooth scene transitions overall. the number of scene jumps 7 can measure the logic of the song. (4) overall: the overall scoring of a song's lyrics. results: we sample 100 lyrics from the source dataset and generate three imitation lyrics for each source lyric. we invite 3 professional lyricists to score each of the 300 lyrics based on thematic, fluency, logic, and overall. the score ranges from 7 scene jumps occur when consecutive sentences describe different things or switch abruptly between different sensory perspectives, resulting in an unnatural or jarring transition. 1-5, with 5 being the best and 1 being the worst. the results are shown in ## demonstration this section demonstrates how the sudowoodo system works. the user interface for this demo is shown in figure 2 . as an imitation demo, it offers limited interaction with the user. the sudowoodo system operates in two modes: lyrics and songs. in lyrics mode, the user is required to select the source how long has it been since i heard from you? where the hell are you, honey? was it ever so kind to me? the torrential rain that fell from high altitude. the heavy rain that fell in the city. blown away by the wind, i lost you. i am very fortunate to have encountered this love. although even the most unforgettable memories will eventually expire. the heavy rain that i remember from my memories. led me to a strange city. it's been a long time since i've heard from you. but there has never been a trace left behind. lyrics and the desired sentiment for the generated lyrics. additionally, the user may provide keywords, which are typically space-separated phrases such as "自由 爱情 (freedom love)" or, alternatively, left blank. when generating lyrics, the sudowoodo system takes into account the writing style of the selected source lyrics, including its theme, rhymes, and provided keywords, as well as the desired sentiment. the provided keywords are highlighted for easy identification. note that not all provided keywords are necessarily used in the generated lyrics. in songs mode, the user can select the name of the source lyrics to hear the generated lyrics as a song. due to technical limitations, the lyrics are rendered offline. in this paper, we apply three different ai singers to provide the sounds. finally, the user can click "generate!" to produce the output. next, we show some generated examples in figure 3 . lyrics: the leftmost column of the display lyrics represents the source lyrics selected by the user, while the three columns on the right show the generated imitation lyrics. if the user has entered keywords, these will be highlighted in red within the generated lyrics. this demo can generate smooth, high-quality lyrics in a format and writing style similar to the source lyrics for each generation. songs: figure 4 shows the results in songs mode. as the real-time rendering of songs is a challenging task, we have performed offline render-ing for this demo. a player is provided above the generated lyrics, which can be clicked on to hear the resulting song after rendering with the imitation lyrics. in the future, we aim to integrate real-time rendering of songs to create a true lyric imitation system that can take source lyrics and generate corresponding songs. more experiences are available in sudowoodo. ## conclusion in this paper, we describe sudowoodo, a chinese lyric imitation system that supports two modes: lyrics and songs. in lyrics mode, users can input keywords to generate imitated lyrics based on existing lyrics. in songs mode, sudowoodo uses an unspecified technology to generate music that accompanies the imitated lyrics to create a complete song. to address the lack of a lyric-to-lyric alignment corpus, we propose a novel training framework structure to construct a parallel corpus for lyric imitation. additionally, we apply chinese pre-trained gpt-2 for initialization. to improve the quality of the generated lyrics, we employ a post-processing module to sort the generated results and select the highest quality ones. finally, we audio-aligned some of the imitation lyrics to form songs!