D eep R apper: Neural Rap Generation with Rhyme and Rhythm Modeling (2021)

Abstract: Rap generation, which aims to produce lyrics and corresponding singing beats, needs to model both rhymes and rhythms. Previous works for rap generation focused on rhyming lyrics, but ignored rhythmic beats, which are important for rap performance. In this paper, we develop DeepRapper, a Transformer-based rap generation system that can model both rhymes and rhythms. Since there is no available rap datasets with rhythmic beats, we develop a data mining pipeline to collect a large-scale rap dataset, which includes a large number of rap songs with aligned lyrics and rhythmic beats. Second, we design a Transformer-based autoregressive language model which carefully models rhymes and rhythms. Specifically, we generate lyrics in the reverse order with rhyme representation and constraint for rhyme enhancement, and insert a beat symbol into lyrics for rhythm/beat modeling. To our knowledge, DeepRapper is the first system to generate rap with both rhymes and rhythms. Both objective and subjective evaluations demonstrate that DeepRapper generates creative and high-quality raps with rhymes and rhythms.

## introduction rap is a musical form originating from america in 1970s, and has quickly developed as one of the mainstream music genres in the world @xcite . with the rapid development of artificial intelligence, automatic rap lyrics generation has drawn attention from academia @xcite @xcite . generally speaking, rap lyrics need to be semantically meaningful and fashionable to convey interesting stories or express feelings. different from natural language or other artistic genres (e.g., lyrics or poetry), rap has distinctive characteristics: 1) it usually contains complex rhyme patterns among several consecutive sentences, which are the key to form a good flow; 2) it needs to align with the singing beat since rap lyrics are usually rapped according to some rhythmic accompaniments. therefore, how to generate rap lyrics with good rhymes and rhythms is a troublesome problem. previous works @xcite @xcite for rap generation mainly focused on lyric generation and some of them developed strategies for rhyme modeling. @xcite directly added a "<@xmath0. nikolov et al. (2020) applied a two-step strategy, which first generates rap lyrics and then adds rhyme tokens to the end of generated lyrics. however, these methods cannot guarantee the rhyme patterns for every lyric line and only care the rhyme on the last token. although many works have studied rhyming modeling in other artistic genres (e.g., poetry) @xcite @xcite , they are not suitable for rap generation due to the complex rhyme structure in rap. for example, poetry needs to rhyme with only the last word in each sentence, while rap rhymes with multiple consecutive tokens at the end of each sentence. no previous works have studied rhythm modeling (i.e., beats in rap), to our knowledge. one of the main reasons is the lack of rap datasets with beat-lyric alignment. consequently, the generation of lyrics without rhythmic beats cannot be regarded as a full rap generation. in this paper, we develop deeprapper, a transformer @xcite based rap generation system which can model both rhymes and rhythms. to build the system, since there is no available rap datasets with aligned rhythmic beats, we design a data mining pipeline and collect a large-scale rap dataset for rhythm modeling. specifically, we first crawl many rap songs, each song with both rap lyrics and audios, from the web. for each crawled rap song, we perform a series of data preprocessing steps to extract rhythmic beats as well as beat-lyric alignment. to better model rhyme, we generate the words in a rap sentence from right to left in an autoregressive manner. doing so we can easily identify the last few words of a sentence (now become the first words of the reverse sentence) to rhyme with. additionally, we incorporate several rhymerelated representations into our language model to further improve the rhyming quality, and encourage the n -gram rhyme in generated rap lyrics through rhyme constraint during inference. we use a special token @xcite to represent the rhythmic beat and insert it into lyrics right before the corresponding word. in this way, we can model the beat in the lyric sequence both in training and generation. inspired by the success of pre-trained language models @xcite @xcite @xcite , we incorporate pre-training into our system. to obtain large-scale data for pre-training, we also use our data mining pipeline to collect another two datasets: 1) non-rap songs with aligned beats, which can be larger than rap dataset since non-rap songs are more general than rap songs; 2) pure lyrics, which can be even larger than non-rap songs. in the pre-training stage, we pre-train our deep-rapper model based on the above two datasets. then we fine-tune our pre-trained model on the rap songs with aligned beats. the fine-tuned model is used for final rap generation. both objective and subjective evaluations verify the advantages of deeprapper in generating rap lyrics with rhymes and rhythms. our main contributions can be summarized as follows: ## background since deeprapper generates rap lyrics with both rhyme and rhythm modeling, in this section, we briefly introduce the related background: lyric generation, rhyme modeling and rhythm modeling. lyric generation broadly speaking, lyric generation can cover rap lyric generation @xcite @xcite , song lyric generation @xcite @xcite , general poetry generation @xcite @xcite and etc. different from previous works that leverage language model to generate lyrics similar to natural language, in this paper, we introduce a novel language model for rap generation, with well-designed rhyme and rhythm modeling to fit the characteristics of rap lyrics. additionally, inspired by the successes of pre-trained language models @xcite @xcite @xcite in nlp applications, we also incorporate pre-training into our model to further improve the quality of rap generation. rhyme modeling rhyme modeling plays an important role in rap generation, which requires the last few tokens in consecutive sentences to have the same rhyme pattern. existing rap generation systems either directly add a special token "<@xmath1, or introduce a two-step strategy for rhyme modeling that first generates rap lyrics and then adds rhyme tokens after the generated lyrics @xcite . however, these works only focused on unigram rhyme while rap appreciates more for n-gram rhyme. although a lot of works have explored rhyme modeling in other genres, most of them cannot be directly used for rap generation. for example, poetry generation @xcite @xcite usually used pre-defined format to control the rhyme pattern since poetry usually has fixed number of words and only cares the rhyme pattern for the last word. however, rap lyrics have diverse rhyme structures across multiple consecutive sentences and most importantly multiple con- data crawling vocal and accompaniment separation vocal and lyric alignment beat detection lyric and beat alignment dataset audio lyric vo cal , lyr ic acc om pa nim en t be at tim est am p lyr ic tim est am p secutive words. therefore, we introduce n -gram rhyme modeling in deeprapper to handle the distinctive rhyme patterns in rap. besides, we also train our language model in a reverse order (i.e., right to left), similar to previous works (van de cruys, 2020), to better model rhymes since they always occur at the end of sentence. rhythm modeling rhythm modeling is usually used in music generation @xcite @xcite which generates the duration of notes along with the note pitch to form rhythmic beats in melody and accompaniment generation. different from music generation, rap cares more about rhythmic beats instead of note pitches (i.e. melody). in this way, the generated rap lyrics need to align with the corresponding rhythmic beats in order to be rapped, otherwise it cannot be regarded as a complete rap. however, to the best of our knowledge, none of previous works have studied the rhythm modeling in rap generation. in this paper, we introduce a novel beat modeling strategy in deeprapper for rhythm generation. ## rap dataset mining previous works @xcite @xcite for rap generation usually used rap datasets with only lyrics, without considering the rhythmic beat information. to model rhythm in rap generation, the rap dataset should contain lyrics with aligned rhythmic beats. however, beat alignments are quite difficult to obtain, since their annotations require musicians with professional knowledge to identify stressing syllable in rap songs. to handle this problem, we design a data mining pipeline to automatically extract beatlyric alignments. in this section, we introduce the details of the data mining pipeline and our mined dataset based on this pipeline. ## rap generation model in this section, we introduce the architecture of our rap generation model, and the details of its rhyme modeling and rhythm modeling. ## experimental results results ## conclusions in this paper, we develop deeprapper, a novel transformer-based rap generation system, which leverages rhyme modeling, rhythm modeling and pre-training for rap generation. considering there is no available rap dataset with aligned rhythmic beats for rhythm modeling, we propose a data mining pipeline to mine a rap dataset with beat-lyric alignments. we leverage right-to-left generation, rhyme representation and rhyme constraint to better model rhyme and encourage n-gram rhyme, and explicitly model beat information by insert beat token beside the corresponding word in the lyric sequence. to our knowledge, deeprapper is the first system to generate rap with both rhymes and rhythms. both objective and subjective evaluations demonstrate that deeprapper generates high-quality raps with good rhymes and rhythms. thanks to the design of deeprapper, we can further build another rap singing system to sing out the raps according to the rhymes and rhythms, which we leave as future work. we also leave multilingual deeprapper as future work. medium figure 8 provides a rap generated by deeprapper with medium beat frequency, which the frequency is 2.6. the rap praises the times we live in. the following is the translation of texts in figure 8 . the following is the translation of texts in figure 9 .