Rapformer: Conditional Rap Lyrics Generation with Denoising Autoencoders (2020)

Abstract: The ability to combine symbols to generate language is a defining characteristic of human intelligence, particularly in the context of artistic story-telling through lyrics. We develop a method for synthesizing a rap verse based on the content of any text (e.g., a news article), or for augmenting pre-existing rap lyrics. Our method, called Rapformer, is based on training a Transformer-based denoising autoencoder to reconstruct rap lyrics from content words extracted from the lyrics, trying to preserve the essential meaning, while matching the target style. Rapformer features a novel BERT-based paraphrasing scheme for rhyme enhancement which increases the average rhyme density of output lyrics by 10%. Experimental results on three diverse input domains show that Rapformer is capable of generating technically fluent verses that offer a good trade-off between content preservation and style transfer. Furthermore, a Turing-test-like experiment reveals that Rapformer fools human lyrics experts 25% of the time.

## introduction automatic lyrics generation is a challenging language generation task for any musical genre, requiring story development and creativity while adhering to the structural constraints of song lyrics. here we focus on the generation of rap lyrics, which poses three additional challenges specific to the rap genre: (i) a verse in rap lyrics often comprises multiple rhyme structures which may change throughout a verse @xcite , (ii) the number of words in a typical rap verse is significantly larger when compared to other music genres @xcite , requiring modeling of long-term dependencies, and (iii) the presence of many slang words. 1 we created a song with lyrics generated by rapformer using the abstract of this paper as input, available in the supplementary material, and at https://bit.ly/3kxgitd . ## conditional generation of lyrics our approach to conditional generation of rap verses consists of three steps (figure 1 ). 1. given a dataset of rap verses, we apply a stripping approach to extract from each verse a set of content words that aim to resemble the main content of the original text, omitting any specific stylistic information. ## experimental setup datasets. we conduct experiments using three datasets. as our rap dataset, we use 60k english rap lyrics provided by musixmatch. 5 we split each lyric into verses (in the dataset, each verse is separated by a blank line), remove verses shorter than 4 lines in order to filter for song choruses and intros, and reserve 2k song lyrics for validation and testing. we use two datasets as our out-of-domain inputs: (1) the summaries from the cnn/dailymail news summarization dataset @xcite and ( 2 ) a subset of the cmu movie plot summary corpus @xcite . since some of the movie summaries are very long, for this dataset, we filter summaries longer than 140 tokens and shorter than 40 tokens. ## automatic evaluation we conduct an automatic evaluation of rap-former, using the test sets of each of our three datasets. our focus is on measuring two components that are important for generating fluent conditional rap verses: preserving content from the input text to the output, and maintaining rhyming fluency during generation. ## human evaluation due to the limitations of automatic metrics for text generation, we also perform four human evaluation experiments using three raters, who are trained to translate lyrics. due to limited resources, we evaluate only the rapformer variant with the shuffle stripping approach and rhyme enhancement, which achieved the highest content overlap in our automatic evaluation. the first two human experiments (in to 5 (this could be from existing rap lyrics), which measures the capacity of our models to preserve the style. ## example model outputs in tables 5 , 6 and 7, we also display a few manually selected example model outputs (additional examples are available in the appendix) produced after inputting content words extracted from each of our input text styles (existing rap lyrics, movie plot summaries and news article summaries). when using existing rap lyrics as input, many outputs seem coherent and of higher quality in comparison to outputs produced using news/movie inputs. for news/movie inputs, the models are still capable of integrating the input content words into a rhyming verse that preserves some of the overall meaning of the original text (e.g., "the film also follows the adventures of lucius the slave escaping via the underground railroad to freedom" → "slave, run from lucius slavery; battle of freedom and liberty"). furthermore, in ## conclusion we have proposed a novel approach to generation of rap verses conditioned on a list of content words. we showed that our method is capable of generating coherent and technically fluent synthetic verses using diverse text types as input, including news articles, movie plot summaries, or original rap verses. the fluency of our outputs is further improved through a novel rhyme enhancement step. our approach is particularly effective when rephrasing the content of existing rap lyrics in novel ways, making it a potentially useful tool for creative writers wishing to explore alternative expressions of their ideas. the generality of our approach to conditional text generation makes it applicable to generation of creative texts in other domains, such as poetry or short stories. future work could explore other approaches to extracting content words, including combining several stripping approaches, and could explore the utility of large-scale pretrained models (e.g., @xcite ) for this task. another direction is to extend our question 45 of 100 lyrics (a) lyrics (b) waka waka: i say na correct eye i take waka this waka they say na blind eye, take it far but after i've got you i blind pata pata i've got it on my own, my own oche du no dum no oda du num doka oche num, oda du, doka dum so anybody try you i go shoot the murderfker if anybody ever try go shoot the almighty ever blazing you amazing blazing so amazing which of these lyrics was written by a human? correct answer: (b) question 72 of 100 lyrics (a) lyrics (b) vegas on the third floor, like lamar with the cardio out in vegas like lamar, third floor tropicana fascinated by the cars smokin' dope in the casino fascinated with the cars, smokin' dope in the phantom despise the propaganda rise, higher teflon's on the rise, i despise propaganda mac-11 camouflage for example, that's why i never set fires camouflage mac-11, i should set an example i walk with a flame that never match my desires never baptized, as i walk through the fires take a pic, cause the pain is higher the pain and the flame never match my desires i'm rich as a coupe, light it up with kelly crucified cause i'm rich, in the coupe, take a pic phone sucker, my friend, it's a blessing on the phone at the light, kelly rowland's a friend benz, plaques, wall, and g6's catfish in the benz, manti teo's a sucker -'em all, hustler say the victim plaques on the wall, hustler so i can say "-'em" ciroc and bel air -bel air for the -, ciroc in the pool april -'s -, her name so my -is a -, her name is april's a fool work to end-to-end generation with an integrated rhyming loss function, which could potentially be tackled using reinforcement learning @xcite . moreover, the task of generating coherent lyrics from a set of content words could be naturally modeled as a text-editing task @xcite @xcite instead of a sequence-to-sequence task. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. input: hoping to improve his financial lot , petty thief hawk chovinski hires a dancing instructor to teach him how to bear himself like a gentleman . his lessons completed , hawk then poses as a european nobleman , intending to trap a wealthy wife . yolande cowles sees through hawk ' s pose but falls in love with him anyway . drop + re: i improve a grizzly lot of petty thief times dancing in the middle of the night i am the man who can teach you how to bear it like a gentleman with diamonds i'm a superheroic, i can be your wife yolande cowles tonight falls in love anyway synonym + re: hoping that you can improve a financial lot of petty use mortarboard chovinski engage dancing with the snake teach her how to settle like a gentleman lessons are shackled by a bullet sit in european imagine in the trap with a wealthy wife yolande hood sees the sky when the pose falls in line anyway, no, not me shuffle + re (rapformer): you teach me petty dancing like bear thief chovinski, intersect, be a lot of financial gentleman hoping he can improve somebody wife, nobleman, the trap is so polished wealthy hawk lessons, european hawk lessons yolande cowles anyway, sees him pose when he says hawk love! input: the film follows the lives of several west point cadet classmates who find themselves on opposite sides of the war . the film also follows the adventures of lucius the slave escaping via the underground railroad to freedom with the film cutting between the first battle of bull run and the birth of a lucius ' child born in slavery . drop + re: film of the west point where they can find the opposite sides of ours film also and they will be a slave escaping me from the underground, and we will not be the same if we are not the maker this is a film cutting first bull from birth to child's slaver. synonym + re: film to succeed our lives in several zones our head is the most likely to find our own we are not the same as the other side of ever film also follows adventure the lucius slave, the escaping via underground motorical, freedom, film out first battle bull, then feed him birth golden child, born in order shuffle + re (rapformer): this is the opposite of war follows lives on both sides several point film from the west to the light find a child born escaping via immigration film the underground cutting off the film of all the complications slave, run from lucius slavery battle of freedom and liberty also the first bull follows luc-up!