P oe LM : A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation (2022)

Abstract: Formal verse poetry imposes strict constraints on the meter and rhyme scheme of poems. Most prior work on generating this type of poetry uses existing poems for supervision, which are difficult to obtain for most languages and poetic forms. In this work, we propose an unsupervised approach to generate poems that follow any given meter and rhyme scheme, without requiring any poetic text for training. Our method works by splitting a regular, non-poetic corpus into phrases, prepending control codes that describe the length and end rhyme of each phrase, and training a transformer language model in the augmented corpus. The transformer learns to link the structure descriptor with the control codes to the number of lines, their length and their end rhyme. During inference, we build control codes for the desired meter and rhyme scheme, and condition our language model on them to generate formal verse poetry. Experiments in Spanish and Basque show that our approach is able to generate valid poems, which are often comparable in quality to those written by humans.

## introduction despite the impressive generative capabilities of large language models (lms) @xcite @xcite automatic poetry generation remains a challenging problem. formal verse poetry, in particular, imposes strict constraints on the meter and rhyme scheme of poems (figure 1 ), which cannot be directly controlled in conventional lms. prior work on generating formal verse poetry has primarily focused on supervised approaches, leveraging existing poems to train lms. this is often combined with additional techniques to impose the meter and rhyme constraints at inference time, such as using finite-state automata to discard invalid candidates @xcite , or generating text right-to-left to better control the rhyming word the poem is the first stanza of a spanish sonnet, which must have 4 lines of 11 syllables and follow an abba rhyme scheme. we use control codes to describe such constraints, and train a language model that can generate text conditioned on them. @xcite @xcite . however, these approaches require poetic text for training, which is difficult to obtain for most languages and poetic forms. in this paper, we propose an unsupervised approach to generate formal verse poetry. our poetic language model (poelm) can be conditioned to follow any desired meter and rhyme scheme, without requiring any poem for training. as illustrated in figure 2 , the key idea behind our method is that any text can be divided into phrases, which will each have a certain number of syllables and end in a certain sound that can make it rhyme with other phrases. while this structure will not follow a regular pattern for standard text, as it would for poetry, we can still annotate it automatically, and train a language model that can be conditioned on such structure descriptors. at inference time, we build a structure descriptor for the desired meter and rhyme scheme, and condition our language model on it to generate formal verse poetry. to improve results, we generate multiple candidates, which are automatically filtered and re-ranked. our experiments in spanish and basque show that our method is able to generate high quality poems meeting the desired meter and rhyme constraints, with human evaluators ranking our system higher than other humans in more than one third of the cases. our code is available at github. 1 en estos años se verá, debido al cambio climático, una subida de temperatura. ## background: formal verse poetry poetic traditions differ across languages and cultures. in this work, we focus on formal verse poetry in spanish and basque, 2 which impose strict meter and rhyme constraints as follows: ## proposed method as described in §2, we want our system to be able to generate text that adheres to a specific structure. the key idea behind our approach is that, similar to formal verse poetry, any text adheres to a certain implicit structure. in the case of non-poetic text the structure will not follow any regular pattern, but we can still extract it and build a structure descriptor for it. we can then augment the non-poetic corpus with these descriptors, and train a regular lm on it (figure 2a ). the model thus learns to respect the structure provided in the descriptor, which allows us to generate formal verse poetry at inference time, by conditioning the model on the appropriate structure descriptor (figure 2b ). we next describe the two main components of our method: structure-aware training ( §3.1) and inference with filtered re-ranking ( §3.2). to last syllable onwards, and the following consonant groups are considered to sound the same for the purposes of rhyme: {p,t,k}, {n,m}, {s,z,x}, and {b,d,g,r}. ## experimental design we run experiments on spanish and basque. we next detail the training details ( §4.1) and the automatic and human evaluation setup ( §4.2 and §4.3). ## results we next discuss our main results for the automatic ( §5.1) and human evaluation ( §5.2). ## analysis we further analyze our system by quantifying at which portion of the poem its perplexity gain is highest ( §6.1), experimenting with manual re- ## related work we next review relevant literature in poetry generation ( §7.1), as well as controllable generation ## conclusions and future work in this work, we present an unsupervised approach to generate formal verse poetry. we identify and extract the latent structure in non-poetic corpora, and feed this information along with the text to a transformer lm, allowing us to control the structure of the text at generation time. our system is capable of generating formal verse poetry with flexible meter and rhyme schemes, without requiring any sort of poetic text to train. the required structure can be easily altered by changing the descriptor, allowing us to generate different types of poetry without needing to re-train the system. automatic and human evaluations show that our model learns to leverage the provided structure information to better predict the text, and is capable of generating short poems that are often preferred to those created by a human. in future work, we would like to extend our framework to be able to control other aspects of the generated text in addition to meter and rhyme.