Acrostic Poem Generation (2020)

Abstract: We propose a new task in the area of computational creativity: acrostic poem generation in English. Acrostic poems are poems that contain a hidden message; typically, the first letter of each line spells out a word or short phrase. We define the task as a generation task with multiple constraints: given an input word, 1) the initial letters of each line should spell out the provided word, 2) the poem’s semantics should also relate to it, and 3) the poem should conform to a rhyming scheme. We further provide a baseline model for the task, which consists of a conditional neural language model in combination with a neural rhyming model. Since no dedicated datasets for acrostic poem generation exist, we create training data for our task by first training a separate topic prediction model on a small set of topic-annotated poems and then predicting topics for additional poems. Our experiments show that the acrostic poems generated by our baseline are received well by humans and do not lose much quality due to the additional constraints. Last, we confirm that poems generated by our model are indeed closely related to the provided prompts, and that pretraining on Wikipedia can boost performance.

## introduction poetry, derived from the greek word poiesis ("making"), is the art of combining rhythmic and aesthetic properties of a language to convey a specific message. its creation is a manifestation of creativity, and, as such, hard to automate. however, since the development of creative machines is a crucial step towards real artificial intelligence, automatic poem generation is an important task at the intersection of computational creativity and natural language generation, and earliest attempts date back several decades; see @xcite for an overview. with this paper, we add a new task to this research area: acrostic poem generation in english. acrostic poems, or simply acrostics, are a special type of poetry, in which typically the first letter of each line spells out a word or message, as in the example in figure 1 . while this is the only formal characteristic of an acrostic, we here define the task of acrostic poem generation as generating poems such that, additionally, poems should also both rhyme and relate to the topic of their hidden word, e.g., the content of the poem in figure 1 should be related to the word "poet". as far as meter is concerned, we are interested in free verse poems. acrostic poem generation, as we define it, is a challenging constrained generation task with multiple constraints: semantic ones (the content of the poem should follow a given topic), and structural ones (the poem should rhyme, and the first letters should spell out a given word). we further propose a baseline model for the task, which we call the neural poet. it is a generative model, which consists of two components: a conditional neural language model, which generates an acrostic poem based on a given word, and a rhyming model, trained on sonnets, which generates rhyming words for the last position in each line. furthermore, two acrostic-specific challenges need to be solved: (i) generating such that the first letters of all lines spell out the defined word, and (ii) making sure that the resulting poem relates to the topic of that word. we address the first challenge by limiting the choices in the output softmax during sampling from the language model. for the second challenge, we feed the word embedding of the topic to the language model at each time step. since no large datasets for the task are available, we scrape poems for given key words from the web and train a separate discriminative model to predict topics, which we then use to predict silver standard topics for a larger poem dataset. our final model is trained on a combination of poems with predicted and gold topics. human evaluation on fluency, meaningfulness, and poeticness @xcite shows that our additional constraints hardly reduce performance as compared to unconstrained generation. further, our poems relate to the acrostic word, even if the topic does not appear in the training set. finally, we show that model performance-in terms of perplexity on a held-out validation set-can be improved by pretraining on wikipedia. ## datasets to train the baseline model for our new task, we make use of 4 datasets, which we will describe here, before explaining the actual model in the next section. knowntopicpoems. we scrape poems from the web 1 in order to create our first dataset (known-topicpoems). the poems on this site are a good match for our task, since they are sorted by topics. our resulting dataset contains 144 topics, and a total of 32,786 poems. these poems are all of different lengths, but we aim at generating poems of up to 8 lines. 2 thus, we split all longer poems such that they contain at least 4 and a maximum of 8 lines in the following way: first, we split poems on empty lines, since those usually mark the end of a semantic unit. second, if any of the resulting partial poems are still too long, we split them at full stops, but only use the beginning, since not every new sentence makes for a meaningful start of a poem. we add all possible options. this step increases the number of poems to 53,972, belonging to 144 topics. this dataset is used to train the language model of our neural poet, and to train the topic prediction model which is used to create additional training data for poem generation. we use 80%, 10%, and 10% of the data for training, development, and test, respectively. we tokenize all poems with the nltk wordtree-bank tokenizer package @xcite . unknowntopicpoems. we further make use of another poem dataset taken from @xcite foot_0 (unknowntopicpoems), since it is larger than knowntopicpoems. however, this dataset does not explicitly state the topics of individual poems. thus, we automatically predict a topic for each poem with the help of our topic prediction model, which will be described in subsection 3.3. the poems in unknowntopicpoems are again broken down into poems with 4 to 8 lines, and, for our main experiments, this dataset is combined with known-topicpoems, increasing the number of samples to 119,252 poems for 144 topics. sonnets. the last poem dataset we make use of is the sonnet poem dataset introduced by @xcite . this dataset consists of shakespearean sonnets. since those differ significantly in style from the poems in knowntopicpoems and unknown-topicpoems, we do not train our language model directly on them. however, we make use of the fact that sonnets follow a known rhyming scheme, and leverage them to train a neural model to produce rhymes, which will be explained in detail in subsection 3.2. as for the previous two datasets, poems are tokenized using nltk. wikipedia. finally, we utilize a large english wikipedia corpus foot_1 for pretraining. while this corpus does not consist of poems, we expect pretraining a language model on english text to help the overall coherence of the generated output. again, we tokenize all sentences using nltk. ## the neural poet we now describe all models that are either part of our baseline for acrostic poem generation or used for data preprocessing. an overview of our final neural poet is shown in figure 2 . ## related work automated poetry generation has long been getting attention from researchers at the intersection of artificial intelligence and computational creativity. even before the advent of deep learning, researchers used stochastic models and algorithms to generate poems @xcite @xcite . with the advancements in deep learning, more and more researchers are exploring possibilities of training neural networks to generate poems which mimic human creativity. the authors of @xcite trained a model on generating shakespearean sonnets. they used a hybrid word-character lstmbased recurrent neural network to generate poems, and used separate rhythmic and rhyming models to enforce sonnet structure on the poems generated. all three component were trained in a multi-task fashion. their crowd-work and expert evaluations suggested that the generated poems conformed to the sonnet structure, however lacked readability and coherent meaning. we make use of explicit representations of topics to address the poem coherence and readability concern: as our poems are generated based on a topic, we expect them to be more coherent. authors of @xcite generated chinese poems based on images, rather than topic words. they used a combination of a convolutional neural network (cnn) and a gated recurrent unit (gru) to generate poems which related to the target image. they also generated acrostic poems, but used character-level modelling to achieve this -which was simpler than in our case, since they worked with chinese text where characters often correspond to entire words. our preliminary experiments on english showed that character-level models learn easily to generate acrostics by themselves, however do not follow the topic as coherently as word-level models. zhang and lapata (2014 @xcite built a model to generate poems based on topics in a similar fashion to ours. however, they chose words related to a given topic to be the last words in each line -and to rhyme. for this, they built rhyming classes for an input topic first, from which rhyming pairs were chosen. the most obvious differences to our work are, however, that they produced poems following predefined stress patterns, while we are interested in free verse poems and that we generate acrostic poems, while they did not. finally, some additional work on poem generation includes @xcite , who applied reinforcement learning to the problem, in order to overcome the mismatch of training loss and evaluation metric. ## conclusion we introduce a new task in the area of computational creativity: acrostic poem generation in english. the task consists of creating poems with the following constraints: 1) the first letters of all lines should spell out a given word, 2) the poem's content should also be related to that word, and 3) the poem should conform to a rhyming scheme. we further present a baseline for the task, based on a neural language model which has been pretrained on wikipedia and fine-tuned on a combination of poems with gold standard and automatically predicted topics. a separate rhyming model is responsible for generating rhymes. we perform a manual evaluation of the generated poems and find that, while human poets still outperform automatic approaches, poems written by our neural poet obtain good ratings. our additional constraints only slightly decrease fluency and meaningfulness and, in fact, even increase the poeticness of the generated poems. furthermore, our model's poems are indeed topic-wise closely related to the acrostic word. our neural poet is available at https://nala-cub.github.io/resources as a baseline for future research on the task.