Evaluating Diversity in Automatic Poetry Generation (2024)

Abstract: Natural Language Generation (NLG), and more generally generative AI, are among the currently most impactful research fields. Creative NLG, such as automatic poetry generation, is a fascinating niche in this area. While most previous research has focused on forms of the Turing test when evaluating automatic poetry generation — can humans distinguish between automatic and human generated poetry — we evaluate the diversity of automatically generated poetry (with a focus on quatrains), by comparing distributions of generated poetry to distributions of human poetry along structural, lexical, semantic and stylistic dimensions, assessing different model types (word vs. character-level, general purpose LLMs vs. poetry-specific models), including the very recent LLaMA3-8B, and types of fine-tuning (conditioned vs. unconditioned). We find that current automatic poetry systems are considerably underdiverse along multiple dimensions — they often do not rhyme sufficiently, are semantically too uniform and even do not match the length distribution of human poetry. Our experiments reveal, however, that style-conditioning and character-level modeling clearly increases diversity across virtually all dimensions we explore. Our identified limitations may serve as the basis for more genuinely diverse future poetry generation models.

## introduction a key aspect of creative language generation is the ability to create new, original and interesting text, cf. @xcite @xcite . to date, extremely little attention has been given to the evaluation of originality and creativity in recent creative text generation models such as those for automatic poetry generation, despite renewed interest in the context of recent llms @xcite . in fact, existing automatic poetry generation models are typically not evaluated regarding how different generated poems are from existing poems in the training set but with the turing test: can humans distinguish whether a poem is human authored or automatically generated @xcite @xcite ? however, this form of turing test and other similar forms of human evaluation may contain an overlooked risk of failure: namely, if the automatically generated instances are (near-)copies of training data instances. in this work, we fill this gap and evaluate, for the first time, (fine-tuned) automatic poetry generation systems in terms of their diversity. as human evaluation is generally not well suited to assess diversity @xcite , we automatically measure diversity by comparing distributions of generated and existing poems along formal, semantic and stylistic dimensions. this yields much better evidence of the models' creative capabilities in contrast to being mere 'stochastic parrots'. our main contributions are: (i) we conceptualize diversity of poetry generation systems along different dimensions: diversity on the structural (e.g., length), stylistic (e.g., rhyming), lexical and semantic level; (ii) we assess different types of automatic poetry generation systems for diversity: general purpose word-and character-level llms, both unconditioned and style-conditioned ones, on the one hand, and poetry-specific models, on the other hand; (iii) we evaluate each class of model for diversity across the different dimensions, by comparing the distribution of the human authored training data set to the distribution of generated poems. we find that on a distributional level, generated poems are considerably different from human ones. character-level style-conditioned general-purpose llms are most diverse. our work prepares the groundwork for truly creative generative ai models @xcite ) and also has implications for the detection of generative ai @xcite . ## related work our work connects to research on diversity and automatic poetry generation, which we now discuss. diversity building systems able to generate diverse output has been a long-standing concern in nlg research @xcite @xcite ) and remains a central issue in neural nlg @xcite . the need for careful analysis of nlg systems' diversity -beyond an assessment of the quality or fluency of single-best generation outputs -has been widely acknowledged @xcite @xcite @xcite . a wellknown finding from this line of research is that neural nlg systems typically face a quality-diversity trade-off @xcite @xcite : their outputs are either wellformed and fluent or diverse and variable. work on evaluating diversity of nlg typically uses automatic metrics that quantify to what extent different outputs by the same system vary @xcite . in practice, though, evaluations of diversity in nlg differ widely across tasks @xcite and even adopt different notions of diversity @xcite . at the same time, most of these notions focus on lexical or semantic aspects of diversity, e.g., local lexical diversity. for instance, @xcite compare decoding methods in dialogue generation and image captioning, assessing lexical overlaps in n-best nlg outputs for the same input. @xcite simply measure the local lexical diversity in automatic generated poems in terms of distinct unigrams. global lexical diversity, on the other hand, measures whether the nlg system generates different outputs for different inputs. for instance, @xcite define the global diversity of image captioning systems as their ability to generate different captions for a set of inputs, using metrics like the number of types in the output vocabulary, type-token ratio, and the percentage of novel descriptions. @xcite view diversity as related to the model's ability to generalize beyond the training set, i.e., generate novel sentences. besides lexical diversity, work on open-ended or creative text generation tasks has been interested in diversity at a more general semantic level. for instance, @xcite and @xcite aim at building dialogue systems that generate entertaining and semantically diverse responses in chit-chat dialog. here, semantic diversity has been measured, e.g., with the help of embeddingbased similarity @xcite . in our work on diversity in poetry generation, we complement both lexical and semantic aspects of diversity with aspects of formal diversity. we thus explore whether automatic poetry generation systems are able to capture the 'full bandwidth' of realizations of poetry found in the data distribution with which they have been trained, focusing mostly on global diversity. ## diversity in poetry generation we first conceptualize diversity in poetry generation using formal and semantic criteria. memorization. in poetry, as in other forms of art, creativity @xcite plays a central role. a basic aspect of creativity is the models' ability to generate poems that are different from the training data, i.e. have not been memorized as a whole. to examine memorization, we proceed as in @xcite . we apply the ratcliff-obershelp similarity @xcite to compare each poem in a sample with poems in the training corpus. if a generated quatrain exhibits a similarity score @xmath0.7 with a quatrain in the training data, we classify it as memorized. a quatrain can be divided into 4 verses or 2 couplets; thus, we also inspect memorization at the verse and couplet levels by comparing each verse or couplet in a sample to those in the training data. higher thresholds for classification are used for these finer-grained comparison levels, as shorter texts have higher chances of being more similar in general. specifically, a verse with a similarity @xmath1.9 or a @xmath2.8 is considered as memorized. we define the memorization score of a sample as the proportion of memorized quatrains in that sample. how much llms memorize from their training data has been a question of central concern recently @xcite . poem length. within a sample of generated poems, we consider differences at the level of poem length, i.e., their number of tokens, as a basic aspect of diversity at the formal or structural level. we analyze to what extent the length distribution of generated poems differs from the distribution in the training data. we define the length of a quatrain as the number of tokens contained: we eliminate all punctuation symbols and split the remaining text by white space. we report mean length, standard deviation, minimal and maximal length of samples. we additionally deploy distance measures between training data distribution and generated samples, in particular, a metric called histogram intersection @xcite , which measures the intersection area of two normalized histograms (and therefore returns values between 0 and 1). rhyme patterns. as a more complex dimension of formal diversity, we consider rhyming as a central aspect that characterizes the structure of a poem. diversity can then be assessed by comparing rhyme distributions between generated samples and training data. in order to classify rhymes in our samples, we use the same classifier used to annotate quatrain @xcite . we distinguish between true rhymes, which involve different words, and repetitions, which refer to rhymes based on the same word. ## evaluation we first report the results of diversity evaluation in §5.1, which is our main focus, followed by an examination of the relationship between diversity and overall quality through human evaluation in §5.2. ## conclusion our work is the first and most comprehensive automatic evaluation of poetry diversity, yielding several interesting observations: for example, we find that style-conditioning enhances virtually all measures of diversity and that character-level modeling also increases diversity, including reducing memorization. our evaluations also shed light on the fact that none of the state-of-the-art poetry generators is able to match the level of diversity in human poems. thus, we find overall that an automatic assessment of the diversity of generated poems covers an important blind spot of existing studies. future work should aim for more diverse automatic poetry generation systems as a prerequisite of general computational creativity.