BP o MP : The Benchmark of Poetic Minimal Pairs – Limericks, Rhyme, and Narrative Coherence (2021)

Abstract: We adapt BLiMP (Benchmark of Linguistic Minimal Pairs) language model evaluation framework to the context of poetry, introducing the first of a series of tasks titled Benchmark of Poetic Minimal Pairs (BPoMP). The tasks presented herein use one genre of English-language poetry, the limerick (five-lines, rhyme scheme AABBA). Following the BLiMP schema, the BPoMP tasks use 10,000 minimal pairs of limerick/corrupted limerick. The latter is created by (1) shuffling two rhyming end-of-the-line words, (2) shuffling two rhyming lines, (3) replacing end-of-the-line word by a non-rhyming synonym. Our general task is detection of the original limerick, which we believe tests a language model’s capacity to utilize “end rhymes”, a common feature of poetry. We evaluate Transformer-based models by checking if they assign a higher probability to the non-corrupted limerick in each minimal pair. We find that the models identify the original limerick at rates better than chance, but with a nontrivial gap relative to human accuracy (average of 98.3% across tasks). The publicly available curated set of limericks accompanying this paper is an additional contribution. In general, we see this as a first step to create a community of NLP activity around the rigorous computational study of poetry.

## introduction machines -i.e., artificial intelligence -continue to make great progress toward the challenges of machine-assisted and machine-written literature. examples range from word auto-completion algorithms to deep learning-based methods that (sometimes) produce turing test-passing text @xcite and even credible sonnets @xcite . evaluating such work uses a range of techniques, including automatic metrics such as bleu as well as -rightly or wrongly -turing test-inspired human evaluation. this paper focuses on testing if large models pretrained on gigabytes of text, such as gpt-2 @xcite and transformer-xl @xcite ), are capable of "discovering" the rhyming and basic narrative information inherently present within a simple poetic form, the limerick, a fiveline poem, usually humorous, with rhyme scheme aabba and regular -albeit not fixed -metrical structure. in particular, we test transformer models on various (suitably contextualized) forms of the line completion problem for the limerick as well as a fundamental narrative-oriented task. these tests should be viewed as a first step toward a broader goal of framing the rigorous interrogation of the poetic (or more broadly, literary) capabilities of language models. given their formulaic structure (e.g., aabba rhyming scheme) and the relatively small size of each unit, limericks present a good testing ground as compared to longer or more complex poems, such as shakespearean sonnets (see e.g., @xcite ), or most broadly, "literature". to this end we gathered a small data set of 99,000 limericks (filtered down further for quality purposes), which we use to test three poetic feature: understanding the concept of a rhyme framed as two kinds of line completion test, and one test of narrative structure, framed as a line ordering test, given the full limerick. the primary contri-butions of this paper are the instantiations of these tests as "minimal pair" tasks and a curated and publicly available dataset of limericks. the former are designed in the spirit of the blimp (benchmark of linguistic minimal pairs) tests @xcite that are used to interrogate more general linguistic capabilities of language models. our limerick-based test is a first construction that will be part of a larger collection of tests of machine poetry capabilities, given the title bpomp, or benchmark of poetic minimal pairs. as for the latter contribution we hope that the availability of this curated data set inspires other computational studies of this and other poetic forms. recognizing the basic ingredients of a limerick (or any poetic form) are also part of being able to produce a limerick and thus, we also see this as a first step toward another larger goal: machine composition of high-quality limericks, or any prescribed poetic form. ## related work this paper is a part of the growing body of literature devoted to the formulation and execution of tasks for probing language models, including the "bertology" literature that focuses on bert (see e.g., @xcite @xcite . in formulation, it is directly modeled on the blimp framework @xcite , wherein the machine is a given a pair of instances, one correct, and another "minimally" modified (i.e., "corrupted" in some very minor, but systematic way) and the language model then outputs probabilities (or some derivative thereof) signifying which is the most likely. our line completion tasks produce line completion-based metrics, and thus are related to the large body of work already devoted to the sentence completion task (see e.g., @xcite ). our simple exploration of narrative structure, interrogated through the minimal pair of limericks, one an original and the other with two rhyming lines swapped, is related to, but different from work on next sentence prediction @xcite , which presents two sentences of text in original and swapped orders (but without the fuller context that our minimal pair of limericks provides) and produces probabilities for both occurrences. our work also drew partial inspiration from poetry generation literature @xcite @xcite . however, in this work we do not aim to create machine poetry, but rather show the way in which poetry can be used as a prism to evaluate models which purport to be accurate models of a wide variety of human-produced texts, i.e. texts written in a variety of genres and registers. ## problem statement in the spirit of blimp @xcite , we present the language model with a five-line limerick and its corrupted counterpart and compare the probabilities the model assigns to each of these text blocks. note that for each test we always include a full limerick and full limerick with corruption (which thus may or may not still be a limerick). the corrupted limerick differs from its source in several possible ways (in the order of increasing difficulty): (1) two rhyming words swap places, (2) two rhyming lines swap places, (3) one word is changed, always an end-of-the-line rhyming word which has been replaced by a synonym (see below for details) that removes rhyming information. each of the three defines its own task. the difficulty ordering is based on the amount of corruption introduced into the limerick-from somewhat obvious to more subtle. note however, that this doesn't necessarily correspond to empirical human rating of difficulty. we mostly focus on causal models, which allows us to infer the probability of sequence (i.e. limerick), by multiplying estimated probabilities of each token using previously seen tokens as context (in practice we sum log-probabilities) @xcite : in some cases (i.e. bert) we use a modified pseudo-likelihood formulation we adapted from @xcite , to compute "probabilities" using bidirectional context: this produces a simple experiment (and 3 tasks): given two sequences, a limerick and its corrupted original shuffled rhyming word shuffled line corrupted rhyming word television or radio stations television or radio nations television or radio stations television or radio stations may broadcast to several nations. may broadcast to several stations. may broadcast to several nations. may broadcast to several countries. one can tell them apart one can tell them apart through the call letters' art-one can tell them apart through the call letters' art-through the call letters' art-one can tell them apart through the call letters' art-which provides their identifications. which provides their identifications. which provides their identifications. which provides their identifications. ## results our general experimental framework is as follows: we ask if the models assign higher probability to the uncorrupted limerick of the pair. again, in each test the "minimal pair" is a pair of five-line texts, one the original limerick, the other a corruption, which may or may not still be a limerick. we compare our models using simple count-based accuracy: total number of correctly guessed pairs (i.e., pairs where the higher probability is assigned to the uncorrupted limerick) versus total number of (minimal) pairs. next we take a look at the task of swapping lines that rhyme. since the semantic issues here are not as apparent, we expected performance of all models to dip compared to our first task. this largely has proven to be true, as accuracy drops by an average of 11% across all models. here, as well, gpt-2 emerges to be superior in terms of performance. finally, we assess results on the most difficult task, where we preserve the semantic information but diminish rhyming information, by replacing one word with a non-rhyming synonym. the accuracy drops significantly, compared to first two tasks, and the gap between human and model performance is considerable. here, bert is the clear "winner" is that it outperforms the favorite gpt-2 by almost 5 percentage points. we theorize that the better performance comes from bert's increased bidirectional context. based on these three experiments, rhyming information seems to matter to a non-trivial extent for these models. ## conclusion in this paper we have introduced a first instance of a benchmark of poetic minimal pairs (bpomp), a framework for the testing of the poetic "knowledge" of a language model. using a curated set of limericks we interrogate the abilities of four transformer-based language models on three poetry-based tasks (end-of-the-line rhyme replacement, rhyming word swapping, and rhymed line swapping) framed as distinguishing a limerick from a minimally corrupted version. in each case the 4-grams. 10 we find a few outliers (with negative frequency difference) farther from the tail end, where our hypothesis fails. transformer-based models succeed at a rate better than chance, but much worse than the perfect and near-perfect human baseline. the limerick dataset is made publicly available. future near-term contributions to the bpomp framework include meter, assonance, and alliteration tested in more complex short poetry. while the limerick may appear to be a "simple" textual form, it is in that way a perfect kind of "model organism" for a more general study of poetry and even literature. a five-line poem of this form can still articulate a range of narrative structure and sophisticated literary devices, but is of a size that admits relatively easy modification and interrogation -e.g., through the minimal pairs framework. we hope that this paper and the concomitant data set release inspire such future work. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.