Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets (2024)

Abstract: Large language models (LLMs) can now generate and recognize poetry. But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry—poetic form—which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms—such as sonnets, sestinas, and pantoums—with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.

## introduction writing free verse is like playing tennis with the net down. ## data we curate more than 4.1k poems, mostly in the english language, which were categorized with their poetic forms by human annotators, and either published online or collected in books. ## auditing pretraining data for poems online resources like poetry foundation are valuable in large part because they make thousands of poems available on the internet for free. however, this also means that these specific poems are more likely to be present in the training data of llms, leading to memorization issues that could affect performance on our form classification task. prior work has found significant amounts of poetry memorization in large models like @xcite . we perform experiments to probe pretraining datasets for our poems, relying on both prompt-based model probes and direct searches of released pretraining data. to directly search for poem texts, we rely on the dolma open pretraining dataset @xcite . dolma is a "three-trillion-token english corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials." we query the dolma dataset using the what's in my big data (wimbd) platform @xcite , 4 which allows us to search for exact strings and returns all matches along with their associated metadata, including the data source, the original web domain, the surrounding text, and other information. we split each poem into lines, and we remove lines with fewer than four whitespace-delimited tokens to avoid unspecific matches. we truncate lines at 20 tokens for query efficiency. we release our search results publicly. ## establishing difficulty of task to establish the difficulty and significance of the poetic form classification task, we conduct a small formative study 5 with human experts, and we also test a fine-tuned llm and a supervised machine learning classifier. through these initial experiments, we confirm that this task cannot be accomplished easily by human experts, by basic fine-tuned models, or by traditional lexical approaches. ## conclusion our work audits current poetic capacities and training data in leading llms. we contribute the poetry evaluation task and release to the research community a dataset of 1.4k+ annotated public domain poems with accompanying metadata about their prevalence in popular training datasets. we also join orr and kang (2024) and others in cautioning the benchmark/task as the be-all and end-all framework for nlp research. poetry is a good example of a human output that purposely troubles neat categorization. we encourage more work that builds nuance and ambiguity into humanistic benchmarks such as this one, as well as work that places value beyond this orientation. further research is also needed to study llm poetic capacities in languages beyond english and to evaluate impacts on human creators.