Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization (2023)

Abstract: While research in natural language processing has progressed significantly in creative language generation, the question of whether language models can interpret the intended meaning of creative language largely remains unanswered. Poetry as a creative art form has existed for generations, and summarization of such content requires deciphering the figurative patterns to find out the actual intent and message of the poet. This task can provide the researchers an opportunity to evaluate the creative language interpretation capacity of the language models. Unlike typical text, summarization of poems is a challenging task as poems carry a deeper meaning, which can be easily lost if only the literal meaning is considered. That being said, we propose a new task in the field of natural language understanding called ‘Poem Summarization’. As a starting, we propose the first-ever dataset for this task, named ‘PoemSum’, consisting of 3011 samples of poetry and its corresponding summarized interpretation in the English language. We have benchmarked the performance of different state-of-the-art summarization models and provided observations on their limitations. The dataset and all relevant code used in this work have been made publicly available.

## introduction poetry has been used for centuries as a powerful means of expressing the feelings, thoughts, ideas, and emotions of human beings @xcite . it holds a profound significance in human culture and expression, transcending time and boundaries. poem summarization refers to the process of reducing a poem to its core ideas to provide a concise and informative overview of the content @xcite . this not only gives us an idea about the main theme but also describes the style, rhyme, techniques, and metaphors used in the poem. automatic summarization of such creative language can make poetry more accessible to a wider range of audiences and applications @xcite . in recent years, there has been notable research conducted on text summarization in the field of natural language processing (nlp) @xcite @xcite . however, to the best of our knowledge, no such work has been done in the domain of poem summarization yet. while the summarization process of poems seems quite similar to the generic text summarization, there are some major differences between the two. text summarization, in general, is a technique that reduces the number of sentences and words in a document by extracting its important information into a concise paragraph without changing its meaning @xcite . most of the summarization models being proposed in recent years have been trained on news-related datasets @xcite @xcite , which mostly contain facts, so they do not have to look for deeper meaning while generating a summary. the same goes for datasets that are built using financial @xcite or medical data @xcite . this process only considers the literal meaning of words, and it does not focus on the deeper meanings. applying this summarization technique directly to poems may not be helpful in capturing the essence. hence, it is necessary to study if baseline nlp models @xcite are sufficient for poem summarization tasks. summarizing literary work poses lots of challenges. for instance, the artistic style of each writer is different, and the literature can be interpreted in many ways. moreover, to understand the figurative language of poems, the model has to understand the interconnection between sentences and the deeper meaning conveyed by the poem as a whole. in order to facilitate research in this domain, we have introduced a new task in the field of natural lan-guage understanding named 'poem summarization' and proposed a novel poem summarization dataset. the main contributions of our paper are as follows: ## related works the text summarization task can be broadly divided into two categories: extractive @xcite and abstractive @xcite methods. the extractive summarization approach highlights the key points in the source text, which are then taken out and compiled into a succinct summary. on the other hand, the abstractive approach uses a different collection of words compared to the original text to construct a paraphrase of the main ideas of a given text. for poem summarization, abstractive summary generation is required as only direct extraction of words from poems would not be enough to convey the actual meaning of the poem. in the abstractive summarization domain, @xcite introduced a dataset called 'x-sum' by gathering 226,711 online articles from the british broadcasting corporation (bbc). in the same year, @xcite proposed a summarization dataset named 'newsroom' containing 1.3 million articles and summaries authored by writers and editors in the newsrooms of 38 major news publications that combine both abstractive and extractive summarization methods. the 'xl-sum' dataset proposed by @xcite covers 44 languages, and it is the largest abstractive summarization dataset in terms of the number of languages represented and the number of samples gathered from a single source. although summarization has made great strides in recent years @xcite , most works have used news articles as data source @xcite , where the focus is mostly on extracting key information, forming a shorter passage. poems, on the one hand, do not benefit from such types of summarization, yet there are no studies aimed at exploring the prospects of poem summarization. apart from poetry, there has been a recent insurgence of datasets in other types of creative language @xcite with the objective of natural language interpretation (nli). for instance, @xcite ied the ability of language models to complete narratives containing idioms and similes. in other related research @xcite , metaphors are generated based on conceptual metaphor theory @xcite . while creative language generation has seen the limelight in the last few years @xcite @xcite , creative language summarization has been mostly left behind. recent endeavors in the field include 'creativesumm' @xcite : a shared task on summarization of creative documents like movie scripts, television scripts, etc. however, poems were not used in any of the sub-tasks. other types of tasks that work with poems include poem classification @xcite , image-to-poem generation @xcite , and developing better large language models for poem generation through human collaboration @xcite . however, work focusing on the summarization of poems is missing in prior literature. to this end, we introduce a novel poem summarization dataset. ## models we evaluated the performance of some of the standard summarization models like t5, bart, pegasus, and mt5 on poemsum, as they have demonstrated impressive performance in various summarization datasets in recent years @xcite . t5: a versatile transformer-based model @xcite that uses a unified text-totext transfer learning framework that can be used for diverse tasks like translation, summarization, question answering, etc. @xcite . pegasus: a transformer-based model specifically designed for text generation tasks like summarization, employing pre-training with extractive supervision @xcite . bart: a sequence-to-sequence model based on the transformer architecture pre-trained on document denoising @xcite . the pre-trained models were fine-tuned with a learning rate of 0.0001, and adamw optimizer @xcite with a batch size of 10. the dataset was splitted into the train, validation, and test sets following the ratio of 80:10:10. ## results and discussions to evaluate the model-generated summaries against the gold reference summaries, we have used the rouge-1 (r1), rouge-2 (r2), rouge-l (rl), and bertscore (bs) metrics. the results are shown in six miles from earth, loosed from its dream of life, i woke to black flak and the nightmare fighters. when i died they washed me out of the turret with a hose. the death of the ball turret gunner' by randall jarrell describes the death of a ballturret gunner who was shot in the head. the poem begins with the speaker stating that he fell into the state of alabama and hunched in its belly until his wet fur froze. it was six miles away from earth, freed from its dream of life. when he died, it was washed him out of the turret with a hose . rouge @xcite : it measures the overlap between actual and generated summary considering unigrams, bigrams, and longest common subsequence in the case of rouge1, rouge2, and rougel, respectively. bertscore @xcite : instead of considering exact matches, bertscore analyzes token similarity using contextual embeddings @xcite . the results show that the t5 model is the bestperforming one, both in terms of the three types of rouge scores and bertscore, while the mt5 performed the worst. the key takeaway here is that the mt5 model that we use in this paper foot_4 was trained on a news summarization dataset. it is performing the worst as per our initial hypothesis. news data is mostly void of creative elements, which leads to an inability to express and summarize creative language even after fine-tuning. appendix a.2 contains further explanation on this issue. furthermore, we have critically analyzed the results and found a few pitfalls of the evaluated models. a sample summary generated by the t5 model is mentioned in apart from lines being carried over from the poems to the summary without plausible explanation, there are a number of other figurative language patterns like metaphors, symbolism, personification, etc., that language models are unable to comprehend @xcite . appendix a.2 contains a detailed insight into some of these lackings in the generated summaries. ## conclusion in this work, we have introduced a novel dataset for poem summarization and assessed the creative language understanding capability of the popular summarization models. we have also discussed how creative language differs significantly from formal language in its structure and expression. hence, we require different approaches when dealing with creative language, at least in the domain of summarization. after conducting several baseline experiments with the state-of-the-art summarization models, we observed that even the best-performing model was unable to produce summaries that capture the main idea of the poems in certain cases. we also used a model trained on a popular news article summarization dataset and observed that it performed worse compared to all other models we have finetuned. our findings have led us to the conclusion that existing text summarization approaches and summarization models are not well-suited for poem summarization. we believe that the proposed po-emsum dataset can serve as a valuable resource for evaluating the performance of nlp models in understanding, interpreting, and generating figurative language within the context of poetry. meanwhile, the evaluation of large language models (llms) @xcite @xcite @xcite in this task could be a good direction for future work.