Poetry in RAG s: M odern G reek interwar poetry generation using RAG and contrastive training (2025)

Abstract: In this paper, we discuss Modern Greek poetry generation in the style of lesser known Greek poets of the interwar period. The paper proposes the use of Retrieval-Augmented Generation (RAG) to automatically generate poetry using Large Language Models (LLMs). A corpus of Greek interwar poetry is used and prompts exemplifying the poet’s style with respect to a theme are created. These are then fed to an LLM. The results are compared to pure LLM generation and expert evaluators score poems across a number of parameters. Objective metrics such as Vocabulary Density, Average words per Sentence and Readability Index are also used to assess the performance of the models. RAG-assisted models show potential in enhancing poetry generation across a number of parameters. Base LLM models appear quite consistent across a number of categories, while the RAG model that is furthermore contrastive shows the worst performance of the three.

## introduction the advent of large language models (llms) has greatly increased the capabilities of nlp systems to deal with generation issues. poetry generation has been one of them, with llms having the ability to generate poetry that is sometimes indistinguishable from human-made poetry by non-experts @xcite . to some extent, this is to be expected. developing an aesthetical taste for poetry requires expertise, and similarly to other art forms, like music, non-experts can find it hard to distinguish ai vs. human-made poetry. however, despite their achievements and quick pace of improvement, llms do not perform as well in languages and/or styles that are not well represented in terms of freely (and even non-freely) available data. highly stylized poetry in a lower resourced language, like interwar poetry in modern greek, can provide a powerful case study. such cases require a targeted use of limited resources to enhance the performance of llms. one method is retrieval-augmented generation (rag), while another is based on contrastive learning. rag has been shown to provide very positive results in enhancing llm performance across a number of nlp tasks like information extraction @xcite , machine translation @xcite , question answering @xcite and dialogue systems @xcite , among many other tasks.see @xcite for a full survey on rag methods in nlp. the idea in contrastive learning is to provide both positive (poems in the target style) and negative examples (similar content but different style), in order to help the model better understand and maintain the distinctive stylistic features of a particular poet or poetic school. recent work in style representation learning @xcite has shown that contrastive methods are able to disentangle content from style; the generation of a highly specific poetic style, such as interwar greek poetry with its slight authorial variations, will provide a litmus test. in this paper, we focus on modern greek poetry of the interwar years, and implement a system to compare the results between rag and contrastive learning in generating poems of the distinctive style. we use a dual retrieval system that is able to not only find poems with similar themes by the target poet but also retrieve examples from other poets that are contrastive. the results show that rag-assisted models show potential in improving poetry generation across a number of parameters. the base llm models are quite consistent across a number of categories, while the contrastive rag model shows the worst performance of the three. 1 ## related work the issue of poetry generation is not new to nlp. it has a history that includes a variety of approaches to generate poetry: hand-crafted symbolic rules @xcite , using statistical rules based on statistical machine translation @xcite , vanilla neural network approaches @xcite , and transformer architectures. llm architectures have shown impressive performance in a variety of tasks, poetry generation notwithstanding. attempts to use these architectures for poetry generation include approaches that finetune gpt-2 for poetry generation @xcite , zero-shot approaches @xcite , fine-tuning of more advanced models like bygpt5 @xcite . the main take away in all these approaches is that fine-tuning helps the models in the task of poetry generation and the absence of fine-tuning is detrimental to the models' performance on more specific tasks, e.g. generating poetry in a specific style @xcite . @xcite introduce a multi-agent framework for poetry generation, using llms. the research suggests incorporating non-cooperative dynamics in ai systems for enhancing creative diversity in a way similar to how human artists often deliberately differentiate their work from others. however, @xcite report that current ai poetry still lacks in diversity, rhyming and semantic complexity, noting however, that style conditioning and character level modelling can help remedy these deficiencies to some extent. ## the dataset this paper uses an open-access dataset created by the second author with the help of a group of undergraduate students at the philology department, university of crete. the slightly modified and richer corpus used here comprises over 600 poems in txt. format by a group of interwar greek poets, namely tellos agras, fotos giofyllis, romos filyras, kostas 1 github of the paper material can be found here: https: //github.com/stergioscha/rag-poetry karyotakis, napoleon lapathiotis, kostas ouranis, mitsos papanikolaou, and maria polydouri. with the notable exception of kostas karyotakis, the most prominent figure of this group who is recognized as a major greek poet, the interwar poets are often referred to collectively, with an emphasis on their shared features. melancholy, pessimism, and existential anxiety, stemming among other sources from the frustration of national expansionist aspirations and the dire sociopolitical reality of greek interwar, as well as an added emphasis on nostalgia and a sotto voce quality, all of which are ascribed to neoromanticism and/or neo/post-symbolism @xcite , are the most frequently repeated features of these lyrical poets @xcite . ## the models the first model we use is based on retrieval-augmented generation. the main idea is to use external resources to augment the performance of llm models. in our case, the system takes a theme and the name of the poet as input, and then tries to search through a collection of poems (in our case, using our dataset of interwar poetry) in order to use them as examples to prompt llms. search is performed using a multilingual model (paraphrase-multilingual-minilm-l12-v2). each poem is converted into vector embeddings that are then stored in a faiss vector store. faiss is an effective library for effective similarity search and clustering. when a query is received, it is converted into the same vector space as the input poems. similarity is computed using cosine distance, with the the model trying to match poems that are thematically similar to the query. the poems are then filtered according to the poet, trying to ensure that the retrieved examples match both the theme and the poet's style. after this filtering, the retrieved poems are used to construct the prompt for the generation model. a prompt example can be seen at the appendix. the pipeline is shown in 1: the second model we use combines this basic rag system with a contrastive approach. while maintaining the same embedding and similarity search infrastructure, the system now retrieves two distinct sets of examples: poems by the target poet that match the theme, and poems about the same theme written by different poets. this dual retrieval process uses the same multilingual embeddings and faiss vector store, but applies different filtering criteria to create contrasting sets. the system first performs a broader similarity search to find thematically relevant poems, then splits these into positive examples (by the target poet) and contrastive examples (by other poets). these two sets are then incorporated into an enhanced prompt structure that explicitly guides the llm to follow the stylistic patterns of the target poet while avoiding the stylistic features present in the contrasting examples. in total we had 8 poet/theme pairs using gpt4turbo with two poems each for base, rag-assisted, and rag-assisted contrastive generation (total of 48 poems) and 7 poet/theme pairs for gpt4o (total of 42). ## results and discussion two expert evaluators were used for the gpt4o generated poems and three expert evaluators for gpt4turbo. the evaluators were only shown the resulting poems without the corresponding prompts, and were asked to assess the closeness of the generated poem to the style and versification of the target poet, as well as evaluate the poem's relevance to its proclaimed theme and the level of creativity shown. the results of inter-annotator agreement show moderate agreement when using spearman correlation (approx. 0.4). the agreement becomes moderate to strong when taking into account the relativity of judgments using nornalized z-scores (0.6). the results are shown in: the table below shows the results of evaluation on several poems pairing themes and poets across a number of parameters as this was done by experts on modern greek poetry and tested on gpt-4-turbo and gpt-4o: as we can see in figures 5 and 3 the rag-model scores the highest for style and theme when using gpt4o and ties with base llm in terms of theme in the gpt4-turbo case. overall, the rag system is marginally better w.r.t style and theme but the base model fares better w.r.t versification and creativity compared to the base models. the rag plus contrastive model has the worst overall scores. this does not mean that the contrastive approach is not useful, but, probably, that the contrastive examples given to the system were not effective, because they were not distinctive enough, given that they were by poets of the same poetic school. the theme superiority of rag is to be expected given that the retrieved poems are retrieved according to thematic fit. versification is lacking in all approaches, however the base llm outerperforms the enhanced approaches across the versification category. besides the expert evaluators, we also reverted to some metrics to assess the performance of the models vs. the original corpus, such as vocabulary density (vd), average words per sentence (awps) that would be more readily recognized as distinctive of style, hence gaining more credibility with the evaluators. ## conclusions and future work the paper has produced mixed results, showing some potential for the use of rag in poetry generation, particularly as regards a recognizable style. rag also seems slightly better at developing a theme consistently throughout a poem, as well as maintaining a style closer to the target poet, while it also has an edge in creativity in one of the two models. still, base llms are quite consistent across a number of categories. the consistency in versification and the fact that they score higher in this dimension might have to do with their ability to maintain an internal vers. style theme creativity 0 2 4 6 1.69 2.58 4.19 3.01 1.76 2.46 4.13 3.07 1.58 2.44 3.87 2.71 score (7) rag base llm rag+con rhythm and also being more successful at rhyming than the assisted models. this does not necessarily have to do with an understanding of the poetic style asked to generate. most probably, this is the result of being trained on simple poems and/or song lyrics that have a sense of rhythm and rhyming. this is an interesting avenue to explore, using rag models that are also improvements over this dimension. this might need a more nuanced approach where the retrieved poems are retrieved across a number of dimensions and not only thematic fit. for example, poems in this style rely heavily on rhyme and, as such, improving a model on this dimension needs a rag system that is not only sensitive to meaning similarity but also to rhyme-sensitive meaning similarity. this is definitely one avenue that needs to be further explored. as far as contrastive training is concerned, future work might include working first with starkly contrastive poetic styles (eg. modernist, or surrealist) and then move on to train the model to the more nuanced differences within a poetic school.