Generation of R ussian Poetry of Different Genres and Styles Using Neural Networks with Character-Level Tokenization (2025)

Abstract: Automatic poetry generation is an immensely complex task, even for the most advanced Large Language Models (LLMs) that requires a profound understanding of intelligence, world and linguistic knowledge, and a touch of creativity.This paper investigates the use of LLMs in generating Russian syllabo-tonic poetry of various genres and styles. The study explores a character-level tokenization architectures and demonstrates how a language model can be pretrained and finetuned to generate poetry requiring knowledge of a language’s phonetics. Additionally, the paper assesses the quality of the generated poetry and the effectiveness of the approach in producing different genres and styles. The study’s main contribution is the introduction of two end-to-end architectures for syllabo-tonic Russian poetry: pretrained models, a comparative analysis of the approaches, and poetry evaluation metrics.

## introduction automatic poetry generation is a challenging task that requires systems capable of handling multiple levels of language understanding, including deep comprehension of text, linguistic and world knowledge, common sense, creativity, and an awareness of syllabic and rhythmic structures. as a form of artistic expression, poetry has been produced in numerous languages, each with its own unique poetic traditions and forms. while most poetry generation systems focus on english and chinese, there are also efforts targeting other languages @xcite @xcite . however, the task of automatically generating poetry in russian remains underexplored and presents unique challenges. to address this gap, we explore neural network architectures for the automatic generation of syllabo-tonic * foot_0 russian poetry. specifically, we investigate whether transformer-based models can effectively handle end-to-end generation of russian syllabo-tonic poetry across various genres, styles, and forms. our analysis reveals that mainstream byte pair encoding (bpe) tokenization often fails to align well with the structural units of russian syllabo-tonic versification. to address this, we propose and evaluate language models with characterand syllable-level tokenization, training and testing their performance on the poetry generation task. we also conduct a detailed study of poetry metrics (subsection 5.1) and share our experiences using existing methods to assess the quality of generated poems. these methods include automatic evaluation ( the contributions of our work are as follows: ## related work creativity has been shown to be closely linked to human intelligence @xcite , making computational creativity a compelling area of research @xcite , including the study of creativity in llms @xcite . generative poetry, related to artistic creativity @xcite , differs from other natural language generation domains @xcite by its special lexical and phonological constraints, as well as specialized metrics to evaluating the quality of generated poems (see @xcite as an example). recent advancements in llms have significantly improved the quality of poetry generation, to the extent that humans often cannot reliably distinguish between poems authored by humans and those generated by llms @xcite . tokenization approaches. despite the progress of current generative models, there remains potential for further improvement in the quality of poetry generation. one area of research is alternative tokenization methods for lms that circumvent the shortcomings of the currently mainstream bpe tokenization. in the case of syllabic or syllabo-tonic poetry, improvements can be achieved by using character-or syllable-level tokenization @xcite @xcite . character-and byte-level tokenization has been used in various systems for automatic poetry generation based on recurrent neural networks @xcite @xcite @xcite . after the invention of the transformer architecture, its applicability with character-based text representation for poetry generation was also investigated @xcite . the need to train the transformer language model from scratch limits the availability of such experiments. in the case of english language, there are open-source foundation models pretrained on vast corpora: ca-nine @xcite and byt5 @xcite . canine is a family of encoder transformer models with tokens corresponding to unicode codepoints. this model was utilized by @xcite in melody-to-lyrics generation system. byt5 implements an encoder-decoder architecture with byte-level tokenization. an example of its use for generating czech poetry is available in @xcite . syllable-level tokenization is a specialized variant of subword unit tokenization. its effectiveness for generating poetry has been studied for several languages: italian @xcite , czech @xcite , vietnamese @xcite . similar to characterlevel tokenization, syllable-level tokenization necessitates either resource-intensive pretraining of a language model from scratch or additional finetuning of a pretrained model with byte-pair encoding tokenization. ## data to train and evaluate poetry generation models, we required a substantial amount of russian poetry data. however, publicly available datasets, such as @xcite ; @xcite , are limited in size and insufficient for training generative models, particularly those based on transformer architectures. to address this limitation, we collected a large volume of amateur poetry from various internet sources (appendix e). these sources often lack editorial oversight, leading to frequent spelling and punctuation errors that can negatively impact model performance. to mitigate this issue, we developed a rule-based spelling correction algorithm to address the most common errors. further details about this algorithm are provided in appendix b. the collected poems also frequently exhibit defects in adhering to poetic meter * and rhyme * . since these defects cannot be automatically corrected, we excluded such samples from the finetuning dataset. to identify meter-and rhyme-related defects, we used our custom library, described in 3.3. ## pretrained models the goal of this paper was to investigate whether using a character-level tokenizer and pretraining with it could improve automatic poetry generation. we hypothesized that character-level tokenization would represent text more accurately for poetry generation compared to byte-pair encoding. to test this hypothesis, we used two model architectures, which are described in detail below. ## results the results of the experiments are shown in tables 3 for 1000 lyrics quatrains and the metrics indicate that poorly written poems can have lower perplexity, while human-authored poems have higher perplexity. as noted by @xcite , it is essential to focus not only on the absolute value of perplexity but also on how well the obtained perplexity value fits within the range of values typical for works written by people. it can be helpful to approximate the corresponding distribution with a gaussian distribution with a specific mean and variance. the automatic evaluation results show that the fine-tuned rugpt3-large and byt5 models performed poorly in poetry generation, while mistral-7b-v0.1 achieved better scores. however, mistral-7b-v0.1's generated poems had higher perplexity and included many out-of-vocabulary words, likely due to its limited pretraining on russian texts. despite this, mistral outperformed other models using bpe and byte-level tokenization, coming close to specialized character-level models. the fred-t5-1.7b-based model performed slightly worse in terms of technical quality and rhyme but produced texts with fewer language errors, as shown by its lower perplexity and fewer out-of-vocabulary words. experiments with stgpt demonstrated that transformer models using syllable-level tokenization achieved the highest technicality scores among all models. for sonnets, stgpt even surpassed humanwritten poems in terms of technicality. however, this tokenization method has several limitations, as discussed in section d. additionally, while these models excel in technicality, they often produce texts with grammatical and fluency issues. these flaws do not affect technicality or rhyming metrics but reduce the overall quality of the poetry. due to these limitations, we chose not to scale these models to a capacity comparable to charllama-1.3b, and no full-scale side-by-side evaluations were performed. human side-by-side evaluation results. we used expert side-by-side evaluations to assess poem quality, applying the bradley-terry model @xcite from the choix library foot_16 . this model was used to compare poems written by humans with