B y GPT 5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models (2023)

Abstract: State-of-the-art poetry generation systems are often complex. They either consist of task-specific model pipelines, incorporate prior knowledge in the form of manually created constraints, or both. In contrast, end-to-end models would not suffer from the overhead of having to model prior knowledge and could learn the nuances of poetry from data alone, reducing the degree of human supervision required. In this work, we investigate end-to-end poetry generation conditioned on styles such as rhyme, meter, and alliteration. We identify and address lack of training data and mismatching tokenization algorithms as possible limitations of past attempts. In particular, we successfully pre-train ByGPT5, a new token-free decoder-only language model, and fine-tune it on a large custom corpus of English and German quatrains annotated with our styles. We show that ByGPT5 outperforms other models such as mT5, ByT5, GPT-2 and ChatGPT, while also being more parameter efficient and performing favorably compared to humans. In addition, we analyze its runtime performance and demonstrate that it is not prone to memorization. We make our code, models, and datasets publicly available.

## introduction end-to-end fine-tuning of pre-trained language models like gpt-2 @xcite or t5 @xcite on downstream tasks has been an immensely popular training paradigm for textgeneration in the last few years @xcite . endto-end models learn to complete a task by directly learning all steps, without intermediary algorithms such as hand-crafted rules or post-processing. this approach has proven to be highly effective on a wide range of problems such as dialog generation @xcite , summarization @xcite @xcite , and machine translation @xcite 1https ://github.com/potamides/uniformers the sweet wild strain, the sudden start, a which shakes the perfumed altar's flame, b to make its shrine a sacred name, b and sing its praise in every heart. ## background in formal verse poetry, poems have to follow strict patterns and rules of language which we term styles. our goal is to train an end-to-end poetry generation system which learns to adhere to specified styles by itself. we refer to this as style-conditioned poetry generation. in our work, we focus on generating quatrains and conditioning on the following defining styles of formal verse poetry (cf. figure 1 ): rhyme a rhyme is the repetition of the same or similar sounds in the final accented syllables of words, which must be preceded by differing consonants @xcite . if all conditions are met, we speak of perfect rhymes, and if some of them are violated, for example, because the final sounds are different or the words are identical, we speak of imperfect rhymes. in a quatrain with abab rhyme scheme, the first and third line endings rhyme, as do the second and fourth lines. meter meter refers to the rhythmic pattern within a verse. in modern poetry, this rhythm is usually accented-syllabic, that is, the succession of stressed ( ) and unstressed syllables ( ) occurs at regular intervals @xcite . the rhythmic unit is also known as a foot and the meter of a verse can thus be described as a sequence of feet. in english poetry, common feet are iambic ( ), trochaic ( ), anapestic ( ), and dactylic ( ). for conditioning on meter, we consider all metric feet appearing in our datasets (cf. appendix a). alliteration @xcite define alliteration as the repetition of the same consonant sounds or any vowel sounds at the beginning of words or syllables that are close together in a verse. in formal verse, alliteration is secondary to rhyme and meter, follows less strict constraints, and is therefore not as easily classified. in this work, we thus consider the level of alliteration instead, which we classify as either low, medium, or high (cf. §5). name size params enc/dec token-free bygpt5 small 73.5m decoder ✓ base 139.2m decoder ✓ medium 289.1m decoder ✓ gpt-2 base 124.4m decoder ✗ medium 354.8m decoder ✗ byt5 small 300m enc-dec ✓ mt5 small 300m enc-dec ✗ ## models we induce a range of end-to-end poetry generation systems for english and german by fine-tuning pretrained transformer models @xcite . for conditioning on style, we consider two architectural variants-encoder-decoder transformers @xcite and decoder-only transformers @xcite . as explained in §1, we focus on token-free models, but also consider subword-level models for comparison. we do not experiment with models with more than 400 million parameters since they exceed the capacity of our available gpu resources. encoder-decoder for encoder-decoder models, we initialize the encoder with a joint triple of rhyme scheme, meter, and, alliteration level and generate a quatrain with the decoder. we represent each style by a special token which we add to the model vocabulary. we use byt5 @xcite , a token-free pre-trained encoder-decoder model, as a baseline. for comparison with subword-level approaches, we fine-tune mt5 @xcite . decoder-only as the input for encoder-decoder models is a relatively short sequence of styles, this could lead to an underutilization of the encoder. we thus hypothesize that a decoder-only model, with styles supplied as a prompt string, would be better suited for our task. on the subword-level, multiple models, such as gpt-2 @xcite , are readily available. however, to our best knowledge, no such model exists at the characterlevel yet, which is why we train our own. since our new model shares some similarities with the gpt family of models, but has its origin in byt5 (see §4), we refer to it as bygpt5. an overview of all models we use can be seen in training steps 2.00 2.25 2.50 2.75 3.00 3.25 3.50 perplexity english b gpt5 (small) b gpt5 (base) b gpt5 (medium) 12500 25000 37500 training steps german b gpt5 (small) b gpt5 (base) b gpt5 (medium) figure 2: perplexity on the training data when pretraining bygpt5 for english and german. ## bygpt5 for pre-training our own token-free decoder-only model bygpt5, we start by modifying the architecture of byt5 and discard its encoder component. we then initialize the weights with the decoder of byt53 to warm-start the training process @xcite . we repeat this for the three smallest variants of byt5. because byt5 has an asymmetrical architecture, the resulting models retain only 25% of its parameters. we refer to their model sizes as small, base, and medium. as training data, we use openwebtext2 @xcite for english and cc100 @xcite for german. for hyper-parameters, we follow @xcite and @xcite and use adam with a weight decay of 0.1, a batch size of 512, varying learning rates depending on model size, and train on a causal language modeling objective for 50k steps following @xcite . we provide loss curves in figure 2 . as might be expected, the perplexity of larger models is generally lower than that of smaller counterparts. ## datasets we collect a range of labeled and unlabeled datasets of english and german poetry (cf. verses may occur in multiple pseudo-quatrains. for automatically labeling rhyme and meter, we leverage the available labeled data and train a range of classifiers. we evaluate them on held-out gold data and subsequently use the best performing classifier for each style (cf. appendix a). meter classification is a multiclass classification problem with a single verse as input, while rhyme classification is a binary classification problem with two verses separated by a special token as input. we classify the meter of a quatrain by choosing the dominant meter among the verses4, and the rhyme scheme by determining which verses rhyme and which do not. as no readily available poetry datasets include labels for alliteration, we approach the problem in a different way. the quantification of the level of alliteration in a document is a long known research problem @xcite @xcite . let 𝑣 𝑖 be the atomic units of sound in verse 𝑣, blain (1987) quantify alliteration as ## experiments for fine-tuning, we use the same hyperparameters as in §4 for all models, but reduce the batch size to 128 (for efficiency reasons). we induce separate models for each language in quatrain and train for 10 epochs.5 we conduct both automatic ( §6.1) and human evaluation ( §6.2). examples of generated quatrains can be found in appendix d. ## analysis we continue with a deeper analysis and look into low-resource training ( §7.1), quantify memorization ( §7.2), evaluate the performance of token-free models on non-character-based, high-level tasks ( §7.3), introspect the models' understanding of style when predicting tokens ( §7.4), and compare bygpt5 with chatgpt ( §7.5). ## related work as indicated in §1, competing poetry generation systems usually consist of model pipelines and/or inject prior knowledge. @xcite , for example, propose a system for modeling quatrains consisting of three components: one model encodes previous verses, a second one reduces them to a single context vector, and a third one generates verses, one at a time. during decoding, phrases which violate style constraints are discarded. similarly, deep-speare @xcite consists of a language model that generates a set of sample verses in reverse order, a model that reinitiates sampling as long as rhyme constraints are not met, and a final model that ranks the samples according to how well they adhere to iambic pentameter. van de cruys (2020) also induce prior knowledge into a generic language model, but they do it by modifying output probability distributions directly. @xcite put their focus on actually learning rhyme and train a sonnet and limerick generator through adversarial training. the model is hierarchical, i.e., it first generates a sequence of line endings which are subsequently completed in reverse. while the model manages to learn the meaning of rhyme to an extent, the authors still filter outputs using pronunciation dictionaries. more in line with our research, @xcite train a model on the phonetic representation of poetry using the international phonetic alphabet (ipa) as a character-level vocabulary. during inference, a second model translates sounds back to human-readable text. although promising, the model did not generalize well, and 6after experimenting with prompt engineering and incontext learning, we settled on a straightforward template: "generate a quatrain @xmath0." an additional model enforces rhythmic constraints in their final approach. @xcite target unsupervised poetry generation by training a system on prosaic text only and conditioning it on structural information such as line endings and syllable counts. during inference, the system can generate poetry when conditioned on rhyming line endings and metric syllable counts. nonetheless, since this information must be crafted manually, it is still supervised in a slightly different sense. in contrast, our model is supervised during training as it requires poetry to learn from, but unsupervised during inference as it is able to independently incorporate poetic elements. ## conclusion in this work, we implement end-to-end styleconditioned poetry generation systems for quatrains in english and german. unlike other work, our systems are able to generate poetry without the need for human supervision, except for the use of poetic training data. in particular, we present bygpt5, a novel token-free decoder-only language model, and show that fine-tuning it on a custom poetry corpus outperforms other models, such as gpt-2, mt5, and byt5, on average, while also performing favorably against human poets in our constrained setting. our key findings are that (i) tokenization algorithms matter, i.e., token-free language models generally perform better at generating character-level styles than subword-level transformers, and (ii) large datasets are crucial for successful training. we further show that bigger models do not necessarily perform better and that decoderonly architectures work best, i.e., we can discard the encoder of byt5 (75% of parameters) while still improving downstream performance. we also demonstrate that token-free transformers perform competitively on tasks not tied to character-level styles, and are less susceptible to memorization of the training dataset. in addition, we conduct a visual analysis of token-level attributions during quatrain generation that is consistent with human perception of styles. in future work, we want to to extend our system to other poetic forms such as sonnets, limericks, or villanelles.