Generating Classical A rabic Poetry using Pre-trained Models (2022)

Abstract: Poetry generation tends to be a complicated task given meter and rhyme constraints. Previous work resorted to exhaustive methods in-order to employ poetic elements. In this paper we leave pre-trained models, GPT-J and BERTShared to recognize patterns of meters and rhyme to generate classical Arabic poetry and present our findings and results on how well both models could pick up on these classical Arabic poetic elements.

## introduction arabic poetry dates back to the sixth century, making it the earliest form of arabic literature. it's often divided into two categories; classical and modern poetry. the classical arabic poetry refers to the poetry written before the 20th century, more specifically poetry that adheres to the rules of classical prosody ( al-'arūd ); following meters or patterns of syllabic pulses, and a rhyme ( al-qafiya ). modern poetry, on the other hand, has liberated itself gradually from these rules. classical poetry is also called vertical poetry in reference to the vertical parallel structure of its two parts known as hemistichs. a classic poem is versified where each verse consists of two halves, each is called shatr 'hemistich'. each verse in a poem follows a meter. meters fall into fifteen different categories collected by the grammarian and prosodist al-farahidi. later, one of his students, al-akhfash, discovered one more meter making them sixteen. ## related work generating poetry is not a straightforward task as there are rules that need to be maintained to ensure the presence of poetic elements such as rhythm and rhyme, whereby these constraints tend to be added * contributed to the work equally. as a part of the architecture of the model. to model constraints during training, @xcite converted their training corpus into its corresponding phonetic encoding and trained a long-short term memory (lstm) trained on these encodings. they also introduced another approach that had a character-level lstm model trained on a generic corpus of poetry an upon outputting a word, it gets approved or rejected by a finite state acceptor (fsa) classifier which ensures that only meter abiding words can be a part of the final poem. @xcite created hafez, a program trained to generate topical poetry. their system relied on recurrent neural networks (rnns) for coherence and finite state machinery to constraint rhyme and rhythm. from prosaic text, van de cruys (2020) generated english and french poetry by having gated recurrent units (grus; @xcite ) in an encoder-decoder setting and an added layer of general attention @xcite . to ingrain their output with poetic elements, they applied a prior probability distribution to their network's probability output, where probabilities relating to words abiding by rhyme and topic constraints were boosted. rnns have also been used without constrain, for example, to build on encoder-decoder architecture @xcite created a network that constructs a poem during each iteration, which gets fed to the network during the following iteration, hence, each poem takes part in constructing the next. on the other hand, zhang and lapata (2014) reserved one rnn for building hidden representations for a current line of poetry which was then fed to another rnn that sequentially predicted words of the next line of poetry. pre-trained models have been put to use to the same task as well. for instance, beheitt and hmida (2022) trained gpt-2 @xcite on arabic news then fine-tuned the model on arabic poetry. @xcite made use of an encoder-decoder architecture to generate modern french poetry, where the encoder is initialized from a pre-trained roberta @xcite checkpoint while the decoder is based on a pre-trained gpt-2 checkpoint. they scraped a corpus of french poems, and used it to train their model for sequence-to-sequence generation, where it predicts a verse given a previous verse in a poem. they also conditioned beam search on rhymes during the generation phase. in our work we dedicate another two pre-trained models of different architectures to explore how effective they are at recognising patterns of classical arabic poetry through the poems they are trained to generate. ## data initially, we compiled datasets of ashaar foot_0 and the arabic poetry dataset foot_1 . combined, each data sample was comprised of a poem's era/country of origin, verses, author, meter and a poem's topic(s). additional scraping was done from al-diwan foot_2 to fill in missing values and to add a new column to the dataset that is rhyme. furthermore, the dataset was tweaked to only contain poems from eight eras, the abbasid, ayubi, ottoman, umayyad, andalusian, mamluk eras and the pre-islamic period. we target these eras as they have more structure compared to the free verse poetry which has no musical pattern or rhyme. we also chose to centralise our poems around 15 out of the main 16 meters of arabic poetry and thus we removed meters variants from our corpus. ## models we experiment with training two different transformer-based architectures: encoder-decoder model and a decoder-only model to generate poetry based once on a prompted meter, and again on a prompted topic. ## evaluation some of the generated poems of our models are shown in ## conclusion & future work we have presented two approaches to classical arabic poetry generation. we have experimented with two transformer-based architectures: the encoderdecoder model initialized with bert-based checkpoints, and the decoder-only architecture: gpt-j. we fine-tuned both models on a classical arabic in generating more fluent and coherent poems, but gpt-j model could capture the rhymes much better. in the future, we aim to incorporate human evaluation in the loop in a reinforcement learning environment, where the model should learn to generate the poetry based on corrected faulty poems.