A rabic Automatic Story Generation with Large Language Models (2024)

Abstract: Large language models (LLMs) have recently emerged as a powerful tool for a wide range of language generation tasks. Nevertheless, this progress has been slower in Arabic. In this work, we focus on the task of generating stories from LLMs. For our training, we use stories acquired through machine translation (MT) as well as GPT-4. For the MT data, we develop a careful pipeline that ensures we acquire high-quality stories. For our GPT-4 data, we introduce crafted prompts that allow us to generate data well-suited to the Arabic context in both Modern Standard Arabic (MSA) and two Arabic dialects (Egyptian and Moroccan). For example, we generate stories tailored to various Arab countries on a wide host of topics. Our manual evaluation shows that our model fine-tuned on these training datasets can generate coherent stories that adhere to our instructions. We also conduct an extensive automatic and human evaluation comparing our models against state-of-the-art proprietary and open-source models. Our datasets and models will be made publicly available at https://github.com/UBC-NLP/arastories .

## introduction storytelling is an essential human skill that serves to transmit knowledge, impart values, and connect individuals through tales of daily experiences. it is utilized in education, where teachers harness children's natural affinity for stories to foster cognitive and literacy development. additionally, stories and legends, viewed as cultural heritage, are passed down through generations by parents, enriching the culture and preserving traditions. the role of storytelling also extends beyond its traditional roots; it acts as a vital connection between the primitive oral language skills in early childhood and the advanced language abilities as-1 gpt-4 refers to gpt-4-0125-preview sociated with literacy. as such, the task of automatic story generation presents numerous benefits across different fields. in entertainment, it allows for the efficient creation of diverse narratives @xcite . in education, tailored stories can be crafted to address the unique needs of learners. in gaming, interactive storytelling significantly enhances user engagement and enjoyment @xcite . and these are only a few application domains. progress in natural language processing (nlp) technologies, particularly with large language models (llms) such as gpt-4 and gemini, has made automatic story generation both viable and effective, producing stories with notable fluency and coherence. while substantial efforts have been made to advance automatic story generation in english using generative models, the development of such technologies for arabic has been limited due to a scarcity of arabic short story data and minimal focus from the research community. in this study, we present a novel approach to automatic story generation utilizing the powerful arabic llm, arallama @xcite . we enhance arallama through fine-tuning with both translated and synthetic datasets to optimize its story-generating capabilities. we explore two fine-tuning strategies: one involving direct application of a synthetic dataset produced by gpt-4, and another beginning with an analogous synthetic dataset translated from english. additionally, we extended the model's utility by fine-tuning it with data from two arabic dialects, enabling the generation of stories in both modern standard arabic (msa) and these two dialects. the efficacy of our model is assessed through human evaluation, which confirmed its ability to produce coherent and fluent narratives as per specified instructions. our contributions are manifold, summarized as follows: the remainder of this paper is organized as follows: section 2 provides a review of prior studies focusing on the task of automatic story generation. section 3 details the creation of our datasets. in section 4, we outline our prompt design. in section 5 we detail our different experiments. results and key insights from our comparative analysis of our fine-tuned models against various commercial and open-source models are discussed in section 6. the paper concludes with section 7. ## data collection we compile data from different resources. we first translate 1.13m english stories generated by gpt-4 alongside their prompts from the tinystories dataset (eldan and li, 2023) using google translate api. 4 to ensure that we have only highquality translation, we apply a filtering strategy based on multilingual sentence embeddings @xcite and remove the story pairs whose embedding similarity is less than 92%. ## prompt design prompting is an approach employed by users to interface with llms @xcite . it functions as the primary mode of communication with these models, effectively serving as the input language that llms are designed to interpret and respond to. the efficacy of the generated output is significantly correlated with the quality and structure of the input prompt. this relationship underscores the critical role that prompt engineering plays in optimizing llm performance and output relevance. in our context, the prompt can be conceptualized as a set of instructions or parameters that guide the llm's for the arabic story generation process. the prompt's composition, including its specificity, clarity, features, and relevance to the desired output, directly influences the model's ability to generate appropriate and accurate stories that adhere to our instructions. this causal relationship between prompt quality and output quality highlights the importance of developing sophisticated prompting strategies to fully leverage the capabilities of llms for arabic story generation. ## experiments we conduct two sets of experiments: the details of each experiment are described next. ## evaluation you are an expert in arabic language, its dialects, and storytelling. i would like your help in evaluating a story written by a student based on a set of instructions. you are expected to give a score out of five based on the following features: **fluency:** how smooth and natural the text is, including appropriate grammar, vocabulary, and sentence structure. **coherence:** the logical connection and flow of sentences and ideas, making the text easy to understand. **following instructions:** how well the text adheres to the provided instructions or task requirements. **consistency:** how consistently accurate and uniform the information and style are throughout the text. **variety:** how well does the model generate story in the required arabic variety. give the scores directly without explanations or additions. i will first give you the instructions on which the story was based, followed by the story written by the student. remember, i want the evaluation directly without explanation. figure 2 : the prompt we pass to gpt-4-turbo to evaluate stories generated with different models. evaluating generative tasks remains an open problem in the ai community. however, in our study, we follow previous works such as @xcite in adapting gpt-4 as an evaluator for model performance. we also conduct an extensive human evaluation trying to understand different model capabilities and how the evaluation of gpt-4 compares to that of human judges. we next describe our two evaluation strategies. arabic variety specified in the prompt or not, and to which degree. figure 2 demonstrates the prompt we pass to gpt-4 for evaluation. we evaluate 20 new prompts by asking the models to generate 20 corresponding stories and then pass the prompt plus story to gpt-4 for evaluation. we compare our two models against three other open and proprietary models. namely, we compare against gpt-3.5, command-r 5 , and acegpt-7b-chat @xcite . it is pertinent to note that we experimented with other strong open-source models, such as llama-3-70b-chat @xcite and mixtral-8x7b @xcite (accessed through an api), but these models failed to adhere to our instructions. this failure highlights the superiority of our models over these strong baselines. furthermore, we could not compare our models against larger arabic llms, such as jais-30b @xcite and acegpt-13b @xcite , due to computational constraints. in other words, we limited our comparisons to 7b models unless an api was available. model b which was exposed to additional training steps on translated data, performs better than model a on almost all metrics across dialects, which proves indeed the intuition behind training on more data does help. comparative results suggest that there might be opportunities for further fine-tuning or learning from stronger models such as command-r since this model shows strong performance across multiple metrics. our results demonstrate that command-r is the strongest baseline, and the most consistent model across metrics and varieties. even in the variety feature where the performance of other models falls, command-r achieves a score as high as 3.27. command-r outperforms even gpt-3.5 while being only a fraction 5 https://dashboard.cohere.com/playground/chat? of its size, suggesting that the model size is not everything and the quality of data does help. we can see from ## conclusion in this paper, we present the first llm-based study on automatic arabic story generation. our study takes as its target msa and two arabic dialects (egyptian and moroccan). for our purpose, we translate and generate datasets based on a custom prompt template. we fine-tune our models on these datasets, comparing against both equallyand bigger-sized models. through extensive automatic and human evaluation, we empirically show our models' superiority to strong baselines. in the future, we plan to fine-tune bigger models on larger datasets. we also plan to include more dialects in our training, for wider coverage.