On the Automatic Generation and Simplification of Children’s Stories (2023)

Abstract: With recent advances in large language models (LLMs), the concept of automatically generating children’s educational materials has become increasingly realistic. Working toward the goal of age-appropriate simplicity in generated educational texts, we first examine the ability of several popular LLMs to generate stories with properly adjusted lexical and readability levels. We find that, in spite of the growing capabilities of LLMs, they do not yet possess the ability to limit their vocabulary to levels appropriate for younger age groups. As a second experiment, we explore the ability of state-of-the-art lexical simplification models to generalize to the domain of children’s stories and, thus, create an efficient pipeline for their automatic generation. In order to test these models, we develop a dataset of child-directed lexical simplification instances, with examples taken from the LLM-generated stories in our first experiment. We find that, while the strongest-performing current lexical simplification models do not perform as well on material designed for children due to their reliance on large language models behind the scenes, some models that still achieve fairly strong results on general data can mimic or even improve their performance on children-directed data with proper fine-tuning, which we conduct using our newly created child-directed simplification dataset.

## introduction large language models (llms), such as gpt-3 or chatgpt, are able to produce stories that are far more coherent and fluent than stories generated by state-of-the-art models from even a couple of years ago, such as graphplan @xcite and plan-and-write @xcite . however, most of the already limited work on automatic story generation focuses on stories for an adult audience. children's stories are not frequently a topic of interest, despite how crucial early literacy is to future success @xcite . as we will describe in more detail in section 2, children's stories are important for both entertainment and education. automatic story generation for children increases their potential for broader impact by making it possible to personalize stories, making them increasingly relevant for each individual child. as one example, tailoring stories to a specific child's interests could allow them to become more easily interested in reading, which could improve their literacy skills. another possible application could be the teaching of specific target words to preschoolers via stories. in this paper, we will keep the latter use case in mind. while, at first look, stories that have been generated by llms seem (and generally are) better than previous attempts, it has not been systematically evaluated if their children's stories adhere to what one would expect from the genre. in this paper, we focus on the simplicity of generated stories as measured by the age of acquisition of their individual words. specifically, we assume that we are interested in generating stories to teach words with an age of acquisition (aoa) from 6 to 9 to children who are preschool-aged (around 2-5). including other complex words or concepts, however, can make story-based vocabulary learning more difficult, as they detract attention from target words and complicate the ascertainment of context clues. thus, we want to include the target words in the stories, but we do not want any other words to have an @xmath0. for the first experiment conducted in this paper, found in section 4, we use 3 different llms for generation and ask the following research questions: (rq1) what is the simplicity of generated stories for different llms? (rq2) how do different prompts, with different descriptions of the target age group, influence story simplicity for different models? we find that, in spite of llms' growing abilities for story generation, they struggle to control for age-appropriate simplicity; in comparison to our dataset of human-generated stories for the same demographic, the models' stories exhibit scores that are over 17% worse for all readability metrics tested (flesch reading ease, flesch-kincaid level, gunning-fog index, and automated readability index). motivated by those findings, in section 5, we then turn towards simplification models. simplification models have mostly foot_0 been developed for text directed at adults. here, we investigate two leaders of the tsar-2022 shared task on multilingual lexical simplification @xcite in english, unihd @xcite and uofm&mmu @xcite , with regards to their ability to generalize to the challenging out-of-domain setting of children's stories. for this, we generate a new dataset of annotated instances for lexical simplification, using age of acquisition as a metric for identifying complex words and pulling examples from our newly created corpus of llm-generated stories for human annotators to simplify. we then use this dataset to evaluate the models' performance on automatically generated children's stories. our experiments show that, while unihd performs considerably worse, achieving an accuracy of only 30.52% in comparison to the 42.89% accuracy it achieves on the tsar-en dataset for general lexical simplification, the ordinarily weaker uofm&mmu model is actually able to perform better on the child-directed dataset when finetuned, achieving an accuracy of 47.37% compared to its 28.95% on tsar. we conclude that simplification models trained on adult data are suitable to simplify automatically generated children's stories only when properly finetuned, i.e., the bestperforming models for adult text do not generalize well without additional training, but fintuning is effective for our domain, even with limited data. to summarize, our contributions are as follows: (1) we examine and compare the ability of several llms to adjust for age-appropriate levels of readability and lexical simplicity in automatically generated and individualized educational stories. in child development, small early differences can compound into big long-term effects. one example of this is the relationship between early vocabulary size, literacy, and later academic achievement. early vocabulary size is strongly related to reading ability in 2nd and 3rd grade @xcite , and even when controlling for vocabulary size in kindergarten, reading ability in 4th grade is associated with vocabulary growth through 10th grade @xcite . although there is a recognition of this relationship, large gaps in vocabulary size persist into elementary school and beyond: e.g., @xcite estimate the vocabulary of 4th graders in the lowest quartile to be less than half the size that of 4th graders in the highest quartile, and a similarly sized gap is found in empirically-based estimates throughout adulthood @xcite . many vocabulary enrichment programs based on shared reading with a caregiver have been developed to try to address this gap, with mixed success. early vocabulary interventions are generally based on storybook reading with a parent or teacher. a meta-analysis focusing on vocabulary intervention studies on children in pre-k and k concluded that although such interventions may increase oral language skills, they are not powerful enough to close the gap, even when implemented at this early age @xcite . experts agree that intensive, individual-level interventions would be necessary to make a difference but acknowledge that the infrastructure investment required for something on that scale would be substantial @xcite . with recent improvements in llms, the concept of using natural language processing techniques to automate the child-by-child customization of educational materials has become increasingly realistic. in particular, state-of-the-art models have proven to be effective for generating individualized reading materials without the labor cost of having a human modify them by hand, making this process more practical in lower-resource settings facing educational discrepancies. developing efficient pipelines for personalized vocabulary-enriching story generation is one way ito address this so called "achievement gap" in early childhood education, providing opportunities for individualized education where they otherwise may not be available. there are several challenges that should be addressed in order to make the automatic generation of individualized educational children's stories a more feasible idea. one primary concern is ensuring that the stories are simple enough to be understood by the target demographic. stories which are too complex or which contain too many unknown words could make understanding the meanings of their target words based on the context provided more difficult. additionally, studies show that having fewer unfamiliar detractors allows children to focus on and retain target words more effectively @xcite . as such, it is important that automatic story generation models be able to control for the simplicity and readability of the stories outside of the words they intend to teach. because of the potential personalized stories have for increasing both specific educational potential and overall engagement in reading, this paper focuses on how llms and lexical simplification models can be used for this purpose. ## experiment 1: assessing readability first, we investigate the readability of automatically generated stories for 1) different models and 2) a variety of prompts. we generate a total of 250 stories for each model: 50 per model-prompt combination. ## experiment 2: story simplification the results shown in section 4.5 demonstrate a serious need for improvement concerning ageappropriate simplicity in automatically generated children's stories. thus, in our second experiment, we examine to what extent current state-of-the-art lexical simplification models generalize to the domain of children's stories, as well as how this could be beneficial in the automatic generation of educational children's stories by llms. ## conclusion in this paper, we investigate the ability of several current llms to generate age-appropriately simplified stories for children, as well as an examination of how modern lexical simplification models generalize to the domain of children's stories to enhance their educational potential. we demonstrate that, in spite of their growing capabilities, modern llms are unable to generate children's stories with age-appropriate simplicity, particularly in comparison to their human-written counterparts. because of these shortcomings found in the automatically generated stories, our second experiment (section 5) focuses on whether or not ordinary lexical simplification models generalize to the domain of children's stories, due to the lack of current ls models that focus on children-directed corpora. we find that some models which are ordinarily lowerperforming than their llm-powered counterparts have the potential to perform well in the domain of simplifying child-directed text, when properly finetuned. over the course of our experiments, we further create a corpus of vocabulary-driven llmgenerated children's stories as well as an annotated lexical simplification dataset, cds, intended specifically for the domain of children's text and using examples taken from this above-mentioned automatically generated stories. we provide these datasets publicly in order to promote further research in this area. in future work, we hope to further improve the automatic generation of customized children's stories by adding models for other tasks to our generation pipeline, such as one that can detect coherence errors or one that can improve readability.