PG -Story: Taxonomy, Dataset, and Evaluation for Ensuring Child-Safe Content for Story Generation (2024)

Abstract: Creating children’s stories through text generation is a creative task that requires stories to be both entertaining and suitable for young audiences. However, since current story generation systems often rely on pre-trained language models fine-tuned with limited story data, they may not always prioritize child-friendliness. This can lead to the unintended generation of stories containing problematic elements such as violence, profanity, and biases. Regrettably, despite the significance of these concerns, there is a lack of clear guidelines and benchmark datasets for ensuring content safety for children. In this paper, we introduce a taxonomy specifically tailored to assess content safety in text, with a strong emphasis on children’s well-being. We present PG-Story, a dataset that includes detailed annotations for both sentence-level and discourse-level safety. We demonstrate the potential of identifying unsafe content through self-diagnosis and employing controllable generation techniques during the decoding phase to minimize unsafe elements in generated stories.

## introduction in recent years, large language models such as chatgpt @xcite , llama @xcite , and palm 2 @xcite , have showcased impressive text generation capabilities. these models have opened up exciting possibilities for neural story generation @xcite @xcite . however, the real-world implementation of story generation models remains limited due to concerns about their uncontrollable and unpredictable outputs @xcite , particularly when creating content for children @xcite . with today's children spending more time online, ensuring access to safe digital content has become paramount. while digital technologies have brought benefits, they've also exposed children to * work done while at amazon. potential risks, including harmful content, misinformation, and violence. previous efforts to ensure the safety of children's digital content have primarily focused on video and audio, addressing issues such as sexual hints, graphic nudity, abusive language, weapons, violent scenes, horror sounds, and scary scenes @xcite @xcite . however, despite extensive research on toxic and offensive machine-generated language in social media @xcite @xcite and online conversations @xcite , ensuring content safety in machine-generated stories, especially for children, remains largely unexplored. ensuring safety in story generation goes beyond avoiding toxic language; it involves considering vocabulary, tone, implications, and overall suitability of the plot for children. in this work, we aim to address part of this gap by introducing a taxonomy designed to assess content safety for children in narrative text, specifically focusing on identifying disturbing and problematic textual elements-an under-explored area. we find that existing safety evaluation tools, whether open-source or open api (e.g. perspectiveapi 1 ), fall short in safeguarding children from potentially disturbing content. additionally, widespread generative ai assistants such as chatgpt also exhibit such potential risks. to rectify this, we have developed a specialized classifier to detect potentially unsafe content based on our child-centric safety taxonomy. to promote research in this direction, we have compiled pg-story, a test bed dataset comprising diverse stories from various sources, such as daily life, movie scripts, and fairy tales, enriched with safety annotations at both the discourse and sentence level. furthermore, we employ a modelin-the-loop approach to facilitate human annotation from a wide range of stories. our contributions can be summarized as follows. we propose a child-centric digital safety taxonomy specifically tailored for narrative text and assess the ## curating the pg-story corpus in this section, we introduce pg-story, a dataset annotated according to our taxonomy, encompassing a wide range of stories on various topics. 3 while there are existing datasets focused on children's content, such as the children stories text corpus 4 and children's book test foot_3 , sourced from project gutenberg and suitable for young readers, they have limited coverage of content safety evaluation. other story datasets like rocstories lack a specific focus on children's content. additionally, despite numerous datasets addressing toxic language, none are tailored for evaluating content safety in narrative text. to bridge this gap, we have curated the pg-story dataset. it aims to address limitations associated with existing datasets and serves as a valuable resource for evaluating content safety in story generation models. our pg-story dataset includes 1,000 human-annotated short stories or excerpts from longer narratives, and an additional 100,000 data points are generated through semi-supervised methods. data source for pg-story. we collected stories from a diverse range of sources, including short and long narratives, covering various themes. ## experiments our experiment addresses two key research questions: we conduct experiments using the pg-story dataset, which we randomly split into train (80%), dev (10%), and test (10%) sets. the plan-to-story generation model is trained using a pre-trained bart model 7 , fine-tuned on the story datasets listed in we assess our model's story generation using a variety of metrics: fluency (measured by perplexity and bert-f1), diversity (evaluated using dist-n ), semantic correctness (measured by the keywords matching ratio, kmr), and content safety. to evaluate content safety, we utilize the perspective api toxicity score for automatic evaluation and conduct human evaluation to gauge the model's ability to generate child-safe stories. ## conclusion in conclusion, we have introduced a comprehensive content safety taxonomy tailored for children's narrative text and curated a dataset, pg-story, enriched with safety annotations for children's story generation. our proposed safe story generation framework, equipped with self-diagnosis and rewrite capabilities, demonstrates the ability of models trained on our dataset to produce child-safe stories. we invite researchers in both the nlp and childhood development domains to leverage pg-story as a valuable resource for advancing story generation models and enhancing nlp technologies to ensure the digital safety of children.