Story Co-telling Dialogue Generation based on Multi-Agent Reinforcement Learning and Story Highlights (2023)

Abstract: Retelling a story is one way to develop narrative skills in students, but it may present some challenges for English as Second Language (ESL) students who are learning new stories and vocabularies at the same time. The goal of this research is to develop a dialogue module for story co-telling for ESL students in order to help students to co-narrate an English story and enhance their narrative skills. However, story co-telling is a relatively underexplored and novel task. In order to understand the story content and select the right plot to continue the story co-telling based on the current dialogue, we utilize open domain information extraction techniques to construct a knowledge graph, and adopt multi-agent reinforcement learning methods to train two agents to select relevant facts from the knowledge graph and generate responses, jointly accomplishing the task of story co-telling. Compared to models that reply on chronological order, our model improves the performance from 67.0% to 70.8% through self-training with reward evaluation, achieving an increase of approximately 3.8%.

## introduction story retelling is one of the methods to enhance students' narrative abilities. however, due to weaker language proficiency, difficulty in organizing complex plots, or encountering obstacles in expressing ideas and emotions, not every student can fully elaborate on a story independently. to address this issue, we propose the task of story co-telling based on the concept of scaffolding theory @xcite to assist students in story retelling. the notion of scaffolding theory draws an analogy from construction, where temporary support is provided during building construction, and it is removed once the construction is complete or learning is mature. similar to training wheels when learning to ride a bicycle, story co-telling offers necessary support to students when needed and gradually reduces assistance as their narrative skills improve. the objective of this study is to develop a story co-telling dialogue module aimed at assisting esl students in collaboratively narrating lengthy english stories to foster narrative abilities. to refine the study's focus, we constrain the dialogue module to engage only in conversations related to story co-telling, rather than purposeless chitchat. thus, our dialogue module is designed as a supportive story chatbot, which, based on the student's ongoing narrative, determines the next plot to be told, achieving the collaborative narration of the story between two participants. story co-telling is a relatively less explored and novel task, distinct from common story generation tasks. while story generation concentrates on generating logical subsequent plots, story co-telling is grounded in the content of the original story. this difference necessitates a reconsideration of model design and training methods. since story co-telling is an interactive process between two participants, we anticipate employing reinforcement learning techniques to implement the story co-telling module. however, designing a story co-telling dialogue system based on reinforcement learning presents four primary challenges. first, it can be timeconsuming and costly if we would train a dialogue system through online reinforcement learning, where the system learns from actual interactions with people. second, utilizing offline reinforcement learning requires suitable dialogue corpora for story co-telling, which currently do not exist, necessitating the generation of relevant datasets. third, the efficacy of reinforcement learning models hinges on well-defined reward functions. the task of determining how to establish appropriate environmental rewards for each dialogue round constitutes a significant challenge. finally, when dealing with long story texts, how the agents can comprehend the entire content and choose the next coherent story plot or event is a significant challenge. ## related work the application of dialogue robots in education has garnered widespread attention. various educational practitioners hold diverse expectations for the roles and functionalities that educational robots should embody. for instance, the education team at the university of california, irvine developed a system named storybuddy that accompanies parents and children in reading stories together. during the reading process, this system integrates question-and-answer interactions to enhance parent-child engagement @xcite . they introduced the fairy-taleqa dataset @xcite and employed question answer generation (qag) to address the challenge of generating questions for parents. through experiments, it was found that implementing companion-based reading through questioning and answering enhances children's comprehension when responding to questions @xcite . on the other hand, chu and min (2021) developed a dialogue robot specifically for retelling elementary school english storybooks. this dialogue robot assists learners in retelling stories by asking questions and utilizes rule-based mechanisms to determine whether each scene has been accurately recounted. for instance, if the first scene has been correctly mentioned, the robot prompts the student to narrate subsequent scenes. if a scene hasn't been correctly mentioned, the student is asked to retell it. through this iterative process, students are guided step by step to independently retell the entire story. the aforementioned approach demonstrates the potential of story dialogue robots in promoting parentchild interactions, cultivating reading interests, and enhancing narrative skills. continuing with the theme of enhancing children's narrative abilities, recent research has also focused on utilizing information extraction techniques to comprehend and analyze long-text narratives. these techniques aim to transform unstructured textual data into structured information. for instance, @xcite developed a document-level narrative event chain extraction toolkit (nece). this approach employs technologies such as semantic role labeling (srl) to extract relevant information about characters and events from stories. furthermore, a specific tf-idf algorithm is used to identify the most important events. through this framework, the narrative structure within lengthy textual stories can be effectively parsed, enabling the extraction of essential elements like characters and events. similarly, andrus et al. (2022) address the challenge of understanding long-text narratives using dynamic knowledge graphs. unlike static commonsense knowledge graphs that involve real-world information, andrus et al. (2022) utilize openie (open information extraction) technology to construct dynamic knowledge graphs. these dynamic knowledge graphs are then applied to tasks such as question answering and story completion. this approach proves effective in overcoming the limitations imposed by language model input constraints when dealing with lengthy documents, and its effectiveness has been demonstrated. ## method the marl structure for story co-telling based on knowledge graph construction is shown in figure 1. we will start from how to convert a long text story into a knowledge graph and introduce how the agent uses conversation history and knowledge graph to select the plot to be told next. secondly, we explain how to construct a dialogue history evaluation model for evaluating the current performance of story co-telling. finally we will explain how to use reinforcement learning to integrate the above parts into a story sharing dialogue module that can make decisions based on the current dialogue history. ## experiment in this study, we chose stories from fairytaleqa @xcite as the designated story set for story co-telling. these stories are classic fairy tales suitable for readers below the ninth grade, with clear narrative structures. the average text length of stories used in fairytaleqa exceeds one thousand words. additionally, with the pre-designed question-answer pairs available in fairytaleqa, we can evaluate the diversity of co-telling content through question answering. to ensure the effectiveness of agent training, we set some termination conditions for the environment. firstly, by limiting the conversation rounds to be no more than 20, we avoid resource wastage and increased training time caused by excessively lengthy dialogues. additionally, when one of the participants introduces an ending keyword, it signifies an appropriate endpoint for the conversation. furthermore, we set the exhaustion of all facts in the knowledge graph as one of the ending conditions. this configuration ensures efficient utilization of information during the conversation and prevents the repetitive use of the same facts. while (e 1 is not done) and (e 2 is not done) do 13 c t ← φ(d, g); 14 s t ← {µ(d), µ(c t )}; 15 a t ← argmax(q t%2 (s t , ε)); 16 d t ← c t [a t ]; 17 append d t to d; 18 score ## conclusion and future work in this study, we designed a dialogue module for story co-telling with the aim of enhancing esl students' english narrative abilities. by training two agents to select optimal responses from the knowledge graph based on dialogue history, our model is capable of making wiser choices among candidate responses generated by different decision actions. through self-training reward evaluation, we observed that our model's performance improved from 67.01% to 70.81% compared to responding based solely on chronological order. for future work, the knowledge graph is still limited by the completeness and coverage of ope-nie performance. therefore, we can try chatgpt to enhance information extraction. furthermore, while our current approach centers on action design guided by coherence, alternative strategies, such as considering story coverage, could also be employed to shape these actions. overall, there is still a lot of room for improvement in this research.