P ea C o K : Persona Commonsense Knowledge for Consistent and Engaging Narratives (2023)

Abstract: Sustaining coherent and engaging narratives requires dialogue or storytelling agents to understandhow the personas of speakers or listeners ground the narrative. Specifically, these agents must infer personas of their listeners to produce statements that cater to their interests. They must also learn to maintain consistent speaker personas for themselves throughout the narrative, so that their counterparts feel involved in a realistic conversation or story. However, personas are diverse and complex: they entail large quantities of rich interconnected world knowledge that is challenging to robustly represent in general narrative systems (e.g., a singer is good at singing, and may have attended conservatoire). In this work, we construct a new large-scale persona commonsense knowledge graph, PeaCoK, containing ~100K human-validated persona facts. Our knowledge graph schematizes five dimensions of persona knowledge identified in previous studies of human interactive behaviours, and distils facts in this schema from both existing commonsense knowledge graphs and large-scale pretrained language models. Our analysis indicates that PeaCoK contains rich and precise world persona inferences that help downstream systems generate more consistent and engaging narratives.

## introduction interlocutors or storytellers in narrative scenarios often exhibit varying behaviours, which are affected by their own diverse personas, but also the personas of the counterparts they are interacting with. for example, an adventurous architect may be interested in talking about outdoor explorations with his friends who have similar hobbies, but may prefer to discuss architectural design ideas with his i am a singer who want to release an album study music at college good at singing write songs win a grammy award have more fans characteristic experience routine or habit goal or plan i am a student who major in music composition i am a famous pianist relationship skilled in playing the piano talk with music mentor routine or habit routine or habit relationship experience characteristic figure 1: illustration of world persona knowledge grounded on commonsense reasoning. this goal of modeling diverse persona attributes is at the heart of research in the areas of personagrounded dialogue @xcite @xcite , story generation @xcite and narrative understanding @xcite . however, the complex nature of real-world personas, which involve rich world knowledge, and the countless ways in which they might interact, is challenging to reliably learn purely from data. for instance, as shown in figure 1 , a singer preparing an album may have studied music at university at one point, which would allow them to share their experience with a student majoring in composition, who may study music as a daily routine. prior work takes first steps at improving the persona knowledge representations available in narrative systems. @xcite extract self-comments from reddit websites to expand the scale of background persona profiles that can be used in downstream narrative settings. however, their collected profiles are fragmented and ignore the interconnections between personas that govern interactions. meanwhile, majumder et al., 2020 use knowledge generators @xcite to expand the persona profiles with commonsense inferences, but these commonsense expansions are limited to general social commonsense @xcite , and do not form a systematic personacentric knowledge frame. consequently, the lack of world-level persona commonsense knowledge resource hinders progress in learning the systematic persona representations necessary to sustain consistent and engaging narratives. in this work, we propose a persona-grounded commonsense knowledge graph (kg), peacok, which represents world-level persona knowledge at scale. building off the persona concept initially proposed in human-computer interaction @xcite @xcite and on behaviour analysis literature for human leisure conversations @xcite , we define a persona frame that formalizes five common aspects of persona knowledge: characteristics, routines and habits, goals and plans, experiences, and relationships. using this knowledge frame, we construct a large-scale graph of persona commonsense knowledge by extracting and generating persona knowledge from both existing hand-crafted commonsense kgs and large-scale pretrained language models (lms). we validate the knowledge graph via a joint human-ai majority voting scheme that integrates large pretrained lms into the loop of crowdsourcing, and efficiently mediates the disagreements between human annotators. our resulting kg, peacok contains ∼100k high-quality commonsense inferences (i.e., facts) about personas whose connectivity in the kg reveals countless opportunities to discover common ground between personas. a neural extrapolation from the kg @xcite also shows that peacok's annotated personas enable the development of effective persona inference generators. finally, the extended knowledge provided by pea-cok enables a downstream persona-grounded dialogue system to generate more consistent and engaging responses in conversations, particularly when more interconnections between the interlocutor personas are found in peacok. ## related work commonsense knowledge graphs commonsense kgs such as conceptnet @xcite , atomic @xcite , anion @xcite and atomic 20 20 @xcite are widely used in nlp applications that involve integrating implicit world knowledge, e.g., question answering @xcite @xcite ) and text generation @xcite . however, despite the importance of persona knowledge in modeling human behavior -a crucial component for building reliable narrative systems @xcite -no commonsense kg explicitly focuses on representing human persona knowledge. we present pea-cok to open the field of developing commonsense knowledge graphs around personas. persona-grounded narratives integrating personas to improve consistency and engagement of narratives is an important goal in dialogue @xcite and storytelling @xcite systems. one representative work that greatly contributed to the development of faithful persona emulation, persona-chat @xcite , constructs a crowdsourced dialogue dataset by asking participants to perform conversations based on their assigned persona profiles -five statements of selfintroduction. more recent work improves persona modeling in narrative systems by generating persona profiles from online resources @xcite , training persona detectors @xcite and predictors @xcite , and distilling persona knowledge from commonsense inference engines @xcite . however, while these works align characters in narratives with persona profiles, they only implicitly model the areas of interaction between personas. in contrast, peacok explicitly represents interconnections between persona profiles, enabling persona interaction modeling in narrative systems. ## peacok knowledge frame to construct a systematic representation of persona knowledge, we distill five common aspects of personas from classical persona definitions. in the field of human-computer interaction, a persona is a fictitious example of a user group that is conceptualized to improve interactive design in areas such as marketing, communications, and atomic 𝟐𝟎 𝟐𝟎 actor (human entity) personx becomes a movie star (event) 1. persona selection + xneed + xwant … … atomic 𝟐𝟎 𝟐𝟎 comet good at acting, be more famous … or instructgpt-3 + guess an experience + guess a(n) … not persona: … characteristic: good at acting, … routine or habit: … goal or plan: be more famous, … experience: graduated from drama school, … relationship: … graduated from drama school … 2. potential attribute induction 3. relation classification kg-based approach instructgpt-3 + lm-based approach majority voting service product development @xcite . from the perspective of goal-directed design @xcite , personas encapsulate user needs and goals when interacting with a product, along with their intrinsic character traits and past experiences @xcite ) that contextualize the interaction. using these attributes of goals, traits, and experiences as the foundation of personas, we also leverage prior studies in human conversational behaviour that explore which topics of conversation are often broached in relaxed human social interactions. after conducting observational studies, @xcite categorized the topics of human conversations into bins: personal relationships (i.e., experiences or emotions rising from social interactions), personal experiences (i.e., factual events or circumstances experienced by a person), future activities (i.e., arrangements of meetings or events), leisure activities (e.g., hobbies), interests (e.g., culture, politics, religion), and work (e.g., daily routines). to select our persona dimensions, we discard certain controversial categories from the above studies (i.e., culture, politics, and religion), as well as temporary dimensions of persona (i.e., emotion, which is well covered by prior work; @xcite @xcite . our final persona frame consists of five relations for each persona, each with multiple attributes attached to it. we describe the five relations below: characteristics describe an intrinsic trait, e.g., a quality or a mental state, that the persona likely exhibits. for example, as shown in figure 1 , good at singing describes a talent of a singer, which is one of the singer's characteristics. routines or habits describe an extrinsic behaviour that the persona does on a regular basis, e.g., a singer may regularly write songs. goals or plans describe an extrinsic action or outcome that the persona wants to accomplish or do in the future, e.g., a singer may aim to win a grammy award some day. experiences describe extrinsic events or activities that the persona did in the past. for instance, a singer may have studied music at college. relationships encode likely interactions of the persona with other people or social groups. note that this relation can be overlapped with other relations in peacok. for example, a singer may want to have more fans, which connotes a relationship between singer and fans, but also a future goal or plan of singer. ## peacok construction we use our persona frames to construct a knowledge graph of persona commonsense where personas are treated as head entities in the graph, frame relations constitute edge type relations, and attributes are tails in a (head, relation, tail) structure. then, we devise a three-step procedure to construct the frames that make up peacok, as shown in figure 2 . first, we search existing commonsense kgs to select entities that can serve as head personas. then we query these kgs and prompt pretrained lms to collect tail attributes that are potentially associated with the personas via the five relations defined in sec. 3. finally, we use crowdsourcing with large lms in the loop to classify whether these persona inferences are valid. ## peacok analysis our statistics of the final peacok relations are shown in persona interconnectivity in addition to containing diverse knowledge from multiple sources, peacok also contains interesting interconnections among personas, which potentially indicate engaging points of common ground for characters of narratives. for example, as shown in figure 1 , a professional singer's experience of studying music at college is also the routine of a music-major student, which shows a common topic for these two persona to discuss. among 40,665 distinctive attributes in peacok, we find that 9,242 attributes are connected to two or more personas, forming 239,812 bridges, i.e., pairs of personas connected via a shared common attribute. foot_4 ## generalizing persona knowledge following the neural kg analysis method proposed by @xcite , we assess whether pea-cok could be used to train inference generators that hypothesize persona knowledge. we train a bart-based @xcite comet @xcite knowledge generator (comet-bart) based on a held-out training set (∼65k facts) of peacok, where the model learns to generate the tail attribute of a fact given its head persona and relation. we evaluate comet-bart on a test set from peacok containing 3030 facts with unique head-relation combinations. as baselines, we compare to a few-shot gpt-3 @xcite ) that uses 5 randomly sampled training facts (with same relation as the testing fact) to prompt the tail knowledge generation and a zero-shot gpt-3.5 (text-davinci-003) baseline model. these baselines compare peacok training to larger lms that use both in context-learning and instruction tuning. we conduct both automatic and human evaluations on the knowledge generators, with results shown in tables 5 and 6 . 10 compared to few-shot gpt-3, comet-bart trained on peacok achieves overall better automatic evaluation results on various nlg metrics, despite being a much smaller model. 11 in the human evaluation, we find that facts generated by comet-bart receive a high acceptance rate by all above results indicate that peacok can serve as a reliable persona knowledge base, which enables light-weight lms to learn knowledge generation capabilities comparable to large-scale lms. ## enhancing dialogue systems as our knowledge graph peacok covers rich world persona knowledge, we validate whether acccess to this knowledge enables better persona modeling in downstream narrative systems. using peacok, we augment a persona-grounded dialogue model p 2 bot @xcite developed on the convai2 @xcite ) persona-chat @xcite dataset. we link facts from peacok to persona-chat dialogues, thereby extending p 2 bot's persona perception and augmenting its dialogue response generation. 12 we evaluate our models based on both original and revised interlocutor profiles provided in the convai2 persona-chat dataset, and measure the perplexity (ppl), word-level f1, and cumulative 4-gram bleu @xcite of the generated responses compared to the references. we also follow convai2 to measure hits@1, i.e., the probability that real response is ranked the high12 downstream application details are in appendix d. persona knowledge linking we link peacok knowledge to interlocutors based on both their persona-chat profiles and their utterances in the dialogue. for each interlocutor, we extract all statements in their profile, as well as first-person sentences in their utterances. then, we follow a commonsense fact linking benchmark, comfact @xcite , to link relevant facts from pea-cok to each extracted statement or sentence. we remove linked facts that are labeled as generic in the distinctiveness dimension, i.e., have little effect on distinguishing this persona from others. for each interlocutor, we randomly sample 5 peacok facts that are linked to their persona-chat profile, 13 and convert them into natural language statements to form their extended persona knowledge. 14 our augmented model is denoted as p 2 bot + peacok. to compare peacok's persona-centric knowledge augmentations with general commonsense augmentations, we also evaluate another baseline model p 2 bot + atomic 20 20 , where we follow majumder et al., 2020 to extend interlocutor personas with 5 randomly sampled commonsense inferences from the comet-atomic 20 20 model @xcite . ## conclusion in this work, we propose a persona commonsense knowledge graph, peacok, to complement the real-world picture of personas that ground consistent and engaging narratives. peacok consists of ∼100k persona commonsense inferences, distilled from existing kgs and pretrained lms, across five dimensions of persona knowledge identified in prior literature on human interactive behaviours. our analysis and experiments demonstrate that peacok contains high-quality inferences whose connectivity provides many instances of common ground between personas, improving the consistency and engagement of downstream narrative systems.