RPTCS : A Reinforced Persona-aware Topic-guiding Conversational System (2023)

Abstract: Although there has been a plethora of work on open-domain conversational systems, most of the systems lack the mechanism of controlling the concept transitions in a dialogue. For activities like switching from casual chit-chat to task-oriented conversation, an agent with the ability to manage the flow of concepts in a conversation might be helpful. The user would find the dialogue more engaging and be more receptive to such transitions if these concept transitions were made while taking into account the user’s persona. Focusing on persona-aware concept transitions, we propose a Reinforced Persona-aware Topic-guiding Conversational System (RPTCS). Due to the lack of a persona-aware topic transition dataset, we propose a novel conversation dataset creation mechanism in which the conversational agent leads the discourse to drift to a set of target concepts depending on the persona of the speaker and the context of the conversation. To avoid scarcely available expensive human resource, the entire data-creation process is mostly automatic with human-in-loop only for quality checks. This created conversational dataset named PTCD is used to develop the RPTCS in two steps. First, a maximum likelihood estimation loss-based conversational model is trained on PTCD. Then this trained model is fine-tuned in a Reinforcement Learning (RL) framework by employing novel reward functions to assure persona, topic, and context consistency with non-repetitiveness in generated responses. Our experimental results demonstrate the strength of the proposed system with respect to strong baselines.

## introduction due to the abundance of conversational corpora, there has been a great interest in building open- § equal contribution 1 codes and dataset available at https://github.com/ zishan-ahmad-nlp/persona-topic-shift domain conversational systems foot_0 @xcite . while capable of producing fluent responses, these systems also frequently generate generic responses devoid of any useful information @xcite . human discourse is complex and difficult to replicate as it often covers a wide range of topics @xcite . hence, the key to a multi-turn conversation's success is striking a balance between effectively changing the subject and keeping it on-topic @xcite . it is also seen that even if open-domain conversational systems employ topic shift, they are not able to engage user due to absence of knowledge of user's own perceptions. therefore, to achieve a robust topic shift guiding conversational system, it has to evolve away from the prior factual information systems @xcite towards a novel blend of both task-oriented and nontask-oriented systems @xcite . however, they may fail to engage user in difficult multi-turn information-seeking tasks or dialogues @xcite and may also suffer from "anomalous state of knowledge" @xcite where the user has vague information requirements and is often struggling to articulate it with enough precision in a conversation. thus, we require context-sensitive user guidance that does not assume a rigid hierarchy of the user's plans and work objectives. such a guidance, can be achieved by utilising persona information of user with the ongoing dialogue. this will drive the conversation from one topic to a different one in consonance with the user's interests. focusing on these two aspects, we propose a reinforced persona aware topic guiding conversational system (rptcs). due to scarcity and expensiveness of human resource, first, employing minimal manual intervention, we present a novel persona aware topic guiding conversational dataset (ptcd) created employing a novel automatic conversation creation mechanism. majority of the topic shift guiding conversational systems leverages only the concept space knowledge to represent the potential conversation flow @xcite @xcite @xcite . these approaches are predicated on a restricted defined logical premise: individuals would be interested in talking about any random concept. this presumption can be too straightforward to mimic topic flows in real human conversations. in this work, we take a significant step toward developing a proactive conversational agent that is more attuned to a user's personality with a clear conversation aim. we aim at training a system that is able to decide on topic shifts and generate responses that reflect these logical topic shifts, while also considering the user persona. figure 1 shows an example of this task. since bringing an open-domain conversation to a task-oriented domain is one of the key use cases for this system, we chose the concepts popular in the latter domain @xcite . using ptcd, we first train a maximum likelihood estimation (mle) loss-based conversational employing a language model (lm) gpt-2 small @xcite . this trained model is then fine-tuned in a reinforcement learning (rl) framework using novel rewards designed to assure persona, concept, and context consistency with nonrepetitiveness in the generated responses. the key contributions of this work are four-fold (i). proposal of a new task to build a persona-aware conversation system for targeted topic transitions and creation of a novel datatset ptcd to tackle the task in-hand; (ii). a novel semi-automatic persona aware topic guiding corpus creation method using pre-trained models (can be adapted to the other similar tasks in the broad area of dialogue systems), (iii). designing a novel reward function to built the proposed system rptcs in an rl framework. (iv). performed extensive automatic and human evaluation by designing two novel evaluation metrics to evaluate topic and context consistency to demonstrate strength of the proposed system. ## related work to have a human-like conversation and avoid offtopic response generation, several methods have been explored @xcite . the research community has employed conceptnet @xcite to mimick concept transitions in human conversations, @xcite , such as: zhang et al. (2020) used concept relations with 2-hop nodes, to account more thoroughly human concept shifts. however, none of the works have taken into account the persona of the user to guide the topic shift in the conversation which may act as a crucial component to have effective communication. some benchmark conversation datasets has been proposed to assess the conversation focusing on different personal attributes such as: @xcite predicts the next topic based on social and cultural situation, ethics dataset constitutes the conversation based on the concept of morality @xcite . @xcite presents emotion labelled emotionlines dataset, @xcite proposes a social relation inference based dialo-gre dialogue dataset. @xcite proposed persona-chat dataset to make chitchat dialogues more engaging by conditioning them on user's profile information. most of these datasets are collected through crowdsource workers. to obtain such a large human resource is expensive, time-consuming, and can be infeasible. we investigate an automatic data creation approach requiring minimal manual intervention as an alternative to this. recently, @xcite proposed a multiturn topic-driven naturalconv dataset ensuring a smooth topic shift in conversations. our work here is different from the existing topic-shift guiding conversational systems in three aspects. first, we followed an automatic dialogue data curation process. second, the topic shift in the conversations is guided using conceptnet as well as the persona of the user. this ensures both goals of an effective communicator viz. completing the task and maintaining the face of the user. third, we built a persona aware topic guiding conversational system in an rl framework using novel efficient and effective reward function. to the best of our knowledge, our proposed setup is novel and has not been tried before. ## dataset creation to tackle the persona-aware concept shift to a target concept, we create and propose a novel dataset. we come up with a novel semi-automatic data creation technique involving prompting a gpt-j model and human intervention for quality control. the entire dataset creation consists of the following steps: obtaining seed data: to start the few-shot dialogue generation, we required a seed utterance from a user with some assigned persona. to build in persona-chat-grounded topic-shifting conversations"the first utterance of a conversation (focused on a persona-profile) in the persona-chat @xcite is selected. in persona-cha", persona profiles are defined as "profiles that are natural and descriptive, and contain typical topics of human interest that the speaker can bring up in conversation". the concepts in the seed utterance is obtained by extracting words with 'nouns', 'adjectives', and 'verbs' part-of-speech (pos) tags. we call these first utterance concepts as the source concept. the objective here is to guide the conversation away from the source concept and toward a target concept. concepts like 'travel', 'restaurant', 'shopping', 'electronics', etc. are chosen as the system's target concepts. ## persona-aware topic-guiding conversational system we build our proposed system rptcs in two phases. in first phase, an mle-loss-based conversational system (mlcs) is trained to learn the user and agent's utterances distribution and obtain natural language interaction. in the second phase, this mlcs is fine-tuned in an rl framework employing proximal policy optimization (ppo) @xcite ss method to generate persona-aware topic guiding utterances. ## results and analysis human evaluation: ## conclusion for an open domain conversation to be successful, proper concept transition befitting the context is essential. further, persona awareness of the user may lead to user adaptive response generation, hence, resulting in more engaging and interactive conversations. therefore, a conversational agent should be able to transition the concept efficiently as well as build a rapport with the user by understanding his/her persona. to encompass both of these aspects, we proposed here a reinforced persona-aware topic-guiding conversational system (rptcs). first, we create a personaaware topic transition dataset (ptcd) by leveraging the few-shot prompt feature of the language model. second, employing gpt-2 small, we train an mle loss-based conversational model (mlcm) on ptcd. lastly, using a novel designed reward function to ensure aspects of persona, topic, and context consistency with non-repetitiveness, we fine-tune mlcm adopting ppo loss optimizer in an rl framework. automatic and human evaluation results strengthen the design and use of rewards and concludes that our proposed model rptcs achieves state-of-the-art performance compared to the strong baselines. our results also concludes that rptcs is able to retain and facilitate persona awareness, naturalness, and consistency at par in an ongoing dialogue. in the future, we would like to look into personality traits such as age, gender, etc. to model a persona-aware conversational system. when interacting with the system, if users continuously state short and direct responses such as 'yes', 'i don't know', 'no', 'i can', 'okay', then the system first tries to respond by inquiring about topic like 'restaurant', 'job', 'shopping' or 'travel' but after two or three turns starts deviating and generating out of the context or hallucinated responses. it is also seen that sometimes model starts attending persona statements of the user frequently and generate most of the time only persona awared responses which tend to be out of context. hence, the model should be forced to generate only relevant responses and persona attention should be controlled. this opens up the door for future studies to build a controlled persona aware topic guiding conversational system.