Active Listening: Personalized Question Generation in Open-Domain Social Conversation with User Model Based Prompting (2024)

Abstract: Large language models (LLMs) capable of casual conversation have recently become widely available. We hypothesize that users of conversational systems want a more personalized experience, and existing work shows that users are highly receptive to personalized questions (PQs). Question Generation tasks, however, focus on factual questions from textual excerpts. To create a PQ generator, we first identify over 400 real user interests by anonymously aggregating ~39K user models. We then populate prompt templates with these 400 interests and use an LLM to generate PQs customized to user interests. The result is PerQs, a novel corpus of ~19K question/answer pairs. We evaluate PerQs at scale in the unique context of the Alexa Prize. Our results show significant positive effects on perceived conversation quality. We then fine-tune, deploy, and evaluate PerQy, a neural model that generates PQs in real-time. When evaluated against several competitive LLM baselines, PerQy produced the most natural and engaging responses.

## introduction large language models (llms) capable of casual conversation have recently become widely available, leading to an increase in research in social open-domain dialogue @xcite @xcite @xcite inter alia. in addition, challenges like the alexa prize socialbot challenge (henceforth ap) @xcite @xcite have given real users the ability to access and evaluate spoken conversational systems in their home. we hypothesize that users of such conversational systems want a more personalized experience @xcite @xcite . user interest personalized question art (dpq) have you ever had a piece of art make you emotional or feel a strong connection to it? what was the piece and how did you feel? mythology (hyp) if you could have a conversation with any mythological character, real or fictional, who would it be and what would you ask them? cooking (wyr) would you rather cook an amazing dinner or the perfect dessert? figure 1: a deep personalized question (dpq), hypothetical personalized question (hyp), and would you rather personalized question (wyr) from the perqs corpus. ## related work user modeling, also called user profiling or personas, is used to adapt dialogue systems to individual users or user groups. many different techniques have been used to create user profiles during a conversation @xcite @xcite @xcite , with some work aiming to detect explicit self-disclosure @xcite . traditional information extraction approaches struggle with conversation because implicit self-disclosures are easily missed by direct pattern matching @xcite . other approaches base user models on big-5 personality traits @xcite or reddit personas @xcite . most similar to our work are approaches grounded in traditional user modeling techniques, e.g., rule-based extraction skimmers @xcite , regular expressions @xcite , and other slot extraction mechanisms @xcite . our user modeling mixes elements from these approaches but uses these models to generate pqs. previous work on user modeling and personalized follow-up content in open-domain dialogue systems has personalized both topic-level and subtopic-level interests @xcite @xcite @xcite @xcite . other work has used user models to affect follow-up content and ask more personal questions to increase rapport @xcite @xcite @xcite . none of this work has produced the training data needed to fine-tune a model that generates pqs in real-time. previous work on question generation (qg) focused on fact-based questions whose answers can be found in text excerpts, e.g., wikipedia and gutenberg @xcite @xcite , or domain-specific questions targeted at a specific information need @xcite . other conversational information-seeking question corpora, e.g., ccpe-m @xcite , elicit user preferences to inform a recommender system, while the pqg task is not intended to recommend or sell any product or service. resources related to pqg are subsequently unsuitable for recommender tasks or tasks attempting to persuade users to take some action. ## generating personalized questions our personalized questions (pqs) are generated by instantiating prompt-templates with user interests, such as hobbies, topical interests, and other relevant sub-topic attributes, e.g., their favorite movie genre or pet's names. the user model does not store any personally identifiable information except for detecting if the user is a child, which is necessary to guarantee age-appropriate content. to personalize the experience to each specific user, we focus on extracting values most representative of the user's interests, including common hobbies and activities (e.g., dancing), popular topics (e.g., movies), esoteric interests (e.g., mythology), and sub-topic information (e.g., their favorite genre of music is pop). we instantiate prompt templates with these interests and use an llm to generate three types of personalized questions: would-you-rather (wyr), hypothetical (hyp), and deep personalized (dpq) questions. wyr presents the user with two options, while hyp is designed to be open-ended. these strategies are based on prior work that shows that they increased open-domain dialogue system performance @xcite . they are intended to build rapport between the system and the user @xcite @xcite . dpqs differ by focusing on long user-centric questions that aim to increase engagement and intimacy by provoking long responses @xcite . we used openai's chatcompletion api with the gpt-3.5-turbo model @xcite . the generation pipeline is illustrated in appendix c. we experimented with different prompts; the final prompts are listed in appendix b. we provided one entire prompt/response sequence as context to ensure high-quality outputs. providing additional context led to longer dpqs. this distinction is shown in appendix b. @xcite if you had the chance to travel to any planet in the universe, which one would you choose and why? [u1] maybe neptune @xcite opting for neptune would be captivating. its striking blue color, icy winds, and position as the farthest planet from the sun make it a mysterious and intriguing destination. if i had the chance to travel to any planet in the universe, i would choose mars. figure 3 : the dialogue policy changes when a predicted user answer is detected. the cyan segments are the system's answer-specific acknowledgment, while the yellow segment indicates the system's answer. when the user provides a predicted answer in u1, the system prefixes the answer-specific acknowledgment to a shortened version of its answer. otherwise, as in a8, the system prefixes a short acknowledgment before providing a longer answer. we then feed the generated pqs back into gpt-3.5 to generate the utterances needed for the dialogue policy, e.g., figure 2 . namely, we generate predicted user answers in order to generate answer specific acknowledgments, which are useful for signaling system understanding and increasing engagement @xcite @xcite . figure 3 shows how the acknowledgments dialogue policy changes when a predicted answer is detected. we also generate system answers to the pqs so that the system can self-disclose and support a dialogue policy that includes an exchange of opinions. while wyr pqs predict the two presented choices, content type samples flagged mtld personalized questions 18,901 0.74% 48.2 system answers 18,901 1.32% 86.3 user acknowledgments 114,328 1.39% 82.0 we evaluated the quality of perqs manually. two native english-speaking authors and one external annotator annotated 300 randomly sampled pqs (100 of each type) from perqs and found that 97% of samples were safe, high-quality, and matched the target interest. samples that did not meet this criterion were arguably too vague for the target interest, e.g., a question about "books" but not specifically "action books". we post-processed the generated perqs using a list of keywords to remove inappropriate content. after this process, canary @xcite , a dialogue safety detection module that predicts whether there is risk, @xmath0.4% of pqs as need_intervention. when inspected, most flagged pqs are innocuous (appendix g). we also calculate the lexical density via mtld to estimate the variation in topics, which is an important corpus characteristic @xcite . ## personalized question generator we create the personalized question generator, perqy, by fine-tuning redpajama-incite-base-3b-v1 (ai, 2023; @xcite with perqs and template-based instruction prompts populated with user interests. the fine-tuning process is shown in figure 22 . we employed 4 nvidia a5000 graphics cards, setting a learning rate of 2e-5 and a batch size of 8. to optimize training time (∼ 6 hours), we applied the parameter efficient finetuning method, lora (hu et al., 2021a), with a standard 85/15% data split. each training target specifies the pq type and a single user interest. then, during conversations with real users, the template-based instruction prompt is filled in real time with user interests extracted from the user model. a diagram demonstrating this process can be found in figure 23 . we integrate perqy in two states of the dialogue policy with the goal of utilizing it frequently to acquire a good sample of real-time test generations. the first use of perqy in the policy occurs just after the system's user model has acquired values for @xcite what do you like to do on weekends. we also found that perqy's pqs were statically significantly longer than perqs' pqs (on average, 16.52 words vs. 11.46 words); possibly this length difference is a result of perqy being able to mix multiple interests into a single question -a level of flexibility that is infeasible for pre-generated pqs. both groups of pqs maintain a similar mtld lexical density (54.73 vs. 55.77, respectively). despite being compact, perqy produces nearly as lexically diverse content as the much larger gpt-3.5. z-score human evaluation. in the second human evaluation, we assess the effect on the associated topical flows before and after integrating perqy. we adopt the paradise evaluation methodology to predict each topic's score based on user ratings @xcite . we report an increase in z-score from 3.13 to 3.84 for animals and .30 to .96 for hobbies (follows the introduction's pq). this difference is suggestive rather than significant; these topical flows already perform well and ask questions in their sub-dialogues, so the fact that adding a single new pq improves performance at all suggests perqy's possible impact. ## conclusion and future work personalized question generation (pqg) is a unique task focused on generating pqs in conversations. we use an llm to generate perqs, a corpus of ∼19k personalized questions and answers based on real user interests. we evaluate perqs at scale using a robust open-domain dialogue system competing in the amazon alexa prize socialbot grand challenge. we report statistically significant improvements in user engagement, intimacy, and perceived conversation quality. we used perqs to fine-tune a redpajama 3b-based pq generator, perqy. our results show that perqy is significantly better than a set of competitive llm baselines. we believe that providing perqs, perqy, and our evaluation corpus will be very useful to the spoken dialogue community. while perqy demonstrated surprising adaptability when combining interests and supporting unseen topics, future work should explore approaches enabling perqy to combine increasingly dissimilar interests and account for different aspects of the user model and multi-session conversations. we report that vicuna-33b outperforms perqy regarding consistency and engagement when evaluated by mechanical turkers. this highlights the importance of future work to explore larger models @xcite , as well as combining perqs and dialogue snippets for incontext learning.