Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps? (2023)

Abstract: Longitudinal Dialogues (LD) are the most challenging type of conversation for human-machine dialogue systems. LDs include the recollections of events, personal thoughts, and emotions specific to each individual in a sparse sequence of dialogue sessions. Dialogue systems designed for LDs should uniquely interact with the users over multiple sessions and long periods of time (e.g. weeks), and engage them in personal dialogues to elaborate on their feelings, thoughts, and real-life events. In this paper, we study the task of response generation in LDs. We evaluate whether general-purpose Pre-trained Language Models (PLM) are appropriate for this purpose. We fine-tune two PLMs, GePpeTto (GPT-2) and iT5, using a dataset of LDs. We experiment with different representations of the personal knowledge extracted from LDs for grounded response generation, including the graph representation of the mentioned events and participants. We evaluate the performance of the models via automatic metrics and the contribution of the knowledge via the Integrated Gradients technique. We categorize the natural language generation errors via human evaluations of contextualization, appropriateness and engagement of the user.

## introduction the state-of-the-art dialogue systems are designed for assisting the user to execute a task, holding limited chit-chat conversations with shallow user engagement, or information retrieval over a finite set of topics. the personalization in these systems is limited to a stereotypical user model. this user model is implicitly inferred from conversations with many users, or is limited to a superficial list of persona statements (e.g., "he likes dogs") @xcite . the dialogue sessions are disconnected and the shared information across sessions is negligible and close to none. longitudinal dialogue (ld) is one of the most challenging types of conversation for human-machine dialogue systems. lds are multi-session interactions that encompass user-specific situations, thoughts, and emotions. dialogue systems designed for lds should interact uniquely with each user about personal life events and emotions over multiple sessions and long periods of time (e.g. weeks). through each session in lds, the dialogue system must learn about the user's personal space of events and participants and social interactions, and engage the user in personal dialogues regarding their thoughts, feelings, and personal and world events. figure 1 shows an example of three types of human-machine dialogues: task-based, opendomain chit-chat and ld. the user dialogues with the tasked-based dialogue system consists of either independent command-and-control exchanges such as on day 1, or a task-driven dialogue such as on day 2. the user model in this system is not personal as it adopts a stereotypical model -implicitlyinferred from dialogue corpora with multiple users. in the open-domain chit-chat dialogue, the dialogue does not include the execution of any explicit task, and the model engages the user in a conversation about movies and news. a common characteristic of task-based and open-domain dialogues is the fact that there is no personal information carried to the next dialogue session. the system does not update/modify the user model with each dialogue session and the level of personalization is intact from one interaction to the other (personalization in the natural language processing and dialogue models could be added based on the voice user interface requirements and could include the exploitation of personal information such as contact directory, preferences, etc.). in contrast, the model designed for the ld must account for three main differences compared to the other two systems; a) the contents of the ld are not about general information or knowledge matters as lds encompass personal emotions, user and time-specific situations, and participants; b) the sessions are not disconnected dialogues and we can not model them as stand-alone interactions. in contrast, they belong to a multi-session interaction unique to the individual user, where the information shared in each interaction creates a common ground between the machine and the user. for each interaction, the system must engage the user in a dialogue respecting the common ground based on the information shared in the previous interactions, as well as the novel information in the new dialogue history; c) the machine has to extract the personal information presented in the user responses to construct and update the user model and respond coherently. similar to a natural interaction between human speakers, the model has to gradually become acquainted with the user throughout the dialogues and not from a superficial list of sentence-based persona descriptions. there has been limited research on personal conversations with users over a long period of time. engaging the user to elaborate on personal situations and emotions is a challenging task and designing appropriate collection/elicitation methodologies is not straightforward. as a result, research on multi-session dialogues resorts to crowd-sourcing datasets with superficial persona statements and pretended longitudinality @xcite . meanwhile, studies on lds have been limited to inferring user's attributes such as age and gender @xcite , or next quickresponse selection from a candidate set of "yes," "haha," "okay," "oh," and "nice" @xcite . in this work, we study the task of response generation in lds. response generation in lds is subject to appropriateness and accuracy as well as personalization and engagement of the user. the level of personalization in lds is beyond a set of personal preferences and can not be learned from a limited set of persona statements ("i like cars" does not necessarily imply that i like to talk about cars in my interactions). the generated response needs to respect individuals' states, profiles, and experiences that vary among users and dialogue sessions. therefore, we can not collect a massive knowledge base of user models that can suit all individuals and scenarios. the dialogue system should learn about each user and derive the individual user model through/from the previous dialogue sessions to generate a personal response that is coherent with respect to the dialogue context as well as the previous dialogue sessions. we investigate the applicability of generalpurpose pre-trained language models (plm) for grounded response generation in lds. we study whether plms can generate a response that is coherent with respect to the dialogue history and grounded on the personal knowledge the user has shared in previous interactions. we conversationally fine-tuned two recent plms, geppetto (gpt-2) (de mattei et al., 2020) and it5 @xcite , using a dataset of lds about real-life events, feelings, and situations that the user has experienced. we use the responses each individual user shared in the previous dialogue sessions with the system as personal knowledge and evaluate whether grounding the generation on such knowledge results in more appropriate and personal responses. in previously published research on grounded generation, the knowledge sequence is provided to the model as-is. in this work, we experiment with three different representations of the knowledge piece; a) raw as unprocessed text, similar to the previously published research; b) bag of head nouns as a distilled syntactic representation of the knowledge; c) graph representation of the events and participants mentioned in the user responses @xcite . an example of a dialogue and different representations of the corresponding personal knowledge is shown in figure 2 . we evaluate the performance of the models and the impact of different knowledge representations through automatic and human evaluations, as well as explainability studies using the integrated gradients technique @xcite . our contributions can be summarised as follows: grounded response generation plms have achieved comparably well performance for opendomain chit-chats @xcite , goaloriented agents @xcite and question answering @xcite . however, such models can generate inappropriate and/or generic responses which can lead to ethical problems and low user engagement @xcite . research to address this problem and improve the generation quality includes grounding the generation on external knowledge content. the selection of the knowledge source to ground the generation has been studied as an individual component @xcite , as well as a joint task along with response generation @xcite . ## evaluations the fine-tuning of the models was done using 80% of the dialogues (640 second-session dialogues, 1284 samples with different turn levels), while the remaining data was split into 10% (80 dialogues, 160 samples with different turn levels) as the validation set for parameter engineering and earlystopping, and 10% as unseen test set. each split was sampled at the dialogue level to guarantee no history overlap among splits. an example of a second dialogue session and the generated responses are presented in appendix ## conclusion we studied the task of response generation in longitudinal dialogues (ld), where the model should learn about the user's thoughts and emotions from the previous dialogue sessions and generate a personal response that is coherent with respect to the user profile and state, the dialogue context, as well as the previous dialogue sessions. we finetuned two state-of-the-art plms for italian, using a dataset of lds in the mental health domain. we experimented with grounded generation using user responses in the previous dialogue session as userspecific knowledge. we investigated the impact of different representations of the knowledge, including a graph representation of personal life events and participants mentioned previously by the user. our evaluations showed there is still a huge gap between the performance of the general-purpose plms with knowledge grounding and the ground truth. nevertheless, we observed that a) refined representations of the knowledge (such as boh and psg) can be more informative and less noisy for a grounded generation; b) the encoder-decoder model exhibited more diversity in the outputs compared to the auto-regressive model; c) knowledge grounding reduces the cases of genericness in response, though it can result in more hallucinated responses.