“In-Dialogues We Learn”: Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning (2024)

Abstract: Personalized dialogue systems have gained significant attention in recent years for their ability to generate responses in alignment with different personas. However, most existing approaches rely on pre-defined personal profiles, which are not only time-consuming and labor-intensive to create but also lack flexibility. We propose In-Dialogue Learning (IDL), a fine-tuning framework that enhances the ability of pre-trained large language models to leverage dialogue history to characterize persona for personalized dialogue generation tasks without pre-defined profiles. Our experiments on three datasets demonstrate that IDL brings substantial improvements, with BLEU and ROUGE scores increasing by up to 200% and 247%, respectively. Additionally, the results of human evaluations further validate the efficacy of our proposed method.

## introduction recently, there has been growing interest in personalized dialogue systems @xcite @xcite @xcite . such systems are often adept at incorporating special personal characteristics into responses. consequently, they offer enhanced flexibility, enabling them to adapt more effectively to a wide range of conversational scenarios, such as personal assistants or chatbots 1 . a common practice in personalized dialogues is to condition a dialogue model on a pre-defined profile that explicitly depicts the personality traits one aims to portray with a textual description. while there have been extensive studies along this line @xcite @xcite , we explore the problem from a different angle: instead of using a brief profile to describe a person's personality, we leverage multiple conversations between the individual and others to build a personalized dialogue model. consequently, the model can generate personalized dialogues without the need for pre-designed profiles which could be both time-consuming and labor-intensive. furthermore, as dialogue history accumulates, past conversations may provide more personalized information than a static profile. we introduce in-dialogue learning (idl), a two-stage framework that directly learns persona information from dialogue sessions, and leverages the learnt insights to synthesize responses that exhibit explicit personality characteristics (cf., figure 1 ). idl comprises a mutual supervised learning (msl) stage and a deep personalized alignment (dpa) stage. the objective of msl is to equip a dialogue model with persona knowledge conveyed in dialogue sessions. to this end, one can simply select one dialogue as the target and take the remaining as the reference to perform few-shot learning to optimize the dialogue model. such a straightforward implementation, however, suffers from two major problems: (1) unified reference dialogues normally contain abundant irrelevant information to the target dialogue, which increases the difficulty of learning; and (2) incoherent transition in multiple dialogues could cause disruption in the dialogue structure. to address the problems, we propose static persona identification (spi) and dynamic persona identification (dpi) to cluster and re-order dyadic dialogues between a target person and the other interlocutors for effective idl. spi divides the dialogues of the person into multiple persona-relevant clusters, ensuring that the target dialogue can easily access inter-session personalized information from reference dialogues from each cluster. dpi further re-orders the reference dialogues by minimizing the gaps in these dialogues, which is measured by conversational edit distance (conved) @xcite . to better align responses with the target persona @xcite @xcite , we introduce direct preference optimization with criterion (dpoc), an optimization method derived from dpo @xcite to mitigate preference degradation problem with a criterion-based penalty. this approach ensures that responses are more closely aligned with the target persona learned from reference dialogues. we conduct experiments on several personalized dialogue datasets to evaluate the effectiveness of idl. evaluation results show that idl achieves performance comparable to very strong profile-based methods, without utilizing any pre-defined profile information. in comparison to traditional personalized dialogue approaches, idl demonstrates significant improvements, highlighting the benefits of leveraging large language models for personalized dialogue. furthermore, idl shows significant improvement over in-context learning (icl) when both utilize large language models, with bleu and rouge scores increasing up to 200% and 247%, respectively. this suggests that, unlike icl, which primarily learns from data samples (single-turn), idl is more effective at incorporating persona information within dialogues (multi-turn). our contributions are threefold: ## method as shown in figure 2 , in-dialogue learning (idl) involves two stages: mutual supervised learning (msl) and deep personalized alignment (dpa). in the msl stage, we propose static and dynamic persona identification to cluster and re-order the dialogues of the target person, and then organize these dialogues into an end-to-end form to perform supervised learning, endowing the model with the ability to leverage persona information within previous dialogues. in the dpa stage, we further extend the dpo algorithm with criterion (abbreviated as dpoc) to address the issue of preference degradation through the incorporation of criterion examples and penalty terms, facilitating fine-grained personalized learning. ## conclusion in this study, we introduce a framework in-dialogue learning (idl) designed for personalized dialogue generation task. unlike previous approaches, our framework directly derives persona information from dialogues without the need of pre-defined profiles and is widely applicable to llms. the efficacy of idl in producing personalized responses is validated through both automatic and human evaluation results.