Selective Prompting Tuning for Personalized Conversations with LLM s (2024)

Abstract: In conversational AI, personalizing dialogues with persona profiles and contextual understanding is essential. Despite large language models’ (LLMs) improved response coherence, effective persona integration remains a challenge. In this work, we first study two common approaches for personalizing LLMs: textual prompting and direct fine-tuning. We observed that textual prompting often struggles to yield responses that are similar to the ground truths in datasets, while direct fine-tuning tends to produce repetitive or overly generic replies. To alleviate those issues, we propose **S**elective **P**rompt **T**uning (SPT), which softly prompts LLMs for personalized conversations in a selective way. Concretely, SPT initializes a set of soft prompts and uses a trainable dense retriever to adaptively select suitable soft prompts for LLMs according to different input contexts, where the prompt retriever is dynamically updated through feedback from the LLMs. Additionally, we propose context-prompt contrastive learning and prompt fusion learning to encourage the SPT to enhance the diversity of personalized conversations. Experiments on the CONVAI2 dataset demonstrate that SPT significantly enhances response diversity by up to 90%, along with improvements in other critical performance indicators. Those results highlight the efficacy of SPT in fostering engaging and personalized dialogue generation. The SPT model code is [publicly available](https://github.com/hqsiswiliam/SPT) for further exploration.

## introduction personalization in dialogue systems enhances user interaction by creating a coherent and customized experience. it involves adapting conversations to individual preferences, backgrounds, and real-time context, ensuring each dialogue feels personally relevant. this tailored approach fosters a deeper connection between users and technology, making interactions more intuitive and engaging. by understanding and anticipating user needs, personalized dialogues can offer more than just relevant responses; they provide a seamless, conversational experience that mirrors human interaction, enriching the overall quality of digital communication. personachat @xcite has become a pivotal dataset for personalization research in conversational ai, offering persona profiles that detail an interlocutor's preferences and background in four to five sentences. these profiles guide conversational agents in creating dialogues that are both engaging and consistent with the persona's characteristics and prior conversational context. this area has seen diverse approaches for enhancing personalization, such as attention mechanisms @xcite , reinforcement learning with multiple rewards @xcite , and persona profile enrichment through stories @xcite , demonstrating the breadth of innovation in making interactions more personalized and meaningful. recently, the advent of large language models (llms) @xcite has opened new avenues for dialogue generation, offering the potential for creating conversations that align with human preferences. however, fully leveraging llms to achieve the level of personalization showed in personachat is a promising yet underexplored area. currently, llms are primarily guided by direct textual prompts or through parameter-efficient fine-tuning like prompt tuning @xcite that only tunes a few virtual tokens instead of whole llms for specific tasks. however, designing personalized conversational agents with llms faces two main challenges. the primary issue lies in diverse settings in conversations, which encompass a wide array of dialogues, each characterized by unique persona profiles and varying lengths of conversation. this diversity ne-cessitates an understanding of the distinct conversational settings within the data. through textual prompting, it is hard to guide the model to generate desired responses aligned with the target texts. simply fine-tuning llms through prompt tuning without careful conversational setting analysis risks producing responses that lack specificity and depth, resulting in a generic and bland generation. secondly, another equally critical challenge arises from the limitations inherent to the datasets used for persona-based dialogue generation. typically small and lacking in diversity, these datasets can restrict the model's exposure to a wide range of conversational scenarios. when llms (e.g., llama2-7b @xcite ) are tuned through trainable soft prompts on personachat, they risk overfitting to specific persona profiles. this overfitting manifests in the model's responses, which become repetitive and overly aligned with the persona, often at the cost of dynamic and contextually appropriate interactions. although this might lead to improvements in metrics such as f1 or bleu scores, it detracts from the overall diversity and engagingness of the dialogues, undermining the model's ability to emulate authentic human conversation. to handle those two challenges when designing personalized conversations with llms, we propose a selective prompt tuning (spt) model. specifically, to tackle the first challenge, it is crucial to identify inherent data patterns without explicit annotations. to achieve this, it is intuitive to utilize a group of multiple soft prompts to handle different conversational settings when tuning the model in a parameter-efficient way. however, as previously mentioned, the annotations for the dialogue settings are missing and even hard to discover and annotate. if we naively concurrently tune all prompts without clear distinctions, this would yield only marginal differences compared with tuning one soft prompt. therefore, to build effective multiple prompts to discover the inherent data pattern inside the personalized dialogue, the proposed spt model utilizes a dense retriever to adaptively select a proper soft prompt from the soft prompt group based on the given input context. to distinguish the effectiveness of soft prompts, we utilize the loss from llms as feedback to guide the update of the dense retriever without explicit annotations. based on this, the proposed spt model could discover patterns intrinsically associated with different dialogues. in this way, the retriever and soft prompt group evolve together, benefiting from continuous interactions that enrich their capability to discriminate and generate diverse, contextually relevant responses. to address the second challenge that llm may overfit small-scale datasets such as personachat, the proposed spt method integrates two complementary mechanisms: context-prompt contrastive learning and prompt fusion learning. the contextprompt contrastive learning mechanism ensures diversity by encouraging the use of different soft prompts for varied dialogue contexts, preventing repetitive responses. concurrently, prompt fusion learning aggregates all prompt predictions during back-propagation, optimizing towards a unified output. this dual strategy not only preserves response diversity across contexts but also enhances overall model performance, demonstrating their cooperative effectiveness in tackling overfitting while maintaining the performance. by integrating the above two parts into the spt method, experiments on the convai2 dataset @xcite with llms (i.e., llama2 @xcite and opt @xcite ) not only demonstrate marked improvements in response diversity and engagingness but also indicate enhancements in other key performance metrics. quantitatively, the proposed spt model consistently outperforms baselines across models with various sizes. moreover, spt offers profound insights into different dialogue scenarios, particularly in the model's strategic prompt selection. comprehensive ablation studies highlight the adaptability of different prompts to specific dialogue contexts. overall, our contributions can be summarized as follows. ## methodology in this section, we introduce the proposed spt method. ## experiments in this section, we empirically evaluate the proposed spt model. ## ablation studies in this section, we conduct ablation studies for the proposed spt method. ## conclusion in this paper, we introduce spt, a strategic approach for personalized dialogue generation through selective prompt tuning. by jointly training a soft prompt group and a dense retriever, spt adeptly navigates various conversational scenarios automatically, enriching response diversity while improving both linguistic and neural-based metrics. experiments on the convai2 dataset highlights the capacity of spt to identify intrinsic conversational settings, showing its effectiveness in generating contextually appropriate dialogues.