Advancing Open-Domain Conversational Agents - Designing an Engaging System for Natural Multi-Turn Dialogue (2024)

Abstract: This system paper describes our conversational AI agent developed for the SCI-CHAT competition. The goal is to build automated dialogue agents that can have natural, coherent conversations with humans over multiple turns. Our model is based on fine-tuning the Snorkel-Mistral-PairRM-DPO language model on podcast conversation transcripts. This allows the model to leverage Snorkel-Mistral-PairRMDPO’s linguistic knowledge while adapting it for multi-turn dialogue modeling using LoRA. During evaluation, human judges will converse with the agent on specified topics and provide ratings on response quality. Our system aims to demonstrate how large pretrained language models, when properly adapted and evaluated, can effectively converse on open-ended topics spanning multiple turns.

## introduction developing artificial intelligence capable of natural, multi-turn conversations remains an ongoing challenge in ai research. while recent advances in deep learning and large pretrained language models have accelerated progress in conversational ai, accurately capturing the nuances of human dialogue over extended interactions is still an active area of investigation. although competitions like sci-chat provide a platform for evaluating conversational models through real-time human interactions, our focus is creating an agent for coherent, opendomain dialog across diverse topics and turns. our approach uses large language models finetuned on real-world podcast conversations, immersing them in the natural flow and back-and-forth that defines human dialogue. we leverage the promising lora (low-rank adaptation of large language models) @xcite architecture to equip our agent with the ability to navigate diverse topics and engage in coherent, multi-turn exchanges. beyond technical prowess, we believe these agents hold immense potential to impact fields like education and customer service. imagine a virtual tutor providing personalized learning experiences or a virtual assistant seamlessly understanding your requests and completing tasks effortlessly. however, it's crucial to acknowledge potential ethical considerations surrounding bias and manipulation in advanced dialogue systems. we're committed to developing these technologies responsibly, ensuring they contribute positively to human-computer interaction. participating in open-domain interactions, like those facilitated by competitions, provides invaluable real-world experience and crucial human feedback. these live evaluations, assessing factors like coherence, consistency, and topical relevance over extended dialogues, serve as a crucial testing ground for our models. ultimately, our hope is to contribute to the overarching goal of creating dialogue agents that can converse with humans naturally, paving the way for deeper and more meaningful human-computer interactions. ## model architecture our conversational agent is based on fine-tuning the snorkel-mistral-pairrm-dpo language model for dialogue response generation. snorkel-mistral-pairrm-dpo foot_0 is a large transformer-based language model based on mistral @xcite and fine-tuned on trained on the ultrafeedback dataset, providing a strong foundation for various natural language generation tasks, to efficiently adapt the large language model for our task, we used lora (low rank adaptation of large language models) @xcite . mmlu (massive multitask language understanding) @xcite allows fine-tuning just a small number of extra weights in the model while freezing most of the adapter module and its integration with the transformer @xcite of the parameters of the pretrained network. this saves on the computational cost and time required to retrain the entire massive model, it also mitigates catastrophic forgetting, a phenomenon where models tend to forget their original training due to excessive fine-tuning, by freezing the model's initial weights. specifically, we initialize our model with the 7b parameter version of the pretrained snorkel-mistral-pairrm-dpo model from huggingface. we then perform additional fine-tuning of this model on podcast conversation transcripts to specialize it for multi-turn conversational modeling using lora @xcite , in mmlu fine-tuning, only adapters are trained, introducing additional weights to the models while preserving the original weights and fine-tuning the newly added weights. the architecture of the fine-tuned lora @xcite model uses a modified transformer architecture with added adapter layers1. these adapter layers are inserted after the attention and feedforward stacks. the adapter layer itself has a bottleneck design: it takes the input, reduces it to a lower dimensionality representation, applies a non-linear activation, then restores the original dimensionality. this allows the subsequent transformer layer to effectively process the adapter output. for input, the dialogue history is merged with the latest human utterance using separator tokens. this merged history is fed into the model which then autoregressively predicts the next utterance. during fine-tuning, the input sequences are truncated to fit within the model's context length limitation. for longer dialogue histories, we only include the most recent utterances to provide necessary context. the model is trained to generate the next utterance in the transcript given the truncated dialogue context. during inference, we utilize the fine-tuned lora model and its tokenizer from the hugging face transformers foot_1 and peft foot_2 libraries for text generation. responses are generated based on the conversation history using nucleus sampling from the peft library with @xmath0.9 to achieve a balance of diversity and coherence in the generated text. fine-tuning this large pretrained model on podcast conversations allows us to leverage the rich linguistic knowledge in snorkel-mistral-pairrm-dpo while adapting it to generate natural, topically consistent responses in a conversational setting. the live human evaluation in sci-chat will provide invaluable feedback on how to further improve the model's conversational abilities. ## training data and data preprocessing training conversational ai requires large, diverse dialogue datasets. as suggested by the competition guidelines, we primarily utilized the freakonomics podcast transcripts dataset foot_3 for lora model training. this podcast corpus contains over 477 episodes covering economics, politics, pop culture, sports and more. the wide topical range provides natural conversational data to teach coherent, free-flowing dialogue skills. in total, the training data comprises 1 5,829 context-response pairs extracted from the podcast dialogues. on average, each dialogue contains approximately 12 turns, with 33 words per turn. the context vocabulary spans 10,194 unique terms, while the response vocabulary covers 14,550 distinct words. the combined vocabulary size is 17,338 words, indicating rich, multifaceted linguistic interactions. these metrics characterize the structural complexity and lexical diversity of the dataset, informing the training process and performance evaluation. we scraped the raw podcast transcripts using a python script proivded in this git repo foot_4 , yielding over ten thousand utterances of training data. the data was preprocessed by lowercasing, removing metadata, and filtering out very short, uninformative utterances. in addition, the base model was pretrained on the ultrafeedback dataset, originally containing 64k prompts with 4 model completions each scored by gpt-4 for quality.