Towards Optimizing Pre-trained Language Model Ensemble Learning for Task-oriented Dialogue System (2023)

Abstract: Task-oriented dialogue systems that employ external knowledge to generate informative responses have become an important field of research. This paper outlines our contribution to Track 5 of the Eleventh Dialog System Technology Challenge (DSTC11), which focuses on constructing high-performing, subjective knowledge-enriched task-oriented dialogue systems. Specifically, we investigate the complementarity of various language models to tackle the diverse knowledge selection task that involves multiple external sources. Based on this investigation, we propose pre- and post-generation model ensemble approaches to mitigate potential biases inherent in using a single model for the knowledge selection task. Finally, we utilize the consensus decoding approach to combine fine-tuned ensemble models and improve the performance of the generation system. Our system ranked 1st in human evaluation, even outperforming human annotation.

## introduction external knowledge is critical for task-oriented dialogue (tod) @xcite systems to reduce hallucinations when assisting users with specific tasks. extensive effort has been investigated into knowledge-enriched tod systems. dstc9 track 1 @xcite is the first challenge that focuses on generating system responses using external knowledge. previous works @xcite @xcite mainly utilize bert, roberta, and electra @xcite @xcite to perform knowledge selection, which achieve satisfactory result. the dstc10 track2 challenge @xcite focuses on improving the robustness of the dialogue system in the presence of automatic speech recognition (asr) recognition errors @xcite and speaker disfluencies @xcite . existing works @xcite @xcite combine data augmentation with pre-trained models like gpt2, dialogpt, and plato-xl @xcite @xcite to bridge the gap between written and spoken conversation. in contrast to the previous tasks, dstc11 track 5 proposed a more practical challenge @xcite . in this task, dialogue systems are expected to provide not only factual information but also subjective insights from different knowledge domains. additionally, a single dialogue may involve multiple knowledge snippets. nevertheless, the presence of multiple external knowledge sources, each containing different subjective intents, can confuse the dialogue system and make it difficult to generate an appropriate response. the recent development of the large language model (llm) such as chatgpt foot_0 , llama, and alpaca @xcite become popular in generation tasks. however, the performance of these models on knowledge-enriched tod systems is far from being well-studied. in this paper, we contribute to the challenge from the following perspectives: ## methodology we define the dialogue context of the t-th utterance turn as w @xmath0, • • • , w t-1 , w t }, where u is the window size of the truncated dialogue context. the knowledge snippets in the external knowledge base are defined as @xmath1, • • • , k m }, where m is the size of the knowledge base. ## conclusions in this paper, we present the details of our submission to the dstc11 track 5 challenge, where our system achieved the top ranking among 14 participating teams on human evaluation. our system not only achieved great results in the final evaluation conducted by the organizers but also demonstrated outstanding performance in the extensive experiments that we conducted. we have also explored the potential of the recent large language model on this challenging knowledge-enriched tod task. we believe that our findings and methods can contribute to the advancement of tod dialogue systems and can inspire future research in this area.