Ensemble Method via Ranking Model for Conversational Modeling with Subjective Knowledge (2023)

Abstract: This paper describes our submission to the fifth track of the 11th Dialog System Technology Challenge (DSTC-11), which focuses on “Task-oriented Conversational Modeling with Subjective Knowledge”. We focus on response generation and leverage a ranking strategy to ensemble individual models of BART, Long-T5, and a fine-tuned large language model based on LLaMA. The strategy is supplemented by other techniques like low rank adaptation to maintain efficient utilization of these large models while still achieving optimal performance. The experiments show that the ensemble method outperforms individual models and the baseline method. Our model was ranked 1st place in ROUGE_1, 2nd place in ROUGE_L score and 4th place in human evaluation among a total of 14 participating teams.

## introduction the task of developing effective and engaging taskoriented dialogue systems has been a subject of ongoing research and development in the field of natural language processing (nlp). traditional approaches to task-oriented dialogue systems have focused on providing information and performing actions based on predefined rules or accessing backend databases or apis. however, user requests may sometimes require information outside the scope of these structured resources, requiring the system to incorporate additional domain knowledge from external unstructured sources. past challenges such as dstc9 track 1 @xcite and dstc10 track 2 @xcite have proposed the use of unstructured knowledge from faqs to build a knowledgegrounded task-oriented dialogue system. the tasks proposed in dstc11 track 5 build off past work by including subjective knowledge from customer reviews, which also involves addressing the complexity of differentiating multiple aspects within a single review post, conflicting opinions, and comparison requests between entities. we primarily focus on the third sub-task of this challenge, which involves response generation. our approach combines the utilization of large foundational language models with a ranking strategy for response prioritization. additionally, we employ the method of low rank adaptation (lora) @xcite to keep resource usage low while training and using such large models. to enhance generation performance, we employ an ensemble technique that combines multiple individual models, leveraging the capabilities of @xcite in the process. our final model achieved impressive rankings in the evaluation metrics. it secured the 1st place in rouge_1, the 2nd place in rouge_l score, and the 4th place in human evaluation. this demonstrates the effectiveness and competitiveness of our approach in the challenge. ## related work several methods of improving knowledgegrounded task-oriented dialogue systems have been explored by participants in past dstc challenges. @xcite implemented a noisy channel model to separate the tasks of generating a response and incorporating knowledge by reranking multiple candidate responses. @xcite and @xcite both use large pre-trained language models exceeding 1 billion parameters to generate the final response. the former focuses on incorporating structured knowledge from multiwoz 2.2 @xcite and negative sampling for the knowledge selection task, while the latter explores several approaches to generating more synthetic dialogue data. the scaling up of large language models in recent years has continued to show gains in terms of performance and abilities @xcite . however, the prohibitive cost of fine-tuning all the parameters in these models has led to the exploration of parameter efficient fine-tuning strategies, such as adapter layers @xcite , prompt tuning @xcite @xcite , and lora @xcite . ## task description the aim of this challenge track is to generate a relevant response given a set of user and agent turns, and a knowledge base which may contain information relevant to the user's query. this is further divided into 3 sub-tasks: the training data provided for this challenge includes 28,431 dialogues, of which 14,768 are knowledge-seeking turns which will be processed by the knowledge selection and response generation steps. a knowledge base containing subjective knowledge in the form of user reviews and objective knowledge in the form of faqs is also provided. the dialogues are augmented from mul-tiwoz 2.1 to include additional knowledge-seeking turns which require information from the knowledge base. all dialogues and knowledge snippets belong to either the hotel or restaurant domain. ## methodology figure 1 depicts the overall architecture of our methods. ## results and analysis we present our experimental results and analyses in this section. ## conclusion in this work, we demonstrated an ensemble method that utilizes a ranking strategy to combine outputs from various large models for response generation. our submission to the dstc11 track 5 challenge achieved the highest rouge_1 score and secured the 2nd place for rouge_l score in sub-task 3, despite relatively weaker performance in sub-task 1 and 2. furthermore, we conducted an analysis on domain masking for sub-task 1 and 2 to assess performance in handling requests from unseen domains. our findings highlight the need for further exploration to improve performance when the system encounters domains that significantly differ from the training data.