Leveraging Ensemble Techniques and Metadata for Subjective Knowledge-grounded Conversational Systems (2023)

Abstract: The goal of DSTC11 track 5 is to build task-oriented dialogue systems that can effectively utilize external knowledge sources such as FAQs and reviews. This year’s challenge differs from previous ones as it includes subjective knowledge snippets and requires multiple snippets for a single turn. We propose a pipeline system for the challenge focusing on entity tracking, knowledge selection and response generation. Specifically, we devise a novel heuristic to ensemble the outputs from the rule-based method and neural model for entity tracking and knowledge selection. We also leverage metadata information in the knowledge source to handle fine-grained user queries. Our approach achieved the first place in objective evaluation and the third place in human evaluation of DSTC11 track 5.

## introduction task-oriented dialogue (tod) systems aim to provide users with specific services, such as making restaurant reservation or booking hotel. typically, these systems utilize domain-specific apis, which are limited to predefined functionalities. however, there are scenarios where a user's request go beyond the coverage of domain api. for instance, the user might ask about alcohol availability at a restaurant or the availability of free wifi at a hotel. to handle such cases, tod system that can refer to external knowledge base, or knowledge-grounded tods, has drawn much attention @xcite . previous studies on knowledge-grounded tods have focused on factual knowledge, such as faqs about restaurants or hotels. in practice, users usually show interest not only in factual information but also in subjective information, involving the experiences, opinions, and preferences of other customers @xcite . to address this aspect, dstc11 track 5 specifically focuses on building a dialogue system that can effectively utilize subjective knowledge snippets. the subjective knowledge snippets consist of customer reviews concerned with hotels and restaurants, such as "if only the room was a bit cleaner, it would have been perfect" and "we loved the atmosphere from the moment we walked in." furthermore, in some cases, multiple knowledge snippets need to be retrieved in order to adequately respond to a single user request, adding an additional layer of complexity to dstc11 track 5. to this end, we propose a pipeline system that addresses four subtasks of knowledge-grounded tod: (1) knowledge-seeking turn detection, (2) entity tracking, (3) knowledge selection, and (4) response generation. for entity tracking, we propose a novel heuristic that combines the results of rule-based and neural entity matching models. in knowledge selection, we leverage given metadata to precisely retrieve relevant review snippets when the user's request is highly specific, such as asking about a particular menu of a restaurant. for response generation, we propose a data augmentation method using a large language model (llm) to fine-tune the response generation model, enabling it to handle cases with mixed opinions more effectively. overall schema of our pipeline is illustrated in figure 1 . our approach demonstrates significant improvement over baseline method in terms of the accuracy of knowledge selection and the quality of response generation. furthermore, our submission achieved the top in objective evaluation and the third place in human evaluation of dstc11 track 5. the overall task can be divided into four sequential subtasks, which are as follows: knowledgeseeking turn detection, entity tracking, knowledge selection, and response generation. knowledgeseeking turn detection determines whether subjective knowledge is required to generate the response based on the dialogue context and the user's last utterance. entity tracking is the task of identifying the entity relevant to the user's last utterance, which helps reduce the number of candidate knowledge snippets. knowledge selection involves finding relevant knowledge snippets to user's request from the narrowed-down knowledge pool (reviews, faqs) obtained by the entity tracking. finally, response generation is the task of generating a response of the user's request grounded on the selected knowledge snippets. given that the knowledge snippets often have conflicting opinions about entities, it is crucial to reflect both positive and negative reviews whenever possible. the challenge dataset is structured in the following way. firstly, it includes a dialog dataset that contains conversations between a user and a system. secondly, there is a knowledge dataset comprising reviews and faqs for each domain/entity. in this context, "domain" refers to "hotel" and "restaurant", and "entity" refers to specific names such as a hotel name (e.g. cityroomz) or a restaurant name (e.g. rosa's bed and breakfast). each entity has its own set of review documents and faqs. the review documents comprise sentences and also store metadata that offers additional information about each review, such as the type of traveler (e.g. couples, etc.), specific dishes (e.g. beef wellington, etc.), and beverages (e.g. beer, ale, etc.). the faqs are stored as pairs of questions and answers. ## method in this section, we explain the methods for each subtask: knowledge-seeking turn detection, entity tracking, knowledge selection, and response generation. the overall pipeline of our subtask is described in the figure 1 . ## related work task-oriented dialogue systems are conversational systems designed to solve specific tasks or purposes requested by users. traditional tod systems have solved these problems by dividing them into subtasks using a pipleline architecture @xcite . the pipeline architecture consists of natural language understanding (nlu) @xcite , dialogue state tracking (dst) @xcite , dialogue policy @xcite , and natural language generation (nlg) @xcite ) modules. by solving these four modules sequentially, the system generates dialogues that fulfill the user's objectives. endto-end neural models for task-oriented dialogue (tod) systems, which build a conversational system using a single unified model, have also been continuously researched and developed @xcite @xcite . task-oriented dialogue systems have predominantly relied on pre-defined application programming interfaces (apis) and structured databases to generate user-specific responses @xcite . however, this approach has limitations as it relies on having prior knowledge of user needs and encounters challenges in storing all the necessary ## conclusion in this work, we proposed a pipeline system for dstc11 track 5: task-oriented conversational modeling with subjective knowledge, mainly focusing on entity tracking, knowledge selection and response generation. our approach involves the following methods: (1) an heuristic ensemble method to combine the outputs from the rule-based method and neural model, (2) a knowledge selection method based on metadata information, and (3) data augmentation method to tackle mixed opinions in review snippets. as a final result, our submission ranked the top in objective evaluation and the third place in human evaluation.