A Difference-aware Ensemble Method for Task-oriented Dialogue with Subjective Knowledge (2023)

Abstract: We participate in the 11th Dialog System Technology Challenges (DSTC) track-5 called Task-oriented Conversational Modeling with Subjective Knowledge. Introducing subjective knowledge into task-oriented dialogue (TOD) can help the DS to understand variables of subjective user needs and to suit more dialogue scenarios. Track-5 includes several sub-tasks: 1) knowledge-seeking turn detection; 2) knowledge entity tracking; 3) knowledge entry selection; and 4) use of the selected knowledge entries for response generation. Besides the challenges of each sub-tasks own, there are two challenges across different sub-tasks. The first is that there are multiple valid knowledge entries for each knowledge-seeking turn, the accuracy of the knowledge entry selection is important for the quality of response generation. The second challenge is how to address the unseen dialogue/entities/entries in the validation and the test set. In this paper, we propose a difference-aware ensemble method to address these sub-tasks and the two challenges mentioned above. Our method helps to obtain more robust results and performs well on unseen instances. Among all the submissions for the test set, our method ranks 1st on the knowledge-seeking turn detection task and achieves 3rd on the overall automatic evaluation score. Our code and data will be released on GitHub.

## introduction task-oriented dialogue systems (tods) aim to assist users in domain-specific tasks, such as booking a traveling ticket or a restaurant. the tods is usually based on a domain-specific api or database (db) that can provide the required information for accomplishing the task, such as the price of a ticket or the location of a restaurant. however, these apis or dbs usually do not contain information for subjective user requests (e.g., "is this a good place for meeting colleagues?" or "does the hotel have a good atmosphere?"). tods trained with these apis/dbs can only handle a limited range of scenarios. to address this issue, @xcite propose a novel task called subjective-knowledgebased tod (sk-tod). one sk-tod example is shown in the track-5 (called task-oriented conversational modeling with subjective knowledge) challenge can be divided into four sub-tasks @xcite : 1) knowledge-seeking turn detection. it takes the dialogue context as input and decides whether the response to this context requires selecting external knowledge; 2) knowledge entity tracking. it selects entities from the entity pool of the subjective knowledge; 3) knowledge entry selection. it selects knowledge entries related to the selected entity from the subjective knowledge; 4) response generation uses the selected knowledge entry for generating a response. the knowledge entity tracking and entry selection sub-tasks are performed only when a response turn is detected as a knowledge-seeking turn. each of the four subtasks has its own challenge. however, there are two main challenges in this competition that across different sub-tasks. the first is that there are multiple related subjective knowledge entries for each knowledge-seeking turn. this affects all knowledge entity tracking, knowledge entry selection, and response generation. the second is that there is a big portion of unseen instances in the validation and test set (see in this paper, we propose a difference-aware ensemble method to address the two challenges across sub-tasks. for each sub-task, we start by training a group of models that are good at specific abilities (e.g. unseen instances, noisy environment). then we aim to combine the advantages of each model for different sub-tasks and for unseen instances. in other words, we need to assign a reasonable weight for each model based on the desired metric. to this end, we need to jointly consider three differences: 1) the different requirements of each sub-task; 2) the difference between each trained model; 3) the difference between the validation set and the test set. our ensemble method can automatically balance these differences among tasks/models/datasets 2 there are also a set of faqs besides the subjective review knowledge. however, the dialogue is mainly grounded on subjective knowledge. please refer to ## related work we introduce related work in this section, including knowledge-grounded conversation, task-oriented dialogue, large language models, and ensemble methods. ## our method in this section, we first introduce how we train the models for each sub-tasks in dstc 11 track-5, then introduce the difference-aware ensemble method. ## results and analysis in this competition, each team has up to five submission opportunities on the final test set. we report the submission results and also provide the performance on the validation set. ## conclusion we participated in track-5 of the 11th dialog system technology challenges (dstc 11), called task-oriented conversational modeling with subjective-knowledge. the task includes four subtasks and we propose a difference-aware ensemble method to address two main challenges in this competition. the first challenge is there are multiple reasonable knowledge entries for one context. the second challenge is that the data distribution is different in train/validation/test sets. we first train several expert models that are good at certain aspects for each sub-task. then we use the difference-aware ensemble method to balance the abilities of expert models. experimental results verify the effectiveness of our method and we got third on the final test set. future work includes but is not limited to 1) testing our ensemble method on the knowledge entity tracking task; 2) designing a method to automatically learned the threshold in algorithm 1.