Enhancing Task-Oriented Dialog System with Subjective Knowledge: A Large Language Model-based Data Augmentation Framework (2023)

Abstract: As Task-Oriented Dialog (TOD) systems have advanced, structured DB systems, which aim to collect relevant knowledge for answering user’s questions, have also progressed. Despite these advancements, these methods face challenges when dealing with subjective questions from users. To overcome this, DSTC11 released a subjective-knowledge-based TOD (SK-TOD) dataset and benchmark. This paper introduces a framework that effectively solves SK-TOD tasks by leveraging a Large Language Model (LLM). We demonstrate the proficient use of LLM for each sub-task, including an adapters-based method and knowledge-grounded data augmentation. Our proposed methods, which utilize LLM as an efficient tool, outperform baseline performance and approaches that directly use LLM as a one-step sub-task solver, showing superior task-specific optimization.

## introduction in many task-oriented dialog (tod) systems, to ensure accurate responses to user inquiries, it is often necessary to generate system responses by extracting relevant information from a preprocessed database @xcite . however, this method shows a generalization limitation, making it difficult to extend to other domains or new information. to overcome this limitation, research has been conducted to generate system responses based on factual information using frequently asked questions (faqs), a readily available source of factual information @xcite . however, dstc11 track5 1 presents a new limitation that previous research and datasets cannot handle users' subjective requests (e.g."do they have nice outdoor dining area?"). therefore, they have released a subjective-knowledge-based tod (sk-tod) dialog dataset, which allows for appropriate subjective knowledge (reviews) for user's subjective requests @xcite . dstc11 track5 defines three sub-tasks to generate appropriate system responses through the sk-tod dataset. (1) finding knowledge-seeking turns, (2) selecting knowledge (reviews or faqs) based on the dialog history of each knowledge-seeking turn, and (3) generating system responses based on the selected knowledge and dialog history. the baseline of dstc11 track5 selects optimized models for each pipeline and fine-tunes them on the given dataset. however, about 50 percent of the total dataset requires knowledge selection turns, so from task2 onwards, the dataset for the learning process decreases, which may similarly result in performance degradation. in particular, recent research shows that large language models (llms) demonstrate excellent generalization performance across various tasks @xcite , including conversation-related tasks @xcite , even in few-shot or zeroshot scenarios. however, a disadvantage of llms is that optimizing for specific tasks or domains is difficult. therefore, this paper proposes a framework that primarily fine-tunes task models suitable for each sub-task, while efficiently utilizing llms for performance improvement. by researching the use of llms ideal for the characteristics of each stage, it is shown that the mentioned limitations can be overcome by using llms and fine-tuning models appropriately. as a result, it has outperformed the benchmark baseline performance in all three subtasks of the officially released dstc11 track5 test set. to summarize the contributions of this paper: ## task description the detailed formulation of the sk-tod task in dstc11 track5 follows the paper @xcite . a formal dialog context @xmath0, s 1 , u 2 , s 2 , ..., u t ] is given between the user and the system. each user utterance u i is followed by the system response utterance ..] such as paragraphs, sentences, or sub-sentences. therefore, each sub-task can be redefined as follows: determine if the last user utterance u t in the given dialog requires knowledge access. ## comparison with llm sub-task solver we have shown how to effectively use llm to solve sub-tasks efficiently. by using a fine-tuning methodology through adapters and data augmentation methodology based on llm's knowledge, we are able to significantly increase the performance per task. our proposed methodology does not simply use llm as a one-step sub-task solver, but utilizes llm to improve the performance of the task solver model. to verify that our methodology outperforms llm directly solving tasks, we compare the performance of our proposed methodology with that of llm directly solving tasks. this is summarized in ## conclusion we propose an llm-based framework for sk-tod. in the ktd task, we fine-tune t5 using task-specific adapters. to address the challenge of relatively reduced data classified as knowledgeseeking turns in the ks and rg tasks, we employ knowledge-based data augmentation through llm. our three-stage data augmentation methodology involves last question generation, knowledge generation, and response generation for the rg task, as well as generating good and bad questions for the ks task. experimental results show that our llmbased framework achieves significantly higher performance compared to the baselines in the ks and rg tasks. furthermore, our proposed framework for sk-tod outperforms using llm directly as a task solver, indicating its promising efficiency for sk-tod tasks. further research may be required to explore its efficiency further and understand its potential advantages more comprehensively. we plan to make the augmented data publicly available, hoping to contribute to the advancement of sk-tod and future research in knowledge-based data augmentation.