A Few-shot Approach to Task-oriented Dialogue Enhanced with Chitchat (2024)

Abstract: Large language models (LLMs) tuned for chat have recently been adopted for few-shot end-to-end task-oriented dialogue (TOD), with some success. To further assess this method, we conduct experiments on two, more complex, task-oriented benchmarks that integrate elements of chitchat into the conversation. We enhance a few-shot baseline by adding zero-shot chitchat detection and implementing function calling for dialogue state tracking (DST). We focus on this step in the task-oriented pipeline as it comes first, and errors due to added chitchat at this stage have the most impact on end-to-end performance. We find that this prompting method shows increased resilience to mixed-mode inputs and our enhanced pipeline allows for natural inter-mode conversations, as assessed through human evaluation. Our findings also suggest that the performance gap between few-shot prompting for TOD and supervised task-specific models is narrowing.

## introduction as chat-tuned llms continue to advance in fluency and instruction-following thanks to approaches leveraging human feedback @xcite , the prospect of developing a functional and conversational tod system with a few dialogue examples becomes increasingly plausible. however, these models are predominantly evaluated using benchmarks characterized by short, explicit task-oriented requests, which do not fully reflect the complexity of more natural tod interactions that might include situational details, or preliminary chitchat @xcite @xcite . few-shot prompting for tod recent studies have explored the potential of few-shot and zeroshot prompting approaches for dst @xcite and end-to-end tod systems. @xcite pioneer the exploration of few-shot prompting for dst, with limited results. @xcite adapt a meta-learning scheme for dst to stabilize the model's ability to perform well under various prompts, using previous-generation llms suffering from limited context-lengths. @xcite and @xcite frame dst as a tablebased task, respectively inserting/deleting entries or generating sql queries over tables, using fewshot prompting. for end-to-end tod, @xcite and @xcite propose chained prompting methods for constructing task bots with minimal human effort, highlighting dst as a critical first step. indeed, errors in detecting the user's constraints directly impact database search results and the subsequently generated responses. in this paper, we focus on refining a few-shot, end-to-end approach @xcite for chitchat-enhanced tods. the main enhancements we propose are adding a chitchat detection step, determining if a turn requires a chitchat or task-oriented response, and improving dst over the original approach, by casting it as a function call generation problem. our enhancements aim to more effectively handle the conversational nuances arising from inter-mode contexts. with function calling, we capitalize on the extensive exposure of state-of-the-art llms to code during pretraining, assuming that such a prompt format is well-represented within the model's initial training dataset and will generalize well in few-shot settings. we evaluate our enhancements on two intermode benchmarks (section 3.1). we compare our dst approach with two other dst prompt variants (section 2.2) across both open-and closed-source llms. our evaluation includes both automatic metrics and a human assessment, aiming to establish the effectiveness and robustness of a few-shot prompting approach in inter-mode dialogue. ## conclusion we show that a few-shot tod baseline can be successfully extended to handle inter-mode inputs. we find that casting dst as function call generation is robust and effective across various llms, substantially outperforming other few-shot statetracking methods. human evaluation results show the promising potential of an inter-mode few-shot tod bot, overall successfully balancing chitchat and tod within a single interaction. these findings suggest a significant advancement in simply and swiftly building more versatile and natural dialogue systems. debjoy saha, bishal santra, and pawan goyal. 2022. a study on prompt-based few-shot learning methods for belief state tracking in task-oriented dialog systems. preprint, arxiv:2204.08167. timo schick, jane dwivedi-yu, roberto dessi, roberta raileanu, maria lomeli, eric hambro, luke zettlemoyer, nicola cancedda, and thomas scialom. 2023. toolformer: language models can teach themselves to use tools. in advances in neural information processing systems, volume 36, pages 68539-68551. curran associates, inc. armand stricker and patrick paroubek. 2024. chitchat as interference: adding user backstories to taskoriented dialogues. in proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), pages 3203-3214, torino, italy. elra and iccl. haipeng sun, junwei bao, youzheng wu, and xiaodong he. 2023. mars: modeling context & state representations with contrastive learning for end-to-end taskoriented dialog. in findings of the association for computational linguistics: acl 2023, pages 11139-11160, toronto, canada. association for computational linguistics. zhao tan, xiping liu, qing shu, xi li, changxuan wan, dexi liu, qizhi wan, and guoqiong liao. 2024. enhancing text-to-sql capabilities of large language models through tailored promptings. in proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), pages 6091-6109, torino, italy. elra and iccl. tom young, frank xing, vlad pandelea, jinjie ni, and erik cambria. 2022. fusing task-oriented and opendomain dialogues in conversational agents. in proceedings of the aaai conference on artificial intelligence, volume 36, pages 11622-11629. xiaoxue zang, abhinav rastogi, srinivas sunkara, raghav gupta, jianguo zhang, and jindong chen. 2020. multiwoz 2.2 : a dialogue dataset with additional annotation corrections and state tracking baselines. in proceedings of the 2nd workshop on natural language processing for conversational ai, pages 109-117, online. association for computational linguistics. xiaoying zhang, baolin peng, kun li, jingyan zhou, and helen meng. 2023. sgp-tod: building task bots effortlessly via schema-guided llm prompting. in findings of the association for computational linguistics: emnlp 2023, pages 13348-13369, singapore. association for computational linguistics.