Synergizing In-context Learning with Hints for End-to-end Task-oriented Dialog Systems (2024)

Abstract: End-to-end Task-Oriented Dialog (TOD) systems typically require extensive training datasets to perform well. In contrast, large language model (LLM) based TOD systems can excel even with limited data due to their ability to learn tasks through in-context exemplars. However, these models lack alignment with the style of responses in training data and often generate comprehensive responses, making it difficult for users to grasp the information quickly. In response, we propose SyncTOD that synergizes LLMs with task-specific hints to improve alignment in low-data settings. SyncTOD employs small auxiliary models to provide hints and select exemplars for in-context prompts. With ChatGPT, SyncTOD achieves superior performance compared to LLM-based baselines and SoTA models in low-data settings, while retaining competitive performance in full-data settings.

## introduction the rise of large-language models (llms) has progressed the field of nlp by leaps and bounds @xcite . pre-trained over massive data, llms work remarkably well with just in-context learning for many nlp tasks like natural language inference, summarization, and dialogs @xcite @xcite . one specific domain within dialogs where llms show promise is in building task-oriented dialogs (tod) systems, where they generate agent responses based on the dialog history and taskspecific knowledge. tod systems, in general, can be divided into two types: modular @xcite and end-to-end @xcite . modular systems require domain experts to define dialog states and annotate each train dialog with state annotations. unlike modular, end-to-end systems do * *work done when author was at iit delhi. not require such expensive state annotations. in this work, we focus on end-to-end tod systems. existing end-to-end task-oriented dialogue (tod) systems perform well when a reasonable number of training dialogues are available. however, in many practical scenarios, only a limited number of expert-curated training dialogues are available. figure 2 showcases the performance of maker, a state-of-the-art (sota) end-to-end tod model, on varying training data scales. when training data is limited, existing tod approaches fail to learn the underlying task effectively, leading to a performance drop. in contrast, large language models (llms) with in-context learning perform better than supervised models when the training dataset is limited. the inherent reasoning capabilities of llms help them learn the associated task with just a few examples. unfortunately, llm-based tod systems do not align well with the language and style in the training dialogs, often generating overly comprehensive responses. this alignment is crucial, particularly in scenarios like in-car voice assistants, where responses must be concise and easily consumable without causing distraction. as an illustrative example, see the responses generated by various models in contributions: we propose synergizing in-context learning with hints for tod (synctod), that aligns llms with the stylings of the available training data. in particular, it trains auxiliary models to provide llms (accessed via an api) with hints (such as expected entity types in the response and response length) on how to phrase the response; selecting exemplars conditioned on these hints further improves the alignment of the responses. on three publicly available datasets, synctod consistently outperforms both vanilla prompting and sota supervised models in low-data settings while maintaining competitive performance compared to supervised models in full-data settings. our code is available at https://github.com/dair-iitd/ synctod . ## related work conventional tod systems follow a modular design @xcite @xcite and require annotations for dst, pl and nlg. this work, however, focuses on end-to-end tod systems @xcite @xcite @xcite that alleviate the need for annotations by directly predicting the response given dialog history and knowledge base (kb). though llms have been explored for tod tasks @xcite @xcite , to the best of our knowledge, we are the first to explore them in an end-to-end setting. directional stimulus prompting (dsp), an approach closer to ours, uses keywords and dialog acts as hints for summarization and response generation tasks, respectively @xcite . however, unlike dsp, synctod uses multiple hints -entity types, response length, and dialog closure -relevant to the tod task. further, synctod also uses these hints to improve the incontext exemplars' quality using a retrieve-rerank approach. a natural approach for combining training data with in-context learning is via retrieval-augmented generation (rag) @xcite . here, a retriever model infuses llm input with exemplars from the training that are similar to the test sample @xcite @xcite . although out-of-box retrievers work reasonably well @xcite , many recent works strive to improve the retriever model further. @xcite employ reward-based and contrastive learning to improve retrieval quality. specifically, they use llms to obtain soft rewards to fine-tune the retriever model. @xcite fused multiple retriever models learned from training data with llms for knowledge-based question-answering tasks. what sets synctod apart from rag is its use of hints not only for selecting the informative exemplars but also for steering llm generation from within the prompt. ## synctod let @xmath0, a 1 , u 2 , a 2 , ..., u j ] be a user-agent dialog history with u and a being user and agent utterances respectively. let @xmath1. the task of a tod system is to predict the next system response ŷ given the dialog history c and a knowledge base (kb) k associated with the user's task. let @xmath2, k i , y i )} n @xmath3. in the in-context learning setup, an llm is queried (via api) with an input prompt containing task instructions, a few exemplars, and (c, k) to generate ŷ. a popular technique for leveraging train dialogs in the in-context learning setup is retrieval augmented generation (rag) @xcite . in rag, the exemplars that are most similar to c are retrieved from d and are used for generating ŷ. our proposed approach, synctod, synergizes incontext learning of llms with hints to better align with agent utterances in the training data d. figure 1 shows the overall architecture. synctod has two main components: hint predictors and exemplar selector. the hint predictors output a set of hints ĥ given the dialog history c. these hints are domainagnostic clues, such as the entity types that should be included in the response and the length of the response, that can guide the generation to follow the same style as the train dialogs. the second component, exemplar selector, first retrieves relevant exemplars from d based on c, and then re-ranks the retrieved exemplars based on ĥ. both these components are aimed at aligning the language and style of llm responses to agent responses in the train dialogs d. as the gold responses y are available for the exemplars, we simply infer the corresponding hints from y and add the hints to the exemplars. the predictors are only used to infer hints for the given input dialog with history c. please refer to appendix h for the exact prompt. ## experimental setup datasets for our evaluation, we use the multi-woz2.1 @xcite , stanford multi-domain (smd) @xcite , and bitod (english) (lin et al.) multi-domain datasets. appendix a provides additional details about the datasets. baselines: we compare synctod against the recent baselines -glmp @xcite , fg2seq @xcite , cdnet @xcite , uni-fiedskg @xcite , and maker @xcite . we also compare against rag with baai/bge-large-en-v1.5 model for exemplar retriever. further, we report the performance of chat-gpt (gpt-3.5-turbo) and gpt-4(gpt-4-0613) in a standard few-shot setting with fixed exemplars foot_0 . training details for hint predictors and retrieval of synctod are in appendix d. evaluation metric: for evaluating model performance, we use the entity f1 @xcite and bleu @xcite metrics prevalent in the endto-end tod paradigm @xcite @xcite @xcite . ## results full data setting: unlike entity f1, synctod variants perform poorly in bleu. entity f1 measures whether the system response includes relevant entities from the kb and dialog history. whereas bleu computes n-gram precision between the system response and the gold response. notably, a system response that includes all relevant entities, can still receive a low bleu score due to differences in phrasing. we find that synctod responses are meaningful and include relevant entities, resulting in good entity f1 scores. however, they use different phrasing and have less lexical overlap with gold responses, leading to lower bleu scores. we verify the quality of synctod responses via human evaluations. ## conclusion we propose synctod that leverages llms for endto-end tod. given a dialog history and kb, sync-tod obtains hints about the expected response using auxiliary models. it then uses predicted hints to retrieve quality exemplars and guide llms toward the desired response. with automatic/human evaluation, we showed that synctod outperforms the sota baseline models. further, synctod showcases a strong performance in the low-data setting. we release code for future research at https://github.com/dair-iitd/synctod .