Prompt Learning for Domain Adaptation in Task-Oriented Dialogue (2022)

Abstract: Conversation designers continue to face significant obstacles when creating productionquality task-oriented dialogue systems. The complexity and cost involved in schema development and data collection is often a major barrier for such designers, limiting their ability to create natural, user-friendly experiences. We frame the classification of user intent as the generation of a canonical form, a lightweight semantic representation using natural language. We show that canonical forms offer a promising alternative to traditional methods for intent classification. By tuning soft prompts for a frozen large language model, we show that canonical forms generalize very well to new, unseen domains in a zero- or few-shot setting. The method is also sample-efficient, reducing the complexity and effort of developing new task-oriented dialogue domains.

## introduction task-oriented dialogue systems in conversational ai are challenging for developers to create. the current generation of dialogue frameworks requires developers to define actions (intents) and parameters (slots) that the natural language understanding (nlu) module accepts. this is then used to populate api service calls that operate in the backend to fulfill the user request. casting natural language utterances from the user to a discrete set of intents and slots is often not very intuitive. this in turn leads to a situation where developers rely on hand-crafted rule-based grammars or a large annotated set of training samples for machine learning models to implement a given design. any change to the design of the dialogue system would then require the developers to revisit and modify the implementation which is very often a time-consuming process. in this work, we aim to make dialogue system design easier and more intuitive. manually specifying prompts suffers from sensitivity to phrasing; we get widely varying results based on how we frame the prompt. prompt tuning @xcite and p-tuning @xcite have emerged as strong alternatives to manual prompt designing and they help optimize taskspecific prompt tokens to get the best performance while keeping the language model itself frozen. in this work, we explore the task of intent classification using these large language models and ptuning. generative methods for classification tasks have not been widely adopted because generation is inherently difficult to control and utilize for further downstream tasks. using our experiments on the schema guided dialogue @xcite dataset and the virtual assistant benchmark @xcite , we show that with p-tuning we can achieve promising zero-shot and few-shot generalization capabilities to unseen domains. in the task of intent classification, the intent labels provided as part of the dataset are usually terse and rigid. generative models generalize better when intent labels are more descriptive but structured at the same. we borrow some aspects and terminology from semantic parsing to cast the intent labels to a more compositional format, known as canonical forms. in the traditional sense, canonical forms are paraphrases of the user utterances to convert them to a form that the semantic parser can operate on to output logical representations. in our use case, we loosely use the term, canonical forms, to refer to intent labels that are more descriptive than the discrete ones but are not too verbose, e.g., "transfer_money" → "transfer money to bank account". we manually frame these canonical forms and do not rely on any grammar, simplifying the approach. we observe that using such canonical forms as labels for the intent classification task allows the model to generalize better to domains that are adjacent, but not seen at train time (e.g., flight @xmath0. we also find that it is beneficial to do a two-stage p-tuning for domain adaption, i.e., once we have a p-tuned large language model on a wide set of domains, we can continue p-tuning this model on a small set of labelled samples from the target domain to allow the model to generalize better. we find that this few-shot approach works very well and this has promising implications for developers for dialogue systems; with minimal effort it would be feasible to adapt an existing model pre-trained on multiple domains to a new domain. in summary, our contributions are: ## method in this section, we describe the creation of canonical forms and the prompt tuning technique we adopt for intent classification in the task-oriented dialogue setting. ## experimental setting in this section, we describe the datasets used, the baselines we use for comparison and the evaluation metrics. ## results in this section, we review the quantitative performance of the models for intent classification. ## discussion the results on zero-shot and few-shot settings for unseen domains demonstrate that p-tuning a llm to have intents that are more verbose than discrete labels can be very helpful. in this section, we analyze the impact of the structure of canonical forms, what helps the language model generalize, how sample efficient are these language models and what all this means for a developer of chatbots and dialogue systems. ## conclusion we explore the use of large language models and p-tuning for intent classification in task-oriented dialogue systems. we show framing intent labels into more verbose forms allows lms to exploit the underlying structure better and exhibit impressive zero-shot and few-shot generalization. we also analyze how important the phrasing of the verbose forms are and how many samples are needed to get good quantitative performance. we hope that this work on using sample efficient llms serves to motivate further research in making tod systems simpler and quicker to develop.