Schema-Guided Paradigm for Zero-Shot Dialog (2021)

Abstract: Developing mechanisms that flexibly adapt dialog systems to unseen tasks and domains is a major challenge in dialog research. Neural models implicitly memorize task-specific dialog policies from the training data. We posit that this implicit memorization has precluded zero-shot transfer learning. To this end, we leverage the schema-guided paradigm, wherein the task-specific dialog policy is explicitly provided to the model. We introduce the Schema Attention Model (SAM) and improved schema representations for the STAR corpus. SAM obtains significant improvement in zero-shot settings, with a +22 F1 score improvement over prior work. These results validate the feasibility of zero-shot generalizability in dialog. Ablation experiments are also presented to demonstrate the efficacy of SAM.

## introduction task-oriented dialog systems aim to satisfy user goals pertaining to certain tasks, such as booking flights @xcite , providing transit information @xcite , or acting as a tour guide @xcite . neural models for task-oriented dialog have become the dominant paradigm @xcite @xcite . these data-driven approaches can potentially learn complex patterns from large dialog corpora without hand-crafted rules. however, the resulting models struggle to generalize beyond the training data and underperform on unseen dialog tasks and domains @xcite . a long-standing challenge in dialog research is to flexibly adapt systems to new dialog domains and tasks @xcite . consider a system that has been trained to handle several different tasks (e.g., restaurant reservations, ride booking, weather, etc.). how can figure 1 : in the standard paradigm, data driven models implicitly learn the task-specific dialog policies (i.e., schemas). this precludes generalization to an unseen task at inference time. in contrast, in the schemaguided paradigm, dialog policy is explicitly provided to the model through a schema graph. at inference time, the model is given the schema for the new task and can therefore generalize in a zero-shot setting. the advent of large-scale pre-training @xcite has led to significant progress in domain adaptation across areas in nlp, including natural language understanding @xcite , open-domain dialog @xcite and language understanding for task-oriented dialog @xcite . generalization in end-to-end task-oriented dialog has proven to be significantly more difficult, particularly in zero-shot settings where there is no training data @xcite . we posit that it is inherently difficult to generalize to unseen dialog tasks be-cause of the dialog policy. traditionally, an end-to-end dialog system must perform three distinct tasks. first, it must understand the dialog history and identify any relevant user intents or slots. next, it must decide on the appropriate system action, according to a taskspecific dialog policy. finally, it must generate a natural language utterance corresponding to the system action. in a pipeline dialog system, these three steps are performed by the nlu, dm and nlg respectively @xcite . prior work has exhibited generalizability in language understanding and, to a lesser extent, in language generation. however for end-to-end dialog, the task-specific dialog policy inherently precludes zero-shot generalization. an end-to-end dialog model trained on several tasks, will implicitly learn the dialog policies from the data. however, when generalizing to a new task in a zero-shot setting, the model has no knowledge of the dialog policy for the new task. to address the difficulty of generalizing to new task-specific dialog policies and in order to facilitate zero-shot generalization, we present the schema-guided paradigm. generally, end-to-end neural models implicitly learn the task-specific dialog policies from large corpora. in contrast, in the schema-guided paradigm, we explicitly provide the task-specific dialog policies to the model in the form of a schema graph. the schema graphs define the system's behavior for a specific task (e.g., when the user provides the reservation time, ask them for the number of people). when transferring to an unseen task, the corresponding schema graph is explicitly provided to the model. this enables language understanding and the dialog policy to be decoupled. the model no longer needs to implicitly memorize the task-specific policies from the training data. instead, the model learns to interpret the dialog history and align it to the schema graph. as such, when transferring to a new task, the schema graph serves as an inductive bias that provides the model with the task-specific dialog policy. to address the challenge of zero-shot transfer learning, @xcite presented the star corpus and several baseline experiments. we extend their baselines for the task of next action prediction. we introduce the schema attention model (sam) and thorough schema representations for the 24 different tasks in the star dataset. sam obtains a +22 f 1 score improvement over baseline approaches in the zero-shot setting, validating the schema-guided paradigm and demonstrating the feasibility of zero-shot generalization for taskoriented dialog. our code and model checkpoints are open-sourced and be found at https://github . com/shikib/schema_attention_model. ## task definition we address the problem of transferring dialog models to unseen tasks and domains @xcite . this problem is especially important in real world settings. it is impossible to preconceive every dialog task that users may need (e.g., a covid-19 information dialog system). furthermore, collecting new dialog data for each new task is inherently unscalable @xcite . while rule-based/pipeline dialog systems may be easier to extend to new tasks @xcite , there is a tradeoff between the adaptability of non-neural systems and the performance of neural models. ## methods in order to enable zero-shot transfer to new dialog tasks and domains, the schema attention model (sam) is introduced. it leverages an external dialog policy representation (i.e., the schema) to predict the next system action. this section begins by describing the baseline model for the task of next action prediction. next, the schema-guided paradigm is introduced (figure 1 ). it includes a graph-based representation of the task-specific schema and sam, a model that identifies the next system action by attending to a task-specific schema representation. ## results to validate the effectiveness of sam, a number of next action prediction experiments are carried out on the star dataset @xcite . first, sam is evaluated in the standard experimental settheir original results were obtained on an older version of star, with annotation errors foot_2 that have since been fixed. ## conclusion this paper shows strong results in zero-shot task transfer and domain transfer using the schemaguided paradigm. we hypothesized that the difficulty of zero-shot transfer in dialog stems from the dialog policy. when neural models implicitly memorize dialog policies observed at training time, they struggle to transfer to new tasks. to mitigate this, we explicitly provide the dialog policy to the model, in the form of a schema graph. this paper introduces the schema attention model (sam) and shows improved schema graphs for the star corpus. this approach attains significant improvement over prior work in the zero-shot setting, with a +22 f 1 score improvement. furthermore, the ablation experiments demonstrate the effectiveness of both sam and the improved schema representations. future work may explore (1) improved schema representations to better capture dialog policy, (2) improved model architectures to better align the dialog to the schema, and (3) extensions to other problems (e.g., response generation).