LAD : Language Models as Data for Zero-Shot Dialog (2022)

Abstract: To facilitate zero-shot generalization in task-oriented dialog, this paper proposes Language Models as Data (LAD). LAD is a paradigm for creating diverse and accurate synthetic data which conveys the necessary structural constraints and can be used to train a downstream neural dialog model. LAD leverages GPT-3 to induce linguistic diversity. LAD achieves significant performance gains in zero-shot settings on intent prediction (+15%), slot filling (+31.4 F-1) and next action prediction (+10 F-1). Furthermore, an interactive human evaluation shows that training with LAD is competitive with training on human dialogs.

## introduction a long-standing goal of dialog research is to develop mechanisms for flexibly adapting dialog systems to new domains and tasks @xcite . while the advent of large-scale pre-training @xcite @xcite has brought about significant progress in few-shot and zero-shot generalization across many different problems in natural language processing @xcite , zero-shot generalization in task-oriented dialog remains elusive. a likely reason for this discrepancy is that dialog models require significant data because they need to learn task-specific structural constraints, such as the domain ontology and the dialog policy. while large language models (e.g., gpt-3) exhibit strong language understanding and generation abilities @xcite , they have no a priori knowledge of the structural constraints implied by a specific (unseen) problem setting (e.g., relevant intents, dialog policy, etc.). as such, in order to adapt a pre-trained lm for task-oriented dialog, it is necessary to impose structural constraints on the unstructured figure 1 : prompting must convey the structural constraints through a natural language prompt. in contrast, lad uses large lms to induce diversity in a synthetic dataset. as such, lad conveys structural constraints through both the synthetic data and the inductive biases in the downstream problem-specific models. given the in-context meta-learning abilities of large lms @xcite , prior work has explored prompt-engineering or prompt-tuning @xcite @xcite . well-designed prompts can convey the necessary structural constraints. how-ever, it is challenging to express complex constraints (e.g., a dialog policy) in natural language. prompting also precludes inductive biases in the model (architecture, training algorithm, etc.) and over-relies on the meta-learning abilities of large lms. as such, there is a tradeoff between prompting large lms (i.e., generalizable nlu and nlg) and fine-tuning smaller lms (i.e., problem-specific inductive biases, efficiency). a potential interpretation for the strength of large lms is that they learn the distributional structure of language @xcite ) by observing web-scale data @xcite . motivated by this interpretation, this paper proposes language models as data (lad). lad is a novel paradigm in which large lms are used in a zero-shot domain-agnostic manner to induce linguistic diversity in synthetic data. given a minimal expression foot_0 of the structural constraints (henceforth referred to as a schema), lad (1) creates a seed synthetic dataset using domain-agnostic algorithms, (2) leverages large lms to reformulate utterances, and (3) validates the resulting data to ensure adherence to the schema. the resulting synthetic data, which is sufficiently diverse and expresses the necessary structural constraints, can be used to train neural dialog models. in contrast to prompting, lad facilitates zero-shot generalization by (1) leveraging the sophisticated abilities of large lms (knowledge of the distributional structure of language) to induce linguistic diversity in the synthetic data while (2) maintaining inductive biases (motivated by the structural constraints) in the problem-specific model architectures. the challenge of creating synthetic data that is indistinguishable from human-annotated data, both in its expression of structural constraints and in its diversity, is highly impractical @xcite . instead, the goal of this work is to create synthetic data that is sufficient to train a sample-efficient and robust model. therefore, the claim of this paper is that lad can create synthetic data, conditioned on a minimal expression of structural constraints (i.e., a schema), that can be used to train robust and sample-efficient neural models and induce performance gains in zero-shot settings. to validate this claim, lad is applied to three problems in dialog: intent prediction, slot filling and next action prediction. next action prediction is particularly difficult in zero-shot settings since the structural constraints include the dialog policy. lad demonstrates significant gains across five datasets (+10 to +30 improvements on f-1 and accuracy) in zero-shot settings when evaluating on human-annotated corpora. to further validate the efficacy of lad, an interactive evaluation with humans (over 1600 dialogs) is performed. the results of this interactive evaluation suggest that lad can yield performance comparable to training on human dialogs. the claim of this paper is validated empirically across multiple datasets. lad is shown to generate diverse and accurate synthetic data, which is subsequently used to train neural dialog models and facilitate zero-shot generalization. ## definitions zero-shot generalization can be conceptualized as imposing structural constraints on the unstructured representation space of a pre-trained model, using a given schema (i.e., minimal expression). we begin with a neural network, m, with general language understanding abilities and limited knowledge of task-oriented dialog (e.g., bert @xcite ). the necessary structural constraints that m must learn are implied by the target dialog setting, i.e., the problem (e.g., next action prediction), the domain (e.g., restaurants) and the task (e.g., restaurant reservation). these structural constraints conceptually define the desired properties for the representations of m, i.e., what must be learned by m. in a full-shot setting, the constraints are conveyed by a human-annotated dataset and thereby learned through supervised learning. in contrast, the goal in zero-shot generalization is to learn these structural constraints from a minimal expression, i.e., a schema. the following sections formally define structural constraints and schemas. throughout this paper, zero-shot refers to a setting wherein the only human-annotated data is the schema. since a schema is a minimal expression of the necessary structural constraints, we argue that it is impossible to use less data, without making assumptions about the prior knowledge of a pretrained model. such assumptions would limit the generality of a method for zero-shot generalization. ## lad: language models as data despite exhibiting strong language understanding and generation abilities @xcite , large lms have no a priori knowledge of the structural constraints of task-oriented dialog. furthermore, imposing the necessary structural constraints on large lms is impractical due to (1) the difficulty of fine-tuning (cost, computation, immutable architectures) and (2) the limitations of natural language prompts. as such, language models as data (lad) uses gpt-3 @xcite to generate diverse synthetic data that express the necessary task-specific structural constraints and can therefore be used to train neural dialog models. lad is a framework for inducing zero-shot generalization in task-oriented dialog by creating diverse and accurate synthetic data. lad, visualized in figure 2 , is a three step process: ( §3.1) domainagnostic algorithms generate a seed dataset from a schema, ( §3.2) gpt-3 reformulates utterances in order to induce linguistic diversity, ( §3.3) heuristics are used to validate the reformulated data to ensure adherence to the schema. lad facilitates zero-shot generalization by explicitly leveraging the strengths of large lms (knowledge of the distributional structure of language) without sacrificing the inductive biases (motivated by structural constraints) in the downstream neural dialog models. ## experiments this paper claims that lad can use a schema to create a sufficiently diverse and accurate synthetic dataset, which can be used to train neural dialog models and facilitate performance gains in zeroshot settings. to validate this claim, experiments are carried out on intent prediction, slot filling and next action prediction across five datasets. for each problem, an appropriate model from prior work is identified. the chosen models (1) exhibit strong zero-shot and few-shot generalizability, and (2) are open-source. though lad is not guaranteed to produce perfectly accurate and diverse data, the inductive biases in the chosen models make them more robust to potential errors and limitations in the synthetic data. ## conclusion in an effort to leverage the abilities of large lms to facilitate zero-shot generalization in task-oriented dialog, this paper introduces lad. lad creates diverse and accurate synthetic data, in order to convey the necessary setting-specific structural constraints to neural dialog models. lad achieves significant performance gains on zero-shot intent prediction, slot filling and next action prediction across five datasets. furthermore, lad is shown to perform competitively in interactive human evaluation, without observing human-annotated data.