Meta-Learning for Domain Generalization in Semantic Parsing (2021)

Abstract: The importance of building semantic parsers which can be applied to new domains and generate programs unseen at training has long been acknowledged, and datasets testing out-of-domain performance are becoming increasingly available. However, little or no attention has been devoted to learning algorithms or objectives which promote domain generalization, with virtually all existing approaches relying on standard supervised learning. In this work, we use a meta-learning framework which targets zero-shot domain generalization for semantic parsing. We apply a model-agnostic training algorithm that simulates zero-shot parsing by constructing virtual train and test sets from disjoint domains. The learning objective capitalizes on the intuition that gradient steps that improve source-domain performance should also improve target-domain performance, thus encouraging a parser to generalize to unseen target domains. Experimental results on the (English) Spider and Chinese Spider datasets show that the meta-learning objective significantly boosts the performance of a baseline parser.

## introduction semantic parsing is the task of mapping natural language (nl) utterances to executable programs. while there has been much progress in this area, earlier work has primarily focused on evaluating parsers in-domain (e.g., tables or databases) and often with the same programs as those provided in training @xcite . a much more challenging goal is achieving domain generalization, i.e., building parsers which can be successfully applied to new domains and are able to produce complex unseen programs. achieving this generalization goal would, in principle, let users query arbitrary (semi-)structured data on the web and reduce the annotation effort required to build multi-domain nl interfaces (e.g., apple siri or amazon alexa). current parsers struggle in this setting; for example, we show in section 5 that a modern parser trained on the challenging spider dataset @xcite has a gap of more than 25% in accuracy between in-and out-of-domain performance. while the importance of domain generalization has been previously acknowledged @xcite , and datasets targetting zero-shot (or out-ofdomain) performance are becoming increasingly available @xcite @xcite , little or no attention has been devoted to studying learning algorithms or objectives which promote domain generalization. conventional supervised learning simply assumes that source-and target-domain data originate from the same distribution, and as a result struggles to capture this notion of domain generalization for zero-shot semantic parsing. previous approaches @xcite @xcite facilitate domain generalization by incorporating inductive biases in the model, e.g., designing linking features or functions which should be invariant under domain shifts. in this work, we take a different direction and improve the domain generalization of a semantic parser by modifying the learning algorithm and the objective. we draw inspiration from meta-learning @xcite and use an objective that optimizes for domain generalization. that is, we consider a set of tasks, where each task is a zero-shot semantic parsing task with its own source and target domains. by optimizing towards better target-domain performance on each task, we encourage a parser to extrapolate from source-domain data and achieve better domain generalization. specifically, we focus on text-to-sql parsing where we aim at translating nl questions to sql queries and conduct evaluations on unseen databases. consider the example in figure 1 , a parser needs to process questions to a new database at test time. to simulate this scenario during training, we synthesize a set of virtual zero-shot parsing tasks by sampling disjoint source and target domains foot_0 for each task from the training domains. the objective we require is that gradient steps computed towards better source-domain performance would also be beneficial to target-domain performance. one can think of the objective as consisting of both the loss on the source domain (as in standard supervised learning) and a regularizer, equal to the dot product between gradients computed on source-and target-domain data. maximizing this regularizer favours finding model parameters that work not only on the source domain but also generalize to target-domain data. the objective is borrowed from @xcite who adapt a model-agnostic meta-learning (maml; @xcite technique for domain generalization in computer vision. in this work, we study the effectiveness of this objective in the context of semantic parsing. this objective is model-agnostic, simple to incorporate and does not require any changes in the parsing model itself. moreover, it does not introduce new parameters for meta-learning. our contributions can be summarized as follows. our code is available at https://github. com/berlino/tensor2struct-public . our work is similar in spirit to @xcite , who also attempts to simulate source and target domains during learning. however, their optimization updates on virtual source and target domains are loosely connected by a two-step training procedure where a parser is first pre-trained on virtual source domains and then fine-tuned on virtual target domains. as we will show in section 3, our training procedure does not fine-tune on virtual target domains but rather, uses them to evaluate a gradient step (for every batch) on source domains. this is better aligned with what is expected of the parser at test time: there will be no fine-tuning on real target domains at test time so there should not be any fine-tuning on simulated ones at train time either. moreover, @xcite treat the division of training domains to virtual train and test domains as a hyper-parameter, which is possible for a handful of domains, but problematic when dealing with hundreds of domains as is the case for text-to-sql parsing. meta-learning for nlp meta-learning has been receiving soaring interest in the machine learning community. unlike conventional supervised learning, meta-learning operates on tasks, instead of data points. most previous work @xcite @xcite has focused on few-shot learning where metalearning helps address the problem of learning to learn fast for adaptation to a new task or domain. applications of meta-learning in nlp are cast in a similar vein and include machine translation @xcite and relation classification @xcite . the meta-learning framework however is more general, with the algorithms or underlying ideas applied, e.g., to continual learning @xcite , semi-supervised learning @xcite , multi-task learning @xcite and, as in our case, domain generalization @xcite . very recently, there have been some applications of maml to semantic parsing tasks @xcite @xcite . these approaches simulate few-shot learning scenarios in training by constructing a pseudo-task for each example. given an example, similar examples are retrieved from the original training set. maml then encourages strong performance on the retrieved examples after an update on the original example, simulating test-time fine-tuning. @xcite use matching networks @xcite to enable one-shot text-to-sql parsing where tasks for meta-learning are defined by sql templates, i.e., a parser is expected to generalize to a new sql template with one example. in contrast, the tasks we construct for meta-learning aim to encourage generalization across domains, instead of adaptation to a new task with one (or few) examples. one clear difference lies in how meta-train and meta-test sets are constructed. in previous work (e.g., @xcite , these come from the same domain whereas we simulate domain shift and sample different sets of domains for meta-train and meta-test. ## meta-learning for domain generalization we first formally define the problem of domain generalization in the context of zero-shot text-to-sql parsing. then, we introduce dg-maml, a training algorithm that helps a parser achieve better domain generalization. finally, we propose a computationally cheap approximation thereof. ## semantic parser in general, dg-maml is model-agnostic and can be coupled with any semantic parser to improve its domain generalization. in this work, we use a base parser that is based on rat-sql @xcite , which currently achieves state-of-theart performance on spider. 2 formally, rat-sql takes as input question q and schema s of its corresponding database. then it produces a program which is represented as an abstract syntax tree t in the context-free grammar of sql @xcite . rat-sql adopts the encoder-decoder framework for text-to-sql parsing. it has three components: an initial encoder, a transformer-based encoder and an lstm-based decoder. the initial encoder provides initial representations, denoted as q init and s init for the question and the schema, respectively. a relation-aware transformer (rat) module then takes the initial representations and further computes context-aware representations q enc and s enc for the question and the schema, respectively. finally, a decoder generates a sequence of production rules that constitute the abstract syntax tree t based on q enc and s enc . to obtain q init and s init , the initial encoder could either be 1) lstms @xcite on top of pre-trained word embeddings, like glove @xcite , or 2) pre-trained contextual embeddings like bert @xcite . in our work, we will test the effectiveness of our method for both variants. as shown in @xcite , the encodings q enc and s enc , which are the output of the rat module, heavily rely on schema-linking features. these features are extracted from a heuristic function that links question words to columns and tables based on n-gram matching, and they are readily available in the conventional mono-lingual setting of the spider dataset. however, we hypothesize that the parser's over-reliance on these features is specific to spider, where annotators were shown the database schema and asked to formulate queries. as a result, they were prone to re-using terms from the schema verbatim in their questions. this would not be the case in a real-world application where users are unfamiliar with the structure of the underlying database and free to use arbitrary terms which would not necessarily match column or table names @xcite . hence, we will also evaluate our parser in the cross-lingual setting where q and s are not in the same language, and such features would not be available. ## experiments to evaluate dg-maml, we integrate it with a base parser and test it on zero-shot text-to-sql tasks. by designing an in-domain benchmark, we also show that the out-of-domain improvement does not come at the cost of in-domain performance. we also present some analysis to show how dg-maml affects domain generalization. ## conclusions the task of zero-shot semantic parsing has been gaining momentum in recent years. however, previ-ous work has not proposed algorithms or objectives that explicitly promote domain generalization. we rely on the meta-learning framework to encourage domain generalization. instead of learning from individual data points, dg-maml learns from a set of virtual zero-shot parsing tasks. by optimizing towards better target-domain performance in each simulated task, dg-maml encourages the parser to generalize better to unseen domains. we conduct experiments on two zero-shot textto-sql parsing datasets. in both cases, using dg-maml leads to a substantial boost in performance. furthermore, we show that the faster first-order approximation dg-fmaml can also help a parser achieve better domain generalization. in step 4, the first two terms ∇ θ l bs (θ) + ∇ θ l bt (θ) can be viewed as the gradient of applying θ to both source and target domains. the last term can be viewed as maximize the dot product between gradients of source and target domain with respect to θ . in the same spirit as dg-maml, dg-fmaml also tries to encourage the gradients to generalize between source and target domains.