Grammar-based Decoding for Improved Compositional Generalization in Semantic Parsing (2023)

Abstract: Sequence-to-sequence (seq2seq) models have achieved great success in semantic parsing tasks, but they tend to struggle on out-of-distribution (OOD) data. Despite recent progress, robust semantic parsing on large-scale tasks with combined challenges from both compositional generalization and natural language variations remains an unsolved problem. To promote research in this area, this work presents CUDON, a large-scale dialogue dataset in Chinese language, particularly designed for evaluating compositional generalization of semantic parsing. The dataset contains about ten thousand multi-turn complex queries, and provides multiple splits with different degrees of train-test distribution divergence. We have investigated improving compositional generalization with grammar-based decodering on this dataset. With specially designed grammars leveraging program schema, we are able to substantially improve accuracy of seq2seq semantic parsers on OOD splits: A LSTM-based parser using a Context-free Grammar (CFG) achieves over 25% higher accuracy than a standard seq2seq baseline; a parser using Tree-Substitution Grammar (TSG) improves parsing speed five to seven times over the CFG parser with only a small accuracy loss. The grammar-based LSTM parsers also outperforms BART- and T5-based seq2seq parsers on the OOD splits, despite having less than one tenth of parameters and no pretraining. We also verified our approach on the SMCalflow-CS dataset, particularly, on the zero-shot learning task.

## introduction the task of semantic parsing is to translate a natural language utterance into an executable program in certain meaning representation (mr). in recent years, sequence-to-sequence (seq2seq) models have become the dominant approach for this task @xcite , mainly due to their excellent ability to handle natural language variations. however, evaluations on synthetic datasets such as scan @xcite have shown that seq2seq models do not reliably generalize to those out-of-distribution (ood) utterances with element combinations not seen in the training data. the ability to generalize to novel element combinations is commonly known as compositional generalization. to improve compositional generalization, many recent studies propose specially-designed model architectures targeting scan-like synthetic datasets, and some have achieved perfect accuracy on some of these datasets @xcite @xcite @xcite @xcite . however, most of these approaches have only been tested on synthetic datasets, which are constructed for a specific purpose and cannot represent the full range of challenges that a real-world semantic parser must address. this motivates us to study the compositional generalization problem in a more realistic setting, specifically, in the context of task-oriented dialogue systems, where semantic parsers must handle greater task complexity and navigate through a wider range of natural language variations and ambiguities. the contributions of this work are two-fold: first, on the dataset front, we have created cudon 1 , a chinese dialogue dataset for compositional generalization research. this is a new large-scale task-oriented dialogue dataset, which contains complex multi-turn cross-domain dialogues, created in a semi-synthetic manner, and specially tailored for studying the compositional generalization problem. it also provides 8 train/dev/test splits with different level of distribution divergence. to the best of our knowledge, this is the first large-scale chinese dataset designed for this purpose. and the dataset being in chinese makes it even more valuable since the similar resources are scarce. second, on the modeling front, we find that by exploiting schema of the target program, including the syntax and function argument lists, in the paradigm of grammar based decoding @xcite @xcite , we can substantially improve compositional generalization of semantic parsers in the seq2seq framework. both the parser using a context free grammar (cfg) and the one using a treesubstitution grammar (tsg) @xcite exceed a standard seq2seq baseline using pointer generator network @xcite , by over 20 percent on the ood splits, with the latter being 5-7 times as fast as the former. and they even compare favorably with parsers using large pretrained language models with more than 10 times of parameters. this level of improvement was not reported before in similar works. we also verified that the proposed approach also works in the smcalflow-cs @xcite dataset without much tuning, achieving significant improvement in the zero-shot setting over the baseline, and exceeds previously reported results with pretrained language models in low-resource settings. in addition, we have not seen the tsg formalism used in grammar-based decoding before, although we adopt a model architecture similar to @xcite . ## dataset we have constructed a large-scale multi-turn crossdomain task-oriented dialogue dataset in the financial assistant scenario over seven domains and 52 intents. there is also a common domain for intents such as goodbye and thankyou, which can occur in all dialogues. the raw material of entities and properties, such as stock prices and gains, or the experience and best return of fund managers, was crawled from various websites. throughout the course of dialogue generation, the system may carry out the user-requested tasks by consulting this data to produce realistic and coherent results. the mr we use is called function expressions, which allows arbitrarily nesting of function invocations. there are two types of functions: one with only named arguments, each associated with an intent in the dialogue system, and its arguments associated with the slots of the intent; the other can have both positional and named arguments, and perform non-interactive operations like data manipulation. overall, the dataset has 9996 dialogues, with an average of 41 turns per dialogue and about 44% of the turns involve multiple domains. there are 3876 different delexicalized patterns of the target function expressions with highly unbalanced frequencies. we use the function refer to reference slots filled in previous turns, aided with constraint functions to limit search matching. this idea, borrowed from @xcite , makes utterances seemingly independent to each other in a multi-turn dialogue. figure 1 shows an example of a multi-turn dialogue. ## approach to improve compositional generalization, we exploit the program schema to enhance seq2seq models in a grammar-based decoding framework. the model takes in the user utterance and the context as input, and outputs a sequence of rules and associated values, which can be used to construct the program. intuitively, if the parser is aware of the argument list of each function, it can avoid predicting arguments not belonging to this function. and grammar can also help avoid generating ill-structured outputs. ## experiments in this section we compare parsing accuracy and speed performance of all parsers. we use exact match accuracy as the metric, with minor normalization to filter out non-significant variations and to sort named arguments (order insensitive). source-to-target text translator, using a pointer generator network with two layers of bidirectional lstms as encoder and two layers of lstms as decoder. we found a small gain by using 300dimension pretrained embeddings from fasttext foot_1 for initialization. since we observe large accuracy variations between models trained with different random seeds, to have a clear picture of the degree of variations, we run training 55 times with different seeds to collect the medians and other statistics. we find on tmcd splits, the standard deviation of accuracies range from 6.4 to 11.3 percent. for this reason we want to use large evaluation sets to reduce random effects. more details can be found in appendix i. ## conclusions compositional generalization remains an unsolved problem for real-world semantic parsers. we propose a semi-synthetic large-scale task-oriented dialogue dataset intended to promote research in this area. we find that by designing appropriate grammars, leveraging schema information, and choosing right model architecture we can substantially improve both compositional generalization and parsing efficiency. we will investigate applying grammar-based decoding approach to large pretrained language models on more challenging tasks.