One Semantic Parser to Parse Them All: Sequence to Sequence Multi-Task Learning on Semantic Parsing Datasets (2021)

Abstract: Semantic parsers map natural language utterances to meaning representations. The lack of a single standard for meaning representations led to the creation of a plethora of semantic parsing datasets. To unify different datasets and train a single model for them, we investigate the use of Multi-Task Learning (MTL) architectures. We experiment with five datasets (Geoquery, NLMaps, TOP, Overnight, AMR). We find that an MTL architecture that shares the entire network across datasets yields competitive or better parsing accuracies than the single-task baselines, while reducing the total number of parameters by 68%. We further provide evidence that MTL has also better compositional generalization than single-task models. We also present a comparison of task sampling methods and propose a competitive alternative to widespread proportional sampling strategies.

## introduction semantic parsing is the task of converting natural language into a meaning representation language (mrl). the commercial success of personal assistants, that are required to understand language, has contributed to a growing interest in semantic parsing. a typical use case for personal assistants is question answering (q&a): the output of a semantic parser is a data structure that represents the underlying meaning of a given question. this data structure can be compiled into a query to retrieve the correct answer. the lack of a single standard for meaning representations resulted in the creation of a plethora of semantic parsing datasets, which differ in size, domain, style, complexity, and in the formalism used as an mrl. these datasets are expensive to create, as they normally require expert annotators. consequently, the datasets are often limited in size. multi-task learning (mtl; caruana 1997) refers to jointly learning several tasks while sharing parameters between them. in this paper, we use mtl to demonstrate that it is possible to unify these smaller datasets together to train a single model that can be used to parse sentences in any of the mrls that appear in the data. we experiment with several q&a semantic parsing dataset for english: geoquery @xcite , nlmaps v2 @xcite , top @xcite , and overnight @xcite . in order to investigate the impact of less related tasks, we also experiment on a non-q&a semantic parsing dataset, targeting a broader coverage meaning representation: amr @xcite , which contains sentences from sources such as broadcasts, newswire, and discussion forums. our baseline parsing architecture is a reimplementation of the sequence to sequence model by @xcite , which can be applied to any parsing task as long as the mrl can be expressed as a sequence. inspired by @xcite , we experimented with two mtl architectures: 1-to-n, where we share the encoder but not the decoder, and 1-to-1, where we share the entire network. previous work @xcite @xcite has focussed on a lesser degree of sharing more closely resembling the 1-to-n architecture, but we found 1-to-1 to consistently work better in our experiments. in this paper we demonstrate that the 1-to-1 architecture can be used to achieve competitive parsing accuracies for our heterogeneous set of semantic parsing datasets, while reducing the total number of parameters by 68%, overfitting less, and improving on a compositional generalization benchmark @xcite . we further perform an extensive analysis of alternative strategies to sample tasks during training. a number of methods to sample tasks proportionally to data sizes have been recently proposed @xcite @xcite , which are often used as de facto standards for sampling strategies. these methods rely on the hypothesis that sampling proportionally to the task sizes avoids overfitting the smaller tasks. we show that this hypothesis is not generally verified by comparing proportional methods with an inversely proportional sampling method, and a method based on the per-task loss during training. our comparison shows that there is not a method that is consistently superior to the others across architectures and datasets. we argue that the sampling method should be chosen as another hyper-parameter of the model, specific to a problem and a training setup. we finally run experiments on dataset pairs, resulting in 40 distinct settings, to investigate which datasets are most helpful to others. surprisingly, we observe that amr and geoquery can work well as auxiliary tasks. amr is the only graphstructured, non q&a dataset, and was therefore not expected to help as much as more related q&a datasets. geoquery is the smallest dataset we tested, showing that low-resource datasets can help high-resource ones instead of, more intuitively, the other way around. ## experimental setup in this section, we describe the datasets used, baseline architectures, and training details. ## experiments in section 4.1, we compare several sampling methods for the 1-to-1 and 1-to-n architectures. in section 4.2 we then compare the mtl models with the single-task baselines. we turn to the issue of generalization in section 4.3, where we use a recently introduced benchmark to evaluate the compositional generalization of our models. finally, in section 4.4 we report experiments between dataset pairs to find good auxiliary tasks. ## conclusions we used mtl to train joint models for a wide range of semantic parsing datasets. we showed that mtl provides large parameter count reduction while maintaining competitive parsing accuracies, even for inherently different datasets. we further discussed how generalization is another advantage of mtl and we used the cfq dataset to suggest that mtl achieves better compositional generalization. we leave it to future work to further investigate this type of generalization in the context of mtl. we compared several sampling methods, indicating that proportional sampling is not always optimal, showing room for improvements, and introducing a loss-based sampling method as a competitive and promising alternative. we were surprised to see the positive impact of low-resource (geoquery) and less-related (amr) datasets can have as auxiliary tasks. challenges in finding optimal sampling strategies and auxiliary tasks suggest that they should be treated as hyper-parameters to be tuned.