Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation (2022)

Abstract: We introduce a novel setup for low-resource task-oriented semantic parsing which incorporates several constraints that may arise in real-world scenarios: (1) lack of similar datasets/models from a related domain, (2) inability to sample useful logical forms directly from a grammar, and (3) privacy requirements for unlabeled natural utterances. Our goal is to improve a low-resource semantic parser using utterances collected through user interactions. In this highly challenging but realistic setting, we investigate data augmentation approaches involving generating a set of structured canonical utterances corresponding to logical forms, before simulating corresponding natural language and filtering the resulting pairs. We find that such approaches are effective despite our restrictive setup: in a low-resource setting on the complex SMCalFlow calendaring dataset (Andreas et al. 2020), we observe 33% relative improvement over a non-data-augmented baseline in top-1 match.

## introduction we aim to improve the performance of a semantic parser based on previous user interactions, but without making use of their direct utterances, nor any associated personal identifiable information (pii). such privacy requirements are common in practical deployment @xcite , and semantic parsers are commonly used in real-world systems such as siri and alexa, converting natural language into structured queries to be executed downstream @xcite . constructing semantic parsers can be expensive: annotating examples consisting of natural languagelogical form pairs often requires trained experts. two complementary lines of work has pursued this concern. first, several works @xcite tackle low-resource semantic parsing via approaches such as data augmentation. a 1 *: work done during internship at semantic machines. ## semantic parsing in practice our setup assumes access exclusively to: in a real-world setting, one might hand-annotate the seed dataset d to train a system for initial deployment, while then leveraging u to refine a future version of the system. while our setting is highly restrictive, we argue that it reflects practical constraints. for example, in practice, the grammar for logical forms-as well as the synchronous context-free grammar (scfg) that maps them to canonical utterances-will often be written from scratch, precluding transfer learning methods which leverage a large quantity of similar data in another domain. moreover, in complex domains, one cannot expect to sample useful logical forms directly from a grammar if the grammar is designed for coverage as in e.g., smcalflow @xcite . therefore, other than d, the only additional data (excluding additional manual annotation) are subsequent user inputs in the form of u, with pii masked to preserve privacy. ## related work compared to prior work in low-resource semantic parsing, our task setup's constraints require different approaches. first, we consider semantic parsing on an entirely new grammar for logical forms, rather than adapting to new domains starting from a preexisting grammar @xcite @xcite @xcite . for example, @xcite takes a natural-language-to-sql model for one database to propose language-sql training examples for another database. second, we assume one cannot sample useful canonical utterances directly from the grammar, unlike @xcite and @xcite . for example, @xcite use a backtranslationesque approach leveraging large numbers of unlabeled natural and canonical utterances. moreover, we do not even assume direct access to unlabeled natural utterances, due to real-world privacy considerations @xcite . many works on low-resource figure 1 : illustration of one of our proposed methods for data augmentation (user-rank) in low-resource semantic parsing. we first obtain canonical forms from unlabeled user data using a parser trained on seed data, replacing pii. next, we simulate corresponding natural language for the generated canonical forms. finally, we filter the canonical-natural pairs to obtain our final silver data pairs for augmentation. nevertheless, recent work @xcite @xcite has demonstrated decent performance given just a small seed dataset d, by combining pretrained language models with constrained decoding. for example, @xcite use only 300 labeled examples in the complex smcalflow dialogue domain @xcite . however, using pretrained models to directly generate silver training data, with a method such as dino @xcite , is unsuitable in semantic parsing: the models are unaware of either the underlying grammar or the space of parsable queries. one of our contributions is to explore more effective uses of pretrained models for data augmentation in a practical semantic parsing scenario. finally, the detection of pii in user data is an applied topic of interest @xcite , such as for safely summarizing call transcripts @xcite or the automatic detection of doxing @xcite . in our work we implement a solution meant as a proof of concept for our exploration, based on detecting and replacing named entities. ## practical augmentation while finetuning a pretrained model on the seed dataset d can yield a reasonable parser p @xcite , we aim to increase performance via data augmentation. however, our realistic setup precludes many prior approaches. we propose to generate silver data via three main steps, shown in figure 1: (1) generate a set c of canonical utterances c, (2) simulate a set n of corresponding natural utterances n, and (3) filter the resulting (c, n) pairs. we suggest multiple approaches for these steps, and benchmark their efficacy in sec. 5. the entire procedure can be iterated multiple times as the parser improves. ## experiments tasks. we evaluate on two domains, both english: 1. atis @xcite , a flight booking dataset. we use the break @xcite subset. foot_2 2. smcalflow @xcite , a calendaring dataset, which we view as the most complex and realistic. in each domain, we assume a seed data d of just 30 pairs, conducting several trials with different random samples of seed data to mitigate noise from this selection. we sample 300 unlabeled natural utterances u from the dataset, which must be parsed to canonical forms (using the grammar and scfg of @xcite ) and then pii-masked before use. our implementation of pii masking is based on recognizing and replacing named entities; see appendix a for further details. methods. we evaluate several methods on each task, listed below. pairs via cycle consistency (sec. 4.3). 5. gram-rank, a weak baseline that samples initial c directly from the grammar, which we run only on smcalflow since our atis grammar is too loosely specified for sampling. results. we observe that our best data augmentation methods (user-rank, gpt-rank) double the performance of the baseline finetuning method base on atis, and outperform it on smcalflow by up to 20% relative gain ( of interest is that gpt-rank outperforms base despite using only the seed d, and not extra unlabeled u. moreover, iterating the data augmentation procedure (user-rank-3x, gpt-rank-3x) can further improve performance compared to base (relative 150% on atis, 33% relative on sm-calflow), by improving the initial parser p used for parsing unlabeled u or for filtering pairs (c, n), although we observed in preliminary experiments that further iterations yielded diminishing benefits. in contrast, user-cyc performs poorly on atis, indicating that the cyc filtering is perhaps too restrictive for certain domains. even on sm-calflow where performance is decent in comparison, the successful cycles are overwhelmingly for relatively trivial canonical utterances (e.g., "hello! how are you?"). we additionally observe that nearly one-third of cycles are successful, much more than the actual validation set accuracy of 15%, indicating that our auto-generated user utterances remain less challenging and diverse compared to real user utterances. meanwhile, gram-rank is no better than the unaugmented baseline base: sampling plans directly from a grammar is ineffective in a complex, realistic domain like smcalflow. ## conclusion we have discussed a challenging setting for low resource semantic parsing based on real-world resource and privacy constraints. in addition to a seed dataset, the only resources allowed are unlabeled natural utterances which must be pii-masked. we observe that data augmentation approaches leveraging pretrained language models can still improve over supervised baselines which use only the seed dataset. at the same time, substantial room remains for improvement: there are many alternatives to our reranking procedure for silver data, and our method loses some effectiveness when more labeled data is provided. we hope that our exploratory observations help lay a foundation for further work in realistic data augmentation approaches for semantic parsing.