Semantic Parsing of Brief and Multi-Intent Natural Language Utterances (2021)

Abstract: Many military communication domains involve rapidly conveying situation awareness with few words. Converting natural language utterances to logical forms in these domains is challenging, as these utterances are brief and contain multiple intents. In this paper, we present a first effort toward building a weakly-supervised semantic parser to transform brief, multi-intent natural utterances into logical forms. Our findings suggest a new “projection and reduction” method that iteratively performs projection from natural to canonical utterances followed by reduction of natural utterances is the most effective. We conduct extensive experiments on two military and a general-domain dataset and provide a new baseline for future research toward accurate parsing of multi-intent utterances.

## introduction semantic parsing to map a natural language utterance to its logical form is regarded as a challenging task partly due to a lack of annotated data @xcite @xcite . a promising avenue of research is to generate a set of candidate logical forms paired with their canonical realizations in natural language. then, the canonical utterance that best matches the input is identified by a model, and its logical form is used as output @xcite . a paraphrase/sequence-to-sequence model may additionally be used to translate a canonical utterance to a logical form @xcite @xcite . while the results are promising, most existing works do not handle natural language utterances with multiple intents. we refer to an intent as a goal intended by a user's utterance. multi-intent utterances allow people to communicate core aspects of a situation in a consistent and timely manner, as illustrated in figure 1 . multi-intent semantic parsing is especially suitable for military domains where emphasis is placed on communication skills, terminology, and brevity @xcite . while communication protocols are often published, variations are allowed given the current situation. an area of interest is intelligence, surveillance, and reconnaissance (isr) domains where contact reports (e.g., "arriving at home base and ready to descend") often contain multiple intents, and a system must determine the number of intents, interpret the natural language and predict the exact logical forms for every intent, which can be highly challenging. we investigate new methods for semantic parsing of utterances with multiple intents. importantly, and distinguishing our work from earlier literature @xcite @xcite @xcite , our domain areas have no supervised training data, nor can pseudo-language utterances be created through crowdsourcing due to their sensi-tive nature and requirement of expert knowledge. we thus operate in a weakly-supervised setting by assuming only access to a grammar that generates canonical utterances and logical forms. obtaining a comprehensive collection of natural utterances for military applications is difficult; it can be easier to create a grammar that generates canonical utterances for the application. in addition, there are scenarios where there is insufficient time or funding to obtain supervised data, e.g. quickly building a virtual assistant for a new mobile app. our goal is distinct from related efforts in dialog systems @xcite @xcite ; the parser does not have additional context or interaction but focuses on modeling complex compositional intents. we build on methods that project natural utterances to the canonical space @xcite and investigate novel adaptations for handling multiintent utterances. our contributions are as follows. our findings suggest that a new approach that iteratively projects the natural language utterance to a canonical utterance, followed by a reduction step can achieve the best performance. ## hierarchical projection let x and y be the set of all natural language utterances and logical forms (lf), respectively. given a natural language utterance x ⊆ x, we wish to produce y ⊆ y . we assume only access to a grammar g that defines a set of n production rules whose union forms a canonical set of utterances z. a grammar is assumed to be in the form @xmath0. . . |r n , where each production rule r @xmath1, τ ) is defined as rule expansion α and a tag τ . tags define the semantic content associated with a rule, which can be used to build lfs. figure 2 shows an example grammar. canonical utterances found in z do not cover the full range of variation available in natural language. a viable option, described below, is to develop a projection function π which maps x directly into z and obtain an appropriate y through g. we follow @xcite and use a pretrained language model (lm) to obtain semantic representations of utterances in r d . a distance function δ is used to compute the closest canonical utterance in vector space to the natural language utterance. the projection function is defined as: ## data most semantic parsers are given access to (natural language utterance, lf) pairs during training. our setting, however, assumes no access to these pairs and are only given a grammar to generate canonical utterances and their lfs. we use two proprietary military communication datasets and a general-purpose dataset overnight (see isr intelligence, surveillance, and reconnaissance (isr) subject matter experts were consulted to develop a corpus of known utterances that an intelligence operator would say during a mission. the lfs appear in json format containing an intent and slots to be filled. the utterances are consolidated into a grammar with a relatively deep structure where an intent may contain slots for nested intents, making it closer to semantic parsing datasets like top @xcite . it consists of domain-specific words and acronyms outside of ordinary vernacular, making this dataset particularly challenging. the grammar contains 37 rules, 36 non-terminals, and approximately 60 terminals. heli short commands were collected from helicopter communications and consolidated into a similar grammar to isr. the grammar has a shallow structure and does not contain many nested intents, but each utterance is short (1-5 tokens), which has its own challenges. the grammar contains 48 rules, 47 non-terminals, and approximately 60 terminals. example in fig. 1 . natural language utterances in both datasets are wholly defined by its grammar. for evaluation, we expand to a set of paraphrased canonical utterances using an english-to-@xmath2. overnight @xcite is a semantic parsing dataset over eight domains, including sports, restaurants, and social media. each domain contains a grammar to generate canonical utterances and lfs, as well as natural language paraphrases. as we are interested in weakly-supervised parsing, we ignore natural language utterances in training and only use those in the test set for evaluation. the datasets we use contain grammars and natural language data for utterances with a single intent, but they lack multi-intent data. we create simulated multi-intent utterances by concatenating natural language utterances together, with target lfs as concatenations of the utterances' lfs. we enforce a limit of three intents to keep task difficulty manageable. ## results we present several baselines in our experiments. we train a sequence-to-sequence (seq2seq) model on sequence pairs of the form (canonical utterance, lf). at evaluation, the model is given a natural language utterance associated with a canonical utterance and evaluated based on the original lf. we make use of pre-trained bart @xcite by fine-tuning on task-specific data. proj is the technique of projecting a natural language utterance to a canonical utterance in the grammar, described in section 2. nt-avg is the proposed method of averaging the representations of a non-terminal's possible values. for the single-intent overnight datasets, we display baseline results presented in @xcite . finally, we experiment with two methods on top of nt-avg for multi-intent parsing -meta-grammar and reduction. the overnight dataset contains a more complex grammar and longer utterances and lfs compared to our internal datasets ( all systems evaluated on the multi-intent split of overnight struggle to perform well. a system must be able to determine the number of intents in an utterance, interpret the natural language in each intent, and predict lfs that exactly match the lfs for every intent. accuracies range between 0% and 2% (see supplementary). this demonstrates that it is non-trivial to transfer parsing systems from the single-intent setting to multi-intent. to tease out performance differences between systems, we instead evaluate a system prediction to be correct if at least one predicted lf has an exact match with any one of the gold standard lfs. for multi-intent utterances, reduction achieves the highest accuracies. we believe the long structure of the lfs in overnight provide a challenge for current seq2seq models to generate accurately. meanwhile, grammar-based approaches can easily side-step this issue by producing lfs directly from the grammar, evidenced by the higher accuracies. an additional phenomenon appearing in all datasets is lower layers of bert used for projection perform better than higher layers (figure 3 ). how- figure 3 : lf accuracies for nt-avg system by varying bert layer. lower layers result in higher accuracies, especially in the multi-intent setting. ## conclusion we tackle multi-intent semantic parsing using weakly-supervised methods. our results show that an iterative approach of projecting the natural utterance to a canonical utterance followed by a token reduction step achieves the best performance. potential further improvement could be achieved by fine-tuning the bert model on free text in the desired domain (e.g. military training materials) to create better utterance embeddings. future research includes parsing more complex multi-intent utterances, borrowing ideas from dialogue systems and capturing dependencies between intents (gangadharaiah and narayanaswamy, 2019).