Compositional Task-Oriented Parsing as Abstractive Question Answering (2022)

Abstract: Task-oriented parsing (TOP) aims to convert natural language into machine-readable representations of specific tasks, such as setting an alarm. A popular approach to TOP is to apply seq2seq models to generate linearized parse trees. A more recent line of work argues that pretrained seq2seq2 models are better at generating outputs that are themselves natural language, so they replace linearized parse trees with canonical natural-language paraphrases that can then be easily translated into parse trees, resulting in so-called naturalized parsers. In this work we continue to explore naturalized semantic parsing by presenting a general reduction of TOP to abstractive question answering that overcomes some limitations of canonical paraphrasing. Experimental results show that our QA-based technique outperforms state-of-the-art methods in full-data settings while achieving dramatic improvements in few-shot settings.

## introduction task-oriented parsing (top) takes an utterance as input and generates an unambiguous specification of a task that can be executed by machine @xcite . traditional approaches to top treat the task as an instance of slot filling @xcite , first classifying the intent of the utterance as a whole and then tagging tokens with slot labels that identify relevant entities (such as numbers, persons, dates or times, organizations, and so on). however, this approach only works for simple utterances that have flat rather than compositional semantics. that is, slot-filling approaches cannot produce nested (or "hierarchical") meaning representations, such as the one shown in fig 1 , where slots and intents can be composite and contain other slots (or intents) as proper parts. a straightforward way to handle compositional semantics is to formulate top as a seq2seq prob-lem, where the input sequence is the utterance and the output sequence is a linearized representation of a semantic tree (shown in the bottom part of fig 1 ). the recent development of highperforming transformer-based pretrained language models (plms) @xcite @xcite that can be fine-tuned on specific tasks (such as a particular top dataset) has made this formulation feasible. this approach has achieved state-of-the-art performance on a variety of top datasets @xcite @xcite . however, the output sequences are typically not expressed in natural language but rather in a mixture of natural language and special symbols (such as [sl:) that were not seen during pretraining and have no meaning to the plms. depending on the task, the output sequences may hardly contain any natural language at all (for example, if the outputs are sql queries). it would seem reasonable to conjecture that performance would improve if we could reformulate top as a more conventional nlp task in which both the inputs and the outputs are expressed in natural language, as such a reformulation might be better able to leverage the plm's pretraining. to bridge this gap, @xcite reduced top to a canonical-paraphrasing task. they first fine-tune a plm to map a natural utterance u to another canonical utterance u ′ , where canonical utterances belong to a controlled fragment of the relevant natural language. then u ′ can be translated into the desired meaning representation (semantic tree) via a context-free grammar that can parse all and only canonical utterances. however, this approach has a major limitation: canonical fragments can be defined and parsed by hand-written grammars only in closed-world domains where the set of underlying entities is fixed and known in advance (e.g., in a domain where people ask questions about basketball players, all of whom are utterance: look up directions to the nearest parking near s beritania street. semantic parse tree of the utterance: in:get_directions sl:destination in:get_location sl:location_modifier nearest sl:category_location parking sl:location_modifier in:get_location sl:search_radius near sl:location s beritania street linarized parse tree: [in:get_directions [sl:destination [in:get_location [sl:location_modifier nearest] [sl:category_location parking] [sl:location_modifier [in:get_location [sl:search_radius near] [sl:location s beritania street] ] ] ] ] ] figure 1: a sample utterance and its semantic parse tree in the topv2 dataset, where nodes starting with "in:" are intents and nodes starting with "sl:" are slots. a parse tree always has an intent node at the root. an intent typically corresponds to a verb and can be viewed as an action, with a sequence of slots as its arguments. a slot may have additional intents nested in it, recursively. a linearized parse tree that can be processed by seq2seq models is shown at the bottom. in this paper we focus on scenarios where it is not viable to specify a canonical grammar with a fixed set of rules. we instead propose to formulate top as abstractive question answering, in such a way that answering all questions for a given utterance allows us to reconstruct the parse tree of that utterance. specifically, we introduce singleturn qa (st qa for short), which poses one single query for a given utterance, and multi-turn qa (mt qa), which dynamically constructs multiple questions for each utterance, depending on previous answers. because single-turn qa asks only one question, it has lower latency; however, the model must generate a longer text representing the entire parse tree. by contrast, multi-turn qa generates shorter answers that are more straightforward to parse; however, all questions need to be answered correctly, and if there are dependencies between questions, they can't be run within the same batch. we study these two approaches and their tradeoffs in both full-data and low-resource settings. to summarize our contributions: ## related work task-oriented parsing has been extensively studied in the literature. the most prevalent approach is seq2seq modeling, which maps utterances to their meaning representations, typically expressed as a mixture of natural language and tokens such as brackets and artificial intent and slot identifiers @xcite @xcite @xcite ; we take this approach as our main baseline. few-shot semantic parsing has also attracted wide interest. @xcite applied a different training paradigm; they assumed there are several source domains with labeled data and adopted a first-order meta-learning algorithm, reptile, to train their model. @xcite argued that plms are better suited for directly generating natural language rather than task-specific meaning representations, and thus they fine-tune plms to generate canonical paraphrases, which can then be parsed by a contextfree grammar to produce the corresponding semantic trees. they further improve performance by augmenting input sequences with similar examples as prompts. @xcite push that direction further by leveraging small amounts of unannotated data. we use canonical paraphrasing as one of our baselines. as we already noted in the introduction, canonical paraphrasing is not widely applicable in open-world task-oriented parsing; our qa-based approach overcomes that limitation. @xcite applied modifications to the linearized trees to make them more natural. inspired by the recent success of qa-driven approaches to a wide range of nlp tasks, such as dialogue state tracking @xcite , named entity recognition @xcite , and multi-task learning @xcite , @xcite framed semantic parsing as an extractive qa task. this limits its scope to noncompositional semantic structures. our work is the first to use qa for semantic parsing of arbitrarily nested and complex meaning representations. moreover, in contrast to previous approaches, our formulation usually results in fewer questions. ## reducing top to abstractive qa we now present a general method for reducing compositional top to abstractive qa. given an utterance, our goal is to recover its semantic parse tree by asking questions and parsing the answers returned by a qa model. for this to succeed, all questions associated with an input utterance need to be answered correctly. at a high level, we formulate questions so as to build the required parse tree in a top-down, left-to-right fashion: we first ask a question to determine the root node, and then we recursively proceed towards the leaves. we start by describing multi-turn qa; the single-turn case is discussed in section 3.3. each qa instance is a triple consisting of a context, a question, and an answer. we use the parse tree in fig. 1 to illustrate the corresponding (multiturn) sequence of qa triples shown in fig. 2 . the context is the utterance provided by the user plus general information about the domain and/or state from previous turns; the question corresponds to a particular node of the parse tree; and the answer provides the content of that node. we first extract the top-level intent (get directions), then the corresponding slots (destination), then the value of each slot (destination is the nearest parking near s beritania street), and then we start to recursively repeat this process on the phrase the nearest parking near s beritania street (by asking what is the intent included in that utterance segment). we note that our system is able to handle negative answers: if there isn't a nested intent in an utterance segment, the multi-turn qa: q: a user may intend to get directions, get distance, get estimated arrival time, get estimated departure time, get estimated duration, get road condition information, get route's information, get traffic information, get location, make unsupported navigation, or update directions. a user said, "look up directions to the nearest parking near s beritania street." what did the user intend to do? a: get directions q: the slots for get directions may be locations, arrival datetimes, road conditions to avoid, waypoints, amounts, paths, sources, travel methods, road conditions, waypoints to avoid, departure datetimes, paths to avoid, obstructions to avoid, and destinations. ## experiments we evaluate our method against two state-of-theart seq2seq techniques, one generating linearized parse trees and the other generating canonical paraphrases. we chose topv2 and pizza because they are the only two task-oriented datasets we are aware of with nested meaning representations that cannot be produced by slot-filling approaches. we investigate both multi-and single-turn qa. in the former, a question aims to recover one or several nodes in the parse tree; in single-turn qa, we only ask one question to reconstruct the entire tree. additionally, we perform ablation analysis to evaluate the contribution of each component, as described in sec. 3. we show that our method achieves superior performance, particularly on those topv2 utterances that have more nested semantics. ## conclusion we have presented a reduction of compositional task-oriented parsing to abstractive qa, whereby parse tree nodes are recovered by posing queries to a qa model. we have also proposed to train qa models with the msp (masked span prediction) objective, to better leverage the massive amount of linguistic knowledge gained during pretraining. we experimentally evaluated single-turn qa and multiturn qa on two public datasets, both in full-data and in few-shot settings, with and without msp, and showed that they consistently outperform a number of powerful baseline techniques, including canonical paraphrasing, in both settings. the msp variants perform best on average, with particularly dramatic improvements obtained in the few-shot setting.