A Few-Shot Semantic Parser for W izard-of- O z Dialogues with the Precise T hing T alk Representation (2022)

Abstract: Previous attempts to build effective semantic parsers for Wizard-of-Oz (WOZ) conversations suffer from the difficulty in acquiring a high-quality, manually annotated training set. Approaches based only on dialogue synthesis are insufficient, as dialogues generated from state-machine based models are poor approximations of real-life conversations. Furthermore, previously proposed dialogue state representations are ambiguous and lack the precision necessary for building an effective agent. This paper proposes a new dialogue representation and a sample-efficient methodology that can predict precise dialogue states in WOZ conversations. We extended the ThingTalk representation to capture all information an agent needs to respond properly. Our training strategy is sample-efficient: we combine (1) few-shot data sparsely sampling the full dialogue space and (2) synthesized data covering a subset space of dialogues generated by a succinct state-based dialogue model. The completeness of the extended ThingTalk language is demonstrated with a fully operational agent, which is also used in training data synthesis. We demonstrate the effectiveness of our methodology on MultiWOZ 3.0, a reannotation of the MultiWOZ 2.1 dataset in ThingTalk. ThingTalk can represent 98% of the test turns, while the simulator can emulate 85% of the validation set. We train a contextual semantic parser using our strategy, and obtain 79% turn-by-turn exact match accuracy on the reannotated test set.

## introduction virtual assistants and task-oriented dialogue agents are transforming how consumers interact with computers. this has led to active research on dialogue state tracking networks (ren et al., 2019; zhou and 1 our data and code can be downloaded from https:// oval.cs.stanford.edu/releases/ searchquestion: area; restaurant, @xmath0, @xmath1, ... } { @xmath2, @xmath3, ... } small, 2019; @xcite @xcite , and even full neural networks that track dialogue states, implement dialogue policies, and generate agent utterances @xcite @xcite @xcite . dialogue state tracking on wizard-of-oz taskoriented conversations, where humans are asked to simulate both the agent and the user, has proven to be challenging. for example, despite multiple rounds of manual annotation, the multiwoz multidomain task-oriented dataset still contains significant errors which hamper the development of accurate semantic parsers @xcite @xcite ). an approach to bypass manual annotations is to generate dialogues using a simulator and then manually paraphrase them @xcite . unfortunately, as we shall show in this paper, such dialogue simulators do not exercise many of the possible dialogue flows seen in wizard-of-oz conversations. this gap is likely to widen with real-life conversations. given the many attempts to create accurate semantic parsers for the multiwoz data set, this paper takes a fresh look at the problem of understanding wizard-of-oz conversations. we observe two fundamental flaws with the current approach. previously proposed state representations such as slot-value pairs and the recently proposed hierarchical forms @xcite do not capture critical details in the user utterances, such as logical "or" and negation. even if the semantic parser is 100% accurate, the agent will not be able to satisfy the user's request. second, it is easy to make errors. the existing slot representation is ambiguous, so it is not possible to be consistently correct. this leads to poor quality of annotation. this paper shows that it is possible to create a precise and accurate semantic parser for wizardof-oz conversations in a sample-efficient manner. we introduce the multiwoz 3.0 dataset, a reannotation of the full test set and partial validation set of multiwoz 2.1 @xcite , using a new, more precise formal representation. the contributions of this paper include: we also demonstrate that thingtalk is a complete representation for dialogues. the agent directly executes the thingtalk representation to retrieve the results from the databases and apis, without referring to any of the user utterances. in fact, the same agent code can be used both during simulation and in a real agent deployment. the few-shot training samples in thingtalk help the semantic parser generalize from the simulated dialogues to woz conversations. whereas the simulator can only generate a subset of the states representable by thingtalk, thingtalk can precisely represent nearly all woz data. our novel contextual semantic parser, described in section 5, obtains a turn-by-turn accuracy of 79% on multiwoz 3.0. note that this model generalizes to utterances that fall out of the realm of simulation. recently, @xcite proposed adopting a formal representation for both the user and agent state, using the treedst representation. treedst was built to support only dialogues synthesized and paraphrased from a compatible state machine, while thingtalk supports the full generality of wizard-of-oz conversations. ## the thingtalk dialogue language the thingtalk dialogue language is designed to formally capture all relevant information in taskoriented dialogues to interpret what the user says next. this includes the user utterances, the result of the user requests, as well as the agent's replies. to see why the results and the agent's reply are needed, consider the example in fig. 1 . the user has previously asked for a cheap restaurant, and now asks "do you have anything with indian food?". in the example, the agent noted that there are many cheap restaurants available, so it is likely that the user wants both "indian" and "cheap". this is reflected in the query that the command maps to. conversely, had the agent responded that there are no cheap restaurants, it is likely that the user no longer cares about finding a cheap and only wants indian. the user query thus would be just: this illustrates that the meaning of the user utterance depends on the result and the agent's response, so we must include them in the formal dialogue. the previous slot-based representation captures only what is mentioned by the user; it is not precise enough to handle this example. formally, thingtalk represents (1) the user state @xmath4, (2) the agent state @xmath5, and (3) the formal dialogue @xmath6. in this section, we provide the ## simulator-agent architecture to synthesize data for training, we propose a simulator-agent architecture. the state-based simulator takes the role of the human user. the same agent that would be used at deployment time is used during synthesis. the agent is built based on the semantics of thingtalk, not just the simulator. it can respond correctly to any dialogue @xmath7. on the other hand, the simulator samples a subset space d sim ⊂ d. we refer to dialogues in d sim as in-simulation; other dialogues are out-of-simulation. formally, the architecture has three components: agent(d, u) : d × @xmath8: an agent that accepts a formal dialogue @xmath9, and the user state @xmath10, to produce a new dialogue @xmath11. the agent guarantees that if @xmath12. sim(d) : d @xmath13: a simulator that accepts an in-simulation dialogue @xmath14, and nondeterministically creates a new user utterance @xmath15. csp(d, x) : d × @xmath16, a contextual semantic parsing model that accepts a dialogue @xmath17, which may not be in d sim , and a user utterance @xmath18. in this section, we describe how the components are used to synthesize training data and build a functional dialogue agent. ## evaluation our evaluation attempts to answer these research questions: 1. how well does our thingtalk representation model wizard-of-oz conversations? 2. what accuracy can a model achieve in the task of predicting thingtalk, given our training data acquisition strategy? 3. how well do our dialogue simulator and our dialogue agent approximate real dialogues? ## conclusion this paper presents a sample-efficient methodology, based on the extended thingtalk representation, to predict precise dialogue states in wizard-of-oz conversations. we achieve a turn-by-turn exactmatch accuracy of 79.2% on the multiwoz 3.0 dataset, while using 50x less manually annotated training data than the original multiwoz dataset. the proposed thingtalk dialogue representation is precise, complete, and executable. it is precise enough to cover 98% of the dialogue turns in multi-woz. the precision enables automatic synthesis of dialogues covering 85% of the multiwoz data set. thingtalk is complete and executable, as evidenced by a fully working agent that can simply execute thingtalk queries without referring to the user input. furthermore, the agent can handle dialogue flows beyond those that can be simulated. the accuracy is achieved with a contextual semantic parser (csp) where the dialogue context is represented in thingtalk rather than the natural language dialogue history. it is trained first with auto-paraphrased synthetic data, fine-tuned with the few-shot annotated data, then self-trained. in summary, this paper shows that with thingtalk, we can predict woz dialogues accurately with training data mostly generated from a state machine. our methodology thus combines the best of the woz and m2m approaches, as it can handle the more realistic woz dialogues, while having a low data acquisition cost like m2m.