Guided K-best Selection for Semantic Parsing Annotation (2022)

Abstract: Collecting data for conversational semantic parsing is a time-consuming and demanding process. In this paper we consider, given an incomplete dataset with only a small amount of data, how to build an AI-powered human-in-the-loop process to enable efficient data collection. A guided K-best selection process is proposed, which (i) generates a set of possible valid candidates; (ii) allows users to quickly traverse the set and filter incorrect parses; and (iii) asks users to select the correct parse, with minimal modification when necessary. We investigate how to best support users in efficiently traversing the candidate set and locating the correct parse, in terms of speed and accuracy. In our user study, consisting of five annotators labeling 300 instances each, we find that combining keyword searching, where keywords can be used to query relevant candidates, and keyword suggestion, where representative keywords are automatically generated, enables fast and accurate annotation.

## introduction conversational semantic parsing (csp), which aims to turn an utterance into a meaning representation such as an executable program or logical form, plays an important role in task-oriented dialogue systems @xcite @xcite . in practice, building a complete task-oriented dialogue system requires back-and-forth revision of the meaning representation design. such a mutable nature makes the data collection process difficult and costly. depending on the complexity of the meaning representation, annotators might even need further training to equip them with basic domain knowledge about the task. inspired by computer assisted translation (cat) @xcite when's the lecture scheduled for in may? 2013, 2014), we would like to know if we can accelerate the annotation process with ai-powered human-in-the-loop interfaces. the main difference between our task and the traditional cat task lies in the facts that (i) only a prototype model trained on a small amount of initial data is available (lowresource setting), leading to limited prediction performance; and (ii) annotators have relatively little or no knowledge about the meaning representation. because neither the model nor the annotators are 100% accurate in our scenario, we propose the k-best selection approach as shown in figure 1 , where we (i) generate a set of candidates using a low-resource model; and (ii) ask annotators to traverse the set to select the correct parse and only modify it if necessary. this formulation allows annotators to focus on reading and verification and thus minimizes the need for annotators to write the complicated meaning representation. while the notion of k-best selection is established @xcite , to our knowledge there has been no investigation into optimizing this approach beyond a simple enumeration of candidates ranked by model score. in addition, a standard k-best selection approach may face challenges such as: in this work we demonstrate the validity of these concerns and propose a solution called guided kbest selection, consisting of: (i) a search interface that allows annotators to type keywords and narrow down the k choices, (ii) a keyword suggestion method that guides the exploration of k-best lists for less experienced users. we show that it is the combination of efficient search and guidance that strikes the optimal balance between accuracy and speed while achieving high annotator satisfaction. ## conversational semantic parsing for our study we focus on a version of conversational semantic parsing (figure 1 ), where we are given a user's natural utterance, and the goal of the task is to annotate it into a canonical utterance. the use of canonical utterances formulates semantic parsing as a paraphrasing task that paraphrases a natural utterance into a "canonical" utterance in a constrained language @xcite @xcite . a synchronous context-free grammar (scfg) defines a mapping between task-specific meaning representations and their corresponding constrained languages. that is to say, using such an scfg, a complicated meaning representation can be presented as a human-readable canonical utterance (more similar to natural language) so models can focus on learning how to paraphrase a natural utterance to a canonical utterance. we choose to annotate canonical utterances also because it substantially reduces the task complexity, since annotators no longer need to learn the syntax of the meaning representation itself. we use canonical utterances induced by an scfg defined in @xcite . the corresponding meaning representation is defined in the smcalflow dataset @xcite , which contains 41.5k taskoriented dialogues about calendar events, weather, places, and people. examples of natural and canonical utterances are shown in ## guided k-best selection interfaces in this section, we describe three proposed and two baseline annotation interfaces. figure 2 shows their basic components. the user utterance and its context are given in (a) and the k-best candidate list is provided in (c). (c) and its variants provide different functions to help users efficiently get to the correct parse. note that (a), (b), (d), and (e) are shared across all uis. k. an example of this process is given on figure 3 . finally, the k keywords are re-ranked based on their discriminativeness, or how evenly they split the current candidates, and the 5 most discriminative keywords are shown in the interface. this allows combining keyword suggestion with the search interface. in section 4.3, we compare keyword suggestion with other mechanisms to guide annotators. ## conclusion in this paper, we tackled the challenge of efficient data collection for conversational semantic parsing. in the presence of little available training data, we propose human-in-the-loop interfaces for guided k-best selection, using a prototype model trained on limited data. guided k-best selection interfaces generate a set of possible candidates with functions for fast traversal and ask annotators to select the correct parse. user studies show that combining keyword search functionality with a keyword suggestion system strikes an optimal balance between annotation accuracy and speed.