Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction (2022)

Abstract: Existing studies on semantic parsing focus on mapping a natural-language utterance to a logical form (LF) in one turn. However, because natural language may contain ambiguity and variability, this is a difficult challenge. In this work, we investigate an interactive semantic parsing framework that explains the predicted LF step by step in natural language and enables the user to make corrections through natural-language feedback for individual steps. We focus on question answering over knowledge bases (KBQA) as an instantiation of our framework, aiming to increase the transparency of the parsing process and help the user trust the final answer. We construct INSPIRED, a crowdsourced dialogue dataset derived from the ComplexWebQuestions dataset. Our experiments show that this framework has the potential to greatly improve overall parse accuracy. Furthermore, we develop a pipeline for dialogue simulation to evaluate our framework w.r.t. a variety of state-of-the-art KBQA models without further crowdsourcing effort. The results demonstrate that our framework promises to be effective across such models.

## dataset construction in this section, we describe the workflow for dataset construction following the design of our framework (figure 2 ). we prepare pairs of complex questions and sparql parses predicted by a base semantic parser (section 2.1.1). then, we decompose the gold and predicted parses and determine correction operations (section 2.1.2). the sub-lfs are translated to questions using templates (section 2.1.3) and we employ crowdworkers to paraphrase these questions using natural language (section 2.2). ## dataset analysis in this section, we conduct a thorough quality analysis of inspired dataset and highlight aspects that contribute to overall quality, including paraphrasing characteristics and contextual awareness. overall data quality. in each dialogue, the crowdworker is required to rephrase the original complex question and each templated sub-question. overall, we believe the quality of the data to be high for a few reasons. in the collection process, our crowdworkers read a detailed tutorial, pass two qualification tasks, and have their work spotchecked at each stage of collection. because we keep our pool of workers small, we are able to maintain frequent communication with them throughout the process, giving feedback in an ongoing fashion. furthermore, we use a semi-automatic data cleaning method to identify inaccurate paraphrases for manual repair, resulting in edits to 325 subquestions in total. based on our observation on a held-out subset of the data, we estimate that only 3.1% of all sub-questions still have inaccuracies, after cleaning. more details are in appendix b.1. paraphrasing characteristics. template corpus rephrased corpus avg length 17.3 10.7 unigrams 8,465 9,864 bigrams 21,072 44,085 trigrams 31,838 81,479 more gem metrics can be seen in appendix b.2. in order to better understand how crowdworkers rephrased templates, 100 randomly selected subquestions are studied in terms of lexical relationships between the template and rephrased versions. we find that they are using synonmy, hypernymy and hyponymy in rephrasings of the templates, in addition to changing word order. this analysis can be found in appendix b.3. contextual awareness. additionally, crowdworkers are encouraged to incorporate contextual information of a given sub-question into their rephrasings, thus improving the contextual richness of the dataset. in order to demonstrate contextual awareness, in general, it is natural for human users to consider the context when making utterances in a dialogue. from the perspective of model development, providing contextual information enriches the input by providing relevant information that may not be present in a given sub-question or sub-lf. we provide concrete examples and analysis to show the effect of context dependency in ## experiments in this section, we explore several base semantic parsers and show how we choose one as the initial parser to construct inspired. then, we conduct extensive experiments on those two core sub-tasks (i.e., sub-question generation and parse correction) in our framework. finally, in order to study the promise of our framework for other parsers (beyond the one used to construct inspired) without introducing extra crowdsourcing effort, we simulate dialogues based on our trained models for sub-question generation and parse correction. we train all models on 4 gtx 1080 ti 11 gb gpus. firstly, we explore transformer @xcite , bart-large @xcite and qgg @xcite as base parsers. in the official leaderboard foot_1 of cwq, qgg is the bestperforming method in the line of query graph generation approaches. models like nsm+h @xcite and pullnet @xcite directly output final answers without lfs, which cannot be made more transparent or interactive with our framework. cbr-kbqa @xcite is the sota model on this dataset as of the submission time, but as its code is not available, we choose transformer and bart-large as the two candidate parsers. we input the complex question to these two seq2seq models and output the lf. since entities are masked in the lfs for these models, we provide qgg with gold entities for fair comparison. we report their lf exact match (em) and f1 scores in we finally select transformer as the initial parser because it is neither state-of-the-art nor has overly poor performance. as the intention is to create a dataset that represents a wide range of parsing errors and correction strategies, a "middle-of-theroad" parser is best for achieving good coverage but also being of decent quality. we report the characteristics of errors made by transformer in appendix b.5. we will explore the other two models in in the following two sections, we explore two sub-tasks under our framework. we treat both of them as seq2seq tasks, then present and evaluate several baseline models including seq2seq @xcite , transformer @xcite , bart-base and bart-large @xcite for each task, in which we use inspired for training and testing. after that, we conduct error analysis for both sub-tasks by examining 100 examples respectively. details of the analysis are in appendix c. ## discussion and future work we are planning to conduct a user study to validate our framework's viability for real use. in this study, human users will utilize the framework to correct parsing errors and query a knowledge base for answers in real time. as shown in figure 3 , users can specify edit operations in a couple clicks, then type in the response box to insert, replace or edit a sub-question. note that we add a new 'edit' operation to make it easier for users to enter replacement sub-questions that require only small edits. while we acknowledge that in a spoken dialogue system, pure natural language feedback may be the most natural, such a system may also suffer from errors caused by automatic speech recognition (asr) @xcite . by contrast, our interface design allows the user to partially specify feedback operations through mouse clicks, which can help mitigate this issue. to evaluate the system, we will use parse accuracy after correction to verify the usefulness of human feedback. we will also use survey questions to measure the subjective quality of the generated explanations, intermediate and final answers, accessibility of the system, etc. in this work, the inspired dataset and experiments provide a foundation for many directions of future work. for example, this could take the shape of gains in parse accuracy as well as improvements to the correction strategy through decomposition. the simulation pipeline provided can also be used for further experimentation. other complementary work could include handling errors introduced by named entity recognition and linking. lastly, ap-plying our framework to other query languages like sql could be an exciting direction. ## conclusion we have proposed an interactive semantic parsing framework and instantiated it with kbqa in this work. using this framework, we crowdsourced a novel dataset, dubbed inspired, and experimentally showed that it can greatly increase the parse accuracy of an initial parser. moreover, we designed a simulation pipeline to explore the potential of our framework for a variety of semantic parsers, without further annotation effort. the performance improvement shows interactive semantic parsing is promising for further improving kbqa in general. no identifying information is included. further, all participants are required to read and agree to an informed consent form before proceeding with the task. amt automatically anonymizes crowdworkers' identities as well. compensation to crowdworkers. in order to ensure both quality data collection and fair treatment of our crowdworkers, we carefully review our payment plan for the amt task. after a pilot study we gauge the average amount of time we expect a task to require and adjust the payment amount per task according to the minimum wage amount in our state, resulting in a 70 cent payment per task. further, we ensure compensation for the time spent on the tutorial and qualification task by awarding @xmath010 bonuses upon every 100 tasks they complete. in total, the cost of creating the inspired dataset is approximately $13,300. ziyu yao, xiujun li, jianfeng gao, brian sadler, and huan sun. 2019a. interactive semantic parsing for ifthen recipes via hierarchical reinforcement learning. in proceedings of the aaai conference on artificial intelligence. ziyu yao, yu su, huan sun, and wen-tau yih. 2019b. model-based interactive semantic parsing: a unified framework and a text-to-sql case study. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5447-5458. wen-tau yih, ming-wei chang, xiaodong he, and jianfeng gao. 2015. semantic parsing via staged query graph generation: question answering with knowledge base. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1321-1331. wen-tau yih, xiaodong he, and christopher meek. 2014. semantic parsing for single-relation question answering. in proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers), pages 643-648. wen-tau yih, matthew richardson, christopher meek, ming-wei chang, and jina suh. 2016. the value of semantic parse labeling for knowledge base question answering. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 201-206. tao yu, rui zhang, he yang er, suyi li, eric xue, bo pang, xi victoria lin, yi chern tan, tianze shi, zihan li, et al. 2020. cosql: a conversational textto-sql challenge towards cross-domain natural language interfaces to databases. in 2019 conference on empirical methods in natural language processing and 9th international joint conference on natural language processing, emnlp-ijcnlp 2019, pages 1962-1979. association for computational linguistics. tao yu, rui zhang, michihiro yasunaga, yi chern tan, xi victoria lin, suyi li, heyang er, irene li, bo pang, tao chen, et al. 2019. sparc: cross-domain semantic parsing in context. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 4511-4523. yuan zhang, jason baldridge, and luheng he. 2019. paws: paraphrase adversaries from word scrambling. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1298-1308. question what is the official language of the country that contains al sharqia governorate? sparql @xmath1:location.country. answer modern standard arabic jun-yan zhu, taesung park, phillip isola, and alexei a.