Error-Aware Interactive Semantic Parsing of O pen S treet M ap (2021)

Abstract: In semantic parsing of geographical queries against real-world databases such as OpenStreetMap (OSM), unique correct answers do not necessarily exist. Instead, the truth might be lying in the eye of the user, who needs to enter an interactive setup where ambiguities can be resolved and parsing mistakes can be corrected. Our work presents an approach to interactive semantic parsing where an explicit error detection is performed, and a clarification question is generated that pinpoints the suspected source of ambiguity or error and communicates it to the human user. Our experimental results show that a combination of entropy-based uncertainty detection and beam search, together with multi-source training on clarification question, initial parse, and user answer, results in improvements of 1.2% F1 score on a parser that already performs at 90.26% on the NLMaps dataset for OSM semantic parsing.

## introduction semantic parsing has the goal of mapping natural language questions into formal representations that can be executed against a database. if realworld large-scale databases such as openstreetmap (osm) foot_0 need to be accessed, the creation of gold standard parses by humans can be complicated and requires expert knowledge, and even reinforcement learning from answers might be impossible since unique correct answers to osm queries do not necessarily exist. instead, uncertainties can arise due to open-ended lists (e.g., of restaurants), fuzzily defined geo-positional objects (e.g., objects "near" or "in walking distance" of other objects), or by ambiguous mappings of natural language to osm tags foot_1 , with the truth lying in the eye of the beholder who asked the original question. semantic parsing against osm thus asks for an interactive setup where an end-user inter-operates with a semantic parsing system in order to negotiate a correct answer, or to resolve parsing ambiguities and to correct parsing mistakes, in a dialogical process. previous work on interactive semantic parsing @xcite @xcite ) has put forward the following dialogue structure: i) the user poses a natural language question to the system, ii) the system parses the user question and explains or visualizes the parse to the user, iii) the user generates natural language feedback, iv) the parser tries to utilize the user feedback to improve the parse of the original user question. in most cases, the "explanation" produced by the system is restricted to a rule-based reformulation of the parse in a human intelligible form, whereas the human user has to take guesses about where the parse went wrong or is ambiguous. the goal of our paper is to add an explicit step of error detection on the parser side, resulting in an automatically produced clarification question that pinpoints the suspected source of ambiguity or error and communicates it to the human user. our experimental results show that a combination of entropy-based uncertainty detection and beam search for differences to the top parse yield concise clarification questions. we create a dataset of 15k clarification questions that are answered by extracting information from gold standard parses, and complement this with a dataset of 960 examples where human users answer the automatically generated questions. supervised training of a multisource neural network that adds clarification questions, initial parses, and user answers to the input results in improvements of 1.2% f1 score on map to tags bar and pub that differ in that only the latter sells food; off-license shops can have licenses to sell only wine or all kinds of alcohol. a parser that already performs at 90.26% on the nlmaps dataset for osm semantic parsing. considerable effort has been invested in the creation of large datasets for parsing into sql representations. @xcite created a dataset called spider which is a complex, cross-domain semantic parsing and text-to-sql dataset. their annotation process was very extensive, and involved 11 computer science students who invested a total of 1,000 hours into asking natural language queries and creating the corresponding sql query. extensions of the spider dataset, sparc @xcite , or co-sql @xcite involved even more computer science students. our work attempts an automatic construction of concise clarification questions, allowing for faster dataset construction. ## (multi-source) neural machine translation our work employs as a semantic parser a sequenceto-sequence neural network @xcite that is based on an recurrent encoder and decoder architecture with attention @xcite . given a corpus of aligned data @xmath1, y n )} n @xmath2, standard supervised training is performed by minimizing a cross-entropy objective -1 n n @xmath3,t |y n,<t , x n ), where the probability of the full output sequence @xmath4, y 2 , ..., y n is calculated by the product of the probability for every timestep where p(y|x) = t @xmath5, x). this model can be easily extended to multisource learning @xcite by using not only one, but multiple encoders. this means that there are actually multiple sequences of hidden states. the decoder hidden state is consequently initialized by a linear projection of the average of the last hidden states of all encoders @xmath6, and needs to implement a separate attention mechanism for every encoder. to be able to fine-tune a model with feedback from a user, the standard cross-entropy objective cannot be used because the desired target is not a gold parse, but a parse ỹ predicted by the system, that has been annotated with positive and negative markings by a human user. this can be formalized as assigning a reward δ t that is either positive or negative to every token in the parse (δ t+ = 0.5 and δ t-= -0.5). it is then possible to maximize the likelihood of the correct parts of the parse by optimizing a weighted supervised learning objective while @xmath7, its use is heavily discouraged 5 . furthermore, several mistakes were introduced in the data by the augmentation with the help of a wordlist. for example, an automatically generated natural language question based on this wordlist asks for bars, whereas the gold parse associated to that question asks for pubs instead: conceptually, bars and pubs may not be that different to each other, but osm advises a strict distinction between bars and pubs 6 . while a pub sells alcohol on premise, a pub also sells food, the athmosphere is more relaxed and the music is quieter compared to a bar. lastly, ambiguity was introduced because natural language words now map to multiple different osm tags. this leads to the following data occurrences: the previous examples show that for the same keyword "off license" both @xmath8. finally, since the data was augmented first, and only afterwards split into train, development and test sets, there is a lot of overlap between the train and test data. this is problematic because a proper evaluation should also test for overfitting, which does not work if data is shared between different splits, as shown in the following examples: ## generation of clarification questions on of the goals of error-aware interactive semantic parsing is to alert to user about suspected sources of ambiguity and error by initiating a dialogue. the parser thus needs to detect uncertainty in its output, and generate a clarification questions on the detected source of uncertainty. we use entropy-based uncertainty measures. firstly, entropy per timestep t is measured asỹt p( ỹt |x, @xmath9, @xmath10. this is employed to calculate the entropy of a token as the mean of the character entropies for each of a 7 meta-parameter settings are reported in the appendix. token's characters. 8 based on entropy information, we generate simple questions by employing a template-based method which incorporates the least certain token: "did you mean @xmath0token or $alternative?". 8 a visualization of entropy is reported in the appendix. ## experiments on synthetic dialogues in a first experiment, we generated entire dialogues synthetically, that is, the clarification question from the parser and synthetic user answers. the latter were constructed by checking if either the original token or the alternative is contained in the given gold parse. dataset statistics for train, development and test splits are given in model training is performed by extending the character-based baseline model by additional encoders for the dialogue (question and answer) and the predicted parse hypothesis. experiments show that the character-based multi-source model including hypothesis and dialogue as additional input (line 4) outperforms the baseline (line 1) by more than 1 point in accuracy and f1 score ( ## human interaction study we furthermore performed a small field study where human users interacted with the system. parses for queries from both train and development parts of the dataset were generated and augmented with automatically created clarification questions based on the uncertainty model. examples were then filtered to keep only those parses that contained a parse mistake or parse ambiguity. this resulted in a total of 930 annotation tasks. 9 the annotation interface shown in figure 1 illustrates the system-user interaction: human annotators are presented with a natural language query ("closest off license from lyon"), the parse (shown below in linearized form), and the result of the generated parse (show as the map extract on top of the figure). in addition to the linearized form of the predicted parse, a human-intelligible list format of the key-value pairs in the parse 10 is presented, following the annotation interface of lawrence and 9 both synthetic and user data will be publicly released. riezler (2018). the task of the human users is to mark the errors in the list of keys and values, and to answer or correct the clarification question. the markings are used as feedback in the weighted finetuning objective of @xcite . as the outputs of the model are on character-level, the token-level reward of the annotations is distributed onto them for training. the final model is trained on the weighted objective in a multi-source fashion, taking parse hypothesis, clarification question, and logged user answer as additional inputs. line 5 in ## conclusion ambiguities or errors in real-world semantic osm parsing arise because of different tagging preferences of developers and users, an issue that can only be solved by an interactive setup where a parser is aware of its errors, and a satisfactory answer is found by the user marking parse errors and communicating alternatives. our current work is a first step towards precise communication and offline learning in interactive semantic parsing. an interesting future direction of work is to move to online learning in interactive semantic parsing.