Uncertainty and Traffic-Aware Active Learning for Semantic Parsing (2020)

Abstract: Collecting training data for semantic parsing is a time-consuming and expensive task. As a result, there is growing interest in industry to reduce the number of annotations required to train a semantic parser, both to cut down on costs and to limit customer data handled by annotators. In this paper, we propose uncertainty and traffic-aware active learning, a novel active learning method that uses model confidence and utterance frequencies from customer traffic to select utterances for annotation. We show that our method significantly outperforms baselines on an internal customer dataset and the Facebook Task Oriented Parsing (TOP) dataset. On our internal dataset, our method achieves the same accuracy as random sampling with 2,000 fewer annotations.

## introduction semantic parsing is the task of mapping natural language to a machine-executable meaning representation. supervised semantic parsing models are trained on corpora of natural language utterances with annotated meaning representations. collecting these annotations is an expensive manual process, usually requiring expert annotators who are familiar with both the domain of utterances and the target meaning representation language (e.g. sql). active learning is a method for collecting training data when annotating is difficult or budgets are limited @xcite . in active learning, an algorithm selects examples from an unlabeled set that are predicted to be more useful for the model if labeled. these examples are annotated and the model is retrained in an iterative process. the goal of an active learner is to reach higher performance faster than a random sampling baseline. in this paper, we propose uncertainty and trafficaware active learning, a simple yet effective method to improve a semantic parser. in our setup, we assume access to a set of initially annotated utterances and a large set of unlabeled utterances from customer traffic. we show that by using a combination of uncertainty and utterance frequency from traffic, we can achieve significantly higher performance than baselines on both an internal customer dataset and on the facebook task oriented parsing (top) dataset @xcite . ## related work active learning has been applied to various nlp tasks @xcite @xcite @xcite . @xcite presented one of the first works on active learning for deep semantic parsing and found that selecting low-confidence examples outperformed random examples on two datasets but failed on a third. @xcite experimented with learning to actively-learn for semantic parsing, a method where the active learner is a learned model, but failed to see better performance than random sampling. @xcite proposed a framework where a weakly trained semantic parser was allowed to actively select examples for extra supervision. the authors found that selecting the least confident of the incorrect examples led to the best performance. incorrect examples were identified by executing the predicted query and comparing the predicted answer with an expected answer. in this paper, we experiment with using uncertainty and utterance frequencies from customer traffic, a feature often found in industry logs. ## uncertainty and traffic-aware active learning we propose uncertainty and traffic-aware active learning for semantic parsing. our method is inspired by @xcite , who presented an active learning method for ranking al-gorithms which selects examples that are both informative to the model and representative of the dataset. the authors found that including a representativeness measure helped offset the tendency of informativeness measures to select outliers. in their paper, the authors measured informativeness as permutation probability based on a committee of ranking models, so a query where the most certain committee member had the least confidence was considered more informative. for representativeness, the authors used an lda model to create a feature vector for each query. if a query's feature vector had higher cosine similarity to the average feature vector of all queries, the query was considered more representative. in our method, we also use informativeness and representativeness, but we introduce new ways to measure both that can be applied to semantic parsing tasks. for each utterance u in a set of unlabeled utterances u, we calculate f(u), a sampling weight associated with u, as: the β is a fine-tunable term that weighs the utterance frequency against the confidence. we man-algorithm 1: perplexity to confidence p ← perplexity if @xmath0: then return 1 / (2 + (100 * (p -threshold))); else return 1 -0.5 * (p / threshold); end ually fine-tuned β by training 9 models with different values ranging from 0.1 to 0.9 and compared performance in terms of exact-match accuracy. we found that a β of 0.4 performed the best on our internal dataset and a β of 0.5 performed the best on top, and so we use these β values in this paper. ## datasets we run experiments on both an internal customer dataset and the facebook task oriented parsing ## experiments for controlled experimentation, we simulate active learning by treating a subset of our data as unlabeled. when an unlabeled example is selected, we reveal the label and add it to the training set. all our experiments are run on an nvidia tesla v100 16gb gpu and the results are reported as exact match accuracy. ## conclusion in this work, we present uncertainty and trafficaware active learning, a method that uses model confidence and traffic frequency to improve a semantic parsing model. we show that our method significantly outperforms baselines on both an internal dataset and top. our method achieves the same precision as random sampling with 2,000 fewer annotations on our internal dataset. based on our results, we present our method as a way to improve semantic parsers while reducing annotation costs and limiting customer data shown to annotators.