Did You Mean...? Confidence-based Trade-offs in Semantic Parsing (2023)

Abstract: We illustrate how a calibrated model can help balance common trade-offs in task-oriented parsing. In a simulated annotator-in-the-loop experiment, we show that well-calibrated confidence scores allow us to balance cost with annotator load, improving accuracy with a small number of interactions. We then examine how confidence scores can help optimize the trade-off between usability and safety. We show that confidence-based thresholding can substantially reduce the number of incorrect low-confidence programs executed; however, this comes at a cost to usability. We propose the DidYouMean system which better balances usability and safety by rephrasing low-confidence inputs.

## introduction task-oriented dialogue systems @xcite @xcite represent one path towards achieving the longstanding goal of using natural language as an api for controlling real-world systems by transforming user requests into executable programs, i.e. translating natural language to code. central to the systems' success is the ability to take rational actions under uncertainty @xcite . when model confidence is low and the system is unlikely to succeed, we would prefer it defer actions and request clarification, while at high confidence, clarification requests may annoy a user. relying on model confidence requires it to be well-correlated with accuracy, i.e. it requires a calibrated model. recent work has focused on the calibration of semantic parsing models. specifically, stengel-eskin and van durme (2022) benchmarked the calibration characteristics of a variety of semantic parsing models, finding some of them to be well-calibrated, especially on parsing for task-oriented dialogue. given the relatively well-calibrated nature of these models, we first examine how they could be used in an annotation interface, with a view to balancing the trade-off between annotation cost and correctness. we simulate a human-in-the-loop (hitl) experiment where high-confidence tokens are automatically annotated and low-confidence tokens trigger a dialogue with an oracle annotator who either picks the correct token from a top-k list or manually inserts it. with a small number of interactions we substantially boost annotator accuracy. a similar trade-off exists between usability and safety in task-oriented user interfaces. we examine how sequence-level model confidence scores can be used to balance this trade-off by reducing the number of incorrect programs executed while also minimizing the number of follow-up user interactions and their cognitive burden. we find that thresholding outputs based on model confidence (i.e. rejecting outputs falling below a tuned threshold) reduces the number of incorrect programs executed by 76% compared to the baseline. however, this comes at a cost to usability, as roughly half the correctly-predicted parses are also rejected. to strike a balance between safety and usability, we introduce the didyoumean system (cf. fig. 1 ), which rephrases the input conditioned on the predicted parse and asks users to confirm the accuracy of the paraphrase. in a user study, we obtain an 36% improvement in usability over the thresholded system while maintaining a 58% reduction in the number of incorrect programs executed. ## related work our experiments in section 4 involve a predictive model for human-in-the-loop coding: similar models have been integrated into ides, e.g. @xcite . didyoumean relates to the interactive semantic parsing domain @xcite @xcite , where humans are included in the semantic parsing loop. in this domain, @xcite introduce a confidence-based interactive system in which a parsing agent can ask users for clarification. our work follows in this spirit, but asks the user to confirm a parse rather than generating questions for the user to answer; we also operate in a different parsing domain, preventing us from using yao et al.'s system as a baseline. didyoumean also relates broadly to selective prediction, where a model is expected to abstain from making decisions at low confidence @xcite @xcite @xcite . our system extends selective prediction's setting by including a human-inthe-loop. finally, didyoumean shares a motivation with fang et al. (2022), who introduce a method for reliably summarizing programs. their work provides post-hoc action explanations while we focus on resolving misunderstandings before execution. 3 methods datasets our data is drawn from the smcalflow (semantic machines et al., 2020) task-oriented dialogue dataset, which contains lisp-like programs (cf. appendix a). we follow the same preprocessing as platanios et al. (2021), and use the sm-calflow data splits given by roy et al. (2022): 108,753 training, 12,271 validation, and 13,496 testing dialogue turns. models we use miso @xcite , a well-calibrated model from @xcite . rather than predict the sm-calflow surface form, including syntactic tokens like parentheses, miso directly predicts the underlying execution graph. the graph can deterministically be "de-compiled" into its surface form, and vice-versa. the fact that miso predicts an underlying graph makes it attractive for applications which require confidence scores, as it only predicts content tokens (i.e. functions, arguments) rather than structure markers, like parentheses. for details on miso's architecture, see @xcite and @xcite . for token confidence estimation, we use the maximum probability across the output vocabulary at each timestep. this has been shown to be a relatively robust confidence estimator in classification @xcite . for sequence-level scores, we follow stengel-eskin and van durme (2022) and take the minimum over token-level confidence scores. ## human-in-the-loop simulation production datasets like smcalflow are constantly evolving as new functionalities are added. the expensive and time-consuming nature of annotating data can be mitigated by the use of predictive parsing models which suggest speculative parses for new utterances. however, the model's output can be incorrect, especially given out-of-distribution inputs. we need to ensure that annotators are not introducing errors by overly trusting the model. if the model is well-calibrated, we can use its confidence to reduce such errors. for example, we can alert annotators to low confidence predictions and ask them to intervene @xcite . using a threshold, we can prioritize time or correctness: a higher threshold would result in more annotator-model interactions, decreasing the speed but increasing program correctness (reducing the need for debugging) while a lower threshold would increase speed but also lower the accuracy. since we do not have access to expert sm-calflow annotators, we simulate an oracle humanin-the-loop (hitl) annotator who always provides a correct answer by using the gold annotations provided in the dataset. specifically, for a given input, we decode the output tokens of a predicted program o 0 , . . . o n normally as long as predictions are confident (above a given threshold). if at time t the confidence p(o t ) falls the threshold, we attempt to match the decoded prefix o 0 , . . . , o t-1 to the gold prefix g 0 , . . . g t-1 . if the prefixes do not match, we count the example as incorrect. if they do match, we replace o t with g t , the gold prediction from our oracle annotator, and continue decoding. we consider three metrics in this experiment: (1) the exact match accuracy of the decoded programs (higher is better). (2) the percentage of total tokens for which we have to query an annotator (lower is better, as each query increases annotator load). (3) the percentage of uncertain tokens (below the threshold) for which the gold token g t is in the top 5 predictions at timestep t. here, higher is better, as selecting a token from a candidate list is typically faster than producing the token. results and analysis fig. 2 shows our three metrics as the threshold is increased in increments of 0.1. we see that accuracy grows exponentially with a higher threshold, and that the percentage of tokens for which an annotator intervention is required grows at roughly the same rate. the exponential growth reflects the distribution of token confidences, with most tokens having high confidence. finally, we see that while at low confidence, most tokens must be manually inserted, the rate at which they are chosen from the top 5 list rapidly increases with the threshold. thus, the increased number of annotator interactions required at higher thresholds may be offset by the fact that many of these interactions are a choice from the top-5 list. ## user correction via didyoumean section 4 showed that token-level confidence scores can be used to balance speed and correctness in an annotation interface. we see a similar tradeoff between safety and usability in user interfaces using semantic parsing models. here, we define safety as rejecting unsuccessful programs before executing them. this strict definition is motivated by physical domains: imagine that rather than controlling a digital assistant, a user is guiding a robot via language commands (e.g. @xcite ; lynch and sermanet (2020); stengel-eskin et al. 2022 )). in this setting, actions may have irreversible consequences, so determining safety before execution is key. safety considerations need to be balanced with usability of the system: an unplugged agent would be very safe but unusable. to increase usability, an agent might make follow-up requests to a user, like asking for clarification or confirmation. the types of requests the agent makes have varying cognitive load on the user: for example, providing confirmation takes less effort than rephrasing. we measure how well we can reject incorrect programs before executing them. following past work in selective prediction @xcite .a.), we measure success by coverage and risk, as well as f score w.r.t. program correctness. coverage is the percentage of inputs for which a program is executed and risk is the percentage of executed programs that were incorrect. precision is inverse risk, and recall is the percentage of correct programs which were accepted. in addition to f1, we consider f0.5, which upweights precision (safety) by a factor of 2. a low-coverage, low-risk system may be safer but have more false negatives, i.e. reject more correct programs, decreasing its usability. a high-coverage, high-risk system is more usable at the cost of false positives, i.e. executing incorrect programs. we do not commit to setting an optimal threshold for this trade-off, since it is task-specific. we consider 3 systems. as a baseline, we consider a system that executes everything it predicts (accept); this will result in the highest-possible coverage, but also high risk. we can also use miso's calibrated nature to improve safety outcomes by tuning a sequence-level confidence threshold for rejecting programs (tuned). we tune on the full validation set using f1; we explore the range [0.0, 1.0) in increments of 0.01, finding 0.40 to be optimal. finally, we introduce the didyoumean system for filtering low-confidence programs. for a given utterance, didyoumean shows the user a paraphrase of the input; the user then decides to accept the parse based on this paraphrase. this allows correctly-predicted low-confidence programs to be accepted and executed, while reducing the user load: making a binary choice to accept a paraphrase is a receptive task, while rephrasing an instruction is a more costly productive task. details of didy-oumean's components are given below. glossing model since users are typically unfamiliar with formats like lisp, we need to present the user with a natural language paraphrase -or gloss -of the candidate parse. to train a glossing model, we modify @xcite 's seq2seq benchclamp semantic parsing framework: rather than using the user utterance with the previous turn's context (u 0 , a 0 , u 1 ) as input and a program p as output, we take the context and program (u 0 , a 0 , p) as the input and user instruction u 1 as the output. we use bart-large @xcite and fine-tune it on the smcalflow training split. see appendix a for model details. didyoumean system when a low-confidence parse is detected, didyoumean triggers a dialogue with the user in order to recover some usability over simply rejecting all low-confidence parses. fig. 1 shows the system workflow. didyoumean shows the original utterance u 1 and the gloss û * to the user, who determines whether they are identical or not. if they accept the gloss, we optionally re-parse the gloss û * rather than the original utterance u 1 ; this can remove typos and other idiosyncracies. we call this the re-parsed setting, while choosing the original prediction û is the chosen setting. we predict that allowing users to accept and reject glosses will improve the balance between safety and usability (i.e. f1) over the threshold system by allowing them to accept correct low-confidence parses. in other words, adding human interaction will allow us to achieve a balance which cannot be attained given the tradeoffs resulting from thresholding. ## selection study in section 5 we presented the didyoumean system, which allowed users to confirm or reject a potential gloss. the gloss was chosen from a larger set of candidates; this naturally raises the question of whether users can directly choose a gloss from the candidate list, rather than accepting or rejecting a single gloss at a time. here, we examine to what extent it is helpful to users to choose glosses from a list of options. we take a sample of validation programs for smcalflow, stratified across 10 evenly-spaced confidence bins from 0.0 to 1.0. note that this differs from the setting in section 5, where the maximum bin was 0.6. we then present the top predictions decoded with nucleus sampling @xcite ; the number of predictions depends on the confidence of the model. specifically, we set a cutoff of 0.85 and add predictions until the sum of the sequence-level confidence scores is greater than the cutoff; this could be achieved by a single very confident prediction or multiple low-confidence predictions. to reduce the load on the annotators, we limit the number of predictions seen to a max of 10, even if the cutoff is not reached. annotators were asked to choose one of the top predictions or to manually rewrite the query if none of the predictions were adequate; the chosen or rewritten query was then re-parsed and compared to the gold parse. further annotation details are given in appendix c. of the 100 examples sampled, annotators manually rewrote 7 and chose from the top-k list for the other 93. ignoring the rewritten examples, 39 model predictions were incorrect and 54 were correct; by choosing glosses, annotators correct 5 incorrect predictions. however, they also inadvertently changed 4 correct predictions to incorrect. figure fig. 3 shows the exact match accuracy before and after annotators selected glosses at each confidence bin. at low confidence, we see very minor increases on the order of a single program being corrected. at high confidence, annotators generally have only one or two options, and are able to choose the correct one, resulting in similar performance to nucleus decoding. however, at medium confidence, annotators often choose the wrong gloss, leading to lower performance. qualitatively, many incorrect choices are driven by glosses that appear to be paraphrases, but in fact correspond to subtly different lisp programs. these results suggest that accepting/rejecting glosses is more promising than choosing them. ## conclusion we examine two common trade-offs in semantic parsing, and how a well-calibrated model can be used to balance them. in section 4 we illustrated how token-level model confidences could be used in a simulated hitl task-oriented parsing annotation task. our experiments in section 5 extended these results to sequence-level confidences and nonexpert users; we found that model confidence could be used to improve the usability-safety trade-off and introduced didyoumean, which improved usability by asking users to accept predictions.