ZEROTOP : Zero-Shot Task-Oriented Semantic Parsing using Large Language Models (2023)

Abstract: We explore the use of large language models (LLMs) for zero-shot semantic parsing. Semantic parsing involves mapping natural language utterances to task-specific meaning representations. LLMs are generally trained on publicly available text and code and cannot be expected to directly generalize to domain-specific parsing tasks in a zero-shot setting. In this work, we propose ZEROTOP, a zero-shot task-oriented parsing method that decomposes semantic parsing problem into a set of abstractive and extractive question-answering (QA) problems. For each utterance, we prompt the LLM with questions corresponding to its top-level intent and a set of slots and use the LLM generations to construct the target meaning representation. We observe that current LLMs fail to detect unanswerable questions; and as a result, cannot handle questions corresponding to missing slots. We address this by fine-tuning a language model on public QA datasets using synthetic negative samples. Experimental results show that our QA-based decomposition paired with the fine-tuned LLM can zero-shot parse \approx 16% of utterances in the MTOP dataset.

## introduction large language models (llms) are trained on publicly available text @xcite @xcite and code @xcite and have been shown to attain reasonable zero-shot generalization on a diverse set of nlp tasks @xcite . however, they are not expected to generalize to domain-specific semantic parsing tasks in a similar way, where the inductive bias from pre-training is less helpful. in this work, we propose zerotop that decomposes the semantic parsing task into one of answering a series of extractive and abstractive questions, corresponding to its top-level intent and a set of relevant slots, and leverage the llm's ability to zero-shot answer reading comprehension questions. as illustrated in figure 1 , we cast top-level intent classification as an abstractive qa task. to address llms' bias towards predicting labels common in the pretraining data @xcite , we propose to generate an intent description in an unconstrained manner and infer the intent label most similar to the generated description. we view slot value prediction as an extractive qa problem. most utterances do not mention all the slots. it is therefore essential for the model to abstain from prediction when corresponding slots are not mentioned. through our analyses, we observe that most llms frequently hallucinate text for missing slots with high confidence, resulting in poor performance. to address this, we fine-tune an lm on a collection of public qa datasets augmented with synthetic unanswerable samples. we call our trained model abstainer, as it is capable of identifying unanswerable questions and abstaining from prediction. we hierarchically prompt for nested slots using the abstainer, and infer nested intents if their corresponding slots are detected. we empirically show that this qa based decomposition of zerotop is an effective way to leverage llms for domain specific semantic parsing, outperforming several strong baselines in the zero shot setting. ## related work llms are increasingly used for semantic parsing in low-data scenarios utilizing canonical representations @xcite , and prompt-tuning @xcite . the closest work to ours is @xcite where they decompose parsing into qa tasks. however, they assume access to some annotated data whereas we focus on a strict zero-shot setting where only the schema information is available along with some natural language prompts for c1 get_contact get_school … cn … s1 sn … possible nested intents slots what is the relation? where is the school located? abstainer abstainer "please send that message to my mom on whatsapp" what did the user intend to do? send message recipient who is the recipient? my mom name_app which app? whatsapp ordinal which one? n/a mom n/a [in:send_message [sl:recipient [in:get_contact [sl:contact_related my ] [sl:type_relation mom ] ] ] ] schema entities. our work is also related to approaches towards zero-shot dialog state tracking using llms @xcite . specifically, @xcite uses an abstainer to handle missing slots. our method differs in that, we focus on semantic parsing where the abstainer needs to be applied multiple times along with intent detection to create nested meaning representations. ## zerotop: zero-shot task-oriented semantic parsing problem formulation we focus on taskoriented parsing with hierarchical intent-slot schema. let @xmath0, i 2 , . . . , i n } and @xmath1, s 2 , . . . , s m } be the set of all possible toplevel intents and slots respectively. each intent i j has a set of slots s @xmath2, s j 2 , . . . , s j n } that can be filled. possible slots in an intent are represented by the intent-to-slot mapping i2s: @xmath3, where p(•) is the powerset operator. similarly, the inverse slot-to-intent mapping is represented by s2i: @xmath4. the input in our setting consists of i2s and s2i, but no annotated data. zerotop requires users to provide a question per slot @xmath5, q s 2 , . . . , q s k }, that represents their purpose. in a real-life setting, this can be obtained from a domain developer. unconstrained generation for zero-shot intent classification we view zero-shot intent classification as an abstractive qa problem. one intuitive way is to prime the llm with a qa prompt and then constrain the generation to search over only valid intent labels @xcite . however, llms are known to be biased towards text sequences @xcite more common in pretraining data. for example, in the mtop dataset, the t0-3b model predicts create_call (make algorithm 1: zerotop: our proposed zero-shot semantic parsing method. call) as intent for 92% of the data in the call domain. therefore, we propose to first generate an intent description in an unconstrained fashion by priming the lm with the following prompt. answer the following question depending on the context. context: a user said, {utterance}. question: what did the user intend to do? answer: then, we choose the intent label that is most similar to generated answer using roberta sentence similarity @xcite . unlike @xcite , our approach does not require enumerating choices in the prompt allowing us to handle a large number of intents and slots as found in datasets like mtop. leveraging qa datasets for slot value prediction slot value prediction involves extracting phrases for a slot from the user utterance. we cast this as an extractive qa problem. all slots might not be mentioned in an input utterance. for example, in the mtop dataset, on average, only onethird of possible slots are mentioned per utterance. the qa model needs to abstain from prediction for such missing slots. to analyze the abstaining capability of pre-trained qa models, we consider a few top-performing zero-shot llms t0-3b, gpt-3, and codex with their corresponding prompts and experiment on a 500 sample subset of unanswerable questions from the squad dataset @xcite . we observe the accuracy of all models to @xmath6. in section 4, we also consider a loglikelihood-based threshold for abstaining and show that this threshold is difficult to tune using public qa datasets. to address this challenge, we leverage multiple publicly available qa datasets 1 to train abstainer, a qa model capable of abstaining from prediction. specifically, we generate synthetic unanswerable training samples by modifying existing qa data, and train a qa model jointly on existing datasets and synthetic unanswerable questions. for every (question, answer, context) triplet, we generate synthetic unanswerable questions by either (1) removing the sentence containing the answer span from the context, or (2) randomly sampling a context that doesn't have the same question. after training the abstainer, we prompt it for each slot with its corresponding question for slot value prediction, in the following format: answer the following question depending on the context. context: a user said, {utterance}. question: {slot question} answer: nested intents to identify nested intents, we assume knowledge of candidate nested intents that can be accommodated by each slot, represented by the slot-to-candidate-nested-intent mapping s2ni: @xmath7. our method assumes that depth of output representations is at most 4 i.e. nested intents cannot further have more nested intents. one intuitive way is to prompt the llm for nested intent with the intent prediction prompt. however, our unconstrained generation-based intent model would predict many false positive nested intents. we instead use abstainer to prompt for their respective slots. if any slot value is identified, we consider its corresponding intent via s2i to be present as well. 1 the qa datasets details are mentioned in appendix a.1 zerotop: putting it all together the pseudocode of zerotop is mentioned in algorithm-1. zerotop employs a top-down, greedy prompting strategy, where we first prompt for intent and then, its respective slots. first, we obtain the top-level intent using the intent model. based on the predicted intent, we prime the abstainer for corresponding slots using their respective questions as prompts. for each identified slot value, we prompt the abstainer for slots of candidate nested intents. we use the same prompt format for this step with the identified slot value now considered as the input utterance. finally, we combine predicted intent, identified slot values, and nested intents to create the meaning representation. ## experiments we experiment on the english language subset of mtop @xcite dataset. mtop is a multilingual task-oriented semantic parsing dataset comprising data from 6 languages and 11 domains. the test set has 4386 samples with 113 distinct intents and 74 slots. on average, each intent has 3.6 slots and 33% of possible slots are filled per utterance. experiment settings. we evaluate on the zeroshot setting, therefore we have no training data. we manually create questions for slots q using one example per slot. for training abstainer, we finetune t0-3b on the extractive and abstractive qa datasets for 1 epoch with a constant learning rate of 10 -4 . we use complete meaning representation match accuracy as the performance metric. more details in appendix a.2. baselines. we compare zerotop with constrained t0-3b, gpt-3 and codex as intent models where they are primed with intent generation prompt and are constrained to search over valid intent labels. we also compare with calibrated constrained t0-3b @xcite whose logits are adjusted to counter lm biases. we consider an ablation of zerotop where we assign intent labels based only on their similarity with user utterance using roberta sentence transformer. ## conclusion in this paper, we propose zerotop that decomposes semantic parsing into abstractive and extractive qa tasks. zerotop identifies top-level intent by generating in an unconstrained fashion and inferring the intent label most similar to the generated description. we train abstainer using public qa datasets, that is capable of identifying unanswerable questions and abstaining from prediction.