Towards Zero-Shot Frame Semantic Parsing with Task Agnostic Ontologies and Simple Labels (2023)

Abstract: Frame semantic parsing is an important component of task-oriented dialogue systems. Current models rely on a significant amount training data to successfully identify the intent and slots in the user’s input utterance. This creates a significant barrier for adding new domains to virtual assistant capabilities, as creation of this data requires highly specialized NLP expertise. In this work we propose OpenFSP, a framework that allows for easy creation of new domains from a handful of simple labels that can be generated without specific NLP knowledge. Our approach relies on creating a small, but expressive, set of domain agnostic slot types that enables easy annotation of new domains. Given such annotation, a matching algorithm relying on sentence encoders predicts the intent and slots for domains defined by end-users. Experiments on the TopV2 dataset shows that our model trained on these simple labels have strong performance against supervised baselines.

## introduction frame semantic parsing is an important subproblem with many applications, and in particular is critical for task-oriented dialogue assistants to identify the desired action (intent) and specific details (slots) @xcite . this is typically modeled as a semantic parsing problem (usually solved via a combination of ml and rules) with custom ontology that reflects capabilities of the system. creation of this custom ontology, and annotation of consistent data is highly non-trivial, and typically requires specialized skills @xcite . this limits extension of the ontology and generation of parsing data to a small group of experts. on the other hand, intent-slot concepts map well to functions and arguments of an api call, a paradigm well understood by software developers. therefore, making extension to the parser is the primary blocker to enabling support for new capabilities (domains) within a task-oriented assistant system. as shown in figure 1 , our goal is to enable non-nlp experts to define allowed intentslot combinations, and provide a small amount of non-nlp specialized labels, which we call simple labels. these data enable the creation of a parser for those intent-slot combinations. this new problem definition lies somewhere between zero-shot and few-shot learning. it requires zero fully annotated semantic parse examples, but does require some human produced labels. to this end, we develop a framework called open frame semantic parser (openfsp). open-fsp takes as input the developer's defined functions and their annotations to augment an existing assistant system for new tasks. underlying open-fsp is a two module model consisting of a general semantic parser and a matching module. the gen-eral semantic parser can identify the intent and slots according to the pre-defined domain agnostic ontology, while the matching module will take this intermediate representation and match them to the specific function and arguments defined in the domain specific ontology. in summary, our contributions are: (1) we formalize a new framework, namely openfsp, that allows for easy development of new domains without much expertise knowledge from software developers (2) we define a general-purpose domain agnostic ontology by analysing the semantic similarity of slots from topv2 @xcite , a well established task-oriented semantic parsing dataset (3) we propose an approach consisting of a parser and a matching module that can outperform strong baselines in the simple labels setting. multiple low resource ic/sf approaches were proposed @xcite @xcite . all of these approaches either rely on a non-trivial amount of training data (hundreds to thousands of examples), or use a fixed set of intents and slots, making it harder to adapt to new domains. our matching module shares some similarities with retrieval based systems @xcite , however these methods learn from standard utterance and semantic frames, instead of simple textual labels for each slot and intent. this is an important differentiator, as the annotation of even a small number of semantic frames requires overcoming a significant knowledge barrier. sentence encoders for language understanding one key aspect of our matching module is to encode the textual spans using sentence encoders. these models have the advantage of working well in low resource settings, and have been used in many applications including natural language inference @xcite , semantic textual similarity @xcite , dense passage retrieval @xcite , natural language explanations @xcite , and many others. the idea of using sentence embeddings for text classification has been previously explored @xcite . most notably, the work of @xcite proposed setfit, a prompt-free model that can learn text classification tasks from a handful of examples. however, none of these works directly applied to semantic parsing which require multiple consistent predictions from the input utterance. task-oriented ontologies task-oriented dialogue systems are natural language interfaces for a system that parses user utterances. a common approach in these systems is the use of ontologies to represent domain information @xcite . in general, such ontologies can be created manually using rule-based techniques, which is a highly accurate but time consuming and expensive approach that usually requires domain experts @xcite , or via ontology induction approaches using machine learning @xcite . while these approaches both involve trade-offs, curated ontologies can also be created via simplification of an existing ontology based on custom needs @xcite . on the other hand, our work simplifies the ontological creation by abstracting the existing functions and arguments defined by the application itself. ## problem definition the standard frame semantic parsing has two main tasks which maps the input utterance x with tokens x 1 , . . . , x n to some structured output frame y. for the slot-filling task, the output frame f consists of a set of m non-overlapping spans and their respective labels @xmath0, e i , l i )} m @xmath1:e i ] has label l @xmath2, where l is the set of possible labels (e.g., the text span "noon tomorrow" has the label sl:date_time). the intent classification assigns a label to the whole utterance. for simplicity, we assume that the intent can be thought of as another slot filling with s @xmath3, with l i as the intent type. da parse training data new domain simple labels + target frames da ontology utterance input x (string) ds frame output y (intent + slots) da frame intermediate y' (intent + slots) dap (parser) dsm (matching) [in:frame [sl:scope_temporal tonight ] [sl:scope_loc san francisco ] ] "what's happening in san francisco tonight?" [in:get_event [sl:date_time tonight ] [sl:location san francisco ] ] tology with a set of domain-agnostic labels l a , where |l a | << |l|. these domain-agnostic labels can be interpreted as generic "label types", with an existing many-to-one mapping ψ : @xmath4. this mapping is further described in appendix a.1. in our proposed simple label setting, training data for new domains will not include x, but only a subsequence of the tokens defined by the frame spans. the number of examples for each slot will be relatively small, with 5 to 50 examples per slot type. the set of all possible domain specific target frames f * (i.e., defined functions and their arguments) is assumed to be known a priory. ## approach our model is comprised of two main components, the domain agnostic parser (dap) and the domain specific matching (dsm). the system overview is shown in figure 2 . the dap is trained to take the input utterance x and output a domain agnostic frame f a , where span labels belong to l a . afterwards the dsm module will score the potential domain specific frames from f * according to their similarity to the domain agnostic frame f a . the dap module can be any frame semantic parser. ours is built on a span-pointer network @xcite , which is a nonautoregressive parser that replaces the decoding from text generation with span predictions. it is trained on domain agnostic data obtained from existing domain specific data using the slot label function ψ. the dsm module has to learn a similarity function between a text span and its slot label (same applies to intents). since the slot label scoring function has to be done in a few-shot setting, the dsm module uses a sentence encoder ϕ (reimers and gurevych, 2019) with a classification head on top. the module modifies a pre-trained transformer language model fine tuned to output semantically meaningful sentence embeddings. the sentence encoder ϕ is tailored to work on generic text, minimizing the distance between semantically similar sentences while maximizing the distance of dissimilar sentences. we use a classification head h over the produced sentence embedding of a given text span x. therefore, the probability score of a label l i can be computed as follows: note that the input training data does not contain fully annotated domain specific frames (only text and their intent-slot labels). for this reason, the dsm module has to aggregate these individual intent and slot probability scores to predict the full frame scores. the similarity score for a target domain specific frame note that for each new domain, only a very small set of parameters are trained (namely, the classification head h), which makes this approach computationally efficient. ## conclusion in this work we propose openfsp, a framework designed to simplify the process of adapting an existing task-oriented dialogue system to new domains. this framework enables non-experts to automatically build new domain ontologies from well defined software engineering concepts such as functions and arguments. we define a general-purpose domain agnostic ontology, that when combined with textual examples of new slots and intents (which we call simple labels), provides sufficient data to adapt the system to a new domain. finally, we propose a two-module system that can use these simple labels to reasonably parse input utterances into the domain specific frames. our experiments show that the proposed model outperforms strong baselines and is able to obtain results comparable with a fully supervised model (achieving 77.9% of its semantic frame accuracy). we hope that our work will facilitate the development of new assistant capabilities, allowing end-users to interact with more software applications through natural language. limitations domain-agnostic (da) slots represent the overal semantic space covered by underlying topv2 slots. given their coarse-grain nature, da slots are likely to be distributed more or less evenly across all domains. this assumption is key when training the parser on data that does not contain a particular target domain. in addition, we find that our approach is also sensitive to the types of linguistic structures accounted for by each da slot and works best when these structures are consistent across domains. more specifically, we conducted a series of leave-one-out (loo) experiments where a separate parser was learned for each domain using training data from all other domains and then tested on test data from the domain in question exclusively. error analysis of 100 randomly selected predictions in our loo model for the "alarm" domain revealed that 32% of the errors were utterances such as "i want alarms set for next monday at 6.00am and 7.00am" where the model predicted [sl:scope_temporal for next monday at 6.00am ] and [sl:scope_temporal 7.00am ] ] whereas a compound [sl:scope_temporal for next monday at 6.00am and 7.00am ] ] slot was expected. upon closer inspection, we observed that these [sl:scope_temporal x and y] compound nominal constructions appear predominantly (83.3%) in the alarm domain, hence causing inaccurate predictions in a loo model for this domain. for these types of errors to be mitigated in our approach, [sl:scope_temporal x and