Cross- TOP : Zero-Shot Cross-Schema Task-Oriented Parsing (2022)

Abstract: Deep learning methods have enabled taskoriented semantic parsing of increasingly complex utterances. However, a single model is still typically trained and deployed for each task separately, requiring labeled training data for each, which makes it challenging to support new tasks, even within a single business vertical (e.g., food-ordering or travel booking). In this paper we describe Cross-TOP (Cross-Schema Task-Oriented Parsing), a zero-shot method for complex semantic parsing in a given vertical. By leveraging the fact that user requests from the same vertical share lexical and semantic similarities, a single cross-schema parser is trained to service an arbitrary number of tasks, seen or unseen, within a vertical. We show that Cross-TOP can achieve high accuracy on a previously unseen task without requiring any additional training data, thereby providing a scalable way to bootstrap semantic parsers for new tasks. As part of this work we release the FoodOrdering dataset, a task-oriented parsing dataset in the food-ordering vertical, with utterances and annotations derived from five schemas, each from a different restaurant menu.

## introduction propelled by deep learning, task-oriented parsing has made significant strides, moving away from flat intents and slots towards more complex treebased semantics that can represent compositional meaning structures @xcite @xcite . however, most semantic parsing systems remain task-specific: they can only produce representations with the set of intents and slots seen during training. to support multiple tasks, this approach requires collecting data, training, and maintaining a model for each task separately. this is costly when multiple tasks need to be supported, as is usually the case for digital voice assistants such as alexa and google assistant, which may need to support hundreds or thousands of different tasks in a given business vertical (e.g., restaurants in the food-ordering vertical, hotels in the travel vertical, and so on). in this paper we present cross-top, a method for building a single semantic parsing model that can support an arbitrary number of tasks in a given vertical. user requests pertaining to the same vertical have lexical and semantic similarities; their main differences lie in their unique schemas. in the food-ordering domain, for example, a customer may request a main dish with various options and possibly a drink and a side. however, depending on the specific restaurant menu, the output semantic representations can differ greatly; see figure 1 . cross-top makes use of a powerful pretrained transformer-based encoder-decoder language model, with schema-specific context added to the input along with the utterance. in this way, the model learns to generate parses for a new, unseen task, by attending to the schema in the input rather than by needing to see it during training. we show that this approach is quite effective and provides a quick solution to the practical problem of bootstrapping semantic parsers for new tasks within a vertical, using a single model in production. the parser is trained on a number of initial tasks, where each task has some training data available. moreover, we assume that every task has a unique schema. that schema consists of all possible intents and slots for the task at hand; both intents and slots can be arbitrarily nested @xcite . for every slot s, the schema also includes naturallanguage phrases for the various values of s. all schemas for the five tasks in our dataset can be found in appendix b. cross-top uses constrained decoding to ensure that it generates well-formed parses that can be resolved to executable representations that can be directly used by the back end. most zero-shot cross-schema semantic parsing work has been in the context of the text-to-sql task @xcite @xcite @xcite . cross-schema taskoriented parsing introduces its own challenges. in sql, the schemas are database schemas, and the parser is trained on some initial databases and then evaluated on another database. there is a lot of invariant structure across different tasks in the output space (since output sequences are always sql queries), as well as common patterns in how sql structures tend to align with natural language. however, for the schemas defined in task-oriented parsing for the food-ordering domain, the only invariant structures are the parentheses and some lexical overlap among the intents and slots. therefore, cross-schema parsing in general is more challenging for task-oriented parsing. however, restricting the scope to a given vertical imposes more common structure that can prove helpful. to evaluate our methodology, we focus on tasks in the food-ordering domain, where each task contains examples from a restaurant with the schema generated from its menu. our main contributions are as follows: ## model our method trains a single schema-aware model to serve multiple tasks and bootstrap new ones from the same business vertical in a zero-shot setting. it leverages the transfer learning capabilities of a transformer-based pretrained encoder-decoder language model. terminology each task is defined by a unique schema consisting of intents, slots belonging to those intents, and catalogs enumerating the possible slot values for each slot. for example, in the pizza-ordering task the topping slot belongs to the pizzaorder intent, and values for this slot could be mushrooms, pepperoni, and so on. in our predefined catalogs, multiple slot values could refer to the same slot value entity, for example peppers and green peppers can both be mapped to topping_peppers-or perhaps topping_35in the back end. cross-top predicts parse trees that contain slot values, which are then entity-resolved into those unique back-end identifiers through this many-to-1 mapping. a task schema can optionally define invocation keywords for each intent, to identify how these are expressed in natural language, for example {drink, drinks} for a drink_order intent. this is used for augmenting model input with fuzzymatched schema elements later on. fuzzy stringmatching algorithms compute lexical similarity between strings. if some schema elements have a significant overlap with certain utterance tokens, then there is a "match" and that schema element will be appended to the input utterance before encoding. model inputs as just mentioned, to achieve zero-shot cross-schema parsing, we append fuzzymatched schema elements to input utterances. given an utterance u, assume our fuzzy-matching process (described later) determined that the intents i 1 and i 2 are present in the request, with slot/slotvalues s 1,1 /v 1,1 for i 1 , as well as s 2,1 /v 2,1 and s 2,2 /v 2,2 for i 2 foot_0 . the input to cross-top is then serialized into the following format: model outputs the model is trained to generate a linearized parse tree similar to the target shown in figure 2 , which is reminiscent of the top decoupled notation @xcite . top decoupled is itself derived from the top notation @xcite by removing tokens that are not direct children of slot nodes. unlike top decoupled, leaf nodes in our output semantics are not tokens copied from the source utterance, but instead must be valid slot values belonging to the task's catalogs. as exemplified in figure 2 , the fuzzy-matched slot value for the utterance segment large-size is the catalog entry large. it can happen that utterance token and catalog value are identical, as is the case for peppers here. by predicting slot values instead of unresolved utterance tokens, cross-top jointly learns to perform semantic parsing and entity resolution, thus eliminating the need to train and maintain a separate entity resolution system for every new task. ## the foodordering dataset we release a dataset for cross-schema zero-shot task-oriented parsing: the foodordering dataset, foot_5 comprising five food-ordering tasks for five fictitious restaurants: pizza, sub, burrito, burger and coffee. ## experimental setup our experimental setup reflects the practical scenario of having to scale a technology to service multiple applications under constrained production resources. we consider a single model to serve all tasks, so we train with synthetic data for only three of the tasks (pizza, burrito and sub), and test zero-shot generalization on two unseen tasks (burger and coffee). training details in this work we use bart-large @xcite , a transformer-based pretrained encoder-decoder language model. we finetune the publicly available 24-layer bart-large checkpoint foot_7 totaling 406m parameters, using the transformers codebase. we expand the target vocabulary by adding special tokens for input markers [i], [s] and [v]. the training dataset was created by concatenating synthetic data from the three training tasks. models are trained for 50 epochs with early stopping patience of 4, using cross-entropy sequence loss and the adamw optimizer. we use the human-generated data of the three training tasks as our development set for early stopping and hyperparameter tuning. hyperparameter tuning is described in appendix e. our best model uses a batch size of 16, learning rate 1e-05 and linear learning rate scheduler with warm-up ratio of 0.1. the hyperparameter no_repeat_ngram_size was disabled by setting it to 0. formance. it checks for an exact match between the golden and predicted trees, where sibling order does not matter. the golden parse trees are executable representations (ready for consumption by an appropriate back end) that contain resolved entity names instead of slot values identified by utterance segments. these entities are fully determined by the many-to-1 mapping mentioned in section 2. validation performance is computed on the aggregated validation sets for the three training tasks. test performance is reported for tasks individually. we used a beam size of 6 for validation and testing. pre-processing and post-processing when appending the schema elements to the input utterance we do not include the slot/slot-value pair number, 1 from the fuzzy matching process if it's the only quantity matched. 9 this choice was made after observing that the slot values a/an can easily trigger false positives in fuzzy matching. for example, in the utterance an order of two sprites, the numeric quantity to extract is two, but the token an would trigger an extra unnecessary match. at inference time, if no number was generated for an intent, we add back (number 1) as a default to the predicted parse tree. before computing unordered em scores, all slot values are resolved into the appropriate entity names using the many-to-1 mapping mentioned earlier. ## results and analysis in the zero-shot setting, cross-top achieves 73% unordered em on burger and 55% on coffee ( schema-aware encoding enables zero-shot transfer learning. the main strength of cross-top is training and maintaining a single model that can serve multiple tasks within the same business vertical, and bootstrapping new tasks without retraining. the zero-shot results in for completeness, we show that the zero-shot ability does not simply come from the conjunction of constrained decoding and bart's extensive pretraining: we perform an ablation exercise where the input to bart-large contains no schema information at all, but constrained decoding is enabled. as can be seen in the second row of schema-aware decoding ensures proper executable parses. schema-aware constrained decoding ensures that cross-top generates fully executable parse trees. without this component, performance drops by 20 absolute points, as shown in the third row of cross-top improves as more training tasks are added. while our main result shows that training with only few tasks allows zero-shot transfer to new tasks with no retraining, a realistic production scenario would be to periodically retrain the model by incorporating new training data. to quantify the benefits of adding more tasks, we compare training cross-top using one, two or three tasks. the results for training on one task are an average over three models, one trained on pizza only, one on burrito and one on sub. likewise, results for training on two tasks are an average of three models, one trained on pizza+burrito, one on burrito+sub and the other on pizza+sub. as shown in dependency on fuzzy matching cross-top relies on the quality of the fuzzy-matching process that determines which schema elements are encoded along with the utterance tokens. it can be challenging to recover from a fuzzy matching failure that ends up omitting a slot value from the input. in burger, such failures account for only 1% of all test utterances. in coffee that phenomenon is more prominent, with 7% of test utterances presenting at least one missing element from the fuzzy-matched schema. these limitations can be addressed by making the fuzzy-matching algorithm more robust and/or by adding unrecognized slot values as extra entries in the slot's catalog. the latter option is appealing, as it involves no model retraining, but does not suffice, as there is no obvious way to automate it. we upper-bound the impact of any candidate fix by providing an oracle schema for all utterances, and observe in the last row of 6 related work slot filling traditionally, task-oriented parsing for flat intents and slots has been framed as a combination of intent classification and slot labeling @xcite , possibly with an additional domain classification component. several authors have addressed zero-shot solutions in this field. qasf @xcite is a qa-driven approach that extracts slot-filler spans from utterances using a question-answering model. both @xcite and @xcite tag words with slots using slot descriptions and context-aware representations of the utterance. these solutions don't apply to structured (compositional) semantic representations, or to multiple intents in a single utterance, both of which are handled by cross-top. task-oriented parsing in the more general area of task-oriented parsing, where hierarchical representations are featured, the authors are not aware of other zero-shot cross-schema work. there is some work in the few-shot setting @xcite , where data from multiple domains is used during an additional stage of fine-tuning combined with meta-learning. text-to-sql some of the most relevant related zero-shot work is in text-to-sql semantic parsing. in this area, a challenging dataset, spider @xcite , is the most common dataset used to test zero-shot solutions. the gazp @xcite ) method generates synthetic training data for the new schema environment and requires a retraining of the neural parser, not making it as convenient of a zero-shot method. rat-sql @xcite moves away from needing to retrain the parser, and focuses on jointly encoding the schema and utterance tokens. bridge @xcite is the main inspiration for our work, as it encodes the utterance and schema together, and augments the input with anchor texts, which are database values from tables, designed to better bridge utterance tokens to database tables, columns and values. another notable contribution intended to bridge the gap between natural language and machineexecutable representations is the work of @xcite , which leverages an intermediate representation to go from text to sql. ## conclusion we presented cross-top, a zero-shot method for cross-schema task-oriented parsing that eliminates the need to retrain and maintain a new model for every new task in a business vertical. we released a new dataset illustrative of five real-world applications in the food-ordering vertical. we showed that cross-top reaches up to 73% em accuracy in zero-shot transfer, making it a viable technique for quickly bootstrapping a parser for a new task. future work could further enrich the joint encoding of utterances and task schemas, while an additional thread of work could study how to best leverage limited annotated data that may be available for a new task.