An Empirical Analysis of Leveraging Knowledge for Low-Resource Task-Oriented Semantic Parsing (2023)

Abstract: Task-oriented semantic parsing has drawn a lot of interest from the NLP community, and especially the voice assistant industry as it enables representing the meaning of user requests with arbitrarily nested semantics, including multiple intents and compound entities. SOTA models are large seq2seq transformers and require hundreds of thousands of annotated examples to be trained. However annotating such data to bootstrap new domains or languages is expensive and error-prone, especially for requests made of nested semantics. In addition large models easily break the tight latency constraints imposed in a user-facing production environment. As part of this work we explore leveraging external knowledge to improve model accuracy in low-resource and low-compute settings. We demonstrate that using knowledge-enhanced encoders inside seq2seq models does not result in performance gains by itself, but jointly learning to uncover entities in addition to the parse generation is a simple yet effective way of improving performance across the board. We show this is especially true in the low-compute scarce-data setting and for entity-rich domains, with relative gains up to 74.48% on the TOPv2 dataset.

## introduction fostered by nlp advances, virtual assistants such as google home or alexa are becoming increasingly competent to address complex yet natural, everyday user needs. while requests as simple as "turn off the living room lights when the movie starts" could not be fulfilled with legacy systems that assigned a single user intent to each utterance and a single slot label to each token in an utterance @xcite , recent works on task-oriented semantic parsing (gupta * the work was done while at amazon alexa ai. @xcite represent utterance semantics with arbitrarily nested trees (figure 1 ), thus handling the above use-case among others (e.g. multiple intents, cross-domain intents, compound entities, etc.). the research community tackles this task with success by treating it as a seq2seq generation task where a linearized semantic tree is predicted iteratively @xcite ), but such approaches fall short when confronted by real-life constraints such as strict run-time latency and scarcity of quality training data. manual data annotation of training examples is a costly and error-prone process, which is exacerbated as utterance target representations become richer (more nested). the impact of data scarcity has been quantified in recent years with the introduction of the topv2 benchmark @xcite ) that provides low-resource scenarios for task-oriented parsing. popular approaches to overcome data scarcity include synthetic data augmentation @xcite @xcite , transfer learning @xcite , and meta-learning @xcite @xcite . in this paper, we explore if we can model richer token representations for mentions by leveraging external knowledge, as mentions are fundamental to generating the correct parse. the backbone motivation lies in the observation that several everyday nlp applications involve real-life entities referenced in knowledge bases (for e.g. street names, sports events, or public figures). this information can be utilized for enhancing downstream nlp tasks. for example the request "play the green line" could refer to either a movie name or a song name, modeling this mention appropriately for the decoder could improve performance while generating a parse. this is particularly appealing in the low-data regime, for which rare entities are unlikely to be represented in the training data at all. additionally, building entity embeddings through entity-focused modeling objectives has shown promising results in entity based nlp tasks such as named entity recognition @xcite and entity linking @xcite . while there has been prior work to leverage knowledge for generation tasks @xcite @xcite this largely focused on unstructured text generation tasks such as question-answering or entity linking. to the best of our knowledge, we are the first to investigate its use in seq2seq models for task-oriented semantic parsing, a complex and structured text generation task. we present an empirical analysis of using knowledge to improve accuracy of semantic parsing models, with a special focus on low-latency models such as small-decoder seq2seq models and nonauto-regressive models like rine @xcite . our contributions are as follow: voice assistants typically transform a voice recording into text, that is further mapped to a backend exploitable representation containing the semantics of the request: the user intent, the invoked entities, relations between those entities, etc. task-oriented parsing was popularized with the introduction of the top dataset @xcite , and is usu-ally treated as a seq2seq task where utterance tokens are copied into a semantic tree constructed auto-regressively @xcite . however such models are not always applicable in production environments with strict memory and latency constraints. this limitation is commonly addressed by reducing model sizes @xcite and leveraging non-auto-regressive modeling @xcite @xcite . knowledge-enhanced lms retrieval-based seq2seq models such as realm @xcite and atlas @xcite leverage factual knowledge from a corpus or knowledge-graph during training and inference, hence incur a considerable latency cost, despite attempts to make the retrieval more efficient @xcite . given our low-latency setup, we focus on parametric knowledge that is learnt during the pre-training or fine-tuning process of large language models (llms), resulting in embeddings that do not require explicit knowledge retrieval at inference. knowledge-enhanced pretraining focuses on modeling entities: wklm @xcite learns to determine if an entity was replaced with another entity of the same type in addition to masked language modeling (mlm) and shows gains on downstream knowledge-intensive tasks such as question-answering (qa) and relation extraction (re). luke @xcite explicitly models entity-embeddings through entityembedding prediction during mlm and entityentity self-attention layers during fine-tuning, with gains on named entity recognition (ner), qa and re. kbir @xcite learns to reconstruct keyphrases in a combination and extension of wklm and spanbert @xcite , improving keyphrase extraction/generation tasks. lastly, blink @xcite ) learns entitydisambiguation by aligning entity surface forms to their descriptions resulting in rich entity embeddings. work in the area of parametric knowledgeenhanced seq2seq models is limited to keybart @xcite for keyphrase generation and genre @xcite for entity disambiguation. ## methods we explore two complementary methods for leveraging knowledge: (1) fine-tuning knowledgeenhanced encoders for task-oriented semantic parsing inside seq2seq models, and (2) multi-tasking the parse generation with a mention detection task. task formulation we follow the task formulation of the seq2seq-ptr model as a sequence-tosequence generation setup @xcite . the source sequence is an utterance and the target sequence is a linearized representation of the semantic parse. the target sequence is modified to contain only intent and slot labels or pointers to tokens in the utterance. following @xcite and subsequent work we use the decoupled format that limits prediction to tokens that are leaves of slots foot_0 as it yielded better downstream performance in previous work. we illustrate the format used with an example from the topv2 dataset below: each @ptr i token here points to the i th token in the source sequence. here @ptr 3 corresponds to the word minneapolis. proposed architecture based on the observation that many slot-values present in our task are actual real-life entities, we hypothesize that learning more effective representations of these slot-values may result in generating more accurate semantic parses as mentions play a critical role in understanding the utterance. we use knowledge-enhanced pretrained encoders (as described in section 2) inside the seq2seq-ptr architecture used in @xcite , extended to multitask training of parse generation and training of the encoder to perform token classification (mention detection), as it aligns with classification-based pre-training of the encoder. we anticipate that the multitask training will allow the knowledge-enhanced encoder representations to be attended and leveraged more effectively by the decoder generating the parse. further, by modeling mentions inherently present in the annotated data, this serves well for low-resource use cases since we maximize the potential to learn from the data available. figure 1 illustrates our proposed architecture, whereby for a given input utterance [x 1 , .., x n ] we obtain encoder representation [e 1 , ..., e n ], from which we jointly learn two tasks: a) mention detection and b) parse generation. ## results all our results are source trained + fine-tuned, unless specified otherwise. we perform 3 runs across each experiment setting and report average scores and standard deviations. our findings are as follows: knowledge-enhanced encoders don't improve generative semantic parsing in both the multitasking and non-multitask setting, we see that the best performing model across data-regimes and domains is not consistently the knowledge-enhanced encoder luke. in the full decoder setting, luke-encoder based models perform on par but no better than the vanilla robertaencoder based models. we also note that both these model underperform bart, but that the gap bridges as we add more training samples. in the light-decoder setting, we also see similar trends, however an interesting finding is that bart tends to underperform when compared to roberta and luke, even in the full data setting. this could be attributed to the smaller encoder size for bart. the above findings are contrary to expected performance improvements typically seen using knowledge-enhanced encoders for other entityrelated tasks such as ner, re and qa. we believe the reason for this is that the aforementioned tasks are all classification-based tasks that are able to leverage the entity representations in making decisions on class-types, but in contrast task-oriented semantic parsing is a complex generation task. even though entities play a critical role, the entity representations are not able to effectively guide the from-scratch decoder. this problem is alleviated to a certain extent through the multitask training that we hypothesize is able to jointly learn representations of entities that will guide the decoder, but these jointly learnt representations do not necessarily benefit from the knowledge-enhanced encoder. further, the application of source training potentially wipes out any gains the knowledgeenhanced encoder had over the vanilla counterparts as they have seen sufficient data to negate the gains through knowledge-enhancements as discussed in section 6. however knowledge-enhanced encoders can bring gains when reformulating parsing as a classification task as shown in multitasking with mention detection is an efficient way to leverage knowledge and improves performance across the board on the two topv2 domains with strong entity presence (navigation and event), especially in the lightweight decoder setting (up to 74.48%, source-training is essential as shown in ## case study on knowledge-enhanced encoders to better understand the lack of performance boost by ke encoders we propose a deeper dive on using luke as well as two alternative ke encoders. further enhancements to luke only result in limited gains for our previous experiments we restrict to using only luke's token embeddings to make a fair comparison with roberta. however the original luke encoder is armed with many more parameters, including the entity-entity self-attention that allows us to leverage richer entity embeddings. we explore using the entity em- other ke encoders than luke lead to similar conclusions we explore using other knowledgeenhanced encoders: kbir and blink. kbir is potentially better suited as it is pre-trained to exploit keyphrases, which are closer to slot-values than entities. however ## conclusion & future work we presented an empirical analysis of how we can leverage external knowledge for task-oriented semantic parsing in the low-resource and lowcompute settings, by conducting a rigorous set of experiments. we demonstrated that simply using a knowledge-enhanced encoder is not sufficient to improve performance over baselines for the complex task of sequence generation, but shows promising result when the task is reformulated as a classification task. we presented a multitask learning framework that leverages external knowledge and requires little to no extra data annotation, and demonstrated its effectiveness in the low-data and low-compute settings. future work could probe the type of knowledge learned by this method, and attempt to apply it to other entity-rich tasks, across model architectures. it could also explore an indepth error analysis of where knowledge-enhanced encoders fail in order to address these shortcomings. further we could extend this work for retrievalbased seq2seq models to improve task-oriented semantic parsing.