S lot R efine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling (2020)

Abstract: Slot filling and intent detection are two main tasks in spoken language understanding (SLU) system. In this paper, we propose a novel non-autoregressive model named SlotRefine for joint intent detection and slot filling. Besides, we design a novel two-pass iteration mechanism to handle the uncoordinated slots problem caused by conditional independence of non-autoregressive model. Experiments demonstrate that our model significantly outperforms previous models in slot filling task, while considerably speeding up the decoding (up to x10.77). In-depth analysis show that 1) pretraining schemes could further enhance our model; 2) two-pass mechanism indeed remedy the uncoordinated slots.

## introduction slot filling (sf) and intent detection (id) play important roles in spoken language understanding, especially for task-oriented dialogue system. for example, for an utterance like "buy an air ticket from beijing to seattle", intent detection works on sentence-level to indicate the task is about purchasing an air ticket, while the slot filling focus on words-level to figure out the departure and destination of that ticket are "beijing" and "seattle". in early studies, id and sf were often modeled separately, where id was modeled as a classification task, while sf was regarded as a sequence labeling task. due to the correlation between these two tasks, training them jointly could enhance each other. @xcite propose a joint model using bidirectional gated recurrent unit to learn the representation at each time step. meanwhile, a max-pooling layer is employed to capture the global features of a sentence for intent classification. @xcite cast the slot filling task as a tag generation problem and introduce a recurrent neural network based encoder-decoder framework with attention mechanism to model it, meanwhile using the encoded vector to predict intent. @xcite and @xcite dig into the correlation between id and sf deeper and modeled the relationship between them explicitly. @xcite propagate the token-level intent results to the sf task, achieving significant performance improvement. briefly summarized, most of the previous works heavily rely on autoregressive approaches, e.g., rnn based model or seq2seq architecture, to capture the grammar structure in an utterance. and conditional random field (crf) is a popular auxiliary module for sf task as it considers the correlations between tags. thus, several state-of-the-art works combine the autoregressive model and crf to achieve the competitive performance, which therefore are set as our baseline methods. however, for sf task, we argue that identifying token dependencies among slot chunk is enough, and it is unnecessary to model the entire sequence dependency in autoregressive fashion, which leads to redundant computation and inevitable high latency. in this study, we cast these two tasks jointly as a non-autoregressive tag generation problem to get rid of unnecessary temporal dependencies. particularly, a transformer @xcite based architecture is adopted here to learn the representations of an utterance in both sentence and word level simultaneously (sec. §2.1). the slots and intent labels are predicted independently and simultaneously, achieving better decoding efficiency. we further introduce a two-pass refine mechanism (in sec. §2.2) to model boundary prediction of each slots explicitly, which also handle the uncoordinated slots problem (e.g., i-song follows b-singer) caused by conditional independence attribute. experiments on two commonly-cited datasets the slot results in the green dotted box are refined results by second pass. note that the initial tag embedding "o" added to each inputting position is designed for the two-pass mechanism(sec. §2.2). show that our approach is significantly and consistently superior to the existing models both in sf performance and efficiency (sec. §3). our contributions are as follows: ## proposed approaches in this section, we first describe how we model slot filling and intent detection task jointly by an nonautoregressive model. and then we describe the details of the two-pass refine mechanism. the brief scheme of our model is shown in figure 1 , details can be found in the corresponding caption. note that we follow the common practice @xcite @xcite to use "inside-outside-beginning (iob)" tagging format. ## conclusion in this paper, we first reveal an uncoordinated slots problem for a classical language understanding task, i.e., slot filling. to address this problem, we present a novel non-autoregressive joint model for slot filling and intent detection with two-pass refine mechanism (non-autoregressive refiner), which significantly improves the performance while substantially speeding up the decoding. further analyses show that our proposed non-autoregressive refiner has great potential to replace crf in at least slot filling task. in the future, we plan to extend our nonautoregressive refiner to other natural language understanding (nlu) tasks, e.g., named entity recognition @xcite , semantic role labeling @xcite , and natural language generation (nlg) tasks, e.g., machine translation @xcite , summarization @xcite .