Discovering Dialogue Slots with Weak Supervision (2021)

Abstract: Task-oriented dialogue systems typically require manual annotation of dialogue slots in training data, which is costly to obtain. We propose a method that eliminates this requirement: We use weak supervision from existing linguistic annotation models to identify potential slot candidates, then automatically identify domain-relevant slots by using clustering algorithms. Furthermore, we use the resulting slot annotation to train a neural-network-based tagger that is able to perform slot tagging with no human intervention. This tagger is trained solely on the outputs of our method and thus does not rely on any labeled data. Our model demonstrates state-of-the-art performance in slot tagging without labeled training data on four different dialogue domains. Moreover, we find that slot annotations discovered by our model significantly improve the performance of an end-to-end dialogue response generation model, compared to using no slot annotation at all.

## introduction task-oriented dialogue systems typically use annotation based on slots to represent the meaning of user utterances @xcite . slots are attributes relevant to completing the task (e.g., price, food type, area). the sets of slots and their values typically need to be designed in advance by domain experts. slots and their values are tracked over the course of the dialogue, forming dialogue state, which allows a dialogue system to plan the next actions effectively @xcite . getting raw data for dialogue system training is not difficult, especially if we restrict the target domain. a requirement for dialogue state labels makes this process much more costly. however, both traditional pipeline systems @xcite and end-to-end task-oriented architectures @xcite typically require such annotation. while some systems use implicit, latent state representation and do not require annotation @xcite , the behavior of such systems is hard to interpret or control. there are several works aiming at keeping interpretability and reducing the annotation needs by automating it @xcite or transferring annotation across domains @xcite ), but they still require significant manual effort. in this paper, we present a novel approach to discovering a set of domain-relevant dialogue slots and their values given a set of dialogues in the target domain (such as transcripts from a call center). our approach requires no manual annotation at all in order to tag slots in dialogue data. this substantially simplifies dialogue system design and training process, as the developer no longer needs to design a set of slots and annotate their occurrences in training data. we discover slots by using unsupervised clustering on top of annotation obtained by domain-independent generic models such as a semantic frame parser or a named entity recognizer (ner). to illustrate our approach, let us consider an example given in figure 1 . find a chinese restaurant that's cheap. ## related work the idea of using weak supervision to perform finegrained language understanding based on domainrelevant (slot-like) attributes was proposed by @xcite , who construct a triplebased database of entity relations based on web search. they exploit the structure of in-domain web pages to obtain semantic annotations. there are also similar works on relation detection @xcite or entity extraction @xcite . this approach is, however, limited by requiring structured web pages as underlying data. chen et al. ( 2014 ) combine semantic frame parsing with word embeddings for weakly supervised semantic slot induction. @xcite also use semantic frames, construct lexical knowledge graphs and perform a random walk to get slot candidates. however, both approaches only output a ranking of potential slot candidates based on frames. since frame annotation is very fine-grained, this produces a huge number of candidates, requiring their manual merging into slots for any practical use. in contrast, we determine domain-relevant slots automatically. coope et al. (2020) focus on a few-shot setting and perform span extraction of slot values using pretrained models. their approach, however, still requires some expert annotation. another direction of research focuses on zero-shot slot filling. @xcite 's recurrent-neuralnetwork-based slot tagger is pretrained on multiple domains and takes a textual description of the target slot on the input in addition to the user utterance. this way, adapting to a new domain only involves providing new slot descriptions. further works extend this idea with more complex architectures @xcite . unsupervised and semi-supervised methods were also investigated for predicting intents (user and our slot tagger (bottom). although "afghan" food is not in the frame parser output, our tagger was able to recognize it. the change in value for slot-1 (corresponding to food type) is successfully captured in the second utterance. this shows that our model can categorize entities (both "afghan" and "asian" relate to the same slot). yang et al. (2014) use semisupervised intent clustering, with manual annotation to seed and interpret the clusters. chen et al. (2016) introduced a model for zero-shot intent embedding prediction based on similarity to known intents. shi et al. (2018) proposed a fully unsupervised intent detection model with the use of sentence clustering based on sentence-level features. most applications of unsupervised or semisupervised methods to end-to-end dialogue response generation avoid explicit dialogue state modeling (e.g., @xcite @xcite . they aim at a non-task-oriented setting, where state interpretability or response controllability are less of a concern. other works in task-oriented dialogues use transfer learning for adapting to low-resourced target domains @xcite , but also keep the dialogue state representation latent. in contrast, @xcite propose to model the dialogue state explicitly, in a semi-supervised way. they extend the end-to-end encoder-decoder sequicity model of @xcite . section 4) by introducing an additional decoder that has access to posterior information about the system response. this allows them to train a state representation with a reconstruction loss on unsupervised examples, using the state as a limited memory for essential concepts (roughly corresponding to slots). their method can be applied in fully unsupervised way, but it still requires some amount of in-domain annotations to achieve good performance. our work aims at explicit dialogue state modeling without the need for any in-domain supervision. ## method our slot discovery method has three main stages: (1) we obtain weak supervision labels from auto-matic domain-generic annotation. (2) we identify domain-relevant slots based on the annotation labels by iteratively (a) merging and (b) ranking and selecting most viable candidates (section 3.2). (3) we use the discovered slots to train an independent slot tagger (section 3.3). ## application in dialogue response generation to verify the usefulness of the labels discovered by our method, we use them to train and evaluate an end-to-end task-oriented dialogue system. we choose sequicity @xcite for our experiments, an lstm-based encoder-decoder model that uses a system of copy nets and two-stage decoding. first, it decodes the dialogue state, so the database can be queried externally. in the subsequent step, sequicity generates the system response conditioned on the belief state and database results. this architecture works with a flat representation of the dialogue state, i.e. the state is represented as a sequence of tokens -slot values. the default sequicity model uses gold-standard dialogue state annotation. however, a compatible state representation is directly obtainable from our labels, simply by concatenating the labels aggregated in each turn from user utterances. whenever a new value for a slot is found in user input by our tagger, it is either appended to the state representation, or it replaces a previous value of the same slot. this artificial supervision thus allows us to provide a learning signal to the sequicity model even without manually labeled examples. ## experiments we evaluate our approach to slot discovery by comparing the resulting slot labels to gold-standard supervised slot annotation. additionally, we evaluate the structure of clusters created during the selection process (section 3.2.2) by comparing it to gold-standard user intents. we also test the use-fulness of our labels in a full dialogue response generation setup (section 4), where we compare to gold-standard dialogue tracking labels. ## results and discussion we first evaluate the main task of slot tagging and include a manual error analysis, then present detailed results for subtasks (slot candidate ranking and merging) and additional tasks (intent clustering and full response generation). slot tagging is evaluated in ## conclusion we present a novel approach for weakly supervised natural language understanding in dialogue systems that discovers domain-relevant slots and tags them in a standalone fashion. our method removes the need for annotated training data by using off-theshelf linguistic annotation models. experiments on five datasets in four domains mark a signifi-cant improvement in intrinsic nlu performance over previous weakly supervised approaches; in particular, we vastly improve the slot recall. the usefulness of slots discovered by our method is further confirmed in a full dialogue response generation application. code used for our experiments is available on github. 16 a drawback of our approach is the reliance on existing linguistic annotation models. we show that the method is able to combine multiple annotation sources and create a tagger that functions as a standalone component, generalizing better than the original annotation and thus lowering this dependency. nevertheless, the results are still somewhat limited by the input annotation structure and quality. in future, we plan to further improve the model by unsupervised selection of slot candidates via keyword extraction and clustering, as well as by taking context information from preceding dialogue turns into account. we also want to focus more on the intent detection aspect of our work.