Unsupervised Slot Schema Induction for Task-oriented Dialog (2022)

Abstract: Carefully-designed schemas describing how to collect and annotate dialog corpora are a prerequisite towards building task-oriented dialog systems. In practical applications, manually designing schemas can be error-prone, laborious, iterative, and slow, especially when the schema is complicated. To alleviate this expensive and time consuming process, we propose an unsupervised approach for slot schema induction from unlabeled dialog corpora. Leveraging in-domain language models and unsupervised parsing structures, our data-driven approach extracts candidate slots without constraints, followed by coarse-to-fine clustering to induce slot types. We compare our method against several strong supervised baselines, and show significant performance improvement in slot schema induction on MultiWoz and SGD datasets. We also demonstrate the effectiveness of induced schemas on downstream applications including dialog state tracking and response generation.

## introduction defining task-specific schemas, including intents and arguments, is the first step of building a taskoriented dialog (tod) system. in real-world applications such as call centers, we may have abundant conversation logs from real users and system assistants without annotation. to build an effective system, experts need to study thousands of conversations, find relevant phrases, manually group phrases into concepts, and iteratively build the schema to cover use cases. the schema is then used to annotate belief states and train models. this process is labor-intensive, error-prone, expensive, and slow @xcite @xcite . as a prerequisite, it hinders quick deployment for new domains and tasks. we therefore are interested in developing automatic schema induction methods in this work to create the ontology foot_0 from conversations for tod tasks. most existing approaches for slot schema induction rely on syntactic or semantic models trained with labeled data @xcite @xcite . our proposed method, on the other hand, is completely unsupervised without requiring generic parsers and heuristics, and hence portable to new tasks and domains seamlessly, overcoming the limitations of previous research. analogous to human experts, our procedure is divided into two general steps: relevant span extraction, and slot categorization. fig. 1 provides an overview of our approach. we introduce a bottomup span extraction method leveraging a pre-trained language model (lm) and regularized by unsupervised probabilistic context-free grammar (pcfg) structure. we also propose a multi-step auto-tuned clustering method to group the extracted spans into fine-grained slot types with hierarchy. we demonstrate that our unsupervised induced slot schema is well-aligned with expert-designed reference schema on the public multi-woz @xcite and sgd @xcite datasets. we further evaluate the induced schema on dialog state tracking (dst) and response generation to indicate usefulness and demonstrate performance gains over strong supervised baselines. meanwhile, our method is applicable to more realistic scenarios with complicated schemas. ## related work schema induction from dialog logs has not been studied extensively in the literature and developers resort to a patch work of tools to automate parts of the process. we first introduce related work on schema induction for dialog, and then discuss previous research on span extraction as part of schema induction for slots. schema induction for dialog motivated by the practical advantages of unsupervised schema induction such as reducing annotation cost and avoiding human bias, @xcite ; @xcite propose to induce spoken dialog grammar based on n-grams to generate fragments. different from studying semantic grammars, @xcite @xcite ; @xcite propose to utilize annotated framenet @xcite to label semantic frames for raw utterances @xcite . the frames are designed on generic semantic context, which contains frames that are related to the target domain (such as "expensiveness") and irrelevant (such as "capability"), while other relevant slots such as "internet" cannot be extracted because they do not have corresponding defined frames. this line of work focuses on ranking extracted frame clusters and then manually maps the top-ranked induced slots to reference slots. instead of framenet, @xcite extract features such as noun phrases (nps) using part-ofspeech (pos) tags and frequent words and aggregate them via a hierarchical clustering method, but only about 70% target slots can be induced. in addition to the unsatisfactory induction results due to candidate slot extraction, most of the previous works are only applicable to a single domain such as restaurant booking with a small amount of data, and require manual tuning to find spans and generate results. these methods are not easily adaptable to unseen tasks and services. the most comparable work to ours is probably @xcite , which is not bounded by an ex-isting set of candidate values so that potentially all slots can be captured. they propose to mix pos tags, named entities, and coreferences with a set of rules to find slot candidates while filtering irrelevant spans using manually updated filtering lists. in comparison, our method does not require any supervised tool and can be easily adapted to new domains and tasks with self-supervised learning. in addition to flexibility, despite our simple and more stable clustering process compared to their variational embedding generative approach @xcite , our method achieves better performance on slot schema induction and our induced schema is more useful for downstream tasks. we survey schema induction work for other natural language processing tasks in appendix a.11. ## unsupervised slot schema induction our proposed method for slot schema induction consists of a fully unsupervised span extraction stage followed by coarse-to-fine clustering. the resulting clusters can be mapped to slot type labels. ## experiments to examine the quality of our induced schema, we perform intrinsic and extrinsic evaluations. our intrinsic evaluation compares the predicted schema with the ground truth schema by measuring their overlap in slot types and slot values. this indicates how well our induced schema aligns with the expert annotation. the extrinsic evaluation estimates the usefulness of the induced schema for downstream tasks, for which we consider dialog state tracking and response generation tasks. experiments are conducted on @xcite and sgd @xcite datasets following previous research. see appendix a.1 for implementation details. we also apply and evaluate our method for both intent and slot schema induction on realistic scenarios (see section 5). baselines we compare our proposed approach with different setups against dsi @xcite , which uses supervised tools and heuristics. we evaluate different span extraction methods including using parsers only, leveraging distance functions from lms, and combining lms with unsupervised pcfg. specifically, np extracts all noun phrases foot_3 , dsi cand. uses the same candidates phrases as @xcite extract phrases from an unsupervised and supervised structure respectively by taking the smallest constituents above the leaf level. these baselines solely rely on parsers. for our bottom-up attention-based lm methods (section 3.2), we compare spans extracted using representations from bert @xcite , @xcite , tod-bert @xcite , and our span-based tod pre-training from masking random spans (tod-span). lastly, we combine the lms with unsupervised pcfg structures. due to space constraints, we show results on multiwoz in this section. observations on sgd can be found in the appendix. ## conclusion in this paper, we propose a fully unsupervised method for slot schema induction. compared to previous research, our method can be easily adapted to unseen domains and tasks to extract target phrases before clustering into fine-grained groups without domain constraints. we conduct extensive experiments and show that our proposed approach is flexible and effective in generating accurate and useful schemas without task-specific rules in both academic and realistic datasets. we believe that our method could also be applied to other languages (since no supervised parser, model, or heuristic is required) and tasks such as question answering where the answering phrase is not explicitly annotated @xcite . in the future, we plan to extend our method to problems with more complex structures and data where slots are less trivial to identify.