QA -Driven Zero-shot Slot Filling with Weak Supervision Pretraining (2021)

Abstract: Slot-filling is an essential component for building task-oriented dialog systems. In this work, we focus on the zero-shot slot-filling problem, where the model needs to predict slots and their values, given utterances from new domains without training on the target domain. Prior methods directly encode slot descriptions to generalize to unseen slot types. However, raw slot descriptions are often ambiguous and do not encode enough semantic information, limiting the models’ zero-shot capability. To address this problem, we introduce QA-driven slot filling (QASF), which extracts slot-filler spans from utterances with a span-based QA model. We use a linguistically motivated questioning strategy to turn descriptions into questions, allowing the model to generalize to unseen slot types. Moreover, our QASF model can benefit from weak supervision signals from QA pairs synthetically generated from unlabeled conversations. Our full system substantially outperforms baselines by over 5% on the SNIPS benchmark.

## introduction automatic slot filling, which extracts task-specific slot fillers (e.g. flight date, cuisine) from user utterances, is an essential component to spoken language understanding @xcite . as shown in figure 1 , the model predicts the slot filler "joe a. pass" for the slot type "artist" given an input utterance. however, fully supervised slot filling models @xcite require labeled training data for each type of slot @xcite . it is even more of a problem for dataintensive models @xcite . this makes the development of new domains in these systems a challenging and resource-intensive task. this has motivated studies in cross-domain zeroshot learning for the slot-filling task (zssf), where * work done during internship at google research. the goal is to achieve good slot-filling performance on new domains without requiring additional training data. previous work @xcite ; @xcite ) often uses a sequence tagging approach (similar to the upper image in figure 1 in a high-level way). to achieve zero-shot domain transfer, they directly encode raw slot descriptions or names, such as "playlist", "music item", to enable models to generalize to slot types unseen at training time. however, slot descriptions are often ambiguous and typically do not encode enough semantic information by themselves. instead of directly encoding slot descriptions and examples, we introduce a qa-driven slot filling framework (qasf) (figure 1 ). inspired by the recent success of qa-driven approaches @xcite @xcite @xcite , we tackle the slot-filling problem as a reading comprehension task, where each slot type (e.g. "artist") is associated with a natural language question (e.g. "who is the artist to play?"). a span-based reading comprehension model is then used to extract a slot filler span from the utterance by answering the question. 1 in this work, we use a linguisti-cally motivated question generation strategy for converting slot descriptions and example values into natural questions, followed by a bert-based qa model for extracting slot fillers by answering questions. as shown in our experiments, this qadriven method is better at exploiting the semantic information encoded in the questions, therefore it generalizes better to new domains without any additional fine-tuning, as long as the questions are meaningful enough. to the best of our knowledge, we are the first to leverage weakly supervised synthetic qa pairs extracted from unlabeled conversations for a second-stage pretraining. drawing insights from @xcite , we create a weakly supervised qa dataset from unlabeled conversations and an associated ontology. the synthetic qa pairs are constructed by matching unlabeled utterances against possible slot values in the ontology. this provides a general and cost-effective way to improve qa-based slot filling performance with easily obtainable data. experimental results show that (1) our qasf model significantly outperforms previous zeroshot systems on snips @xcite and top @xcite ; (2) encoding natural questions help models better leverage weakly supervised signals in the pretraining phase, compared to encoding raw descriptions. ## task definition given an input utterance u, a slot filling model extracts a set of (slot type, span) pairs (s i , a i ), @xmath0, . . . , k where s i comes from a fixed set of slot types s, and each a @xmath1, k), 1 ≤ @xmath2. each slot type is accompanied with a short textual description that describes its semantic meaning ( our goal is to build a slot filling model that performs well on a new target domain with unseen slot types. our training data consists of utterances from n source domains d 1 , d 2 , ..., d n . each domain d i is associated with a set of predefined slot types s i . at test time, utterances are drawn from a new domain d n +1 . the new domain contains both seen and unseen slot types from the source domain. for example, in the snips dataset @xcite , domains "getweather" and "bookrestaurant" both predicate-argument structures are represented as qa pairs. have a slot type called "city", while "condi-tion_temperature" only appears in the "getweather" domain. ## methodology in this section, we describe our framework for question answer-driven slot filling (qasf). the framework consists of (1) a question generation strategy that turns slot descriptions into natural language questions based on linguistic rules; (2) a generic span-extraction-based question answering model; (3) an intermediate pretraining stage with generated synthetic qa pairs from unlabeled conversations, which is before task-specific training. ## conclusion we propose a qa-driven method with weakly supervised pretraining for zero-shot slot filling. our experimental results and analyses demonstrate the benefits of qa-formulation, especially in the setting with synthetic qa pairs pretraining.