Transforming Slot Schema Induction with Generative Dialogue State Inference (2024)

Abstract: The challenge of defining a slot schema to represent the state of a task-oriented dialogue system is addressed by Slot Schema Induction (SSI), which aims to automatically induce slots from unlabeled dialogue data. Whereas previous approaches induce slots by clustering value spans extracted directly from the dialogue text, we demonstrate the power of discovering slots using a generative approach. By training a model to generate slot names and values that summarize key dialogue information with no prior task knowledge, our SSI method discovers high-quality candidate information for representing dialogue state. These discovered slot-value candidates can be easily clustered into unified slot schemas that align well with human-authored schemas. Experimental comparisons on the MultiWOZ and SGD datasets demonstrate that Generative Dialogue State Inference (GenDSI) outperforms the previous state-of-the-art on multiple aspects of the SSI task.

## introduction developing task-oriented dialogue (tod) systems presents the significant challenge of creating and maintaining a slot schema, where each slot defines a type of information that is critical for successfully completing the dialogue task @xcite . traditionally slot schemas are handcrafted, but manually defining each slot is time-consuming, especially when task domains are complicated or the functionality of the dialogue system is frequently updated. to address this, slot schema induction (ssi) has been proposed to automatically generate slot schemas from unlabeled dialogue data @xcite . this task facilitates the automatic analysis of dialogue structure @xcite and identifies key types of information that should be included in dialogue state representations @xcite . by reducing the need for manual schema creation, ssi expedites developing tod systems for new application domains, and enables continual discovery of new slot types to improve the coverage of existing slot schemata. the core challenge of ssi is identifying which information presented in unlabeled dialogue data is important for the task domain and should be included in the dialogue state. once the important information values are identified, a second challenge is defining a minimal set of slots that captures the different types of information the values represent. all previous work on ssi tackles these challenges in an explicit two-step process involving (1) candidate value identification and (2) inducing a slot schema by clustering candidate values into a set of slot clusters. identifying value candidates has been explored using tagging models trained on other tasks such as ner or srl @xcite @xcite , or using token attention distributions produced by a plm to extract syntactic constituents @xcite . inducing slots from value candidates has been explored using out-of-the-box clustering algorithms @xcite , multi-stage clustering pipelines specific to ssi @xcite @xcite , or a neural latent variable model @xcite . unlike all previous approaches to ssi, we are the first to take a generative approach to value candidate identification. 1 candidates are identified using a dialogue state generator model, which is trained to summarize the key task-related information in a given dialogue context as a set of state values. crucially, this state generator also creates a slot name for each value, which serves as a candidate prediction of the name of the slot the value fills. value candidates are then clustered in conjunction with these predicted slot names to induce a uni- i'm looking for a cheap italian restaurant. i found a couple cheaper options on the west side. cuisine italian price range cheap unlabeled dialogues discovered values are there any in the city centre? restaurant price range cheap restaurant area west area preference centre restaurant pricerange cuisine restaurant area encodings slot clusters fied set of slots. the advantage of this approach is that the type semantics of each value candidate are concretely represented using slot name predictions, whereas previous approaches rely on the similarity of value encodings alone to cluster values into slot types. predicting slot names has the additional benefit of enabling automatic naming of each slot cluster in the induced schema. we demonstrate the impact of these benefits by evaluating our approach on the multiwoz @xcite and sgd @xcite datasets. our ssi approach produces slot schemas that better match gold reference schemas when compared to the previous state-of-the-art (sota) approaches. ## approach our ssi approach, generative dialogue state inference (gendsi), induces a slot schema consisting of a set of slot clusters from an unlabeled set of dialogues. the induction procedure is performed in three stages (fig. 1 ). first, a dialogue state generator discovers value candidates for each turn in the dialogue data and jointly predicts a slot name with each value. second, an encoding model produces a dense vector representation for each slot-value candidate. finally, a clustering algorithm uses the encodings to filter and group candidates into a unified set of slot clusters. dialogue state generator our approach formulates the discovery of value candidates from unlabeled dialogue data as a sequence-to-sequence generation task. the input is a dialogue context d * ..t , and the output is a list of slot-value candidates [(s 1 , v 1 ), ..., (s k , v k )] represented by the sequence format s 1 :v 1 ; s 2 :v 2 ; ...s k :v k [eos]. each candidate includes a value v i that is inferred to belong to the dialogue state and a slot name prediction s i to represent the type of v i . to enable the model to generate slot-value pairs that discover important dialogue state information without any prior knowledge of the task domain, we fine-tune a pretrained encoderdecoder transformer on tod data that covers a large variety of domains. section 3 presents an evaluation of the dialogue state generator in which two different training datasets are compared. value candidate encoding each slot-value candidate (s i , v i ) produced by the dialogue state generator is encoded into a single dense vector representation e i . to do this, we concatenate the slot name and value candidate with a separator to form a single token sequence s i :v i . we then use the sbert encoder (reimers and gurevych, 2019) to independently encode each candidate sequence. slot clustering given a complete list of all slotvalue candidates [(s 1 , v 1 ), ..., (s n , v n )] produced by the dialogue state generator across all turns of the dialogue dataset, slot-value candidates are jointly filtered and grouped by applying the hdb-scan algorithm (mcinnes et al., 2017) to the candidate encodings [e 1 , e 2 , ..., e n ]. as demonstrated in previous work (yu et al., 2022), hdbscan is a suitable clustering algorithm because (1) it automatically discovers an appropriate number of slot clusters and (2) it filters out examples in low-density regions of the encoding space, which are likely to represent noisy candidates. the result is a set of slot clusters [s 1 , s 2 , ..., s k ] where each cluster s i is a list of values that fill the slot it represents. ## state generator evaluation since our ssi approach relies mainly on the dialogue state generator component to infer highquality value candidates with appropriate slot names, we first conduct an evaluation of the performance of this component when discovering slotvalues from dialogues in unseen task domains. ## schema induction evaluation to evaluate our ssi approach, we use the benchmark defined by @xcite on the validation splits of multiwoz 2.1 @xcite and sgd @xcite datasets. this evaluation method measures the quality of an induced set of slot clusters by matching it against a gold reference slot schema. matching is performed automatically by computing the centroid of each induced and gold reference slot cluster using bert encodings @xcite of their values. each induced cluster is mapped to the gold slot whose cluster centroid is nearest by cosine similarity, or to no cluster if there is no match of 80% similarity or higher. similarly, in order to evaluate the purity and coverage of clustered values, discovered values are matched against the gold value labels that fill each slot. this value matching is performed between the values that fill each gold slot and the discovered values of all induced clusters mapped to that slot using fuzzy string matching. metrics given the mapping of induced clusters to gold slots, slot precision measures the proportion of induced clusters that were able to be matched to a gold slot, slot recall is the proportion of gold slots that were matched with at least one induced cluster, and slot f1 is their harmonic mean. since multiple induced slots are allowed to map to a single gold slot, the induced slot count is also reported to measure redundancy. value precision is the average proportion of discovered values that matched to gold values, averaged across all gold slots. similarly, value recall is the average proportion of gold values that were matched to a discovered value, and value f1 is the average f1 score across all gold slots. equations defining these metrics are presented in appx. b. models our ssi approach, gendsi, uses a t5-3b model trained on the d0t dataset. since d0t contains some task domains that are related to travel domains appearing in multiwoz and sgd, we manually review and filter out 34 domains with overlap and train our dialogue state generator on all d0t dialogues in remaining domains. we also evaluate the performance when using a model trained with all d0t domains (gendsi +all domains), which simulates extending our approach using the d0t data generation method to create synthetic training resources for target domains. additionally, we evaluate a version of our approach where value candidates are encoded without their predicted slot names (gendsi -slot names) to measure the benefit of concretely representing value type information. implementation details provided are in appx. d. finally, we compare to two strong baselines from previous work: ## conclusion this work presents a new sota for ssi, demonstrating the power of a generative approach to value candidate discovery. our dialogue state generator model shows excellent performance for discovering key dialogue state information from unlabeled dialogues without any prior knowledge of the task domain. its ability to label discovered values with appropriate slot names provides rich type information, allowing a simple clustering method to induce a quality slot schema for unseen domains. despite this advancement, there is still room to improve ssi. in particular, current ssi methods produce a far greater number of induced slots compared to human-defined schemas. although our approach reduces the number of induced slots somewhat, future work should aim for ssi with minimal redundancies in induced slots to further improve the utility of ssi in practical settings.