H ierarchical C ontrast: A Coarse-to-Fine Contrastive Learning Framework for Cross-Domain Zero-Shot Slot Filling (2023)

Abstract: In task-oriented dialogue scenarios, cross-domain zero-shot slot filling plays a vital role in leveraging source domain knowledge to learn a model with high generalization ability in unknown target domain where annotated data is unavailable. However, the existing state-of-the-art zero-shot slot filling methods have limited generalization ability in target domain, they only show effective knowledge transfer on seen slots and perform poorly on unseen slots. To alleviate this issue, we present a novel Hierarchical Contrastive Learning Framework (HiCL) for zero-shot slot filling. Specifically, we propose a coarse- to fine-grained contrastive learning based on Gaussian-distributed embedding to learn the generalized deep semantic relations between utterance-tokens, by optimizing inter- and intra-token distribution distance. This encourages HiCL to generalize to the slot types unseen at training phase. Furthermore, we present a new iterative label set semantics inference method to unbiasedly and separately evaluate the performance of unseen slot types which entangled with their counterparts (i.e., seen slot types) in the previous zero-shot slot filling evaluation methods. The extensive empirical experiments on four datasets demonstrate that the proposed method achieves comparable or even better performance than the current state-of-the-art zero-shot slot filling approaches.

## introduction slot filling models are devoted to extracting the contiguous spans of tokens belonging to pre-defined slot types for given spoken utterances and gathering information required by user intent detection, and thereby are an imperative module of task-oriented dialogue (tod) systems. for instance, as shown in figure 2 , given a user utterance "send a reminder for a tire check next week" belonging to reminder domain, the slot filling task is to identify slot entities: "a tire check" and "next week" that correspond to slot types (an alias of slot type is slot), todo and date_time, respectively. supervised slot filling methods @xcite @xcite @xcite have achieved promising performance. nevertheless, these methods are strongly dependent on substantial and high-quality annotation data for each slot type, which prevents them from transferring to new domains with little or no labeled training samples. to solve this problem, more approaches have emerged to deal with this data scarcity issue by leveraging zero-shot slot filling (zssf). typically, these approaches can be divided into two categories: one-stage and two-stage. in one-stage paradigm, @xcite ; @xcite ; @xcite firstly generate utterance representation in token level to interact with the representation of each slot description in semantic space, and then predict each slot type for utterance token individually. the primary weakness for this paradigm is multiple prediction issue where a token will probably be predicted as multiple slot types @xcite . to overcome this issue, @xcite ; @xcite ; @xcite separate slot filling task into two-stage paradigm. they first identify whether the tokens in utterance belong to @xcite entity span or not by a binary classifier, subsequently predict their specific slot types by projecting the representations of both slot entity and slot description into the semantic space and interact on each other. based on the previous works, @xcite propose a two-stage variant, which introduces linguistic knowledge and pretrained context embedding, along with the entity span identify stage @xcite , to promote the effect on semantic similarity modeling between slot entity and slot description. recently, @xcite develop another two-stage variant that applies momentum contrastive learning technique to train bert @xcite and to improve the robustness of zssf. however, as shown in figure 1 , we found that these methods perform poorly on unseen slots in the target domain. although two-stage approaches have flourished, one-pass prediction mechanism of these approaches @xcite @xcite inherently limit their ability to infer unseen slots and seen slots separately. thus they have to adopt the biased test set split method of unseen slots (see more details in appendix f), being incapable of faithfully evaluating the real unseen slots performance. subsequently, their variants @xcite @xcite still struggle in the actual unseen slots performance evaluation due to following this biased test set split of unseen slots @xcite , or the intrinsic architecture limit of one-pass inference @xcite . in another line @xcite , zssf is formulated as a one-stage question answering task, but it is heavily reliant upon handcrafted question templates and ontologies customized by human experts, which is prohibitively expensive for this method to generalize to unseen domains. besides, the problem with multiple slot types prediction that happened to them @xcite @xcite seriously degrades their performance. in this paper, we introduce a new iterative label set semantics inference method to address these two problems. moreover, a downside for these two orthogonal methods is that they are not good at learning domain-invariant features (e.g., generalized tokenclass properties) by making full use of sourcedomain data while keeping target-domain of interest unseen. this may lead them to overfit to the limited slot types in source training domains. actually the current models' performance on unseen slots in target domain is still far from upper bound. to tackle the above limitation, intuitively, through contrastive learning (cl), we can redistribute the distance of token embeddings in semantic space to learn generalized token-class features across tokens, and better differentiate between token classes, and even between new classes (slot- input contextual encoder [cls] emb1 s emb2 s . . . embn s s [sep] emb1 u emb2 u emb 1 �1 . . . embn u u [sep] utterance target slot type transformer layer (x 12) embedding [cls] h1 s h2 s . . . hn s s [sep] h 1 �1 h 2 �1 h �� �� h1 u . . . . . . hn u u [sep] contextual representation fine-tuning loss token embeddings gaussian transformation network (��, �ʃ) gaussian transformation network (��, �ʃ) output coarse-contrastive linear classifier crf negative log likelihood fine-contrastive entity-level cl token-level cl hierarchical contrast representation transformation token-level cl utterance1 utterance2 b-todo i-todo slot labels: b-date_time i-date_time b-location i-location outside reduce divergence utterance (� 8 �1 , ʃ 8 �1 ) (� 9 �1 , ʃ 9 �1 ) (� 4 �1 , ʃ 4 �1 ) (� 6 �1 , ʃ 6 �1 ) (� 7 �1 , ʃ 7 �1 ) increase divergence (� 7 �2 , ʃ 7 �2 ) (� 8 �2 , ʃ 8 �2 ) (� 5 �2 , ʃ 5 �2 ) (� 6 �2 , ʃ 6 �2 ) token gaussian embedding (� 5 �1 , ʃ 5 �1 ) (� 4 �2 , ʃ 4 �2 ) emb �� �� . . . send a reminder for a tire check next week what the weather in st paul this weekend [sep] emb 2 �1 all slot types except target slot type iterative label set semantics inference agnostic features) , which is beneficial for new token class generalization across domains @xcite . however, it's hard to train token-level class since its supervised labels are closely distributed, which easily leads to misclassifications @xcite . while training entity-level class is relatively easy since its training labels are dispersedly distributed and it does not require label dependency that exists in token-level sequence labeling training @xcite . we argue that entity-level class knowledge contributes to token-level class learning, and their combination is beneficial for zssf task. entity-level class learning supplements token-class learning with entity type knowledge and boundary information between entity and non-entity, which lowers the difficulty of token class training. hence, we propose a new hierarchical cl framework called hicl, a coarse-to-fine cl approach. as depicted in figure 2 , it first coarsely learns entity-class knowledge of entity type and boundary knowledge via entity-level cl. then, it combines features of entity type and boundary, and finely learns token-class knowledge via token-level cl. in recent years, some researchers have employed gaussian embedding to learn the representations of tokens @xcite @xcite due to their superiority in capturing the uncertainty in representations. this motivates us to employ gaussian embedding in our hicl to represent utterance-tokens more robustly, where each token becomes a density rather than a single point in latent feature space. different from existing slot filling contrastive learners @xcite @xcite that optimize training objective of pairwise similarities between point embeddings, hicl aims to optimize distributional divergence by leveraging effectively modeling gaussian embeddings. while point embeddings only optimize pairwise distance, gaussian embeddings also comprise additional constraint which preserves the class distribution through their variance estimates. this distinctive quality helps to explicitly model entityor token-class distributions, which not only encourages hicl to learn generalized feature representations to categorize and differentiate between different entity (token) classes, but also fosters zerosample target domain adaptation. concretely, as shown in figure 3 , our token-level cl pulls inter-and intra-token distributional distance with similar labels closer in semantic space, while pushing apart dissimilar ones. gaussian distributed embedding enables token-level cl to better capture semantics uncertainty and semantics coverage variation of token-class than point embedding. this facilitates hicl to better model generalized slot-agnostic features in cross-domain zssf scenario. our major contributions are three-fold: ## problem definition zero-shot setting for zssf, a model is trained in source domains with a slot type set a s (i) and tested in new target domain with a slot type set a @xmath0, j and k are index of different slot type sets, a ts (j) are the slot types that both exist in source domains and target domain (seen slots), and a tu (k) are the slot types that only exist in target domain (unseen slots). since a s (i) ∩ a tu (k) = ∅, it is a big challenge for the model to generalize to unseen slots in target domain. task formulation given an utterance @xmath1 the architecture of hicl is illustrated in figure 3 . hicl adopts an iterative label set semantics inference enhanced hierarchical cl approach and conditional random field (crf) scheme. firstly, hicl successively employs entity-level cl and token-level cl, to optimize distributional divergence between different entity-(token-) class embeddings. then, hicl leverages generalized entity (token) representations refined in the previous cl steps, along with the slot-specific features learned in the crf training with bio label, to better model the alignment between utterance-tokens and slotentities. ## conclusion in this paper, we improve cross-domain zssf model from a new perspective: to strengthen the model's generalized differentiation ability between entity-class (token-class) by extracting the most domain-invariant and class-agnostic features. specifically, we introduce a novel pretraining-free hicl framework, that primarily comprises hierarchical cl and iterative label set semantics inference, which effectively improves the model's ability of discovering novel entities and discriminating between new slot-entity classes, which offers benefits to cross-domain transferability of unseen slots. experimental results demonstrate that our hicl is a backbone-independent framework, and compared with several sota methods, it performs comparably or better on unseen slots and overall performance in new target domains of zssf.