Contrastive Zero-Shot Learning for Cross-Domain Slot Filling with Adversarial Attack (2020)

Abstract: Zero-shot slot filling has widely arisen to cope with data scarcity in target domains. However, previous approaches often ignore constraints between slot value representation and related slot description representation in the latent space and lack enough model robustness. In this paper, we propose a Contrastive Zero-Shot Learning with Adversarial Attack (CZSL-Adv) method for the cross-domain slot filling. The contrastive loss aims to map slot value contextual representations to the corresponding slot description representations. And we introduce an adversarial attack training strategy to improve model robustness. Experimental results show that our model significantly outperforms state-of-the-art baselines under both zero-shot and few-shot settings.

## introduction the slot filling task in the goal-oriented dialog system aims to identify task-related slot types in certain domains for understanding user utterances. traditional supervised slot filling models @xcite @xcite @xcite have made great achievements. however, these models require massive amounts of labeled data for a new domain, hindering the rapid development of new tasks. to address the data-intensiveness problem, domain adaptation approaches @xcite @xcite @xcite have been successfully applied. in this paper, we focus on zero-shot cross-domain transfer learning which leverages knowledge learned in the source domains and adapts the models to the target domain without labeled training samples in the target domain. the main challenge of zero-shot slot filling is to identify unseen slot types without any supervision signals in the target domain. typically, the previous methods rely on slot descriptions or example values to bootstrap to new slots by capturing the semantic relationship between slot descriptions and input tokens. these methods can be classified into two categories: one-stage and two-stage. @xcite @xcite conduct one-stage slot filling individually for each slot type. they first generate word-level representations, then interact with the representation of each slot type description in semantic space. finally, the predictions are independent for each slot type based on the fused features. the main drawback is the multiple prediction problem where a word may be predicted as multiple slot types. in contrast, @xcite propose a two-stage slot filling framework. they first predict whether the tokens are slot entities or not by a bio 3-way classifier, then identify their specific slot types based on slot type descriptions. although the two-stage framework helps learn the general pattern of slot entities, it can't directly leverage auxiliary description information to facilitate to detect bio labels as the one-stage framework does. owing to limited labeled training data in source domains, another common issue of zero-shot slot filling is that existing approaches always suffer from weak generalization capability. inspired by the above challenges, in this paper, we propose a contrastive zero-shot learning with adversarial attack (czsl-adv) method for the cross-domain slot filling. to leverage auxiliary slot play portsmouth by paul smith on google music bilstm encoder o b o b i o b i o b o b i o b i avg pooling avg pooling avg pooling track artist service stage 1: bio 3-way classification stage 2: zero-shot slot type classification a) the overall architecture play portsmouth by paul smith on google music play track by artist on service play city by album on object name play year by object name on playlist bilstm encoder input sentence representation c) contrastive representation learning b) adversarial attack for slot type classification avg pooling classification avg pooling classification lossslot + = adversarial attack loss'slot lossadv contrastive loss description information for detecting bio labels, we introduce a contrastive learning loss that maximizes the mutual information between raw input encoded representations and corresponding slot description delexicalized input representations. we aim to map slot value representations to the corresponding slot description representations in the latent space. therefore, slot descriptions can help learn the semantic pattern of related slot entities. to improve the generalization capability of our model, we also propose an adversarial attack training strategy which adds adversarial noise to the inputs in the direction that significantly increases the model's classification loss. the training strategy further improves the adaptation robustness of our method. our main contributions are three-fold: (1) we propose a contrastive zero-shot learning method for the cross-domain slot filling. (2) we introduce an adversarial attack training strategy to improve model robustness. (3) experiments on zero-shot learning and few-shot learning settings show that our proposed czsl-adv outperforms the state-of-the-art models with large margins. we also provide a comprehensive ablation study and further experiment analysis. ## approach fig 1 shows the overall architecture of our proposed czsl-adv model. in the first stage, it predicts bio labels with a contrastive representation loss to help learn the semantic pattern of corresponding slot entities. then in the second stage, it classifies the slot entities into related types with slot descriptions using an adversarial attack training strategy. ## conclusion in this paper, we propose a contrastive zero-shot learning with adversarial attack (czsl-adv) method for cross-domain slot filling. the main contributions are contrastive representation learning and adversarial attack training. the former leverages slot descriptions to help learn the semantic pattern of related slot entities and the latter improves model robustness by augmenting noise inputs. extensive experiments show the effectiveness of our proposed method, especially for the zero-shot learning setting.