Adaptive End-to-End Metric Learning for Zero-Shot Cross-Domain Slot Filling (2023)

Abstract: Recently slot filling has witnessed great development thanks to deep learning and the availability of large-scale annotated data. However, it poses a critical challenge to handle a novel domain whose samples are never seen during training. The recognition performance might be greatly degraded due to severe domain shifts. Most prior works deal with this problem in a two-pass pipeline manner based on metric learning. In practice, these dominant pipeline models may be limited in computational efficiency and generalization capacity because of non-parallel inference and context-free discrete label embeddings. To this end, we re-examine the typical metric-based methods, and propose a new adaptive end-to-end metric learning scheme for the challenging zero-shot slot filling. Considering simplicity, efficiency and generalizability, we present a cascade-style joint learning framework coupled with context-aware soft label representations and slot-level contrastive representation learning to mitigate the data and label shift problems effectively. Extensive experiments on public benchmarks demonstrate the superiority of the proposed approach over a series of competitive baselines.

## introduction slot filling, as an essential component widely exploited in task-oriented conversational systems, has attracted increasing attention recently @xcite @xcite . it aims to identify a specific type (e.g., artist and playlist) for each slot entity from a given user utterance. owing to the rapid development of deep neural networks and with help from large-scale annotated data, research on slot filling has made great progress with considerable performance improvement @xcite @xcite . despite the remarkable accomplishments, there are at least two potential challenges in realistic application scenarios. first is the data scarcity problem in specific target domains (e.g., healthcare and e-commerce). the manually-annotated training data in these domains is probably unavailable, and even the unlabeled training data might be hard to acquire @xcite . as a result, the performance of slot filling models may drop significantly due to extreme data distribution shifts. the second is the existence of label shifts (as shown in the example in figure 1 ). the target domain may contain novel slot types unseen in the source-domain label space @xcite @xcite , namely there is a mismatch between different domain label sets. this makes it difficult to apply the source models to completely unseen target domains that are unobservable during the training process. zero-shot domain generalization has been shown to be a feasible solution to bridge the gap of domain shifts with no access to data from the target domain. recent dominating advances focus on the two-step pipeline fashion to learn the zero-shot model using the metric learning paradigms @xcite @xcite @xcite . nevertheless, besides inefficient inference resulted from non-parallelization, the generalization capability of these models may be limited due to lack of knowledge sharing between sub-modules, and context-independent discrete static label embeddings. although the alternative question-answering (qa) based methods @xcite @xcite are able to achieve impressive results, they need to manually design and construct the questions/queries, essentially introducing detailed descriptive information about the slot types. in this work, we revisit the metric-based zeroshot cross-domain slot filling under challenging domain (both data and label) shifts. we propose an adaptive end-to-end metric learning scheme to improve the efficiency and effectiveness of the zeroshot model in favor of practical applications. for one thing, we provide a cascade-style joint learning architecture well coupled with the slot boundary module and type matching module, allowing for knowledge sharing among the sub-modules and higher computational efficiency. moreover, the soft label embeddings are adaptively learnt by capturing the correlation between slot labels and utterance. for another, since slot terms with same types tend to have the semantically similar contexts, we propose a slot-level contrastive learning scheme to enhance the slot discriminative representations within different domain context. finally, to verify the effectiveness of the proposed method, we carry out extensive experiments on different benchmark datasets. the empirical studies show the superiority of our method, which achieves effective performance gains compared to several competitive baseline methods. overall, the main contributions can be summarized as follows: (1) compared with existing metric-based methods, we propose a more efficient and effective end-to-end scheme for zero-shot slot filling, and show our soft label embeddings perform much better than previous commonly-used static label representations. (2) we investigate the slotlevel contrastive learning to effectively improve generalization capacity for zero-shot slot filling. ## methodology in this section, we first declare the problem to be addressed about zero-shot slot filling, and then elab-orate our solution to this problem. ## conclusion in this paper, we tackle the problem of generalized zero-shot slot filling by the proposed end-toend metric learning based scheme. we propose a cascade-style multi-task learning framework to efficiently detect the slot entity from a target domain utterance. the context-aware soft label embeddings are shown to be superior to the widely-used discrete ones. regarding domain adaptation robustness, we propose a slot level contrastive learning scheme to facilitate the discriminative representations of slot entities. extensive experiments across various domain datasets demonstrate the effectiveness of the proposed approach when handling unseen target domains. our investigation also confirms that semantically richer label representations enable help further boost the recognition performance, which motivates us to further explore external knowledge enhanced soft label embeddings for advancing the metric-based method.