Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network (2020)

Abstract: In this paper, we explore the slot tagging with only a few labeled support sentences (a.k.a. few-shot). Few-shot slot tagging faces a unique challenge compared to the other fewshot classification problems as it calls for modeling the dependencies between labels. But it is hard to apply previously learned label dependencies to an unseen domain, due to the discrepancy of label sets. To tackle this, we introduce a collapsed dependency transfer mechanism into the conditional random field (CRF) to transfer abstract label dependency patterns as transition scores. In the few-shot setting, the emission score of CRF can be calculated as a word’s similarity to the representation of each label. To calculate such similarity, we propose a Label-enhanced Task-Adaptive Projection Network (L-TapNet) based on the state-of-the-art few-shot classification model – TapNet, by leveraging label name semantics in representing labels. Experimental results show that our model significantly outperforms the strongest few-shot learning baseline by 14.64 F1 scores in the one-shot setting.

## introduction slot tagging @xcite , a key module in the task-oriented dialogue system @xcite , is usually formulated as a sequence labeling problem @xcite . slot tagging faces the rapid changing of domains, and the labeled data is usually scarce for new domains with only a few samples. few-shot learning technique @xcite @xcite is appealing in this scenario since it learns the model that borrows the prior experience from old domains and adapts to new domains quickly with only very few examples (usually one or two examples for each class). the few-shot scenario poses unique challenges in learning the emission and transition scores of crf. it is infeasible to learn the transition on the few labeled data, and prior label dependency in source domain cannot be directly transferred due to discrepancy in label set. to tackle the label discrepancy problem, we introduce the collapsed dependency transfer mechanism. it transfers label dependency information from source domains to target domains by abstracting domain-specific labels into abstract domain-independent labels and modeling the label dependencies between these abstract labels. it is also challenging to compute the emission scores (word-label similarity in our case). popular few-shot models, such as prototypical network @xcite , average the embeddings of each label's support examples as label representations, which often distribute closely in the embedding space and thus cause misclassification. to remedy this, @xcite propose tapnet that learns to project embedding to a space where words of different labels are well-separated. we introduce this idea to slot tagging and further propose to improve label representation by leveraging the semantics of label names. we argue that label names are often semantically related to slot words and can help word-label similarity modeling. for example in figure 1 , word rain and label name weather are highly related. to use label name semantic and achieve good-separating in label representation, we propose label-enhanced tapnet (l-tapnet) that constructs an embedding projection space using label name semantics, where label representations are well-separated and aligned with embeddings of both label name and slot words. then we calculate similarities in the projected embedding space. also, we introduce a pair-wise embedding mechanism to representation words with domain-specific context. one-shot and five-shot experiments on slot tagging and named entity recognition show that our model achieves significant improvement over the strong few-shot learning baselines. ablation tests demonstrate improvements coming from both l-tapnet and collapsed dependency transfer. further analysis for label dependencies shows it captures non-trivial information and outperforms transition based on rules. our contributions are summarized as follows: ## problem definition we define sentence @xmath0, x 2 , . . . , x n ) as a sequence of words and define label sequence of the sentence as @xmath1, y 2 , . . . , y n ). a domain as shown in figure 2 , few-shot models are usually first trained on a set of source domains {d 1 , d 2 , . . .}, then directly work on another set of unseen target domains {d 1 , d 2 , . . .} without fine-tuning. a target domain d j only contains few labeled samples, which is called support set the k-shot sequence labeling task is defined as follows: given a k-shot support set s and an input query sequence @xmath2, x 2 , . . . , x n ), find x's best label sequence y * : ## model in this section, we first show the overview of the proposed crf framework ( §3.1). then we discuss how to compute label transition score with collapsed dependency transfer ( §3.2) and compute emission score with l-tapnet ( §3.3). ## experiment we evaluate the proposed method on slot tagging and test its generalization ability on a similar sequence labeling task: name entity recognition (ner). due to space limitation, we only present the detailed results for 1-shot/5-shot slot tagging, which transfers the learned knowledge from source domains (training) to an unseen target domain (testing) containing only a 1-shot/5-shot support set. the results of ner are consistent and we present them in the supplementary appendix b. ## related works traditional few-shot learning methods depend highly on hand-crafted features @xcite . classical methods primarily focus on metric learning @xcite , which classifies an item according to its similarity to each class's representation. recent efforts @xcite propose to leverage the semantics of class name to enhance class representation. however, different from us, these methods focus on image classification where effects of name semantic are implicit and label dependency is not required. ## conclusion in this paper, we propose a few-shot crf model for slot tagging of task-oriented dialogue. to compute transition score under few-shot setting, we propose the collapsed dependency transfer mechanism, which transfers the prior knowledge of the label dependencies across domains with different label sets. and we propose l-tapnet to calculate emission score, which improves label representation with label name semantics. experiment results validate that both the collapsed dependency transfer and l-tapnet can improve the tagging accuracy.