Few-shot Learning for Slot Tagging with Attentive Relational Network (2021)

Abstract: Metric-based learning is a well-known family of methods for few-shot learning, especially in computer vision. Recently, they have been used in many natural language processing applications but not for slot tagging. In this paper, we explore metric-based learning methods in the slot tagging task and propose a novel metric-based learning architecture - Attentive Relational Network. Our proposed method extends relation networks, making them more suitable for natural language processing applications in general, by leveraging pretrained contextual embeddings such as ELMO and BERT and by using attention mechanism. The results on SNIPS data show that our proposed method outperforms other state of the art metric-based learning methods.

## introduction neural networks have been successfully utilized in natural language processing (nlp) applications with a large amount of hand-labeled data whereas they suffer a persistent challenge of low-resource. the approach of learning with few samples, known as few-shot learning -a branch of meta-learning (learn to learn) -has recently been popularized @xcite @xcite @xcite in computer vision. recently, few-shot learning has also been applied to nlp tasks, e.g. natural language understanding @xcite ), text classification @xcite @xcite , machine translation @xcite and relation classification @xcite . in the slot tagging task, we aim at predicting taskspecific values (e.g. artist, time) for slots (placeholders) in user utterances. @xcite propose a two-stage modeling approach to exploit domain-agnostic features to tackle low-resource do-main challenges. besides, the other state of the art techniques e.g. based on external memory @xcite , ranking loss @xcite , encoder @xcite , and attention @xcite have achieved promising results with a wide range of neural networks methods. however as many other nlp applications, the low-resource issue is a tremendous challenge for slot tagging in new domains, although labeled samples exist in related domains. many studies have recently proposed to overcome this low-resource challenge using different techniques, e.g. multitask modeling @xcite , adversarial training @xcite , and pointer networks @xcite . in addition, studies like zero-shot learning has influenced the studies of the domain scaling problem for slots prediction @xcite , eliminating the need of labeled examples for transferring reusable concepts @xcite , and conveying the domainagnostic concepts between the intents @xcite by exploiting label names and descriptions. likewise, @xcite use label semantics within a few-shot classification method tapnet @xcite . we suggest using a small amount of annotated samples from different domains as training input instead of slot descriptions and slot names as in previos zero-shot @xcite @xcite and few-shot @xcite slot tagging studies for two reasons: (1) the creation of slot descriptions needs qualified linguistic expertise and is thus expensive. (2) the relationship between slot names and the corresponding tokens is not constant. to give an example, the relationship between the 'genre' slot name and 'drama' token is hypernymic whereas the relationship between the 'artist' slot name and 'tarkan' token is instance based. hence, it may not be valid to learn only one function to represent the different relationships between names and tokens. in this paper, we provide a new experimental design where the slot tagging task needs to be solved for unseen slot labels. the experimental design mimics previous few-shot learning studies @xcite @xcite . thus, the existing data sources from different domains are used to learn meta-knowledge, whereas unseen labels from low-resource domains are used to evaluate the models. furthermore, we propose a novel modeling approach -attentive relational network, inspired by @xcite @xcite , that leverages contextual embeddings such as elmo and bert and extends the previous relation networks @xcite by learning to attend local and global features @xcite . experimental results on snips data show that the proposed model outperforms other few-shot learning networks. ## conclusion we presented a deep analysis with a wide variety of few-shot learning methods and pretrained (contextual) embeddings for slot tagging. furthermore, we proposed a novel architecture that leverages attention mechanism attending both, local and global features of given support samples. experimental results on snips dataset show that a) pretrained contextual embeddings contributed to high performance and b) our proposed approach consistently outperformed other methods in all setups.