Semi-supervised Meta-learning for Cross-domain Few-shot Intent Classification (2021)

Abstract: Meta learning aims to optimize the model’s capability to generalize to new tasks and domains. Lacking a data-efficient way to create meta training tasks has prevented the application of meta-learning to the real-world few shot learning scenarios. Recent studies have proposed unsupervised approaches to create meta-training tasks from unlabeled data for free, e.g., the SMLMT method (Bansal et al., 2020a) constructs unsupervised multi-class classification tasks from the unlabeled text by randomly masking words in the sentence and let the meta learner choose which word to fill in the blank. This study proposes a semi-supervised meta-learning approach that incorporates both the representation power of large pre-trained language models and the generalization capability of prototypical networks enhanced by SMLMT. The semi-supervised meta training approach avoids overfitting prototypical networks on a small number of labeled training examples and quickly learns cross-domain task-specific representation only from a few supporting examples. By incorporating SMLMT with prototypical networks, the meta learner generalizes better to unseen domains and gains higher accuracy on out-of-scope examples without the heavy lifting of pre-training. We observe significant improvement in few-shot generalization after training only a few epochs on the intent classification tasks evaluated in a multi-domain setting.

## introduction recent developments of large scale pre-trained models, such as bert @xcite , @xcite and xlnet @xcite , have significantly advanced the natural language processing (nlp) techniques. however, these models still rely on fine-tuning on a relatively large number of labeled samples (> 1000) to achieve high accuracy even for tasks seen during training @xcite . recent studies @xcite @xcite have demonstrated that these large language models have the potential to be few shot learners, i.e., capable of adapting to a new task or a new domain by training only on a few examples with the aid of meta-learning. meta-learning tackles the fewshot learning problem through learning a robust yet flexible representation from a variety of tasks in a so-called meta training stage, so that the model can quickly adapt to new tasks with only a few examples. in addition, random sampling is introduced in the design of meta training tasks to avoid memorization, a phenomenon in which the meta learner memorizes a function that directly associates an input with the label when no real learning occurs @xcite . meta-learning approaches such as the optimization-based maml @xcite , the metric-based prototypical networks (protonet) @xcite and etc., have been successfully applied in nlp domain @xcite . @xcite successfully applied maml and its variants to low-resource text classification tasks on the glue dataset @xcite . it showed models trained with maml, first-order maml and reptile @xcite outperform strong baseline models such as bert and mt-dnn @xcite . @xcite developed a method leopard that generalizes maml to handle diverse nlp tasks. they used pre-trained bert @xcite as the underlying task-agnostic base model, coupled with a task-dependent softmax classification parameter generator. the meta trained bert learns better initial parameters, which helped to reach high accuracy across 17 down steam nlp tasks with very few examples per class. however, successful implementations of metalearning depend on the availability of a diverse set of tasks with plenty of labeled data during meta training. to create meta-learning tasks in a data-efficient manner, a number of papers have tried to explore the idea of unsupervised metalearning. these methods explore to learn representations through automatically constructing tasks from unlabeled dataset and utilize learned representation functions for specific task prediction. @xcite proposed to leverage clustering embeddings to construct tasks from unlabeled data and then apply meta-learning method for explicitly optimizing for adapting to new tasks. @xcite proposed to sample objects with synthetic labels from the latent space and generate meta-tasks using generative models. in the domain of natural language processing, @xcite proposed subset masked language modeling tasks (smlmt), which automatically construct self-supervised tasks by masking out certain tokens from sentences as labels to create few shots classification tasks from unlabeled data. the study showed that meta training with these diverse unsupervised tasks can prevent over-fitting to specific supervision tasks, leading to better generalization than pre-training language-model followed by finetuning. in this study, we focus on cross-domain few shot classification with the goal to investigate whether we can meta train a large pre-trained language model (e.g., bert) in a semi-supervised fashion without access to a large number of labeled data or meta training tasks. the resulting representation should generalize and adapt well to a new domain, and provide clear separations between indomain and out-of-scope (oos) examples @xcite . our base meta learner consists of an embedding function (e.g., bert) and protonet @xcite as the general supervised classifier, which can be fine-tuned either using the supervised n -way k-shot classification tasks (supervised meta training) or together with the selfsupervised smlmt tasks (semi-supervised meta training). we compares classifiers with supervised meta-training against classifiers trained without the diverse meta training tasks. we then compare the semi-supervised meta-learner with the supervised approach without adding additional labeled data. the resulting text representations will be evaluated in terms of their few-shot generalization accuracy, their capability to detect oos examples, and their ability to adapt when more training examples are included. while @xcite focuses on the crossproblem transfer capability of smlmt trained with a general-purpose corpus like wikipedia, our study further investigates the cross-domain transfer capability of smlmt within a problem, i.e., whether additional self-supervised training on the unlabeled data from the domain of interest (e.g., dialogues) can help generalize a seen problem to a new unseen domain. moreover, smlmt as a classification task combines well with metric-based meta learners like protonet @xcite . compared to optimization-based meta learners like maml @xcite , protonet is easier to optimize and scale, has a simpler inductive bias therefore works well for very-few-shot classification problems. these properties are complementary to maml and can provide good initialization for the latter @xcite . ## experiments, results and discussion there have been a number of papers that have explored the idea of unsupervised meta-learning, where tasks are constructed automatically from an unlabeled dataset and a meta-learner is pre-trained on these tasks without using any labeled dataset. can we extend these ideas to the case where we have a small number of supervised meta-training tasks rather than zero meta-training tasks, to construct a semi-supervised meta-learner? we hope to explore answers to the following questions through experiments: (a) whether meta training effectively improve domain adaptation? and (b) will the semisupervised approach outperform the supervised meta-learning given the same number of labeled data? ## conclusion we proposed a semi-supervised meta-learning approach for cross-domain few-shot intent classification by incorporating the representation power of pre-trained language model with the fast adaptation capability of protonet enhanced through selfsupervision. this methodology tackles the realistic few shot learning setting where not enough meta training tasks exist and meta learner trained only on supervised tasks suffers from over-fitting on a small number of labeled data. the experiments have shown that meta learner generalizes better to new domains and predicts more accurately on out-of-scope examples if trained with additional meta training tasks created through self-supervision from unlabeled data. compared to pre-training language models using self-supervision, the volume of unlabeled data required for our semi-supervised meta training is rather small and the optimization is much easier. however, it effectively improve the few-shot generalization and out-of-scope accuracy by learning a better cross-domain representation and learning to quickly attend to the right word in new domains. while protonet has its limitations due to simpler inductive bias, the resulting presentation can be used to initialize more sophisticated meta learner and extend beyond the classification problems. future directions include exploring different ways to combine various types of meta learners, different designs of self-supervised tasks as well as validating our algorithms on other datasets.