Classical Sequence Match Is a Competitive Few-Shot One-Class Learner (2022)

Abstract: Nowadays, transformer-based models gradually become the default choice for artificial intelligence pioneers. The models also show superiority even in the few-shot scenarios. In this paper, we revisit the classical methods and propose a new few-shot alternative. Specifically, we investigate the few-shot one-class problem, which actually takes a known sample as a reference to detect whether an unknown instance belongs to the same class. This problem can be studied from the perspective of sequence match. It is shown that with meta-learning, the classical sequence match method, i.e. Compare-Aggregate, significantly outperforms transformer ones. The classical approach requires much less training cost. Furthermore, we perform an empirical comparison between two kinds of sequence match approaches under simple fine-tuning and meta-learning. Meta-learning causes the transformer models’ features to have high-correlation dimensions. The reason is closely related to the number of layers and heads of transformer models. Experimental codes and data are available at https://github.com/hmt2014/FewOne .

## introduction when the labeled data is scarce in practical application, it is struggled to learn a well-performed model using deep learning algorithms. yet annotating data costs much labor and time. few-shot learning (fsl) intuitively addresses this obstacle @xcite @xcite @xcite . fsl learns at the meta-task level, where each meta-task is formulated as inferring queries with the help of a support set @xcite . multiple meta-tasks facilitate the task-agnostic transferrable knowledge. thus it can learn new knowledge fast after being taught only a few samples. despite * hang gao is the corresponding author. this was the worst hotel in vegas. figure 1: meta-task example in few-shot one-class text classification, where 1 denotes a positive instance and 0 denotes a negative one. fsl has been well-studied, its one-class scenario @xcite is less investigated. in this paper, following the one-class trait, we design each meta-task as a binary classification. it consists of a reference instance, a support set, and a query set (see figure 1 ). the reference instance is one known sample of a class, which is exploited to tell whether an instance out of the support/query set belongs to the same class. such purpose is consistent with sequence match, which also makes a decision for two sequences. previous sequence match can mainly be categorized into two promising directions: classical methods, e.g. siamese network @xcite , compare-aggregate (ca) @xcite , and transformerbased method, e.g. distilbert @xcite , bert @xcite . in recent years, transformer models have already beaten classical ones in a wide range of tasks @xcite . we wonder how two kinds of models perform under the few-shot one-class scenario. consequently, it is presented that with metalearning, classical sequence match method can significantly outperform transformer-based models. the classical models require much less training cost. specifically, model-agnostic meta-learning (maml) algorithm @xcite is a subtle bi-level optimizing approach that aims to learn a good parameter initialization. by introducing this algorithm, classical methods act as simple but competitive few-shot one-class learners. furthermore, we make an empirical comparison between classical and transformer-based models under simple fine-tuning and meta-learning. firstly, it is found that maml has a more positive impact on the both sequence match approaches than simple fine-tuning. this suggests that a good parameter initialization is important for both of them. secondly, maml tends to make transformer models extract features with high-correlation dimensions. the bi-level optimization might cause the feature extraction layers of large models less trained. yet the last classifying layer tends to be better learned relatively. we demonstrate that the high correlation is related to the number of heads and layers in the transformer. in summary, our main contributions are as follows: (1) we present a simple but competitive few-shot one-class learner, which is based on the classical sequence match approach and metalearning. extensive experimental results show that this learner achieves significant improvements compared with transformer-based models. this provides new insights in the transformer-dominant era. ## methods in this work, two kinds of sequence match methods, including classical and transformer-based ones, are mainly investigated. compare-aggregate @xcite is a promising classical method. we choose to study its extended version, i.e. bidirectional compare-aggregate (bica) @xcite , introduced in §3.2. the transformer-based sequence match @xcite is also presented in §3.3 briefly. ## conclusion in this work, we revisit the classical sequence match approaches and find that with meta-learning, the classical method can significantly outperform transformer models in the few-shot one-class scenario. the training cost is greatly reduced. furthermore, an empirical study is made to explore the effects of simple fine-tuning and meta-learning. interestingly, although meta-learning is more effective than simple fine-tuning on both sequence match approaches, it makes the transformer features have high correlation dimensions. the correlation is closely related to the number of layers and heads in the transformer models. we hope this work could provide insights for future research on few-shot problems and transformer models.