Few-shot Classification with Hypersphere Modeling of Prototypes (2023)

Abstract: Metric-based meta-learning is one of the de facto standards in few-shot learning. It composes of representation learning and metrics calculation designs. Previous works construct class representations in different ways, varying from mean output embedding to covariance and distributions. However, using embeddings in space lacks expressivity and cannot capture class information robustly, while statistical complex modeling poses difficulty to metric designs. In this work, we use tensor fields (“areas”) to model classes from the geometrical perspective for few-shot learning. We present a simple and effective method, dubbed as hypersphere prototypes (HyperProto), where class information is represented by hyperspheres with dynamic sizes with two sets of learnable parameters: the hypersphere’s center and the radius. Extending from points to areas, hyperspheres are much more expressive than embeddings. Moreover, it is more convenient to perform metric-based classification with hypersphere prototypes than statistical modeling, as we only need to calculate the distance from a data point to the surface of the hypersphere. Following this idea, we also develop two variants of prototypes under other measurements. Extensive experiments and analysis on few-shot NLP tasks and comparison with 20+ competitive baselines demonstrate the effectiveness of our approach.

## introduction constituting cognition of novel concepts with a few examples is crucial for machines to emulate human intelligence, and with the exorbitant cost associated with annotating large amounts of data, fewshot learning has garnered considerable attention in modern deep learning @xcite . despite the success under ample supervision, limited training examples remain a challenge for traditional deep * equal contribution † corresponding authors neural models. consequently, various approaches have been proposed to extend the applicability of deep neural networks to scenarios with limited data availability. one significant area of research within this domain is metric-based meta-learning @xcite @xcite , where models are trained to generate expressive representations and perform classification through predefined metrics. the success of metric-based meta-learning depends on both representation learning and the metrics chosen. one straightforward approach relies on training feature representation and adopts a nearest-neighbor classifier @xcite @xcite . other works introduce additional parameters as class representation to achieve better generalization ability. a naive way to estimate class representation is to use the mean embedding of feature representation, i.e., prototypes @xcite , while some also use second-order moments @xcite or reparameterize the learning process to generate class representation in a richer semantic space @xcite or in the form of probability distribution @xcite . apart from traditional euclidean and cosine distance, a variety of metric functions are also proposed @xcite @xcite . most existing works learn class representation from the statistical perspective, making designing and implementing the metrics more difficult. for example, the proposed covariance metric in covamnet @xcite theoretically requires a non-singular covariance matrix, which is awkward for neural-based feature extraction methods. this paper revisits metric-based meta-learning and demonstrates that geometrical modeling can simultaneously enhance the expressivity of representations and reduce the difficulty of metric calculation, meanwhile yielding surprising per-formance in few-shot classification. specifically, we propose hyperproto , a simple and effective approach to model class representation with hyperspheres. it is equipped with three advantages: (1) characterizing geometrical "area" as manifolds with complex boundaries can often be difficult in deep learning. instead, we only use two sets of learnable parameters: the center and the radius, to represent a hypersphere, which is straightforward and easy to learn. (2) a hypersphere is much more expressive than a single point in the representation space. the introduction of a learnable radius parameter greatly expands the representative power. (3) besides, hyperspheres are suitable for constructing measurements in euclidean space. we can calculate the euclidean distance from one feature point to the surface of the hypersphere in order to perform metric-based classification, which is difficult for other manifolds. along with the simplicity in metric design and the enhanced expressive power is the easiness in optimization and learning. with the metrics designed as distance to the hypersphere surface, both the radius and the center of the hypersphere will appear in the loss function and participate in the backward propagation during optimization. intuitively, for the classes with sparse feature distributions, the corresponding radii of their prototypes are large, and the radii are small otherwise. beyond the euclidean space, we also develop two variants based on the general idea -cone-like prototypes with cosine similarities and gaussian prototypes from the probability perspective (in appendix a). we conduct extensive experiments to evaluate the effectiveness of hyperproto on two classical information extraction tasks, few-shot named entity recognition (ner) @xcite and relation extraction (re) @xcite . despite the simplicity, we find that our approach is exceedingly effective, which outperforms the vanilla prototypes by 8.33 % absolute in average f1 on few-nerd (intra), 6.55% absolute in average f1 on few-nerd (inter), 4.77% absolute in average accuracy on fewrel, respectively. the generality of our approach allows it to be easily integrated with other techniques. we combine our method with prompt-learning and task-specific pre-training to obtain high-quality representations, substantially outperforming many competitive baselines. we believe our approach could serve as a strong baseline for few-shot learning and inspire new ideas from the research community for representation learning. ## problem setup we consider the episodic n -way k-shot few-shot classification paradigm foot_0 . given a large-scale annotated training set d train , our goal is to learn a model that can make accurate predictions for a set of new classes d test , containing only a few labeled examples for training. the model will be trained on episodes constructed using d train and tested on episodes based on d test . each episode contains a support set @xmath0, y i } n ×k @xmath1, with n classes and k examples for each class, and a query set for inference ## methodology this section describes the mechanisms of hypersphere modeling of prototypes. one hypersphere prototype is represented by two parameters: the center and the radius, which are first initialized via estimation and then optimized by gradient descent in conjunction with the encoder parameters. ## experiments to evaluate the effectiveness of the proposed method, we conduct experiments on few-shot named entity recognition (ner) and few-shot relation extraction (re) tasks, both of which are fundamental tasks of information extraction accompanied by well-established datasets. task descriptions, datasets, and implementation details are reported in appendix b. apart from the experimental study in this section, we also carry out additional experiments and analyses of image classification to demonstrate the generality of our method in appendix c. ## conclusion this paper proposes a novel metric-based few-shot learning method, hypersphere prototypes. unlike previous metric-based methods that use dense vectors to represent the class-level semantics, we use hyperspheres to enhance the capabilities of prototypes to express the intrinsic information of the data. it is simple to model a hypersphere in the embedding space and conduct metric-based classification in few-shot learning. our approach is easy to implement and also empirically effective, we observe significant improvements to baselines and compatibility with other techniques on downstream tasks. for potential future work, such modeling could be extended to more generalized representation learning like word embeddings. school (hw2021008), and institute guo qiang at tsinghua university.