F EW - SHOT L EARNING WITH B IG P ROTOTYPES
Anonymous authors
Paper under double-blind review
A BSTRACT
Using dense vectors, i.e., prototypes, to represent abstract information of classes has become a common approach in low-data machine learning scenarios.
Typically, prototypes are mean output embeddings over the instances for each class.
In this case, prototypes have the same dimension of example embeddings, and such tensors could be regarded as “points” in the feature space from the geometrical perspective.
But these points may lack the expressivity of the whole class-level information due to the biased sampling.
In this paper, we propose to use tensor fields (“areas”) to model prototypes to enhance the expressivity of class-level information.
Specifically, we present big prototypes , where prototypes are represented by hyperspheres with dynamic sizes.
A big prototype could be effectively modeled by two sets of learnable parameters, one is the center of the hypersphere, which is an embedding with the same dimension of training examples.
The other is the radius of the sphere, which is a scalar.
Compared with atactic manifolds with complex boundaries, representing hypersphere with parameters is immensely easier.
Moreover, it is convenient to perform metric-based classification with big prototypes in few-shot learning, where we only need to calculate the distance from a data point to the surface of the hypersphere.
Extensive experiments on few-shot learning tasks across NLP and CV demonstrate the effectiveness of big prototypes.
I NTRODUCTION
Learning from few examples, i.e., few-shot learning, receives increasing attention in modern deep learning.
On the one hand, constituting cognition of novel concepts with few instances is a crucial way for machines to imitate human intelligence.
On the other hand, annotating large-scale supervised datasets is expensive and time-consuming (Lu et al., 2020).
Although traditional deep neural models have achieved tremendous success under sufficient supervision, it is still challenging to produce comparable performance when training examples are limited.
Hence, a series of studies are proposed to generalize deep neural networks to low-data scenarios.
One crucial branch of them is meta-learning with prototypes (Reed, 1972; Nosofsky, 1986;
Snell et al., 2017), where models are trained to quickly adapt to current tasks and carry out classification via metric-based comparisons between examples and newly introduced variables, prototypes of those classes.
In a general way, prototypes are designed to represent abstract class-level information and calculated by taking the mean output of a few examples belonging to one same class.
Thus, prototypes are represented as dense vectors with the same dimensions as the embeddings of training examples and can be regarded as center points of those classes in the embedding space.
And for the queried examples that the model needs to predict, the basic idea is to calculate the distances between the queried examples and prototypes and conduct classification based on the distances.
Originated from
Prototypical Network (Snell et al., 2017), sets of derivative prototype-based methods demonstrate the effectiveness in few-shot learning (Ren et al., 2018;
Gao et al., 2019a; Allen et al., 2019; Pan et al., 2019; Ding et al., 2021a).
However, prototypes are estimated from a few sampled examples, which cannot uncover the overall ground truth distribution (Yang et al., 2021).
Such biased distributions may generate biased prototypes and trigger sequent classification errors.
In other words, this modeling approach of prototypes could lack the capability to express the universal class-level information.
Hence, to enhance such expressibility of prototypes, we propose to use areas, i.e., tensor fields, rather than points in the embedding space to represent the class-level information.
In this paper, we propose big prototypes to use hyperspheres in the feature space to abstract the class-level information, and the feature points can be distribued inside or around the big prototypes.
[Figure]
Such modeling is equipped with two obvious advantages: easy to model and easy to calculate the distances.
On the one hand, even if we attempt to use areas but not points to represent class-level information, it is difficult to explicitly characterize manifolds with complex boundaries in deep learning.
But via hyperspheres modeling, we can obtain a big prototype only through two sets of parameters: the center and the radius of hyperspheres.
On the other hand, hyperspheres are suitable for calculating the distances in Euclidean space.
We can simply calculate the distance from one feature point to the surface of the hypersphere to perform metric-based classification, which is also difficult for other manifolds.
Moreover, it is easy to combine these two advantages in few-shot learning, the distances from one feature point to the surface of a big prototype can be formalized as the distance from the point to the center of the hypersphere minus the radius.
Thus, both radius and hypersphere center can appear in the loss function and participate in the backward propagation during optimization.
Intuitively, for the classes with sparse feature distribution, the corresponding radii of their prototypes are large, and the radii are small otherwise.
We conduct extensive experiments in both natural language processing (NLP) and computer vision (CV) to evaluate the effectiveness of big prototypes.
Specifically for NLP, we choose widely used benchmarks in the few-shot named entity (Ding et al., 2021b) and relation extraction (Han et al., 2018; Gao et al., 2019b).
For CV, we use classical image classification datasets (Vinyals et al., 2016) in experiments.
The results demonstrate that, with only a few additional parameters introduced, such modeling significantly outperforms the baseline.
Surprisingly, big prototypes perform extremely well in cross-domain few-shot relation extraction, indicating the promising ability to domain adaptation.
Given that such small changes can bring huge benefits, hopefully, big prototypes can inspire new ideas for the research community of representation learning.
The source code and checkpoints of our models in experiments will be publicly available for reproducibility.
R ELATED W ORK
This work is related to studies of few-shot learning and meta-learning, whose primary goal is to quickly adapt deep neural models to new tasks with few training examples.
To achieve that, two branches of studies are proposed: optimization-based methods and metric-based methods.
The optimization-based studies (Finn et al., 2017; Franceschi et al., 2018; Ravi & Beatson, 2018) regard few-shot learning as a bi-level optimization process, where a global optimization is conducted to learn a good initialization across various tasks, and a local optimization quickly adapts the initialization parameters to specific tasks with few training examples by few steps of gradient descent.
Compared to the mentioned studies, our work is more related to the metric-based meta-learning approaches (Vinyals et al., 2016;
Snell et al., 2017; Satorras & Estrach, 2018; Sung et al., 2018), whose general idea is to learn a metric to measure the similarity between representations and find the closest labeled example (or a derived prototype) for an unlabeled example.
Typically, these methods learn a metric function during episodic optimization.
More specifically, we inherit the spirit that uses prototypes to abstractly represent class-level information, which could be tracked back to cognitive science (Reed, 1972;
Rosch et al., 1976; Nosofsky, 1986), statistical machine learning (Graf et al., 2009) and similar to the Nearest Mean Classifier (Mensink et al., 2013).
In the area of deep learning, Snell et al.
(2017) propose the prototypical network to exploit the average of example embeddings as a prototype to perform metric-based classification in few-shot learning.
In Prototypical Network, prototypes are estimated by the embeddings of instances, and it is hard to find a satisfying location of the prototypes of the entire dataset.
Ren et al.
(2018) adapt such prototype-based networks in the semi-supervised scenario where the dataset is partially annotated.
A set of prototype-based networks are proposed concentrating on the improvements of prototype estimations and application to various downstream tasks (Allen et al., 2019;
Gao et al., 2019a; Li et al., 2019b; Pan et al., 2019; Seth et al., 2019;
Ding et al., 2021a; Li et al., 2020c).
We discuss big prototypes and some other prototype-enhanced methods in Appendix C.
There has also been a series of works that embed prototypes into a non-Euclidean output space (Mettes et al., 2019; Keller-Ressel, 2020; Atigh et al., 2021).
It should be noted that these studies regard hyperspheres or other non-Euclidean manifolds as the embedding space, and our proposed method use hyperspheres to represent big prototypes and conduct metric-based classification in the Euclidean space.
Therefore, the focus of our proposed big prototypes is different from the nonEuclidean prototype-based works.
We evaluate the effectiveness of big prototypes in three downstream tasks across NLP and CV, including few-shot named entity recognition, few-shot relation extraction, and few-shot image classification.
Another technical route to achieve promising results in these downstream few-shot tasks is to use the ability of large-scale pre-trained models (Brown et al., 2020; Han et al., 2021b), or build large-scale data sets to design specific pre-training tasks (Huang et al., 2020;
Soares et al., 2019) (which may face the risk of information leakage).
These techniques are also orthogonal to our contribution.
Generally, the method of big prototypes is model-agnostic.
B IG P ROTOTYPES
This section begins with the problem setup of few-shot learning.
Then, we introduce the metrics, initialization, and learning of big prototypes.
Generally, unlike previous prototype-based models that use estimated dense vectors as prototypes, our approach use hyperspheres to represent the conceptlevel information for classes.
One big prototype is represented by two parameters: the center and the radius of the hypersphere, which are firstly initialized via estimation and then optimized by gradient descent in an end-to-end fashion.
3. P ROBLEM S ETUP
In this work, we consider the episodic N way K shot few-shot learning paradigm.
In this setting, we have a large-scale annotated training set D train , and our goal is to learn a model that could predict for a set of new classes D test , where only a few examples are labeled.
In this setting, there are a support set for learning and a query set for inference in both training and testing stages.
The model will be trained in episodes.
Each episode contains a support set
S =
{ x i , y i } N × K
i
=
with N classes and K examples for each class, a query set Q = { x ∗ j , y ∗ j } N × K ′ j
= of examples in the same N classes.
Each input data is a vector x i ∈ R L with the dimension of L and y i is an index of class label.
For each input x i , let h i =
f ϕ ( x i ) ∈
R D denote the D -dimensional output embedding of a neural network f ϕ : R L → R D parameterized by ϕ .
Beyond the conventional few-shot classification, we also carry out experiments in few-shot named entity recognition.
This is a sequence labeling task where each token in a sequence is asked to be labeled as if it is a part of a named entity.
But as the context is extremely important for this task, the examples are sampled in sequence-level.
Thus the problem setup is slightly different from the traditional N way K shot classification.
We follow the strategy in Ding et al.
(2021b)
and sample sequences in a N way K ∼ 2 K shot manner (see Appendix B).
3. M ETRIC
We now introduce big prototypes, which are a set of hyperspheres in the feature space to abstractly represent the intrinsic features of classes.
No matter what the dimension of embedding space is, one big prototype is represented by p =
( z , r ) , where z ∈ R D is the center of the hypersphere with the same dimension to h
= f ϕ ( x ) , and r ∈ R is a scalar denoting the radius of the hypersphere.
The central idea is to learn a big prototype for each class with limited episodic supervision, and each example in the query set ( x ∗ j , y ∗ j ) is predicted by measuring the distance from the embedding h ∗ to the surface of the hyperspheres.
The distance of two vectors is calculated by a metric function: d : R D × R D → [0 , + ∞ ) .
In the Euclidean space, the metric being
[Equation]
And the distance ˜ d from an embedding to a big prototype p is the distance from the point to the center of the hypersphere minus the radius
[Equation]
Note that in this case, the value of ˜ d ( · ) may be negative, that is geometrically speaking, the point is contained inside the hypersphere, and it does not affect the calculation of loss.
Although more generally, the idea is to use areas but not points in the embedding space to model prototypes, hyperspheres naturally have two obvious advantages.
As stated in § 1, one big prototype could be uniquely modeled by the center and the radius p = { z , r } .
While characterizing manifolds with complex boundaries in the embedding space is difficult.s Furthermore, it is easy to calculate metrics by Equation 2.
We use the distance from a point to the surface as the metric.
In this way, the center and the radius are spontaneously in the loss function and optimized.
In this geometric interpretation, sparse classes will have larger learned radii, while compact classes will have smaller learned radii.
3. I NITIALIZATION
To construct such big prototypes, the first step is the initialization, that is, initialize the center z and the radius r of the hypersphere.
We use subscripts to denote indices of the classes and superscript to denote the indices of the support sets. As stated in § 1, we cannot fathom the distribution of the whole data, thus, for support set of the n -th class S n , we choose a more secure initialization strategy than random initialization
[Equation]
where the center z n is initialized by the mean output of the embeddings of the current class like Snell et al.
(2017), and r n is initialized by the average of all the metrics from the embeddings to the center.
The initialization essentially approximates the center and the sparse degree of the representations.
Since the computational cost of this initialization strategy is small, an alternative way is to use the average of the initialized hyperspheres of multiple episodes as the final big prototype.
In a K -shot episodic training, for the first m support set of the n -th class S Mn , we have
[Equation]
where m is a hyper-parameter.
Equation 3 denotes the situation that when m = 1 .
And if m is equal to the total number of episodes, establishing big prototypes degenerates into a estimation procedure.
3. L EARNING
Once initialized, a big prototype will participate in the training process, where its center and radius are simultaneously optimized.
A query point x is classified by softmax over the metrics to the big prototypes calculated by Equation
[Equation]
And the parameters of f and big prototypes are optimized by minimizing the metric-based crossentropy objective function:
[Equation]
log p ( y | x , p ) . Equation 5 explains the combination of the two advantages of big prototypes, where ˜ d is calculated as r and z will participate in the optimization.
The parameters of the neural network are optimized along with the centers and radii of big prototypes through gradient descent.
To sum up, in the initialization stage, z and r are estimated by the embeddings of examples.
And in the learning stage, z and r are optimized by an independent optimizer, because the final location and shape of the hyperspheres need to serve the performance of classification.
Algorithm 1 expresses the initialization and learning stages of big prototypes in pseudocode.
The goal of few-shot learning is to learn new classes with a handful of examples.
There are two strategies to handle the unseen classes at the episodic evaluation stage.
Since there still exist a support set and a query set in evaluation, we could strictly follow the initialization-learning paradigm.
Alternatively, we could directly estimate the big prototype like vanilla prototypical network.
Keeping aligned with the vanilla prototypes for fair comparison, we use the latter strategy in experiments.
Algorithm 1: Training loss of big prototypes, T is the total number of samples in the training set, N total is the total number of classes in the training set, N is the number of classes for support and query set, K is the number of examples per class in the support set,
K ′ is the number of examples per class in the query set, and m and M are hyper-parameters.
R ANDOM S AMPLE ( S, N ) denotes a set of N elements chosen uniformly at random from set S , without replacement.
Input
// Initialization phase for n = 1 to N total do ˜z n ← 0 , ˜ r n ←for j = 1 to m do
S nj ← R ANDOM S
AMPLE ( D n , K ) ˜z n ← ˜z n + 1 m |S nj | (cid:80) ( x i ,y i ) ∈S nj f ϕ ( x i ) , ˜
r n ← ˜ r n + 1 m |S nj | (cid:80) ( x i ,y i ) ∈S nj d ( f ϕ ( x i ) , z n ) , //
Learning phase
L ←for i = 1 to M do
V ← R ANDOM S
AMPLE ( { 1 , ..., N total } , N ) for n in { 1 , ..., N } do
Q n ← R ANDOM S AMPLE ( D V n , K ′ )
L ← L + 1 MNK ′ (cid:80) ( x i ,y i ) ∈Q n [ d ( f ϕ ( x i ) , z n ) − ˜ r n +log (cid:80) n ′ exp(˜ r n ′ − d ( f ϕ ( x i ) , z n ′ ))]
E XPERIMENTS
To evaluate the effectiveness of the proposed method and compare it with the baselines, we conduct experiments on three common few-shot learning tasks both in NLP and CV, including few-shot named entity recognition (NER), few-shot relation extraction (RE) and few-shot image classification.
This section describes and presents the results of each of the three tasks.
Note that the goal of big prototypes is to provide a new insight for representation learning of concept-level information, not to lead in all the leaderboards.
Since big prototypes are model-agnostic, the central idea of the design for the experiments is to make comparisons with the related baselines.
We use original prototypes as our main baseline and give less attention to other orthogonal techniques like the structures of neural models or the task-specific pre-training.
4. F EW - SHOT N AMED E NTITY R ECOGNITION
We assess the effectiveness of big prototypes on NLP, specifically, the first task is few-shot named entity recognition (NER) and the dataset is F EW -NERD
(Ding et al., 2021b). Different from typical instance-level classification, few-shot NER is a sequence labeling task, where labels may share structural correlations.
NER is the first step in automatic information extraction and the construction [Table] of large-scale knowledge graphs.
Quickly detecting fine-grained unseen entity types is of significant importance in NLP.
To capture the latent correlation, many recent efforts in this field use large pretrained language models (Han et al., 2021a) like BERT (Devlin et al., 2019) as backbone model and have achieved remarkable performance.
The original prototypical network has also been applied to this task (Li et al., 2020b; Huang et al., 2020; de Lichy et al., 2021).
We report the dataset, baselines, and experimental details in Appendix A.1.
Results Table 1 shows the performance of current state-of-art models on F EW -NERD.
Overall, big prototypes have a considerable advantage over prototypes, with an increase of at least 5% in f1score across all settings.
The success on both datasets demonstrates that big prototypes can learn the general distribution pattern of entities across different entity types and thus can greatly improve the performance when little information is shared between training and test set.
It can also be observed that a large portion of the improvement comes from the increase in precision, showing that big prototypes can better distinguish entities from context.
It is possibly because context words are much diverse and modeling them with a hypersphere as in big prototypes is more plausible than a single point as in prototypes.
With respect to number of shots, big prototypes are more advantageous when larger shots are provided and becomes the new state-of-art in the 5 ∼ 10 shot setting.
For the comparison with NNShot, big prototypes remains the superiority under the settings of high-shot (5 ∼ 10), outperforming it by at least 10% of F1-scores.
Interestingly, the performances of NNShot and big prototypes are comparable when it comes to low-shot.
This is because in the sequece labeling task, it is more difficult to infer the class-level information from very limited tokens.
And the use of big prototypes and tokens for metric-based classification play a similar role.
And the mediocre performance on low shot settings indicates that the hypersphere parameters are learned more effectively when more shots are available.
Also since in strict 1-shot setting the initialization of the radius would be zero, we believe a good initialization strategy is important for big prototypes.
4. F EW - SHOT R ELATION E XTRACTION
The other common NLP task is relation extraction (RE), which aims at correctly classifying the relation between two given entities in a sentence.
This is a fundamental task in information extraction.
RE is an important form of learning structured knowledge from unstructured text. We use FewRel Han et al.
and FewRel 2.0 Gao et al. (2019b) as the datasets. In real-world datasets, many of the relations are long-tailed and thus cannot be identified accurately under the common supervised setting.
Traditional methods often alleviate the problem with distant supervision, which would result in wrong labels.
Recent approaches have applied few-shot learning models on the task to learn from a handful of samples, which yield promising results (Gao et al., 2019a).
We report the datasets, baselines and experimental details in Appendix A.2.
Results Table 2 presents the results on two FewRel tasks, we did not compare methods that use additional data or conduct task-specific encoder pre-training.
Big prototypes generally perform
[Table]
better than prototypes across all settings.
In terms of backbone models, when combining with pretrained models like BERT, big prototypes can yield a larger advantage against prototypes.
It shows that the distribution assumption of big prototypes may be closer to the real data distribution, which boosts the finetuning phase.
Meanwhile it also sheds light on the untapped potential ability of large pre-trained models and stresses that the assumptions made about data distribution may help us unlock the potential.
It is also worth noting that big prototypes achieves outstanding performance on Domain Adaptation task, with an increase of about 20% in accuracies when using BERT.
The success signals the great capability of big prototypes in transfer learning and also points to the importance of a better abstraction of data distribution in transfer learning investigation.
Although big prototypes yield surprisingly performance in the domain adaptation task, the standard errors are also high.
This observation suggests that when the domain shifts, the unstable encoding of examples by language model may lead to different results.
Compared to other meta learning baselines, big prototypes also yield promising results on all the settings.
4. F
I MAGE C
Image classification is one of the most classical tasks in few-shot learning research.
Seeking a better solution for few-shot image classification is beneficial in two ways: (1) to alleviate the need for data augmentation, which is a standard technique to enrich the labeled data by performing transformations on a given image; (2) to facilitate the application where the labeled image is expensive.
We use mini ImageNet (Vinyals et al., 2016) as the dataset in our experiment.
The dataset, baselines, and experimental details are reported in Appendix A.3.
Results Table 3 shows the result on mini ImageNet few-shot classification under 2 settings.
Big prototypes substantially outperform prototypes in most settings, displaying their ability in modeling class distribution of images.
We observe that compared to NLP, the results of image classification are more stable both for vanilla prototypes and big prototypes.
This observation may indicate the difference in encoding between the two technologies.
Token representations in BERT are contextualized and changeable around different contexts, yet the image representation produced by deep CNNs aims to thoroughly capture the global and local features.
Different from the NLP experiments, we use 5-way 1-shot and 5-way-5-shot settings in image classification.
Under the 5-way 5-shot setting, the improvements of big prototypes are significant.
Compared to the main baseline of prototypes, big prototypes could yield considerable improvements with all the three backbones. The effective- ness of our method is also demonstrated by the comparisons with other previous few-shot learning methods.
With different backbones, big prototypes achieve better or comparable results than the previous state-of-the-art models.
In particular, with the WideResNet (Zagoruyko & Komodakis, 2016) backbone, big prototypes yield the best results of all the compared methods, suggesting that as the representational power of the model improves, so does the expressive capability of big prototypes.
Compared to the 5-shot setting, the big prototypes achieve less boost in the 1-shot setting of Convnet and ResNet-12 (He et al., 2015), the phenomenon is consistent with the intuition that more examples would be more favourable to the learning of radius
[Table]
4. A NALYSIS
Radius Dynamics In this subsection, the mechanism of big prototypes will be empirically analyzed. We demonstrate the mechanism of big prototypes by illustrating the change of radius for one specific hypersphere.
In the learning phase, the radius of a big prototype is changing according to the “density” of the sampled episode, which could be quantified by the mean distance of samples to the corresponding prototype center.
Practically, since the sampling is random, the value of the mean distance may oscillate at a high frequency in this process, and the radius changes accordingly.
To better visualize the changing of radius along with the mean distance at each update, for each round of training we fix one specific class as the anchor class for mean distance and radius recording and apply a special sampling strategy at each episode.
Specifically, we take FewRel training data and train on the 5 way 5 shot setting with CNN encoder.
While training, each episode contains the anchor class and 4 other randomly sampled classes.
Training accuracy is logged every 50 steps.
After a warmup training of 500 steps, we sample ”good” or ”bad” episodes for every 50 steps alternatively.
A ”good” episode has higher accuracy on the anchor class than the previously logged accuracy, while conversely, a ”bad” episode has an accuracy lower than before.
The mean distance to the prototype center and radius at each episode are logged every 50 steps after the warmup.
Figure 1 shows the changing of mean distance and radius for 8 classes during 600 ∼ 2000 training steps.
Although the numeric values of distance and radius differ greatly and oscillate at different scales, they have similar changing patterns.
Besides, it could be observed that there is often a small time lag in the change of radius, indicating that the change of radius is brought by the change in mean distance.
This is in line with our expectations and perfectly demonstrates the learning mechanism of big prototypes.
The experiment provides a promising idea, if we can control the sampling strategy through priori knowledge, we may find a way to learn ideal big prototypes.
Visualization We also use t -SNE (van der Maaten & Hinton, 2008) to visualize the embedding before and after training.
5 classes are sampled from training set and test set of Few-NERD dataset, respectively, and for each class 500 samples are randomly chosen to be embedded by BERT trained on 5 way 5 shot NER task.
Figure 2 is a graph showing the result of embeddings in a 2-dimensional
[Figure] Position held Publisher Owned by
Contains territorial entity
[Figure]
space, where each distinct color represents one class.
Note that for NER task the representations are obtained on the token level, where the interaction between the target token and its context may result in a more mixed-up distribution compared to instance-level embedding.
It could be seen that after training, the representations of the same class in the training set become more compressed and easier to classify.
It is also worth noting that the trained BERT encoder could even generate a more clustered representation distribution for a novel class.
The visualization demonstrates the effectiveness of big prototypes training in learning discriminative features, especially the success on learning novel class representation that greatly boosts model performance under few-shot settings.
[Figure]
C ONCLUSION
This paper proposes a novel metric-based few-shot learning method, big prototypes .
Unlike previous prototype-based methods that use dense vectors to represent the class-level semantics, we use hyperspheres to enhance the capabilities of prototypes to express the intrinsic information of the data.
It is simple to model a hypersphere in the embedding space than other complex manifolds, we establish two variables, the center and the radius to represent a big prototype.
Such modeling is easy to implement and also empirically effective, we observe significant improvements than vanilla prototype on three tasks across NLP and CV.
For potential future work, big prototypes could be extended to more generalized representation learning like image, word, or sentence representations.
E THICS S TATEMENT
This section states the ethical considerations of our work.
Generally, the method of big prototypes is a tool for few-shot learning, making it intrinsically unbiased.
The impact, no matter positive or negative, is depend on the intentions of specific applications that uses our method.
Nevertheless, since few-shot systems have already been deployed in real-world scenarios, it is crucial to carefully choose the training data to avoid the potential bias.
We strongly encourage the users of our method to be careful about the data selection in practice.
All the data used in our research are granted for research purpose, and it will not result in privacy issues.
In terms of energy saving, we will make all the checkpoints in our experiments publicly available to reduce the carbon emission.
R EPRODUCIBILITY S TATEMENT
We evaluate the effectiveness of big prototypes in three tasks across NLP and CV, which are few-shot named recognition, few-hot relation extraction, and few-shot image classification (§ 4).
For each experiment, we provide an experimental setting part to introduce details for reproduction, including data processing, hyper-parameters, the choice of the neural encoder, and the steps for training, validating, and testing.
Please refer to § 4.1, § 4.2, § 4.3 for the details of the three tasks, respectively.
For the specific initialization and learning of big prototypes, we provide pseudocode in Algorithm 1.
The source code and the checkpoints in our work will be released.
A E XPERIMENTAL D ETAILS
This section reports the experimental details of all the three tasks in our evaluation.
All the experiments are conducted on NVIDIA A100 and V100 GPUs with CUDA.
A. E XPERIMENTAL D ETAILS FOR F EW - SHOT N AMED E NTITY R ECOGNITION
Dataset The experiment is run on F EW -NERD dataset (Ding et al., 2021b).
It is a large-scale NER dataset containing over 400,000 entity mentions, across 8 coarse-grained types and 66 fine-grained types, which makes it an ideal dataset for few-shot learning.
It has been shown that existing methods including prototypes are not effective enough on this dataset.
Baselines NNShot
(Yang & Katiyar, 2020) is a token-level metric-based method that is specifically designed for few-shot labeling.
Note that the main baseline here is the Proto method, which adapts the prototypical network on few-shot named entity recognition.
Implementation Details We run experiments under four settings on the two released benchmarks,
F EW -NERD (INTRA) and F EW -NERD (INTER).
Specifically, we use uncased BERT as the backbone encoder and 1e-4 as the encoder learning rate.
We manually tune the learning rate for the radius parameter and the best result is obtained with 10.
AdamW is used as the BERT optimizer and Adam (Kingma & Ba, 2017) is used to optimize prototype radius.
The batch size is set toacross all settings.
All models are trained for 10000 steps and validated every 1000 steps.
The results are reported on 5000 steps of the test episode.
For each setting, we run the experiment withdifferent random seeds and report the average metrics including precision, recall, f1-score, and the standard error for each.
We use PyTorch (Paszke et al., 2019) and huggingface transformers (Wolf et al., 2020) to implement the backbone encoder BERT base .
A. E XPERIMENTAL D ETAILS FOR F EW - SHOT R ELATION E XTRACTION
Dataset We adopt the FewRel dataset (Han et al., 2018; Gao et al., 2019b), a relation extraction dataset specifically designed for few-shot learning.
FewRel has 100 relations with 700 labeled instances each.
The sentences are extracted from Wikipedia and the relational entities are obtained from Wikidata.
FewRel 1.0 (Han et al., 2018) is released as a standard few-shot learning benchmark.
FewRel 2.0 (Gao et al., 2019b) adds domain adaptation task and NOTA task on top of FewRel 1.
with the newly released test dataset on PubMed corpus.
Baselines In addition to the main baseline, prototypical network Snell et al.
(2017), we also choose the following few-shot learning methods as the baselines in relation extraction.
(1) Proto-HATT Gao et al.
(2019a) is a neural model with hybrid prototypical attention.
(2) MLMAN Ye & Ling (2019) is a multi-level matching and aggregation network for few-shot relation classification.
Note that ProtoHATT and MLMAN are not model-agnostic.
(3) GNN Satorras & Estrach (2018) is a meta-learning model with a graph neural network as the prediction head.
(4) SNAIL Mishra et al.
(2017) is a metalearning model with attention mechanisms.
(5) Meta Net Munkhdalai & Yu (2017) is a classical meta-learning model with meta information.
(6) Proto-ADV Gao et al.
(2019b) is a prototype-based method enhanced by adversarial learning.
(7) BERT-pair Gao et al.
(2019b) is a strong baseline that uses BERT for few-shot relation classification.
We re-run all the baselines, except for MLMAN and report the corresponding performances.
Implementation Details The experiments are conducted on FewRel 1.0 and FewRel 2.0 domain adaptation tasks.
For FewRel 1.0, we follow the official splits in Han et al.
For FewRel2.0, we follow Gao et al.
(2019b), training the model on wiki data, validating on SemEval data, and testing on the PubMed data.
We use the same CNN structure and BERT as encoders.
The learning rate for big prototype radius is 0.1 and 0.01 for CNN and BERT encoder, respectively.
Adam (Kingma & Ba, 2017) is used as radius optimizer.
We train the model for 10000 steps, validate everysteps, and test for 5000 steps.
The other hyperparameters are the same as the original paper.
A. E XPERIMENTAL D ETAILS FOR F EW - SHOT I MAGE C LASSIFICATION
Dataset mini ImageNet (Vinyals et al., 2016) is used as a common benchmark for few-shot learning.
The dataset is extracted from the full ImageNet dataset (Deng et al., 2009), and consists of
Baselines The baselines we choose are as follows: (1) Prototypical network (Snell et al., 2017) is our main baseline; (2) IMP (Allen et al., 2019) is a prototype-enhanced method that models infinite mixture of prototypes for few-shot learning; (3) CovaMNet (Li et al., 2019a) is a few-shot learning method that uses covariance to model the distribution information to enhance few-shot learning peformance.
(4) SNAIL (Mishra et al., 2017) is an attention-based classical meta-learning method; (5) Variational FSL (Zhang et al., 2019) is a variational Beyasian framework for few-shot learning, which contains a pre-training stage; (6) Activation to Parameter (Qiao et al., 2018) predicts parameters from activations in few-shot learning; (7) LEO (Rusu et al., 2018) optimizes latent embeddings for few-shot learning.
(8) TRAML (Li et al., 2020a) uses adaptive margin loss to boost few-shot learning, and Prototypes + TRAML is a strong baseline in recent years.
; (9) Meta-baseline Chen et al.
(2021) is a pre-training & tuning method that serves as a strong baseline in few-shot learning.
Implementation Details The experiments are conducted on 5 way 1 shot and 5 way 5 shot settings.
To ensure validity and fairness, we implement big prototypes with various backbone models including CNN, ResNet-12 and WideResNet (Zagoruyko & Komodakis, 2016) to make it comparable to all baseline results, and we also re-run some of the baselines including prototypical network (Snell et al., 2017), infinite mixture prototypes (Allen et al., 2019), and CovaMNet (Li et al., 2019a) under our settings based on their original code.
Other baseline results are taken from the original paper.
Each model is trained on 10,000 randomly sampled episodes for 30 ∼ 40 epochs and tested on 1000 episodes.
The result is reported with 95% confidence interval.
Note that both ResNet-12 and
WideResNet (Zagoruyko & Komodakis, 2016) are pretrained on the training data, where the pretrained ResNet-12 is identical to Chen et al.
and the pretrained WideResNet follows Mangla et al.
The CNN structure is the same as Snell et al.
(2017), which is composed of 4 convolutional blocks each with a 64-filter 3 × 3 convolution, a batch normalization layer (Ioffe & Szegedy, 2015), a ReLU nonlinearity, and a 2 × 2 max-pooling layer.
We use SGD optimizer for the encoder and Adam (Kingma & Ba, 2017) optimizer for the prototype radius.
The learning rate for the backbone model is 1e-3.
The learning rate for radius is manually tuned and the reported result has a learning rate of 10.
At the training stage, the prototype center is re-initialized at each episode as the mean vector of the support embeddings.
B K ∼ 2K S AMPLING FOR F EW -NERD
In the sequence labeling task F EW -NERD, the sampling strategy is slightly different from other classification task.
We follow the strategy of the original paper Ding et al.
(2021b)
and report it in
Algorithm 2.
Algorithm 2:
Input: Dataset X , Label set Y , N , K
Output: output result
S ← ∅ ; //
Init the support set //
Init the count of entity types for i = 1 to N do
Count [ i ] = 0 ; repeat
Randomly sample ( x , y ) ∈
X ;
Compute | Count | and Count i after update ; if | Count | > N or ∃ Count [ i ] > 2 K then
Continue ; else
S = S (cid:83) ( x , y ) ;
Update Count i ; until
Count i
≥ K for i = 1 to N ;
C O THER P ROTOTYPE - ENHANCED M ETHODS
Here, we discuss the difference between big prototypes with four prototype-enhanced methods in few-shot learning: infinite mixture prototypes (Allen et al., 2019), CovaMNet (Li et al., 2019a),
Variational Few-Shot Learning (Zhang et al., 2019), and Two-Stage Approach (Das & Lee, 2020).
Infinite mixture prototypes (Allen et al., 2019) model each class as an indefinite number of clusters, and the prediction is obtained by computing and comparing the distance to the nearest cluster in each class.
This method is an intermediate model between prototypes and nearest neighbor model, whereas big prototypes alleviate the over-generalization problem of vanilla prototypes with a single additional parameter that turns a single point modeling into a hypersphere.
The essential prototypebased feature of big prototypes allows faster computation and easier training.
CovaMNet (Li et al., 2019a) calculates local variance for each class based on support samples and conduct metric-based learning via covariance metric, which basically evaluates the cosine similarity between query samples and the eigenvectors of the local covariance matrix.
To ensure the nonsingularity of covariance matrix, the feature of each sample is represented with a matrix, generated by a number of local descripters, with each extracting a feature vector.
Compared to big prototypes, both methods attempt to model more variance based on vanilla prototypes, while the idea of big prototypes is more straightforward and requires fewer parameters.
On the other hand, the multichannel features adopted by CovaMNet are less natural for NLP tasks.
Variational Few-Shot Learning (Zhang et al., 2019)
tackles few-shot learning problem under a bayesian framework.
In order to improve single point based estimation, a class-specific latent variable representing the class distribution is introduced and is assumed to be Gaussian.
The method parameterizes the mean and variance of the latent variable distribution with neural networks that take the feature of a single instance as input.
The learning and inference process are both conducted on the latent variable level.
The method adopts variational inference and is built on modeling distribution as a latent variable, where the metric calculation highly relies on the Gaussian assumption.
Big prototypes, on the other hand, models the distribution with a center vector and a radius parameter in the actual embedding space, which is more tangible and easier to calculate.
Two-Stage Approach first trains feature encoder and variance estimator on training data in episodic manner with extracted absolute and relative features.
Then in the second stage, training data are split into ”novel” class and base class, novel class prototypes are learned from both sample mean and base class features.
The classification is carried out with the integrated prototypes.
This method improves on vanilla prototypes by extracting more features and combining information from base class, but still follows single-point-based metric learning.
Big prototypes extends single point to a hypersphere in the embedding space, and therefore better captures within-class variance.