F EW - SHOT T EXT C LASSIFICATION WITH D ISTRIBUTIONAL S IGNATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we explore meta-learning for few-shot text classiﬁcation.
Metalearning has shown strong performance in computer vision, where low-level patterns are transferable across learning tasks.
However, directly applying this approach to text is challenging–lexical features highly informative for one task may be insigniﬁcant for another.
Thus, rather than learning solely from words, our model also leverages their distributional signatures, which encode pertinent word occurrence patterns.
Our model is trained within a meta-learning framework to map these signatures into attention scores, which are then used to weight the lexical representations of words.
We demonstrate that our model consistently outperforms prototypical networks learned on lexical knowledge (Snell et al., 2017) in both few-shot text classiﬁcation and relation classiﬁcation by a signiﬁcant margin across six benchmark datasets (20.0% on average in 1-shot classiﬁcation).
I NTRODUCTION
In computer vision, meta-learning has emerged as a promising methodology for learning in a lowresource regime.
Speciﬁcally, the goal is to enable an algorithm to expand to new classes for which only a few training instances are available.
These models learn to generalize in these low-resource conditions by recreating such training episodes from the data available.
Even in the most extreme low-resource scenario–a single training example per class–this approach yields 99.6% accuracy on the character recognition task (Sung et al., 2018).
Given this strong empirical performance, we are interested in employing meta-learning frameworks in NLP.
The challenge, however, is the degree of transferability of the underlying representation learned across different classes.
In computer vision, low-level patterns (such as edges) and their corresponding representations can be shared across tasks.
However, the situation is different for language data where most tasks operate at the lexical level.
Words that are highly informative for one task may not be relevant for other tasks.
Consider, for example, the corpus of HuffPost headlines, categorized into 41 classes.
Figure 1 shows that words highly salient for one class do not play a signiﬁcant role in classifying others.
Not surprisingly, when meta-learning is applied directly on lexical inputs, its performance drops below a simple nearest neighbor classiﬁer.
The inability of a traditional meta-learner to zoom-in on important features is further illustrated in Figure 2: when considering the target class ﬁfty (lifestyle for middle-aged), the standard prototypical network (Snell et al., 2017) attends to uninformative words like “date,” while downplaying highly predictive words such as “grandma.”
In this paper we demonstrate that despite these variations, we can effectively transfer representations across classes and thereby enable learning in a low-resource regime.
Instead of directly considering words, our method utilizes their distributional signatures , characteristics of the underlying word distributions, which exhibit consistent behaviour across classiﬁcation tasks.
Within the meta-learning framework, these signatures enable us to transfer attention across tasks, which can consequently be used to weight the lexical representations of words.
One broadly used example of such distributional signatures is tf-idf weighting, which explicitly speciﬁes word importance in terms of its frequency in a document collection, and its skewness within a speciﬁc document.
[Figure]
[Figure]
this gorgeous grandma proves beauty has no expiration date
Prototypical network (unseen class) this gorgeous grandma proves beauty has no expiration date
Our model (unseen class) this gorgeous grandma proves beauty has no expiration date
Building on this idea, we would like to learn to utilize distributional signatures in the context of cross-class transfer.
In addition to word frequency, we assess word importance with respect to a speciﬁc class.
This latter relation cannot be reliably estimated of the target class due to the scarcity of labeled data.
However, we can obtain a noisy estimate of this indicator by utilizing the few provided training examples for the target class, and then further reﬁne this approximation within the meta-learning framework.
We note that while the representational power of distributional signatures is weaker than that of their lexical counterparts, meta knowledge built on distributional signatures are better able to generalize.
Our model consists of two components.
The ﬁrst is an attention generator , which translates distributional signatures into attention scores that reﬂect word importance for classiﬁcation.
Informed by the attention generator’s output, our second component, a ridge regressor , quickly learns to make predictions after seeing only a few training examples.
The attention generator is shared across all episodes, while the ridge regressor is trained from scratch for each individual episode.
The latter’s prediction loss provides supervision for the attention generator.
Theoretically, we show that the attention generator is robust to word-substitution perturbations.
We evaluate our model on ﬁve standard text classiﬁcation datasets (Lang, 1995; Lewis et al., 2004;
Lewis, 1997; He & McAuley, 2016; Misra, 2018) and one relation classiﬁcation dataset (Han et al., 2018).
Experimental results demonstrate that our model delivers signiﬁcant performance gains over all baselines.
For instance, our model outperforms prototypical networks by 20.6% on average in one-shot text classiﬁcation and 17.3% in one-shot relation classiﬁcation.
In addition, both qualitative and quantitative analyses conﬁrm that our model generates high-quality attention for unseen classes.
R ELATED W ORK
Meta-learning Meta-learning has been shown to be highly effective in computer vision, where lowlevel features are transferable across classes.
Existing approaches include learning a metric space over input features (Koch, 2015;
Vinyals et al., 2016;
Snell et al., 2017; Sung et al., 2018), developing a prior over the optimization procedure (Ravi & Larochelle, 2016;
Finn et al., 2017; Nichol &
Schulman, 2018; Antoniou et al., 2018), and exploiting the relations between classes (Garcia &
Bruna, 2017).
These methods have been adapted with some success to speciﬁc applications in NLP, including machine translation (Gu et al., 2018), text classiﬁcation (Yu et al., 2018;
Geng et al., 2019; Guo et al., 2018; Jiang et al., 2019) and relation classiﬁcation (Han et al., 2018).
Such models primarily build meta-knowledge on lexical representations.
However, as our experiments show, there exist innate differences in transferable knowledge between image data and language data, and lexicon-aware meta-learners fail to generalize on standard multiclass classiﬁcation datasets.
[Figure]
In this work, we observe that even though salient features in text may not be transferable, their distributional behaviors are alike.
Thus, we focus on learning the connection between word importance and distributional signatures.
As a result, our model can reliably identify important features from novel classes.
Transfer learning Our work is also closely related to transfer learning: we assume access to a large number of labeled examples from source classes, and we would like to identify word importance for the target classiﬁcation task.
Current approaches transfer knowledge from the source to the target by either ﬁne-tuning a pre-trained encoder (Howard & Ruder, 2018; Peters et al., 2018;
Radford et al., 2018; Bertinetto et al., 2019), or multi-task learning with a shared encoder (Collobert & Weston, 2008; Liu et al., 2015;
Luong et al., 2015; Strubell et al., 2018).
Recently, Bao et al.
(2018) also successfully transferred task-speciﬁc attention through human rationales.
In contrast to these methods, where the transfer mechanism is pre-designed, we learn to transfer based on the performance of downstream tasks.
Speciﬁcally, we utilize distributional statistics to transfer attention across tasks.
We note that while Wei et al.
(2017) and Sun et al.
(2018) also learn transfer mechanisms for image recognition, their methods do not directly apply to NLP.
B ACKGROUND
In this section, we ﬁrst summarize the standard meta-learning framework for few-shot classiﬁcation and describe the terminology (Vinyals et al., 2016).
Next, we introduce our extensions to the framework.
Figure 3 and 4 graphically illustrate our framework.
Problem statement Suppose we are given labeled examples from a set of classes Y train .
Our goal is to develop a model that acquires knowledge from these training data, so that we can make predictions over new (but related) classes, for which we only have a few annotations.
These new classes belong to a set of classes Y test , disjoint from Y train .
Meta-training In meta-learning, we emulate the above testing scenario during meta-training so our model learns to quickly learn from a few annotations.
To create a single training episode , we ﬁrst sample N classes from Y train .
For each of these N classes, we sample K examples as our training data and L examples as our testing data.
We update our model based on loss over these testing data.
Figure 4a shows an example of an episode.
We repeat this procedure to increase the number of training episodes, each of which is constructed over its own set of N classes.
In literature, the training data of one episode is commonly denoted as the support set , while the corresponding testing data is known as the query set .
Given the support set, we refer to the task of making predictions over the query set as N -way K -shot classiﬁcation .
Meta-testing After we have ﬁnished meta-training, we apply the same episode-based mechanism to test whether our model can indeed adapt quickly to new classes.
To create a testing episode , we ﬁrst sample N new classes from Y test .
Then we sample the support set and the query set from the N classes.
We evaluate the average performance on the query set across all testing episodes.
[Figure]
Our extension We observe that even though all examples from Y train are accessible throughout meta-training, the standard meta-learning framework (Vinyals et al., 2016) only learns from small subsets of these data per training episode.
In contrast, our model leverages distributional statistics over all training examples for more robust inference.
To accommodate this adjustment, we augment each episode with a source pool (Figure 4b).
During meta-training (Figure 3a), this source pool includes all examples from training classes not selected for the particular episode.
During metatesting (Figure 3b), this source pool includes all training examples.
M ETHOD
Overview Our goal is to improve few-shot classiﬁcation performance by learning high-quality attention from the distributional signatures of the inputs.
Given a particular episode, we extract relevant statistics from the source pool and the support set.
Since these statistics only roughly approximate word importance for classiﬁcation, we utilize an attention generator to translate them into high-quality attention that operates over words.
This generated attention provides guidance for the downstream predictor, a ridge regressor , to quickly learn from a few labeled examples.
We note that the attention generator is optimized over all training episodes, while the ridge regressor is trained from scratch for each episode.
Figure 5 illustrates the two components of our model.
• Attention generator: This module generates class-speciﬁc attention by combining the distributional statistics of the source pool and the support set (Figure 5a).
The generated attention provides the ridge regressor an inductive bias on the word importance.
We train this module based on feedback from the ridge regressor (Section 4.1).
• Ridge regressor: For each episode, this module constructs lexical representations using the attention derived from distributional signatures (Figure 5b).
The goal of this module is to make predictions over the query set, after learning from the support set (Figure 5c and 5d).
Its prediction loss is end-to-end differentiable with respect to the attention generator which leads to efﬁcient training (Section 4.2).
In our theoretical analysis, we show that the attention generator’s outputs are invariant to wordsubstitution perturbations (Section 4.3).
4. A TTENTION GENERATOR
The goal of the attention generator is to assess word importance from the distributional signatures of each input example.
There are many choices of distributional signatures.
Among them, we focus on functions of unigram statistics, which are provably robust to word-substitution perturbations (Section 4.3).
We utilize the large source pool to inform the model of general word importance and leverage the small support set to estimate class-speciﬁc word importance.
The generated attention will be used later to construct the input representation for downstream classiﬁcation.
It is well-documented in literature that words which appear frequently are unlikely to be informative (Sparck Jones, 1972).
Thus, we would like to downweigh frequent words and upweight rare
[Table]
+h5305hY3Nre6e4W9rbPzg8Kh+ftEycasqaNBax7oTEMMEVayJHwTqJZkSGgrXDyf283n5i2vBYPeI0YYEkI8UjTglaq9sfE8y6s0FjUK54VW8hdx38HCqQqz4of/WHMU0lU0gFMabnewkGGdHIqWCzUj81LCF0QkasZ1ERyUyQLRaeuRfWGbpRrO1T6C7c3xMZkcZMZWg7JcGxWa3Nzf9qvRSj2yDjKkmRKbr8KEqFi7E7v94dcs0oiqkFQjW3u7p0TDShaDMq2RD81ZPXoXVV9S03riu1uzyOIpzBOVyCDzdQgweoQxMoSHiGV3hztPPivDsfy9aCk8+cwh85nz/cB5Bu</latexit>
inference
[Figure]
[Table] [Table]
+omnMpKUCjekEfmLDDLXlVLBpqZsaliAd45B1HEqMmQmz+bZTcuacPhko7Z60ZO7+nsgwNmYSR64zRjsyy7WZ+V+tk9rBdZhxmaSWSbr4aJAKYhWZnU76XDNqxcQBUs3droSOUCO1LqCSCyFYPnkVmhfVwPH9ZaV2k8dRhBM4hXMI4ApqcAd1aACFR3iGV3jzlPfivXsfi9aCl88cwx95nz+Lg48Y</latexit
[Figure]
words.
To measure general word importance , we select an established approach by Arora et al.
[Equation]
where ε = 10 − 3 , x i is the i th word of input example x , and P ( x i ) is the unigram likelihood of x i over the source pool.
On the other hand, words that are discriminative in the support set are likely to be discriminative in the query set.
Thus, we deﬁne the following statistic to reﬂect class-speciﬁc word importance :
[Equation]
where the conditional likelihood P ( y | x i ) is estimated over the support set using a regularized linear classiﬁer 3 and H ( · ) is the entropy operator.
We note that t( · ) measures the uncertainty of the class label y , given the word.
Thus, words that exhibit a skewed distribution will be highly weighted.
Directly applying these statistics may not result in good performance for two reasons: 1) the two statistics contain complementary information, and it is unclear how to combine them; and 2) these statistics are noisy approximations to word importance for classiﬁcation.
To bridge this gap, we concatenate these signatures and employ a bi-directional LSTM (Hochreiter & Schmidhuber, 1997) to fuse the information across the input: h =
biLSTM([s( x ); t( x )]) .
Finally, we use dot-product attention to predict the attention score α i of word x i :
[Equation]
where h i is the output of the biLSTM at position i and v is a learnable vector.
4. R IDGE REGRESSOR
Informed by the attention generator, the ridge regressor quickly learns to make predictions after seeing a few examples.
First, for each example in a given episode, we construct a lexical representation that focuses on important words, as indicated by attention scores.
Next, given these lexical representations, we train the ridge regressor on the support set from scratch.
Finally, we make predictions over the query set and use the loss to teach the attention generator to produce better attention.
Constructing representations Given that different words exhibit varying levels of importance towards classiﬁcation, we construct lexical representations that favor pertinent words.
Speciﬁcally, we deﬁne the representation of example x as
[Equation]
where f ebd ( · ) is a pre-trained embedding function that maps a word into R E .
Training from the support set Given an N -way K -shot classiﬁcation task, let Φ S ∈ R NK × E be the representation of the support set, obtained from φ ( · ) , and Y S ∈ R NK × N be the one-hot labels.
We adopt ridge regression (Bertinetto et al., 2019) to ﬁt the labeled support set for the following reasons: 1) ridge regression admits a closed-form solution that enables end-to-end differentiation through the model, and 2) with proper regularization, ridge regression reduces over-ﬁtting on the small support set.
Speciﬁcally, we minimize regularized squared loss
[Equation]
over the weight matrix W ∈ R E × N .
Here (cid:107)·(cid:107) F denotes the Frobenius norm, and λ > 0 controls the conditioning of the learned transformation W .
The closed-form solution can be obtained as
[Equation]
where I is the identity matrix.
Inference on the query set Let Φ Q denote the representation of the query set.
Although we optimized for a regression objective in Eq equation 5, the learned transformation has been shown to work well in few-shot classiﬁcation after a calibration step (Bertinetto et al., 2019), as
[Equation]
7) where a ∈ R + and b ∈ R are meta-parameters learned through meta-training.
Finally, we apply a softmax over ˆ Y Q to obtain the predicted probabilities ˆ P Q .
Note that this calibration only adjusts the temperature and scale of the softmax; its mode remains unchanged.
During meta-training, we compute the cross-entropy loss L CE between ˆ P Q and the labels over the query set.
Since both Φ S and Φ Q depend on φ ( · ) , L CE provides supervision for the attention generator.
4. T HEORETICAL ANALYSIS
Working with distributional signatures brings certiﬁed robustness against input perturbations.
Formally, let ( P , S , Q ) be a single episode, where P is the source pool, S is the support set and Q is the query set.
For any input text x
[Equation]
, the attention generator produces word-level importance given the support set and the source pool:
[Equation]
Now consider a word-substitution perturbation σ that replaces a word w by σ ( w ) .
4 If we arbitrarily swap words, we may encounter nonsensical outputs, as important words may be substituted by common words, like “a” or “the.”
Thus, we consider perturbations that preserve the unigram probabilities of words, estimated over the source pool: P ( w ) =
P ( σ ( w )) for all w ∈ V .
In this way, important words (for one class) are mapped to similarly important words (perhaps for another class).
We use ˜ S and ˜ Q to denote the support and query sets after applying σ word by word.
Theorem 1.
Assume σ : V → V satisﬁes P ( w ) =
P ( σ ( w )) for all w .
If σ is a bijection, then for any input text x ∈ S ∪ Q and its perturbation ˜ x ∈ ˜ S ∪ ˜ Q , the outputs of the attention generator are the same:
[Equation]
The idea behind this proof is to show that general word importance s and class-speciﬁc word importance t are invariant to such perturbations.
Details may be found in Appendix A.2.
Alternatively, one can view this word-substitution perturbation as a change to the input distribution.
Let P ( x | y ) be the conditional probability of the input x given the class label y .
Assuming that σ is a bijection, the conditional after perturbation can be expressed as ˜P( x | y )
=
P (cid:0) σ − 1 ( x ) | y (cid:1) , where σ − 1 ( x ) is the result of applying σ − 1 to every word in x .
Intuitively, Theorem 1 tells us that if a word w is a discriminative feature for the original classiﬁcation task, then after the change of the input distribution, the word σ ( w ) should be a discriminative feature for the new task.
This property makes sense since word-importance for classiﬁcation should depends only on the relative differences between classes.
In fact, the theorem holds when the input to the attention generator is any function of unigram counts.
We also note that the classiﬁcation performance on the query set can be different, as the pre-trained embedding function f ebd ( · ) is generally not invariant under such perturbations.
E XPERIMENTAL SETUP 5. D ATASETS
We evaluate our approach on ﬁve text classiﬁcation datasets and one relation classiﬁcation dataset.(See Appendix A.4 for more details.)
20 Newsgroups is comprised of informal discourse from news discussion forums (Lang, 1995).
Documents are organized under 20 topics.
RCV1 is a collection of Reuters newswire articles from 1996 to 1997 (Lewis et al., 2004).
These articles are written in formal speech and labeled with a set of topic codes.
We consider 71 secondlevel topics as our total class set and discard articles that belong to more than one class.
Reuters-21578 is a collection of shorter Reuters articles from 1987 (Lewis, 1997).
We use the standard ApteMod version of the dataset.
We discard articles with more than one label and consider 31 classes that have at least 20 articles.
Amazon product data contains customer reviews from 24 product categories (He & McAuley, 2016).
Our goal is to classify reviews into their respective product categories.
Since the original dataset is notoriously large (142.8 million reviews), we select a more tractable subset by sampling 1000 reviews from each category.
HuffPost headlines consists of news headlines published on HuffPost between 2012 and 2018 (Misra, 2018).
These headlines split among 41 classes.
They are shorter and less grammatical than formal sentences.
FewRel is a relation classiﬁcation dataset developed for few-shot learning (Han et al., 2018).
Each example is a single sentence, annotated with a head entity, a tail entity, and their relation.
The goal is to predict the correct relation between the head and tail.
The public dataset contains 80 relation types.
5. B ASELINES
We compare our model (denoted as OUR ) to different combinations of representations and learning algorithms.
Details of the baselines may be found in Appendix A.11.
Representations We evaluate three representations.
AVG represents each example as the mean of its embeddings.
IDF represents each example as the weighted average of its word embeddings, with weights given by inverse document frequency over all training examples.
CNN applies 1D convolution over the input words and obtains the representation by max-over-time pooling (Kim, 2014).
Learning algorithms
In addition to the ridge regressor ( RR ) (Bertinetto et al., 2019), we evaluate two standard supervised learning algorithms and two meta-learning algorithms.
NN is a 1-nearestneighbor classiﬁer under Euclidean distance.
FT pre-trains a classiﬁer over all training examples, then ﬁnetunes the network using the support set (Chen et al., 2019).
MAML meta-learns a prior over model parameters, so that the model can quickly adapt to new classes (Finn et al., 2017).
Prototypical network ( PROTO ) meta-learns a metric space for few-shot classiﬁcation by minimizing
[Table]
the Euclidean distance between the centroid of each class and its constituent examples (Snell et al., 2017).
5. I MPLEMENTATION DETAILS
We use pre-trained fastText embeddings (Joulin et al., 2016) for our model and all baselines.
For sentence-level datasets (HuffPost, FewRel), we also experiment with pre-trained BERT embeddings (Devlin et al., 2018) using HuggingFace’s codebase (Wolf et al., 2019).
For relation classiﬁcation (FewRel), we augment the input of our attention generator with positional embeddings (Zhang et al., 2017).
In the attention generator, we use a biLSTM with 50 hidden units and apply dropout of 0.1 on the output (Srivastava et al., 2014).
In the ridge regressor, we optimize meta-parameters λ and a in the log space to maintain the positivity constraint.
All parameters are optimized using Adam with a learning rate of 0.001 (Kingma & Ba, 2014).
During meta-training, we sample 100 training episodes per epoch.
We apply early stopping when the validation loss fails to improve for 20 epochs.
We evaluate test performance based ontesting episodes and report the average accuracy over 5 different random seeds.
R ESULTS
We evaluated our model in both 5-way 1-shot and 5-way 5-shot settings.
These results are reported in
Table 1.
Our model consistently achieves the best performance across all datasets.
On average, our model improves 5-way 1-shot accuracy by 7.5% and 5-way 5-shot accuracy by 3.9%, against the best baseline for each dataset.
When comparing against CNN + PROTO , our model improves by 20.0% on average in 1-shot classiﬁcation.
The empirical results clearly demonstrate that meta-learners privy to lexical information consistently fail, while our model is able to generalize past class-speciﬁc vocabulary.
Furthermore, Figure 6 illustrates that a lexicon-aware meta-learner ( CNN + PROTO ) is
[Figure]
unseen
[Figure]
unseen classes [Figure] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
able to overﬁt the training data faster than our model, but our model more readily generalizes to unseen classes.
Ablation study We perform ablation studies on the attention generator.
These results are shown at the bottom of Table 1.
We observe that both statistics s( · ) and t( · ) contribute to the performance, though the former has a larger impact.
We also note that instead of computing word importance independently for each word using a multilayer perceptron ( OUR w/o bi LSTM ), fusing information across the input with an biLSTM improves performance slightly.
Finally, we observe that restricting meta-knowledge to distributional signatures is essential to performance: when both lexical word embeddings and distributional signatures are fed to the attention generator ( OUR w EBD ), the performance drops consistently.
Contextualized representations For sentence-level datasets (FewRel, HuffPost), we also experiment with contextualized representations, given by BERT (Devlin et al., 2018).
These results are shown in Table 2.
While BERT signiﬁcantly improves classiﬁcation performance on FewRel, we observe no performance boost on HuffPost.
We postulate that this discrepancy arises because relation classiﬁcation is highly contextual, while news classiﬁcation is mostly keyword-based.
Analysis We visualize our attention-weighted representation φ ( x ) in Figure 7.
Compared to directly using general word importance s( x ) or class-speciﬁc word importance t( x ) , our method produces [Figure] [Figure] [Figure]
[Table]
unemployment was 6. unemployment was 6. in december last better separation, which enables effective learning from a few examples. This highlights that the power of our approach lies not in the distributional signatures themselves, but rather in the representations learned on top of them.
Figure 8 visualizes the model’s input and output on the same query example in two testing episodes.
The example belongs to the class jobs in the Reuters dataset.
First, we observe that our model generates meaningful attention from noisy distributional signatures.
Furthermore, the generated attention is task-speciﬁc : in the depicted example, if the episode contains other economics-related classes, the word “statistical” is downweighed by our model.
Conversely, “statistical” is upweighted when we compare jobs to other distant classes.
C ONCLUSION
In this paper, we propose a novel meta-learning approach that capitalizes on the connection between word importance and distributional signatures to improve few-shot classiﬁcation.
Speciﬁcally, we learn an attention generator that translates distributional statistics into high-quality attention.
This generated attention then provides guidance for fast adaptation to new classiﬁcation tasks.
Experimental results on both text and relation classiﬁcation validate that our model identiﬁes important words for new classes.
The effectiveness of our approach demonstrates the promise of meta learning with distributional signatures.
A S UPPLEMENTAL M ATERIAL A. R EGULARIZED LINEAR CLASSIFIER
Given an N -way K -shot classiﬁcation task, the goal of the regularized linear classiﬁer is to approximate task-speciﬁc word importance using the support set.
Let x = { x 1 , . . .
x T } be an input example, and let f ebd ( · ) be an embedding function that maps each word x i into R E .
We compute the representation of x by the average of its embeddings:
[Equation]
i
[Equation]
Since the support set only contains a few examples, we adopt a simple linear classiﬁer to reduce overﬁtting:
[Equation]
where
W ∈ R N ×
E is the weight matrix to learn.
We minimize the cross entropy loss between the prediction ˆ p and the ground truth label while penalizing the Frobenius norm of W .
We stop training once the gradient norm is less than 0 .
Finally, given a word x i , we estimate its conditional probability via softmax( Wψ ( x i )) .
Time efﬁciency Since the support set is very small (less than 25 examples) and the loss function is strongly convex, this linear classiﬁer converges very fast in practice.
7 Note that for larger problems, we can speed up computation by formulating this procedure as a regression problem and solving for its closed-form solution (as in Section 4.2).
A. P ROOF OF T HEOREM
Theorem 1.
Assume σ : V → V satisﬁes P ( w ) =
P ( σ ( w )) for all w .
If σ is a bijection, then for any input text x ∈ S ∪ Q and its perturbation ˜ x ∈ ˜ S ∪ ˜ Q , the outputs of the attention generator are the same:
[Equation]
Proof.
It sufﬁces to show that the general word importance s( · ) and the class-speciﬁc word importance t( · ) are the same under the perturbation φ .
For the former, since φ preserves unigram probability estimated over the source pool, we have
[Equation]
To prove the latter, we need to show that the conditional probability P ( y | x i ) estimated over the support set is also invariant under φ .
Let #( x i ∧ y | S ) denote the occurrences of the word x i in examples with class label y over the support set
S .
Using maximum likelihood estimation, we have
[Equation]
where the second equality is derived from the fact that σ is a bijection.
A. L EARNING PROCEDURE
Algorithm 1 contains the pseudo code for our learning procedure.
We apply early stopping when validation loss does not improve for 20 epochs.
A. D ATASETS
To reliably test our model’s ability to generalize across classes, we consider two data splitting mechanisms in our experiments: 1) easy split : we randomly permute all classes and split them into train/val/test; 2) hard split : we select train/val/test based on the class hierarchy such that train is the number of training classes.
is the number of classes of each few-shot task.
K, L are the number of support and query examples per target class, respectively.
To ease notation, we use S AMPLE ( S, N ) to denote a subset of S with
N elements, chosen uniformly at random.
We use D ( Y ) ⊆
D to denote the set of all elements ( x i , y i ) for which y i ∈ Y . Input (cid:46) classes D pool
T
(cid:46) do
T
Generate attention scores
D (cid:46) Eq
Construct
(cid:46) Eq Compute
S
Φ Q
Update met
Dataset # tokens / example vocab size # examples / cls # train cls # val cls # test cls 20 Newsgroups     7 RCV
[Table]
classes are distant to val and test.
We applied the easy split to one sentence-level dataset (HuffPost) and one document-level dataset (Reuters-21578).
Hard split is used for the other four datasets (details below).
This setting tests the generalization capacity of the algorithm, following Xian et al.
20 Newsgroups Each class in 20 Newsgroups belongs to one of six top-level categories, which roughly correspond to computers, recreation, science, politics, religion, and for-sale.
We partition the set of labels so that no top-level category spans two splits.
Train contains “sci” and “rec,” val contains “comp,” and test contains all other labels.
Amazon The Amazon dataset does not come with predeﬁned top-level categories.
To generate a hard split, we ﬁrst apply spectral clustering to classes based on their word distributions.
Then we select train/val/test from different clusters.
RCV1 We apply the same approach as above
FewRel While FewRel does not provide higher-level categories, we observe that most relations occur between named entities of similar types.
Thus, we extract the named entity type of the head and tail for each example using a pretrained spaCy parser.
For each class, we determine the most common head and tail entity types.
Test contains all classes for which the most common head entity type is W ORK OF A RT .
Train and validation were arbitrarily split to contain the remaining relations.
A. O THER BASELINES
We also consider two other baselines: induction network (Geng et al., 2019) and P-MAML (Zhang et al., 2019). Implementation details may be found in Appendix A.11. [Table]
[Figure]
[Table]
We run P-MAML on text classiﬁcation datasets with shorter documents, for which it is feasible to ﬁnetune BERT (after WordPiece tokenization, documents from ”longer” datasets exceed BERT’s max length of 512 tokens).
[Figure]
• Induction network encodes input examples using a biLSTM with self-attentive pooling (Lin et al., 2017). Based on the encoded representation, it then computes a prototype for each class through dynamic routing over the support set (Sabour et al., 2017).
Finally, it uses a neural tensor layer (Socher et al., 2013) to predict the relation between each query example and the class prototypes.
• P-MAML combines pre-training with MAML. It ﬁrst ﬁnetunes pre-trained BERT representation on the meta-training data using masked language modeling (Devlin et al., 2018).
Based on this ﬁnetuned representation, it trains ﬁrst-order MAML Finn et al.
(2017) to enable fast adaptation. Table 4 shows that learning with distributional signatures ( OUR ) signiﬁcantly outperforms the two baselines across all datasets. This is not surprising: both baselines build meta-knowledge on top of lexical representations, which may not generalize when the lexical distributions are vastly different between seen classes and unseen classes.
Figure 9 shows the learning curve of INDUCTION NET . Similar to Figure 6, we see that INDUCTION NET overﬁts to meta-train classes with poor generalization to meta-test classes.
Figure 10 depicts the perplexity of BERT’s masked language model objective, during ﬁnetuning for P - MAML .
Again, we see that the meta-train and meta-test classes exhibit vast lexical mismatch, as the perplexity improves only slightly on meta-test classes.
A. A NALYSIS OF LEARNED REPRESENTATION
To further understand the rationale behind our performance boost, we provide a more detailed analysis on our model’s empirical behavior.
Visualizing the embedding space We visualize our attention-weghted representation φ ( x ) in 20 Newsgroups (Figure 11) and HuffPost (Figure 12).
We observe that our model produces better separation than the unweighted average AVG and directly using the distributional statistics, s( x ) or t( x ) .
For instance, in 20 Newsgroups, our model recognizes three clusters: { talk.religion.misc, soc.religion.christian } , { talk.politics.mideast } and { talk.politics.misc, talk.politics.guns } , [Figure] [Figure]
Cosine similarity to oracle word importance We also quantitatively analyze the generated attention in (Figure 13).
To obtain a more reliable estimate of word importance [Figure] we train an oracle model over all examples from the N target classes.
This oracle model uses a biLSTM to encode each example from its word embeddings.
It then generates an attention score based on this encoding.
In order to estimate the importance of individual unigram, we use this attention score to weight the “original” word [Figure] embeddings (not the output of the biLSTM ).
A MLP with one hidden layer is used to perform
N -way classiﬁcation from the attention-weighted representation.
Compared to both s( · ) and t( · ) , the attention generated by our model is much closer to the oracle’s, which explains our model’s large performance gains.
Note that the attention generator does not see any examples from the target classes during meta-training.
Visualizing the generated attention Figure 14 visualizes the generated attention for a testing episode in HuffPost.
We observe that our model identiﬁes meaningful keywords from the sentence.
[-AVG s( · ) t( · ) OUR 0 .0 .0 .0 .0 .0 .0 .
-]
A. E FFECT OF SOFTMAX CALIBRATION
In Section 4.2, we use ridge regression with softmax calibration as our downstream predictor due to its efﬁciency and effectiveness.
To study the effect of the softmax calibration, we follow Bertinetto et al.
and compare ridge regression against direct optimization of binary logistic classiﬁers ( LR ) with Newton’s method (ﬁve iterations).
Since Newton’s method admits a closed-form solution at each iteration, the ﬁnal solution is also end-to-end differentiable.
For each N -way K -shot episode, to adapt the binary logistic classiﬁer for multi-class prediction, we train N binary (one-vsall) classiﬁers using the support set.
The output of the N binary classiﬁers are combined to make the ﬁnal predictions on the query set (Bishop, 2006, Chapter 4.1.2).
The results are shown in Table 5.
Overall, RR and LR produce similar results (78.0 vs. 77.1 on 5way 5-shot, 60.1 vs. 58.5 on 5-way 1-shot, respectively), though RR performs slightly better.
When using LR as the downstream predictor, learning with distributional signatures improves 5-way 1-shot accuracy by 6.1% and 5-way 5-shot accuracy by 3.2% over the best baseline.
{+Note that LR requires solving N binary classiﬁcations for each episode, which is less efﬁcient than RR . AVG s( · ) t( · ) OUR 0 .0 .0 .0 .0 .0 .0 .
 .20 Newsgroups a vg c o s i n e s i m il a r it y AVG s( · ) t( · ) OUR 0 .0 .0 .0 .0 .0 .0 .+}
HuffPost Headlines a vg c o s i n e s i m il a r it y
[Figure]
A. F INETUNING WORD EMBEDDINGS DURING META - TRAINING
In Table
[Figure]
we ﬁx word embeddings during meta-training for all experiments. Table 6 studies the effect of ﬁnetuning word embeddings during meta-training. We observe that when the word embeddings are ﬁnetuned, performance drops for nearly all models. To intuitively understand this behavior, we compare the meta-train vocabulary with the meta-test vocabulary.
On Amazon, 15095 of 28591 (52.8%) tokens in the meta-train vocabulary are not present in meta-test, and 21049 of 34545 (60.9%) tokens in the meta-test vocabulary are not present in metatrain. On Reuters, 3604 of 6372 (56.6%) meta-train tokens are not in meta-test, while 2481 of(47.2%) meta-test tokens are not in meta-test. Due to this lexical mismatch, ﬁnetuning will destroy the original geometry of the pretrained word embeddings, leading to poor generalization on unseen tasks. [Table]
[Table]
A. D ISTRIBUTIONAL SIGNATURES WITH OTHER CLASSIFIERS
In this paper, we demonstrate the advantage of learning meta-knowledge on top of distributional signatures.
We focus on a ridge regressor (Bertinetto et al., 2019) as our downstream classiﬁer due to its simplicity and effectiveness.
However, the idea of learning with distributional signatures is not limited to ridge regression; rather, we can combine distributional signatures with other learning methods such as prototypical networks (Snell et al., 2017) and induction networks (Geng et al., 2019).
Prototypical networks To augment a prototypical network with features learned from distributional signatures, we can construct per-class prototypes based on the attention-weighted representation φ ( x ) (Section 4.2).
From Table 7, we see that learning with distributional signatures improves 5way 1-shot accuracy by 9 . 9 % and 5-way 5-shot accuracy by 16 .
7 %, against the best prototypical network baseline for each dataset.
Induction networks We can also augment induction networks with meta-knowledge learned from distributional signatures.
Speciﬁcally, we directly feed our attention-weighted representation φ ( x ) (Section 4.2) to the induction module.
To avoid over-ﬁtting on meta-train features, we replace the meta-learned relation module (see Geng et al.
(2019) for details) by a parameter-free nearest neighbour predictor, similar to that of prototypical networks.
From Table 7, we again see that learning with distributional signatures improves 5-way 1-shot accuracy by 14 . 3 % and 5-way 5-shot accuracy by 25 . 1 % on average.
These empirical results clearly demonstrate the beneﬁt of our approach.
However, both prototypical networks and induction networks perform consistently worse than the ridge regressor across all [Table]
[Figure] experiments. Our hypothesis is that the ridge regressor enables task-speciﬁc ﬁnetuning based on the support set, while the prototypical network and induction network directly compute nearest neighbours from the meta-learned metric space.
A. F EW - SHOT LEARNING V . S . SUPERVISED LEARNING
To understand the extent of few-shot learning’s utility, we compare our results with that of a fullysupervised CNN classiﬁer, as we increase the number of training points. From Figure 15, we see that our model’s 5-shot accuracy is approximately equivalent to a fully-supervised classiﬁer trained on 50 examples. A. I MPLEMENTATION D ETAILS
We detail the implementations of our baselines here.
All baseline code are available in our repository.
CNN For 1D convolution, we use ﬁlter windows of 3, 4, 5 with 50 feature maps each.
We applied
ReLU after max-over-time pooling.
Prototypical Network Prototypical network meta-learns a multi-layer perceptron to transform the input representation into an embedding space that is suitable for few-shot classiﬁcation.
If the input representation is learnable (e.g., CNN ), the parameters for the input representation are also updated using meta-training.
In the experiments, we use a MLP with one hidden layer and ReLU activation.
The dimensions of both the hidden layer and the output layer are 300.
We apply dropout with rate 0.1 to the hidden layer.
MAML MAML meta-learns an initialization such that the model can quickly adapt to new tasks after a few gradient steps.
For prediction on the input representation, we use a MLP with one hidden layer of 300 ReLU units.
We apply dropout with rate 0.1 to the hidden layer.
During the MAML inner loop (adaptation stage), we perform ten updates with step size 0.1 (we empirically found this outperform one-step MAML).
We backpropagate higher order gradients thoughout meta-training.
During the MAML outerloop, we average the gradient across ten sampled tasks and use Adam with learning rate 10 3 to update the parameters.
Finetune Chen et al.
(2019) recently showed that ﬁne-tuning a properly pre-trained classiﬁer can achieve competitive performance when compared with the state-of-the-art meta-learning.
Following their work, we explicitly reduce the intra-class variation during the pre-training stage.
Similar to
MAML, we use a MLP with one hidden layer (300 ReLU units) to make predictions from the input representation (e.g., CNN ).
During ﬁnetuning stage, we re-train the MLP from scratch and ﬁne-tune the learnable parameters of the input representation.
We stop ﬁne-tuning once the gradient norm is less than 10 − 3 .
Induction network Induction network consists of three modules: encoder module, induction module and relation module.
The encoder module uses a biLSTM with self-attention to obtain a ﬁxlength representation of each input example.
The induction module performs dynamic routing to compute the class-speciﬁc prototype.
The relation module uses a neural tensor layer to predict the relation association between each query example and the prototypes.
Following Geng et al.
(2019), we set the hidden state size of biLSTM to 256 (128 for each direction) and the attention dimension to 64.
The iteration number in dynamic routing is set to 3 .
The dimension of the neural tensor layer is set to 100 .
P-MAML P-MAML (Zhang et al., 2019) has two phases: masked language model pretraining (Devlin et al., 2018) and MAML (Finn et al., 2017).
For pretraining, we used Hugging Face’s language model ﬁnetuning code with default hyperparameters and BERT’s pretrained base-uncased model.
We applied early stopping when the validation perplexity failed to decrease for 2 epochs.
After pretraining, we add a softmax layer on top of the representation of the [CLS] token (Devlin et al., 2018) to make predictions.
During the MAML inner loop (adaptation stage), we perform ten updates with step size 10 − 3 .
Following Zhang et al.
(2019), we do not consider higher order gradients.
During the MAML outerloop, we average the gradient across ten sampled tasks and use Adam with learning rate 10 − 5 to update the parameters.
A. R ESULTS
This section contains experimental results with standard deviations.
[Table] [Table]
talk.politics.mideast sci.space misc.forsale talk.politics.misc comp.graphics israel space sale president image armenian nasa cramer graphics turkish launch shipping mr jpeg armenians orbit offer stephanopoulos images israeli shuttle condition people gif jews moon 1st government format armenia henry price optilink ﬁle arab earth forsale myers 3d people mission asking clayton ftp jewish solar comics gay color sci.crypt comp.windows.x comp.os.ms-windows.misc talk.politics.guns talk.religion.misc key window ax gun god clipper xx max guns jesus encryption motif windows fbi sandvik chip server g9v ﬁrearms christian keys widget b8f atf bible security ﬁle a batf jehovah privacy xterm weapons christ government x pl people lord escrow entry 1d waco kent des dos 34u cdt brian rec.autos sci.med comp.sys.mac.hardware sci.electronics rec.sport.hockey car medical mac circuit hockey cars disease apple wire game engine msg centris ground team ford cancer quadra wiring nhl oil health lc voltage play dealer patients monitor battery season callison doctor duo copy games mustang hiv nubus amp 25 com food drive electronics ca autos diet simms audio pit alt.atheism rec.motorcycles
comp.sys.ibm.pc.hardware rec.sport.baseball soc.religion.christian
god bike scsi baseball god atheism dod drive game church atheists ride ide year jesus keith bmw controller team christ livesey com card players sin morality riding bus games
[Table]