A TTENTIONAL META - LEARNERS FOR FEW - SHOT POLYTHETIC CLASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Polythetic classiﬁcations, based on shared patterns of features that need neither be universal nor constant among members of a class, are common in the natural world and greatly outnumber monothetic classiﬁcations over a set of features.
We show that threshold meta-learners, such as Prototypical Networks, require an embedding dimension that is exponential in the number of task-relevant features to emulate these functions.
In contrast, attentional classiﬁers, such as Matching Networks, are polythetic by default and able to solve these problems with a linear embedding dimension.
However, we ﬁnd that in the presence of task-irrelevant features, inherent to meta-learning problems, attentional models are susceptible to misclassiﬁcation.
To address this challenge, we propose a selfattention feature-selection mechanism that adaptively dilutes non-discriminative features.
We demonstrate the effectiveness of our approach in meta-learning Boolean functions, and synthetic and real-world few-shot learning tasks.
I NTRODUCTION
Classiﬁcation meta-learning is typically approached from the perspective of few-shot learning: Can we train a model that generalises to unseen ‘natural’ classes at test time?
For example, in the Omniglot task (Lake et al., 2011) we have access to a labelled set of handwritten characters during training and we are tasked with distinguishing new characters, from unseen writing systems, at test time.
From this perspective each example is associated with a consistent class and members of that class share a common set of properties (e.g. all handwritten characters have the shape of the underlying character class).
Alternatively, we may consider meta-learning over unseen ways of categorising : Can we train a model on character recognition that generalises to alphabet recognition?
or to distinguishing upper from lower case letters?
In this setting, features need to be understood in relation to a given classiﬁcation: For instance, when tasked with distinguishing equids (horses, zebras, donkeys) from big cats, the presence of stripes on both zebra and tigers is irrelevant, and potentially misleading.
On the other hand, stripes are the key to distinguishing horses from zebra.
Understanding features in the context of a classiﬁcation is central to the concepts of monothetic and polythetic classes recognised in the ﬁelds of taxonomy and knowledge organisation.
Monothetic classiﬁcations are based on universal attributes: there is at least one necessary and sufﬁcient attribute for class membership.
Polythetic classiﬁcations are instead based on combinations of attributes, none of which are sufﬁcient in isolation to indicate membership and, potentially, none of which are necessary.
Carl Linneaus, inventor of the binomial nomenclature for species and “father of taxonomy,” recognised that natural orders could not be deﬁned monothetically, lacking features that were unique and constant over families and that, until such features could be found, such classiﬁcations were necessarily polythetic (Hjørland, 2017; Stevens, 1998).
Figure 1 illustrates Linnaeus’ system for classifying plants, which relies on polythetic classiﬁcations and is still in use today.
Consider, for example, that we can distinguish A from B by the number of ﬁlaments, but not
B from C ; we can distinguish B from C based on whether there are split anthers (the ends), but not C from A , and so on.
To recognise classes it is necessary to consider their attributes in the context of other attributes.
Threshold functions are a related concept in Boolean algebra.
A threshold function evaluates positively if a weighted sum of binary inputs crosses some threshold.
Lacking a preferred feature basis, we can identify monothetic classiﬁcations with threshold functions, as threshold functions that are polythetic in one basis, such as logical OR ( x, y ) , may be recast in a basis where they are monothetic e.g. binary OR ( x, y ) evaluates as the unary MIN ( x + y, 1) .
We can compare the frequency of monothetic and polythetic classiﬁcations in this general case.
The number of binary inputs of length n is 2 n and the total number of Boolean functions is the number of binary labellings of these inputs, 2 2 n , whereas the number of threshold functions grows only singly exponentially ( ≤ 2 n 2 ), and therefore monothetic classiﬁcations represent a vanishingly small proportion of the total (see Gruzling (2007); Irmatov (1993)).
This work explores meta-learning for polythetic classiﬁcation.
Speciﬁcally, we
• consider the limitations of widely used threshold classiﬁers, such as Prototypical Networks (Snell et al., 2017)
and how they are able to learn and approximate non-threshold functions in practice; • show that simple alternatives based on attention, such as Matching Networks (Vinyals et al., 2016), are polythetic by default but susceptible to misclassiﬁcation due to excessive sensitivity; • characterise the challenge of spurious correlations in irrelevant features for attentional classiﬁers; • and propose
a simple solution to this challenge that is non-parametric and based on self-attention.
Throughout, we evidence these ﬁndings and the effectiveness of our proposals with experiments on synthetic and real-world few-shot learning tasks.
B ACKGROUND
Problem formulation.
We are interested in few-shot classiﬁcation: provided with a small number of labelled points S
= { x i , y i } i ∈ I S , the support set, with feature vectors x i ∈ R n and labels y i ∈ { 1 , . . .
, K } , we want to predict the labels of the query set Q = { x j } j ∈ I Q .
S k denotes the set of support elements with label k and I (cid:63) is the index set of the subscript e.g. I S is the index set of the support.
The label space is arbitrary and potentially unique to a task (also referred to as an episode) — both the number of classes, k , and assigned labels may vary over tasks — and, importantly, examples that share labels under one categorisation will not necessarily share labels under another.
We will refer to classiﬁcation functions over n features simply as classiﬁcations, and in meta-learning we are often interested in classiﬁcations that only depend on some features, α , and not the remainder, β = n − α .
Classiﬁers.
Deep neural models for problems of this kind are usually equipped with either a threshold classiﬁer or an attentional classiﬁer.
Threshold classiﬁers are based, as the name suggests, on using thresholds to partition a space into regions associated with each class.
Prototypical Networks (Snell et al., 2017) are an example of such a model.
This model embeds examples using a learned neural function, f φ , ﬁnds the average embedding for each class in the support, c k , and classiﬁes queries based on their proximity (measured with a
Prototype, x = ++ + − − + −−
Attention, x = ++ + − − + −−
Prototype, y = − x +2 /++
+ − − + −−
Attention, y = − ln(tanh( x )) /++
+ − − + −−
Prototype,undeﬁned ++ + − − + −−
Attention, y =0, x = ++ + − − + −− f 1 ( x,y ) =
x f 2 ( x,y ) =
AND( x,y ) f 3 ( x,y )
=
XOR( x,y ) distance function d ) to these class prototypes:
[Equation]
The key advantage of such an approach is that salient class features are preserved when forming the prototype while irrelevant aspects of particular examples are washed out.
Attentional classiﬁers instead use a similarity function to directly compare queries with each example in the support.
For example, Matching Networks (Vinyals et al., 2016) also learn embedding functions, but each query is compared with every member of the support to weight a sum over their labels, which we can write simply as ˆ y = (cid:80) i ∈ I S a (ˆ x, x i )
y i .
In the popular terminology of transformers we may write the embedded queries as Q , the embedded support as keys
K and their labels as values V , and dot-product attention classiﬁcation with temperature τ − 1 as
[Equation]
Attentional classiﬁers are more sensitive to variations within a class at the cost of additional computation.
Boolean tasks.
In comparing these classiﬁers we make repeated use of tasks based on Boolean functions, and on exclusive-OR (XOR) in particular.
For a binary feature vector x ∈ {− 1 , 1 } n , the number χ A ( x ) = (cid:81) i ∈
A x i is the parity function or exclusive-or (XOR) over the bits ( x i ) i ∈
A .
We write a parity function of α bits XOR α .
The set of parity functions over n bits form a linearly independent basis (O’Donnell, 2014) and, as such, being able to model the partition functions guarantees that one can model any other Boolean function over that domain.
Put another way, the decision boundaries of XOR α are at least as complicated as those for any other α -variable Boolean function: there is one between every possible pair of feature vectors.
For these reasons, XOR is our polythetic function of choice in derivations, examples, and experiments.
C HALLENGES IN META - LEARNING POLYTHETIC CLASSIFICATIONS
Threshold and attentional classiﬁers have their own strengths and, in meta-learning polythetic classiﬁcations, their own weaknesses.
Threshold classiﬁers are insufﬁciently ﬂexible and attentional classiﬁers are prone to misclassiﬁcation.
Threshold classiﬁers.
Figure 2 shows the decision boundaries formed by a prototypical threshold classiﬁer and an attentional classiﬁer for a selection of 2-variable Boolean functions, highlighting the problem with using threshold classiﬁers for polythetic classiﬁcation: logical XOR ( x, y ) is not a threshold function and the prototypes fail to produce a useful decision boundary.
This is the perceptron problem identiﬁed by Minsky &
[Figure]
Papert (1969).
However, deep networks using threshold classiﬁers can learn XOR.
This is possible because the network can learn additional pseudo-features for the non-threshold functions it observes.
Figure 3 shows an example for 2-variables: one can embed the corners of the square at the corners of a tetrahedron with coordinates (cid:0) x, y, XOR ( x, y ) (cid:1) and produce linear thresholds solutions for every 2-variable Boolean function.
There are two problems with this approach, both of which are exacerbated in the meta-learning context: i) the required number of pseudo-variables grows as (cid:0) nα (cid:1) ∼ O
( n α ) to account for all combinations of α active components, and ii) the method does not generalise to unseen non-threshold functions (see Appendix A).
The right plot in Figure 3 demonstrates these shortcomings for XOR 2 : the required number of pseudo-variables is (cid:0) n 2 (cid:1) ∼ O ( n 2 ) and initially we ﬁnd that a quadratic embedding is able to maintain performance, but for longer sequences the number of threshold-functions unseen in training grows and performance degrades.
Attentional classiﬁers.
Attentional classiﬁers avoid these problems — the required embedding dimension is linear in the number of features and the classiﬁer generalises to unseen classiﬁcations — but suffer from over-sensitivity to irrelevant features.
This results in misclassiﬁcation, which we quantify in the case of
Boolean functions to understand the scaling properties of this problem generally.
Consider classiﬁcations over binary feature vectors x ∈ {− 1 , 1 } n , with the class determined by XOR α over α elements with the remaining β = n − α being irrelevant.
Assume each of the 2 α variations of the active elements are present with equal frequency, r , for a support set S of size | S | = r 2 α , and that the remaining β elements follow a Bernoulli distribution with probability p .
Using an attention classiﬁer of the form ˆ y = (cid:80) i ∈ S softmax i (cid:0) a (ˆ x, x i ) (cid:1)
y i , where a (ˆ x, x i ) is a measure of the similarity of ˆ x and x i , how likely is it that we misclassify a query drawn from the same distribution as S ?
Without loss of generality, we can focus on the positive examples (with label = 1 ) for which a positive output gives the correct classiﬁcation.
Using dot-product attention, the mean and variance of the classiﬁer output are, with ¯ p =
p 2 + (1 − p ) 2 and ¯ q = 1 − ¯ p ,
[Equation]
introducing c and d for compactness.
The mean is positive, as desired, but we are interested in the rate of misclassiﬁcation, which may be interpreted using the scale-free and dimensionless coefﬁcient-of-variation (the ratio of the standard deviation to the mean) where greater variation indicates a greater rate of misclassiﬁcation.
From Equations 3 and 4, we have
[Equation]
Starting with the leftmost term, increasing the number of repetitions, r , reduces the relative variability.
This aligns with intuition and limits, where an empty support set, r = 0 , provides no basis on which to make predictions and at the other extreme, r (cid:29) 2 β , the support set is likely to span the input domain reducing classiﬁcation to look-up.
The term raised to α is approximately 3 . 2 , and so the variability increases with the number of active elements.
An intuitive explanation is that the number of immediate neighbours of each point grows as (cid:0) nα (cid:1) =
α and this reduces the conﬁdence with which the point is classiﬁed, so the barrier to misclassiﬁcation is reduced.
Finally, d > c 2 and so the rightmost term is positive and grows exponentially in β , meaning that misclassiﬁcation increases with the number of irrelevant features.
A full derivation, including alternative attention functions, is provided in Appendix C.
The problem of misclassiﬁcation due to over-sensitivity in attentional classiﬁers was recognised in the work of Luong et al.
(2015) on sequence processing.
There the problem was addressed by attending only to a subset of elements within some distance of the target position.
However, sets do not have such an ordering and so we instead propose a feature-selection method to resolve the problem more generally.
A TTENTIONAL FEATURE SELECTION
A key challenge of meta-learning is that not all features are relevant in all tasks and that the support is unlikely to span the input domain.
The model must choose, using incomplete information, what to focus on and what to ignore by detecting the salient features within and between classes.
For monothetic classiﬁcations this is straightforward: by deﬁnition, averaging highlights necessary features whilst diminishing irrelevant features.
Prototype methods rely on this process.
In the polythetic case, attentional classiﬁers have the advantage of being able to learn non-threshold functions without the need for pseudo-variables but do not beneﬁt, as prototypes do, from ‘washing-out’ irrelevant features through averaging.
Indeed, attentional classiﬁers are susceptible even in the monothetic setting to misclassifying on the basis of closely matching features that [Figure] are not relevant to the problem (putting a zebra with the big-cats because its stripes match those of a tiger in the support set, for example).
Misclassiﬁcation occurs when irrelevant features overwhelm the signal from the active elements.
As this is a problem of highlighting the salient patterns within a set, we propose a self-attention based mechanism for feature selection, presented in Algorithm 1 and illustrated in Figure 4.
Intuitively, the process exploits the over-representation of patterns within features that are relevant to the classiﬁcation as compared to patterns within the irrelevant features.
We ﬁrst standardise the features to prevent those common to the entire support, which are not discriminating, from dominating (Line 1) and stabilise, (cid:15) , to prevent weakly activated features from being excessively scaled-up.
We then repeatedly self-attend within each separate class k of the support set, using dot product attention with scale τ , X k ← DotAttn ( X k , X k , X k , τ ) .
Self-attention maps elements of a set of vectors to the interior of their convex hull.
If every member of a class has some feature in common, the convex hull in that dimension is a point and the features do not change.
In the polythetic case it is patterns of features that matter, and by attending more strongly between elements of the support set with such feature-patterns, these too are preserved.
Figure 4, for example, shows polythetic variations within a class of XOR 3 with three active and three irrelevant features ( α = β = 3 ).
The patterns in the active features self-reinforce, forming cliques of strongly connected elements, whilst the irrelevant features decay.
Finally, features are scored by their dispersion over the support set (Line 5) which indicates how well they have been preserved through the self-attention iterations, and thus how relevant they are.
[Table]
// standardise repeat R times for k ← 1 to K do // for each class
X k ← softmax ( τ X k X Tk )
X k ; // softmax is row-wise 5 f ← dispersion( { x i } i ∈ S i ) ; // mean-absolute-deviation, std.
dev etc.
The scores can be used directly to rescale features across the support and query sets before applying the classiﬁer, as in Figure 5, or in top-k selection.
We focus on rescaling as the method that makes the fewest assumptions about the underlying classiﬁcation, but using top-k is highly effective when the number of active elements is known, as shown in Appendix D.
E XPERIMENTS
We compare the proposed method (FS) with Prototypical Networks (PN)
(Snell et al., 2017), a threshold classiﬁer, and Matching Networks (MN)
(Vinyals et al., 2016), an attentional classiﬁer without featureselection, in a sequence of increasingly complex synthetic and real-world few-shot learning problems.
As our approach is non-parametric and operates directly on high-level features, it is agnostic to the choice of feature extractor, and we are free to choose as appropriate e.g. a convolutional neural network for images or a multi-layer perceptron for tabular data, and in all experiments we use the same embedding model for all methods (see experimental details in Appendix F).
Binary strings.
We consider meta-learning Boolean functions of n = α + β variables.
Labels for inputs x ∈ {− 1 , 1 } n are generated by computing the XOR of a random subset of components of size α .
Each variation in the active elements occurs 5 times in the support, [Equation] [Table] [Figure]
∆accuracy 0 .0 .0 .0 .0 .1 .0 .0 .0 .0 .0 .1 .0 .0 .0 .0 .0 . [Table] is unknown to the meta-learner.
Appendix I shows a concrete example of an XOR 2 task.
Table 1 summarises the performances of Prototypical Networks and our approach.
PN accuracy decreases sharply with sequence length n and the number of embedding units required to effectively solve this problem grows rapidly with n , as shown previously in Figure 3.
This suggests that PN are indeed learning pseudo-variables and demonstrates the limitations of threshold classiﬁers in solving polythetic problems.
Polythetic MNIST.
We evaluate the ability of the models to jointly extract high-level features and identify polythetic patterns.
We build tasks (episodes) using MNIST digits (LeCun et al., 2010), where an example consists of 4 coloured digits (RGB).
An example task and further details are provided in Appendix J. For monothetic tasks, a single high-level feature (e.g. colour of the top-right digit) distinguishes classes.
For polythetic tasks, class membership derives from XOR interactions over a subset of features, and the remainder are problem-irrelevant.
Table 2 shows the performances on three versions of the polythetic MNIST dataset: clean (excluding non-discriminative digits), colourless (task-irrelevant digits but no colour), and full (both taskirrelevant digits and colour).
The models are trained on monothetic tasks and evaluated both on monothetic and polythetic tasks.
Protonets excel at identifying monothetic features and ignoring non-discriminative features, but have a close to random performance on polythetic tasks.
Conversely, matching networks, which are polythetic classiﬁers by default, are highly sensitive to task-irrelevant features.
The proposed approach (FS) can simultaneously detect salient features and perform polythetic classiﬁcations.
Furthermore, as shown in Figure 6, we found our classiﬁer to be robust to the rate of polythetic tasks seen during training in a second experiment.
Omniglot.
The Omniglot dataset (Lake et al., 2011) consists of handwritten characters from 50 writing systems with 20 hand drawn examples of each character.
Training tasks are formed using examples fromof the alphabets and test tasks draw from the other 20.
We compare our method to PN, MN, Inﬁnite Mixture [Table] [Figure]
[Table] Prototypes (IMP) (Allen et al., 2019), and MAML (Finn et al., 2017) with a threshold classiﬁer (Triantaﬁllou et al., 2020).
We train the models for character recognition, and additionally evaluate performance on 3-way alphabet recognition (inherently polythetic).
Our approach can be used in conjunction with most few-shot learning approaches – we speciﬁcally apply FS prior to MN and single-nearest-neighbours (NN), corresponding to MN with softmax converging to argmax (see Appendix B).
Table 3 shows the results of this experiment.
The end-to-end trained model (FS+MN) is competitive with PN and IMP, while performing better than MN and MAML in character recognition.
In alphabet recognition, FS+MN performs better than other methods, while being competitive with IMP.
We further evaluate the performance when using a pre-trained feature extractor (methods marked with *), obtained by training a PN threshold classiﬁer on character recognition.
FS+MN* improves over MN* in both tasks.
Compared to PN, the accuracy of FS+MN* and FS+NN* in character recognition is reduced by at most 0 . 5 pp; yet improved in alphabet recognition by 12 . 6 pp, while performing similarly to the more complex IMP.
TieredImageNet.
TieredImageNet (Ren et al., 2018) is a subset of ILSVRC-12 (Russakovsky et al., 2015) with polythetic characteristics, with classes grouped into categories corresponding to higher-level nodes in the ImageNet hierarchy.
There are 34 categories of 10 to 30 classes each.
We compare our method (FS) to over 500 tasks.
[Table] [Table] PN and MN classiﬁers.
We use a publicly available pre-trained ResNet-12 (Zhang et al., 2020), pre-trained using the training classes in TieredImageNet, as the feature extractor for all models.
Table 4 presents the aggregate accuracy while Table 5 shows the head-to-head results of this experiment. FS leads to signiﬁcant improvements in performance (except C=8/G=5, where the difference between PN and FS is not signiﬁcant) in this full scale, naturally polythetic problem, particularly in the “more polythetic” case with 10 subgroups.
R ELATED WORK
We characterise meta-learning approaches for few-shot classiﬁcation. In addition to evaluating their ability to generalise to unseen classes, we investigate how well they can adapt to polythetic tasks (generalisation to unseen ways of cateogorising). Adaptability in few-shot settings has been studied through different paradigms such as fast weights (Ba et al., 2016), learnable plasticity (Miconi et al., 2018) and meta-learning. Recent work on meta-learning for few-shot classiﬁcation includes approaches that are able to quickly adapt through various mechanisms such as recurrent architectures (Mishra et al., 2018;
Santoro et al., 2016) for learning parameter updates (Ravi & Larochelle, 2017). Other more general optimisation-based approaches (Finn et al., 2017; Nichol et al., 2018; Rusu et al., 2019), tackle these tasks by explicitly optimising the model’s parameters. These, however, are typically model-agnostic and commonly used in conjunction with threshold classiﬁers (Triantaﬁllou et al., 2020), inheriting their limitations in a polythetic scenarios. Our work aligns more closely with metric-learning approaches for few-shot classiﬁcation (Chen et al., 2019) that apply distance functions between queries and the support in a common embedding space (Allen et al., 2019; Oreshkin et al., 2018;
Snell et al., 2017; Sung et al., 2018; Vinyals et al., 2016; Zhang et al., 2020).
Methods that construct class-wise prototypes from the support (Allen et al., 2019; Ren et al., 2018;
Snell et al., 2017) can successfully tackle monothetic tasks, but can struggle with task-adaptiveness in a polythetic context.
Attentional meta-classiﬁers (Hou et al., 2019; Jiang et al., 2020; Kim et al., 2019; Vinyals et al., 2016) adapt to polythetic tasks but lack crucial mechanisms for focusing exclusively on relevant features.
Attending over datapoints has been considered previously.
Luong et al. (2015) introduced dot-product attention in the context of attending over sequences.
Vinyals et al.
(2016) considered ˆ y = (cid:80) i ∈ I S a (ˆ x, x i )
y i and provided the conditions under which such a model carries out kernel density estimation or k -nearestneighbours classiﬁcation.
Vaswani et al.
(2017) used explicit scaling in hidden attention layers, but scaling a classifying softmax by a temperature parameter dates to the work of Boltzmann and later Gibbs (1902).
Plötz & Roth (2018) note that their neural-nearest-neighbours (N 3 ) block recovers a soft-attention weighting when the number of neighbours is set to 1, and deploy their model on an outlier-detection set-reasoning task.
C ONCLUSION
In this work we have articulated the difference between monothetic and polythetic classiﬁcations and considered the limitations of standard meta-learning classiﬁers in the polythetic case.
We have shown that threshold classiﬁers require an embedding space that is exponential in the number of active features and that attentional classiﬁers are overly sensitive and susceptible to misclassiﬁcation.
To address this, we have proposed an attention based method for feature-selection and demonstrated the effectiveness of our approach in several synthetic and real-world few-shot learning problems.
Our approach is simple and can be used in conjunction with most few-shot meta-learners.
We expect polythetic meta-learners to ﬁnd real-world application in domains where data is typically scarce and complex, such as healthcare or bioinformatics.
For example, we envision a use in classifying rare diseases — there are around 7000 rare diseases, affecting ∼ 1/ of the worldwide population — from DNA sequences, where mutations often lead to different phenotypes.
In such scenarios, few-shot learning approaches able to generalise over unseen combinations of mutations (i.e. ways of categorising), in a similar vein to our binary strings experiment, may lead to better performance in diagnosing rare diseases and shed new insights into their molecular mechanisms.
R EPRODUCIBILITY STATEMENT
In Section 3 we discuss the challenges of polythetic classiﬁcation and the limitations of current work on meta-classiﬁers.
Our proposed self-attention feature scoring algorithm is described in detail in Section and in particular, Algorithm 1 (and Appendix H).
To ensure the work is readily reproducible, besides descriptions of the experimental setup provided in Section 5 and the supplementary material (Appendices F, G, I, J) - we also provide code used for producing the results in the paper
A P ARTITIONED XOR PERFORMANCES
We hypothesise that, in order to solve polythetic tasks, prototypical networks need to create pseudo-variables in the embedding space (e.g. XOR for each pair of components for binary strings tasks).
To test this, we train a prototypical network on binary strings tasks (where pairs of active components are always chosen from the 5 ﬁrst components) and evaluate the performance on unseen combinations of components (i.e. combinations of the 5 ﬁrst components not seen during training).
Table 6 shows the accuracies for sequences of length n = 5 + β (where β is the number of variables that are always inactive).
We attribute the better-than-chance performances (i.e. for 100-dimensional embeddings) to the high-dimensionality of the embeddings – by chance, large numbers of non-linear features (e.g. output by a random feature extractor) will include some features that are useful for the task of interest (i.e. similar to extreme learning machines).
Overall, these results highlight the inability of prototypical networks to generalise to unseen combinations and support our hypothesis.
the same expected frequency of and times The inability to generalise in this scenario that protonets need pseudo variables (i e. XOR functions applied to each pair of components) to solve the binary strings task.
[Table]
B T EMPERATURE IN ATTENTION CLASSIFICATION
The softmax in attention mechanisms permits a temperature scaling that interpolates between argmax and uniform (and argmin.)
This is controlled by T = 1 β , as
[Equation]
with softmax converging to argmax as β → ∞ , and to uniform (i.e. all elements equal to the reciprocal of the length of the vector) as β → 0 .
For attention classiﬁers, a decrease in temperature increases the model conﬁdence and can cause decision boundaries to move.
As β → ∞ and the softmax converges to argmax, the classiﬁer tends to the single nearest neighbour classiﬁcation scheme; for β = 0 the classiﬁer returns the support set class balance.
For
Boolean functions, changes in temperature effect the degree to which decision boundaries are axis-aligned.
For example, Figure 2 shows the decision boundary for f ( x, y ) =
AND ( x, y ) using a softmax temperature of
T =++ + − − + −−
T = 1 /++ + − − + −− T = 1 /++ + − − + −−
T = 1 /++ + − − + −−
1 at y = − 12 ln (tanh x ) , which we derive as
[Equation]
The effect of decreasing the temperature on the decision boundary is shown in Figure 7.
C M ISCLASSIFICATION DISTRIBUTION FULL DERIVATION
We ﬁrst present a more detailed derivation for the case of dot-product attention.
We will make repeated use of the Binomial theorem
[Equation]
Recall that we consider classiﬁcations over binary feature vectors x ∈ {− 1 , 1 } n , with the class determined by XOR α over α elements with the remaining β = n − α being irrelevant.
Assume each of the 2 α variations of the active elements are present with equal frequency, r , for a support set S of size | S | = r 2 α , and that the remaining β elements follow a Bernoulli distribution with probability p .
There are (cid:0) αδ (cid:1) strings over the active elements that differ from a given example in δ positions.
XOR ﬂips the classiﬁcation with each difference, so if δ is even the class is the same, if δ is odd the class is different (parity).
The class is determined by the sign of the sum of contributions at an even distance subtract those at an odd distance.
Putting those into a single sum we have
[Equation]
We can factor out the number of repetitions, r , and as it is positive it doesn’t change the sign.
We can then rearrange to match the Binomial theorem form and recover the form in the main text:
[Equation]
Next we give the Binomial distribution of the irrelevant feature contribution as 2 B ( β, ¯ p ) − β with ¯ p = p 2 − (1 − p ) 2 = 2 p 2 − 2 p + 1 .
Something that is important to the behaviour of the mean and variance later, and breaks the usual symmetry of p and q = (1 − p ) , is that ¯ p ≥ 0 .
5 and is quadratic in p deﬁned by the three points ( p, ¯ p ) =
¯ q has the usual deﬁnition 1 − ¯ p , so ¯ q ≤ 0 .
5 and so on.
The contribution at a difference δ is then X ( δ ) ∼ exp ( α − 2 δ
+ 2 B ( β, ¯ p ) − β ) .
The expectation is computed in the usual way
[Equation]
Finding the variance in the traditional way
[Equation]
Var [ X ( δ )]
=
E [ X ( δ ) 2 ] − E [ X ( δ )] 2 , ﬁrst E [ X ( δ ) 2 ] following the derivation for E [ X ( δ )] :
[Equation]
From this we can write the variance
[Equation]
Next we want to ﬁnd the expectation of the sum of the contributions at each difference δ .
As pointed out there are (cid:0) αδ (cid:1) many strings at a difference of δ .
We apply E [ (cid:80) i X i ]
= (cid:80) i E
[ X i ] and Var [ X − Y ] =
Var [ X ]+ Var [ Y ] , and, remembering to change the sign with each increase in δ ,
[Equation]
and
[Equation]
C. D IFFERENT ATTENTION MECHANISMS
For dot-product and cosine-similarity attention, ‘angular’ difference mechanisms, we use (+ , − ) to encode the input variables.
This is because we want the score to be greater when the variables are the same and lesser when they are opposed (if we were to use (1 , 0) we’d have 0 × 0 (cid:54) = 1 × 1 and 0 × 1 = 0 × 0 ).
With these methods we get
[Equation]
[Equation]
The change for cosine-similarity introduces a factor of exp (1 /α ) to the mean and exp (2 /α ) to the variance, and the overall picture doesn’t change
[Equation]
For squared Euclidean distance attention (which coincides with the ‘Laplace attention’ L1 norm in this encoding), if we encode the inputs as (1 , 0) we get f L 2 ( δ ) =
− ( √ δ ) 2 = − δ , introducing a factor of exp ( − α ) to the mean and exp ( − 2 α ) to the variance, which also doesn’t change the overall picture.
D T OP - K FEATURE SELECTION
Top-k feature selection masks out all but the k highest scoring features.
That is x ← m (cid:12) x with m
[Equation]
(cid:40)
[Equation]
if
[Equation]
0 otherwise (23)
As compared to soft feature selection which rescales as x ← s (cid:12) x , or x ← s (cid:48) (cid:12) x with normalised s (cid:48) .
[Table]
Directly on inputs
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
I n a c t i v e c
o m p o n e n t s , β
[Figure] [Figure]
Variant examples, r
[Figure] [Figure]
E C ONVERGENCE OF FEATURE VECTORS ACROSS SELF - ATTENTION ITERATIONS
Figure 9 illustrates how feature vectors converge on the XOR 2 task across self-attention iterations of the feature selection mechanism.
r =[0 .
r =[0 .
r =[0 .
49] r =[0 .
r =[1 . 06 , 0 . 97 , 0 . 23] quickly align by and variation is ‘washed-
’ also seen selection scores
F E XPERIMENTAL DETAILS
We use the same feature extractor architecture and train loop for all the baselines.
Feature extractor.
We leverage a convolutional neural network with 4 blocks as a feature extractor.
Each block consists of a convolutional layer (64 output channels and 3 × 3 ﬁlters), followed by batch normalisation (momentum 0.01), a ReLU activation, and 2 × 2 max pooling:
[Equation]
Then, we ﬂatten the output and apply a linear layer to map the data into a 64-dimensional embedding space (unless otherwise stated).
As explained in the main manuscript, each method then manipulates these embeddings in different way.
Train loop.
We train all models in an episodic manner.
At each training iteration, we follow these steps:
• First, we sample task-speciﬁc support and query sets.
For polythetic MNIST, the support set has 96 samples (2 classes, 2 groups per class, and 24 group-speciﬁc examples per group).
The query set consists of 32 samples (2 classes, 2 groups per class, and 8 group-speciﬁc examples per group).
For Omniglot, the support set consists of 5 examples for 20 classes (20-way, 5 shot).
The query set consists of 15 examples per class.
Second, we compute embeddings using the feature extractor and produce class probabilities for the query points.
The way in which these probabilities are computed depends on the method (e
g. attentional classiﬁcation for matching networks or softmax over prototype distances for prototypical networks).
• Finally, we compute the cross entropy for the query examples and optimise the feature extractor via gradient descent.
We employ an Adam optimise with learning rate 0 . 001 .
We train the models for 10,000 iterations (i.e. tasks) for all experiments, except for full polythetic MNIST (100,000 tasks).
We then compute the performances on a held-out dataset and average the results across 1, tasks.
G
M ULTI - CATEGORICAL PRE - TRAINING
In this experiment we ﬁrst train a classiﬁer in the multi-categorical setting for the full polythetic MNIST task.
In this case there are no task-irrelevant digits or colours as the label describes all four digits with their colours: there are four 10-way labels for the digits ( ∈ R 4 × 10 ) and four 3-way labels for the colours ( ∈ R 4 × 3 ) for a combined multi-hot output vector ∈ R 52 .
The model architectures match that used in the other experiments, see Appendix F, other than using a variation of the MLP head that takes the ﬂattened output from the convolutional network.
The variation has two hidden layers with 512 units each and ReLU activations, before linear layers with softmaxes for each of the label heads.
The model is pre-trained overbatches of 16 examples drawn at random from the label combinations.
In the multi-categorical pre-training, the model achieved a validation accuracy of 95 .
6% on the digit labels and 100% on the colour labels over 1600 validation examples.
The pre-trained model was then used with prototypical and attentional classiﬁers in the polythetic MNIST few-shot classiﬁcation setting discussed in Section 5 and detailed further in Appendix F. We compared performance using the multi-headed softmax activations the model was pre-trained with and an elementwise sigmoid for both the monothetic and polythetic settings.
The results are presented in Table 7, and conform to the trend we see in other experiments: threshold classiﬁcation has the advantage in monothetic tasks but perform no better than chance for polythetic tasks.
Attentional classiﬁers are weaker in the monothetic setting, but more than make up for this defeict in the polythetic setting.
[Table]
H A N ILLUSTRATIVE EXAMPLE OF THE METHOD
Figure 10 provides a more comprehensive overview of the proposed method.
In this example, we are interested in distinguishing between big-cats and equids (horses, donkeys, and in the case zebra).
We ﬁrst extract features from all samples in the support and query sets.
Here, we imagine these features as corresponding to some general ‘cat’ properties, patterns (such as stripes, dots etc.) and general ‘equid’ properties.
Next (as presented in lines 2 : 4 in Algorithm 1), we perform repeated self-attention with respect to the separate classes of the support set, which yields updated features for each support sample.
We then aggregate the resulting support features with an appropriate dispersion metric, e.g. mean absolute deviation or standard deviation, (line 5 in Algorithm 1) to obtain a vector of feature scores.
These scores quantify the relevance of each feature in a given task.
Next, we rescale both the query and (initial) support features, i.e. multiplying by the feature scores, to dilute the task-irrelevant and (potentially) misleading features.
In this particular example, this would correspond to diluting the ‘patterns’ feature, since it is irrelevant when distinguishing between cats and equids.
Finally, we produce class probabilities via an attentional classiﬁer.
I C ONSTRUCTION OF BINARY STRINGS TASKS
For binary strings tasks, we construct training tasks by sampling | S | = 5 · 2 α support examples and | Q | = 5 · 2 α query examples.
Each example consists of α + β bits.
The location of the α active bits is completely random and consistent between the support and query sets within the same task.
The labels are computed as the XOR over the α active components.
The remaining β noisy components are randomly sampled from a Bernoulli distribution with protability 0 .
Figure 11 shows an example of a binary strings task with α = 2 active components, β = 3 noisy bits, and r = 1 repetitions.
[Figure]
[Equation]
[Figure]
[Equation]
J C ONSTRUCTION OF POLYTHETIC MNIST TASKS
For polythetic MNIST tasks, the support set consists of 96 samples, with 48 samples for class 0 andsamples for class 1.
Each class is further divided into 2 groups of 24 samples and each group is deﬁned by a speciﬁc set of traits.
The groups are complementary between classes, e.g. red ones and blue zeros for class 0; and blue ones and red zeros for class 1.
The set of traits is sampled randomly for each task.
The query set is sampled in the same manner, with 2 groups per class and 8 samples per group.
Tables 8 and 9 summarise the details of the support and query sets and Figures 13, 14, and 15 show the whole support set for the clean, colourless, and full versions of polythetic MNIST.
[Figure]
[Table]
[Table]
[Table]