Original: Rethinking Generalization in Few-Shot Classification
Revised: Rethinking Generalization in Few-Shot Classification

Original: Abstract
Revised: Abstract

Original: Single image-level annotations only correctly describe an often small subset of an image’s content, particularly when complex real-world scenes are depicted.
Revised: Single image-level annotations only correctly describe an often small subset of an image’s content, particularly when complex real-world scenes are depicted.

Original: While this might be acceptable in many classification scenarios, it poses a significant challenge for applications where the set of classes differs significantly between training and test time.
Revised: While this might be acceptable in many classification scenarios, it poses a significant challenge for applications where the set of classes differs significantly between training and test time.

Original: In this paper, we take a closer look at the implications in the context of few-shot learning .
Revised: In this paper, we take a closer look at the implications in the context of few-shot learning .

Original: Splitting the input samples into patches and encoding these via the help of Vision Transformers allows us to establish semantic correspondences between local regions across images and independent of their respective class.
Revised: Splitting the input samples into patches and encoding these via the help of Vision Transformers allows us to establish semantic correspondences between local regions across images and independent of their respective class.

Original: The most informative patch embeddings for the task at hand are then determined as a function of the support set via online optimization at inference time, additionally providing visual interpretability of ‘ what matters most ’ in the image.
Revised: The most informative patch embeddings for the task at hand are then determined as a function of the support set via online optimization at inference time, additionally providing visual interpretability of ‘ what matters most ’ in the image.

Original: We build on recent advances in unsupervised training of networks via masked image modelling to overcome the lack of fine-grained labels and learn the more general statistical structure of the data while avoiding negative image-level annotation influence, aka supervision collapse.
Revised: We build on recent advances in unsupervised training of networks via masked image modelling to overcome the lack of fine-grained labels and learn the more general statistical structure of the data while avoiding negative image-level annotation influence, aka supervision collapse.

Original: Experimental results show the competitiveness of our approach, achieving new state-of-the-art results on four popular few-shot classification benchmarks for 5 -shot and 1 -shot scenarios.
Revised: Experimental results show the competitiveness of our approach, achieving new state-of-the-art results on four popular few-shot classification benchmarks for 5 -shot and 1 -shot scenarios.

Original:  Introduction
Revised:  Introduction

Original: Images depicting real-world scenes are usually comprised of several different entities, e g., a family walking their dog in a park surrounded by trees, or a person patting their dog (Figure 1).
Revised: Images depicting real-world scenes are usually comprised of several different entities, e g., a family walking their dog in a park surrounded by trees, or a person patting their dog (Figure 1).

Original: Nevertheless, popular computer vision datasets like ImageNet [38] assign a single image-level annotation to classify their entire content.
Revised: Nevertheless, popular computer vision datasets like ImageNet [39] assign a single image-level annotation to classify their entire content.

Original: Hence, such a label only correctly applies to an often small subset of the actual image.
Revised: Hence, such a label only correctly applies to an often small subset of the actual image.

Original: As a result, models trained on such data via gradient-based methods learn to ignore all seemingly irrelevant information, particularly entities that occur across differently labelled images.
Revised: As a result, models trained on such data via gradient-based methods learn to ignore all seemingly irrelevant information, particularly entities that occur across differently labelled images.

Original: While this might be acceptable for conventional classification methods that encounter a diverse number of training examples for all classes they are expected to distinguish, it poses a major but often overlooked challenge for applications where the set of classes differs between training and test time.
Revised: While this might be acceptable for conventional classification methods that encounter a diverse number of training examples for all classes they are expected to distinguish, it poses a major but often overlooked challenge for applications where the set of classes differs between training and test time.

Original: One such affected area is few-shot learning (FSL) where approaches are expected to correctly classify entirely new classes at test time that have never been encountered during training, just by being provided with a few (e.g., one or five) samples for each of these new categories.
Revised: One such affected area is few-shot learning (FSL) where approaches are expected to correctly classify entirely new classes at test time that have never been encountered during training, just by being provided with a few (e.g., one or five) samples for each of these new categories.

Original: During test time, entities that have not been part of the set of training classes and have possibly been perceived as irrelevant might very well be part of the set of test classes – yet, the method was taught to ignore these.
Revised: During test time, entities that have not been part of the set of training classes and have possibly been perceived as irrelevant might very well be part of the set of test classes – yet, the method was taught to ignore these.

Original: Similarly, a method might overemphasize the importance of certain image patterns learned during training that are however of no relevance for the test classes, resulting in supervision collapse [9].
Revised: Similarly, a method might overemphasize the importance of certain image patterns learned during training that are however of no relevance for the test classes, resulting in supervision collapse [9].

Original: Few previous works [9, 18] partially tackle the above challenges.
Revised: Few previous works [9, 19] partially tackle the above challenges.

Original: CTX [9] proposes to learn the spatial and semantic alignment between CNN-extracted query and support features using a Transformer-style attention mechanism.
Revised: CTX [9] proposes to learn the spatial and semantic alignment between CNN-extracted query and support features using a Transformer-style attention mechanism.

Original: The authors further show that self-supervised learning tasks (i.e., SimCLR) can be integrated into episodic training along with normal supervised tasks to learn more generalized features, which benefits solving unseen tasks and mitigates supervision collapse.
Revised: The authors further show that self-supervised learning tasks (i.e., SimCLR) can be integrated into episodic training along with normal supervised tasks to learn more generalized features, which benefits solving unseen tasks and mitigates supervision collapse.

Original: CAN [18] achieves this in a similar manner by performing cross-attention between class prototypes and query feature maps, highlighting the region of a feature map important for classification during inference.
Revised: CAN [19] achieves this in a similar manner by performing cross-attention between class prototypes and query feature maps, highlighting the region of a feature map important for classification during inference.

Original: While both methods propose important contributions towards tackling supervision collapse, there exist important drawbacks.
Revised: While both methods propose important contributions towards tackling supervision collapse, there exist important drawbacks.

Original: Firstly, both methods build their ideas around aligning prototypes based on each query.
Revised: Firstly, both methods build their ideas around aligning prototypes based on each query.

Original: Such prototypes are merely class-aware and ignore all inter-class information present in the support set – a part that has however been shown to be crucial for few-shot learning [34, 48].
Revised: Such prototypes are merely class-aware and ignore all inter-class information present in the support set – a part that has however been shown to be crucial for few-shot learning [35, 49].

Original: Furthermore, learning query-aligned class representations requires performing the same operation for each query, rendering such approaches rather inefficient at inference time.
Revised: Furthermore, learning query-aligned class representations requires performing the same operation for each query, rendering such approaches rather inefficient at inference time.

Original: Summarizing our and previous works’ observations, we aim to address what we see as the two main criteria: 1) building an understanding about an image’s structure and content that generalizes towards new classes, and 2) providing the ability to interpret the provided samples in context , i.e., finding the intra-class similarities and inter-class differences while jointly considering all available information.
Revised: Summarizing our and previous works’ observations, we aim to address what we see as the two main criteria: 1) building an understanding about an image’s structure and content that generalizes towards new classes, and 2) providing the ability to interpret the provided samples in context , i.e., finding the intra-class similarities and inter-class differences while jointly considering all available information.

Original: Our work.
Revised: Our work.

Original: 2 To alleviate the negative influence of image-level annotations and to avoid supervision collapse, we decompose the images into patches representing local regions, each having a higher likelihood of being dominated by only one entity.
Revised: 2 To alleviate the negative influence of image-level annotations and to avoid supervision collapse, we decompose the images into patches representing local regions, each having a higher likelihood of being dominated by only one entity.

Original: To overcome the lack of such fine-grained annotations we employ self-supervised training with Masked Image Modelling as pretext task [59] and use a Vision Transformer architecture [10] as encoder due to its patch-based nature.
Revised: To overcome the lack of such fine-grained annotations we employ self-supervised training with Masked Image Modelling as pretext task [60] and use a Vision Transformer architecture [10] as encoder due to its patch-based nature.

Original: We build our classification around the concept of learning task-specific similarity between local regions as a function of the support set at inference time.
Revised: We build our classification around the concept of learning task-specific similarity between local regions as a function of the support set at inference time.

Original: To this extent, we first create a prior similarity map by establishing semantic patch correspondences between all support set samples irrespective of their class, i.e., also between entities that might not be relevant or potentially even harmful for correct classification (Figure 1, step (1)).
Revised: To this extent, we first create a prior similarity map by establishing semantic patch correspondences between all support set samples irrespective of their class, i.e., also between entities that might not be relevant or potentially even harmful for correct classification (Figure 1, step (1)).

Original: Consider the depicted support set with only two classes: ‘person’ and ‘cat’.
Revised: Consider the depicted support set with only two classes: ‘person’ and ‘cat’.

Original: The lower-right image is part of our support set for ‘cat’ – and the dog just happens to be in the image.
Revised: The lower-right image is part of our support set for ‘cat’ – and the dog just happens to be in the image.

Original: Now in the query sample that shall be classified, the image depicts a person patting their dog.
Revised: Now in the query sample that shall be classified, the image depicts a person patting their dog.

Original: We will thus correctly detect a correspondence of the two dogs across those two images, as well as between the person patches and the other samples of the person support set class.
Revised: We will thus correctly detect a correspondence of the two dogs across those two images, as well as between the person patches and the other samples of the person support set class.

Original: While the correspondences between the person regions are helpful, there is no ‘dog’ class in the actual support set (i.e., ‘dog’ is out-of-task information), rendering this correspondence harmful for classification since it would indicate that the query is connected to the image with the ‘cat’ label.
Revised: While the correspondences between the person regions are helpful, there is no ‘dog’ class in the actual support set (i.e., ‘dog’ is out-of-task information), rendering this correspondence harmful for classification since it would indicate that the query is connected to the image with the ‘cat’ label.

Original: This is where our token importance weighting comes into play.
Revised: This is where our token importance weighting comes into play.

Original: We infer an importance weight for each token based on its contribution towards correct classification of the other support set samples, actively strengthening intra-class similarities and inter-class differences by jointly considering all available information – in other words, we learn which tokens ‘help’ or ‘harm’ our classification objective (Figure 1, step (2)).
Revised: We infer an importance weight for each token based on its contribution towards correct classification of the other support set samples, actively strengthening intra-class similarities and inter-class differences by jointly considering all available information – in other words, we learn which tokens ‘help’ or ‘harm’ our classification objective (Figure 1, step (2)).

Original: These importance-reweighted support set embeddings are then used as basis for our similarity-based query sample classification (step (3)).
Revised: These importance-reweighted support set embeddings are then used as basis for our similarity-based query sample classification (step (3)).

Original: Our main contributions include the following:
Revised: Our main contributions include the following:

Original: C×H×W
Revised: C×H×W

Original: N·K×M×C ... ...
Revised: N·K×M×C ... ...

Original: N·K×L×D 1×L×D M×C *
Revised: N·K×L×D 1×L×D M×C *

Original: N·K×L×L
Revised: N·K×L×L

Original: Token importance weights generator ...
Revised: Token importance weights generator ...

Original: N·K×L
Revised: N·K×L

Original: N·K×L×L
Revised: N·K×L×L

Original: N·K×C×H×W
Revised: N·K×C×H×W

Original: Logits aggregation low high
Revised: Logits aggregation low high

Original: Class probabilities x
Revised: Class probabilities x

Original: Broadcasting & Reweighting
Revised: Broadcasting & Reweighting

Original: Pair-wise similarity
Revised: Pair-wise similarity

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: We demonstrate that Transformer-only architectures in conjunction with self-supervised pretraining can be successfully used in few-shot settings without the need of convolutional backbones or any additional data.
Revised: We demonstrate that Transformer-only architectures in conjunction with self-supervised pretraining can be successfully used in few-shot settings without the need of convolutional backbones or any additional data.

Original: We show that meta fine-tuning of Vision Transformers combined with our inner loop token importance reweighting can successfully use the supervision signal of provided support set labels while avoiding supervision collapse.
Revised: We show that meta fine-tuning of Vision Transformers combined with our inner loop token importance reweighting can successfully use the supervision signal of provided support set labels while avoiding supervision collapse.

Original: We provide insights into how establishing general similarities across images independent of classes followed by our optimization-based selection at inference time can boost generalization while allowing visual interpretability at the same time, and show the efficacy of our method by achieving new state-of-the-art results on four popular public benchmarks.
Revised: We provide insights into how establishing general similarities across images independent of classes followed by our optimization-based selection at inference time can boost generalization while allowing visual interpretability at the same time, and show the efficacy of our method by achieving new state-of-the-art results on four popular public benchmarks.

Original:  Few-shot classification via reweighted embedding similarity
Revised:  Few-shot classification via reweighted embedding similarity

Original: We start this section by briefly introducing the problem setting we are tackling in this work: inductive few-shot classification.
Revised: We start this section by briefly introducing the problem setting we are tackling in this work: inductive few-shot classification.

Original: We then provide an overview of our proposed method FewTURE – Fewshot classification with Transformers Using Reweighted Embedding similarity (Figure 2) before elaborating on the main elements in more detail.
Revised: We then provide an overview of our proposed method FewTURE – Fewshot classification with Transformers Using Reweighted Embedding similarity (Figure 2) before elaborating on the main elements in more detail.

Original: Problem definition .
Revised: Problem definition .

Original: Inductive N -way K -shot few-shot classification aims to generalize knowledge learned during training on D train to unseen test data D test , with classes C train ∩C test = ∅ , using only a few labelled samples.
Revised: Inductive N -way K -shot few-shot classification aims to generalize knowledge learned during training on D train to unseen test data D test , with classes C train ∩C test = ∅ , using only a few labelled samples.

Original: We follow the meta-learning protocol of previous works [48] to formulate the few-shot classification problem with episodic training and testing.
Revised: We follow the meta-learning protocol of previous works [49] to formulate the few-shot classification problem with episodic training and testing.

Original: An episode E is composed of a support set X s = { ( x nks , y nks ) | n = 1 , . .
Revised: An episode E is composed of a support set X s = { ( x nks , y nks ) | n = 1 , . .

Original: N ; k = 1 , . . .
Revised: N ; k = 1 , . . .

Original: , K ; y nks ∈ C train } , where x nks denotes the k -th sample of class n with label y nks , and a query set X q
Revised: , K ; y nks ∈ C train } , where x nks denotes the k -th sample of class n with label y nks , and a query set X q

Original: = { ( x nq , y nq ) | n = 1 , . .
Revised: = { ( x nq , y nq ) | n = 1 , . .

Original: . , N } , where x nq denotes a query sample 3 of class n with label y n q .
Revised: . , N } , where x nq denotes a query sample 3 of class n with label y n q .

Original: 2.1 Overview of FewTURE
Revised: 2.1 Overview of FewTURE

Original: As depicted in Figure 2, we encode the image patches P s of the support set samples along with the query sample patches p q via f θ and obtain corresponding sets of tokens Z s and z q , respectively 4 .
Revised: As depicted in Figure 2, we encode the image patches P s of the support set samples along with the query sample patches p q via f θ and obtain corresponding sets of tokens Z s and z q , respectively 4 .

Original: It is to be noted that while we choose to illustrate our method via the use of one single query sample, the classification of all query samples is computed at the same time in one single pass in practice.
Revised: It is to be noted that while we choose to illustrate our method via the use of one single query sample, the classification of all query samples is computed at the same time in one single pass in practice.

Original: We retrieve our ‘prior’ correspondence map S expressing the token-wise similarity between the encoded semantic content of the local regions in the query sample and all patches of all support samples, allowing us to consider all available information jointly without incurring information loss due to averaging or similar operations.
Revised: We retrieve our ‘prior’ correspondence map S expressing the token-wise similarity between the encoded semantic content of the local regions in the query sample and all patches of all support samples, allowing us to consider all available information jointly without incurring information loss due to averaging or similar operations.

Original: This ‘prior’ similarity map represents correspondences between regions of samples irrespective of their individual class, i.e. also between entities that might not be relevant or potentially even harmful for correct classification.
Revised: This ‘prior’ similarity map represents correspondences between regions of samples irrespective of their individual class, i.e. also between entities that might not be relevant or potentially even harmful for correct classification.

Original: Using the annotated support set samples, we now infer a task-specific importance weight factor v j for each support token z js representing its contribution to correctly classify other samples in the support set via online optimization at inference time (Section 2.4).
Revised: Using the annotated support set samples, we now infer a task-specific importance weight factor v j for each support token z js representing its contribution to correctly classify other samples in the support set via online optimization at inference time (Section 2.4).

Original: We then reweight the prior similarities to obtain our classification result for the query sample, jointly considering all available information.
Revised: We then reweight the prior similarities to obtain our classification result for the query sample, jointly considering all available information.

Original: 2.2 Self-supervised pretraining against supervision collapse
Revised: 2.2 Self-supervised pretraining against supervision collapse

Original: To overcome the problem of supervision collapse induced by image-level annotations, we split the input images into smaller parts where each region has a higher likelihood of only containing one major entity and hence a more distinct semantic meaning.
Revised: To overcome the problem of supervision collapse induced by image-level annotations, we split the input images into smaller parts where each region has a higher likelihood of only containing one major entity and hence a more distinct semantic meaning.

Original: Since no labels are available for this more fine-grained data, we encode the information of each local region via an unsupervised method.
Revised: Since no labels are available for this more fine-grained data, we encode the information of each local region via an unsupervised method.

Original: We build our approach around the recently introduced idea of using Masked Image Modeling (MIM) [3, 59] as a pretext task for self-supervised training of Vision Transformers.
Revised: We build our approach around the recently introduced idea of using Masked Image Modeling (MIM) [3, 60] as a pretext task for self-supervised training of Vision Transformers.

Original: In contrast to previous unsupervised approaches [5, 7] which focused mainly on global image-level representations,
Revised: In contrast to previous unsupervised approaches [5, 7] which focused mainly on global image-level representations,

Original: MIM randomly masks a number of patch embeddings ( tokens ) and aims to reconstruct them given the remaining information of the image.
Revised: MIM randomly masks a number of patch embeddings ( tokens ) and aims to reconstruct them given the remaining information of the image.

Original: The introduced token constraints help our Transformer backbone to learn an embedding space that yields semantically meaningful representations for each individual image patch.
Revised: The introduced token constraints help our Transformer backbone to learn an embedding space that yields semantically meaningful representations for each individual image patch.

Original: We then leverage the information of the provided labels through fine-tuning the pretrained backbone in conjunction with our inner loop token importance weighting described in the following sections while successfully avoiding supervision collapse (see experimental results in Section 3.2).
Revised: We then leverage the information of the provided labels through fine-tuning the pretrained backbone in conjunction with our inner loop token importance weighting described in the following sections while successfully avoiding supervision collapse (see experimental results in Section 3.2).

Original: 2.3 Classification through reweighted token similarity
Revised: 2.3 Classification through reweighted token similarity

Original: As illustrated in Figure 2, we split each input image x ∈ R H × W × C into a sequence of M = H · W / Ppatches p
Revised: As illustrated in Figure 2, we split each input image x ∈ R H × W × C into a sequence of M = H · W / Ppatches p

Original: = { p i } Mi =1 , with each patch p i ∈
Revised: = { p i } Mi =1 , with each patch p i ∈

Original: R P 2 × C .
Revised: R P 2 × C .

Original: We then flatten and pass all patches of the support and query images as input to the Transformer architecture, obtaining the set of support tokens Z s = f θ ( P s ) with Z s =
Revised: We then flatten and pass all patches of the support and query images as input to the Transformer architecture, obtaining the set of support tokens Z s = f θ ( P s ) with Z s =

Original: { z nks | n =1 , . . .
Revised: { z nks | n =1 , . . .

Original: , N, k =1 , . .
Revised: , N, k =1 , . .

Original: , K } ,
Revised: , K } ,

Original: z nks = { z nkls | l =1 , . . .
Revised: z nks = { z nkls | l =1 , . . .

Original: , L ; z nkls ∈ R D } and query tokens
Revised: , L ; z nkls ∈ R D } and query tokens

Original: z q = f θ ( p q ) with z q = { z lq | l = 1 , . . .
Revised: z q = f θ ( p q ) with z q = { z lq | l = 1 , . . .

Original: , L ; z lq ∈ R D } .
Revised: , L ; z lq ∈ R D } .

Original: Vision Transformers like ViT [10, 46] satisfy L = M
Revised: Vision Transformers like ViT [10, 47] satisfy L = M

Original: whereas hierarchical Transformers like Swin [27] generally emit a reduced number of tokens L < M due to internal merging strategies.
Revised: whereas hierarchical Transformers like Swin [28] generally emit a reduced number of tokens L < M due to internal merging strategies.

Original: Having obtained all patch embeddings, we establish semantic correspondences by computing the pair-wise patch similarity matrix between the set of support tokens Z s and query tokens
Revised: Having obtained all patch embeddings, we establish semantic correspondences by computing the pair-wise patch similarity matrix between the set of support tokens Z s and query tokens

Original: z q as S ∈
Revised: z q as S ∈

Original: R N · K · L × L , where each element in S is obtained by s l s ,l q nk = sim
Revised: R N · K · L × L , where each element in S is obtained by s l s ,l q nk = sim

Original: ( z nkl s s , z l q q ) , where l s =1 , . . .
Revised: ( z nkl s s , z l q q ) , where l s =1 , . . .

Original: , L and l q =1 , . . .
Revised: , L and l q =1 , . . .

Original: , L .
Revised: , L .

Original: Note that local image regions representing similar entities exhibit higher scores.
Revised: Note that local image regions representing similar entities exhibit higher scores.

Original: While any distance metric can be used to compute the similarity ( sim ), we found cosine similarity to work particularly well for this task.
Revised: While any distance metric can be used to compute the similarity ( sim ), we found cosine similarity to work particularly well for this task.

Original: We then use the task-specific token importance weights v ∈ R N · K · L ×inferred via online optimization based on the annotated support set samples (see Section 2.4) to reweight the similarities through column-wise addition and obtain our task-specific similarity matrix as ˜ S
Revised: We then use the task-specific token importance weights v ∈ R N · K · L ×inferred via online optimization based on the annotated support set samples (see Section 2.4) to reweight the similarities through column-wise addition and obtain our task-specific similarity matrix as ˜ S

Original: = S + (cid:2) v · 1 1 × L (cid:3) , with elements ˜ s l s ,l q nk .
Revised: = S + (cid:2) v · 1 1 × L (cid:3) , with elements ˜ s l s ,l q nk .

Original: Note that this addition of our reweighting logits corresponds to multiplicative reweighting in probability space.
Revised: Note that this addition of our reweighting logits corresponds to multiplicative reweighting in probability space.

Original: We temperature-scale the adapted similarity logits with 1 / τ S and aggregate the token similarity values across all elements belonging to the same support set class via a LogSumExp operation, i.e. aggregating K · L 2 logits per class followed by a softmax – resulting in the final class prediction ˆ y q for the query sample x q as
Revised: We temperature-scale the adapted similarity logits with 1 / τ S and aggregate the token similarity values across all elements belonging to the same support set class via a LogSumExp operation, i.e. aggregating K · L 2 logits per class followed by a softmax – resulting in the final class prediction ˆ y q for the query sample x q as

Original: [Equation]
Revised: [Equation]

Original: L (cid:88)
Revised: L (cid:88)

Original: [Table]
Revised: [Table]

Original: (cid:88)
Revised: (cid:88)

Original: [Table]
Revised: [Table]

Original: exp (cid:16)
Revised: exp (cid:16)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (cid:17) 
Revised: (cid:17) 

Original: N n =
Revised: N n =

Original: [Table]
Revised: [Table]

Original:   .
Revised:   .

Original: 2.4 Learning token importance at inference time
Revised: 2.4 Learning token importance at inference time

Original: We use all samples of a task’s support set together with their annotations to learn the importance for each individual patch token via online optimization at inference time.
Revised: We use all samples of a task’s support set together with their annotations to learn the importance for each individual patch token via online optimization at inference time.

Original: As depicted in Figure 3, we formulate the same classification objective as in the previous section but aim to classify the support set samples instead of a query sample.
Revised: As depicted in Figure 3, we formulate the same classification objective as in the previous section but aim to classify the support set samples instead of a query sample.

Original: In other words, we use two versions of the support set samples: one with labels as ‘support’ Z s and one without as ‘pseudo-query’ Z sq , and obtain the similarity
Revised: In other words, we use two versions of the support set samples: one with labels as ‘support’ Z s and one without as ‘pseudo-query’ Z sq , and obtain the similarity

Original: N·K·L×N·K·L N·K·L×N·K·L
Revised: N·K·L×N·K·L N·K·L×N·K·L

Original: Class probability
Revised: Class probability

Original: Class label
Revised: Class label

Original: Loss
Revised: Loss

Original: Logits aggregation
Revised: Logits aggregation

Original: N·K×L ...
Revised: N·K×L ...

Original: Pair-wise similarity
Revised: Pair-wise similarity

Original: Broadcast&addition
Revised: Broadcast&addition

Original: N·K×L
Revised: N·K×L

Original: Masking
Revised: Masking

Original: Update
Revised: Update

Original: Backpropagate
Revised: Backpropagate

Original: Forward pass
Revised: Forward pass

Original: N·K×L×D ... ... N·K×L×D ... ...
Revised: N·K×L×D ... ... N·K×L×D ... ...

Original: matrix S s ∈ R N · K · L × N · K · L .
Revised: matrix S s ∈ R N · K · L × N · K · L .

Original: The token importance weights are initialized to v 0 = 

Revised: The token importance weights are initialized to v 0 =

Original: ∈ R N · K · L ×and column-wise added to form ˜ S s
Revised: ∈ R N · K · L ×and column-wise added to form ˜ S s

Original: = S s + (cid:2) v 0 · 1 1 × N · K · L (cid:3) .
Revised: = S s + (cid:2) v 0 · 1 1 × N · K · L (cid:3) .

Original: The goal is now to determine which tokens of Z s are most helpful in contributing towards correctly classifying Z sq , and which ones negatively affect this objective.
Revised: The goal is now to determine which tokens of Z s are most helpful in contributing towards correctly classifying Z sq , and which ones negatively affect this objective.

Original: To prevent tokens from simply classifying themselves, we devise the following strategy for our N -way K -shot tasks.
Revised: To prevent tokens from simply classifying themselves, we devise the following strategy for our N -way K -shot tasks.

Original: For scenarios with K > 1 samples per class, we apply block-diagonal masking with blocks of size L × L to the similarity matrix ˜ S s – meaning that we enforce classification of each token in Z sq exclusively based on information from other images.
Revised: For scenarios with K > 1 samples per class, we apply block-diagonal masking with blocks of size L × L to the similarity matrix ˜ S s – meaning that we enforce classification of each token in Z sq exclusively based on information from other images.

Original: Since there are no other in-class examples available in 1 -shot scenarios, we slightly weaken the constraint and apply local masking in an m × m window around each patch, forcing the token to be classified based on the remaining information in the image.
Revised: Since there are no other in-class examples available in 1 -shot scenarios, we slightly weaken the constraint and apply local masking in an m × m window around each patch, forcing the token to be classified based on the remaining information in the image.

Original: We found a local window of m = 5 to work well throughout our experiments for both architectures.
Revised: We found a local window of m = 5 to work well throughout our experiments for both architectures.

Original: We use temperature-scaling and aggregate all modified similarity logits across all elements belonging to the same support set class of the annotated Z s for each element z nksq , apply a softmax operation and obtain the predicted class probabilities ˆ y nks for each support set sample ( cf.
Revised: We use temperature-scaling and aggregate all modified similarity logits across all elements belonging to the same support set class of the annotated Z s for each element z nksq , apply a softmax operation and obtain the predicted class probabilities ˆ y nks for each support set sample ( cf.

Original: Equation (1)).
Revised: Equation (1)).

Original: Given that the prediction is now dependent on the initialized token similarity weights v , we can formulate an online optimization objective by using the support set labels y nks as
Revised: Given that the prediction is now dependent on the initialized token similarity weights v , we can formulate an online optimization objective by using the support set labels y nks as

Original: [Equation]
Revised: [Equation]

Original: It is to be noted that by using column-wise addition of v , we share the importance weights of each support token across all ‘pseudo-query’ tokens and thus constrain the optimization to jointly learn token importance with respect to all available information.
Revised: It is to be noted that by using column-wise addition of v , we share the importance weights of each support token across all ‘pseudo-query’ tokens and thus constrain the optimization to jointly learn token importance with respect to all available information.

Original: In other words, task-specific intra-class similarities will be strengthened while inter-class ones will be penalized accordingly.
Revised: In other words, task-specific intra-class similarities will be strengthened while inter-class ones will be penalized accordingly.

Original: We further do not require any second order derivatives like other methods during meta fine-tuning, since the optimization of v is decoupled from the network’s parameters.
Revised: We further do not require any second order derivatives like other methods during meta fine-tuning, since the optimization of v is decoupled from the network’s parameters.

Original:  Experiments and discussion 3.1 Implementation details
Revised:  Experiments and discussion 3.1 Implementation details

Original: Our training strategy is divided into two parts: self-supervised pretraining followed by meta finetuning.
Revised: Our training strategy is divided into two parts: self-supervised pretraining followed by meta finetuning.

Original: It is to be noted that each architecture is exclusively trained on the training set data of the respective dataset that is to be evaluated, and no additional data is used.
Revised: It is to be noted that each architecture is exclusively trained on the training set data of the respective dataset that is to be evaluated, and no additional data is used.

Original: Datasets .
Revised: Datasets .

Original: We train and evaluate our methods using four popular few-shot classification benchmarks, namely mini ImageNet [48], tiered ImageNet [37], CIFAR-FS [4] and FC-100 [34].
Revised: We train and evaluate our methods using four popular few-shot classification benchmarks, namely mini ImageNet [49], tiered ImageNet [38], CIFAR-FS [4] and FC-100 [35].

Original: Architectures .
Revised: Architectures .

Original: We compare two different Transformer architectures in this work, the monolithic
Revised: We compare two different Transformer architectures in this work, the monolithic

Original: ViT architecture [10, 46] in its ‘small’ form ( ViT-small aka DeiT-small ) and the multi-scale Swin architecture [27] in its ‘tiny’ version ( Swin-tiny ).
Revised: ViT architecture [10, 47] in its ‘small’ form ( ViT-small aka DeiT-small ) and the multi-scale Swin architecture [28] in its ‘tiny’ version ( Swin-tiny ).

Original: Self-supervised pretraining .
Revised: Self-supervised pretraining .

Original: We employ the strategy proposed by [59] to pretrain our Transformer backbones and mostly stick to the hyperparameter settings reported in their work.
Revised: We employ the strategy proposed by [60] to pretrain our Transformer backbones and mostly stick to the hyperparameter settings reported in their work.

Original: Our ViT and Swin
Revised: Our ViT and Swin

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Meta fine-tuning .
Revised: Meta fine-tuning .

Original: We use meta fine-tuning to further refine our embedding space by using the available image-level labels in conjunction with our token-reweighting method.
Revised: We use meta fine-tuning to further refine our embedding space by using the available image-level labels in conjunction with our token-reweighting method.

Original: We generally train for up to 200 epochs but find most architectures to converge earlier.
Revised: We generally train for up to 200 epochs but find most architectures to converge earlier.

Original: We evaluate at each epoch on 600 randomly sampled episodes from the respective validation set to select the best set of parameters.
Revised: We evaluate at each epoch on 600 randomly sampled episodes from the respective validation set to select the best set of parameters.

Original: During test time, we randomly sample 600 episodes from the test set to evaluate our model.
Revised: During test time, we randomly sample 600 episodes from the test set to evaluate our model.

Original: Token importance reweighting and classifier .
Revised: Token importance reweighting and classifier .

Original: We use cosine similarity to compute S .
Revised: We use cosine similarity to compute S .

Original: While the temperature τ S used to scale the logits can be learnt during meta fine-tuning, we found τ S = 1 / √ d to be a good default value (or starting point if learnt, see supplementary material).
Revised: While the temperature τ S used to scale the logits can be learnt during meta fine-tuning, we found τ S = 1 / √ d to be a good default value (or starting point if learnt, see supplementary material).

Original: We use SGD as optimizer with a learning rate of 0.1 for the token importance weight generation.
Revised: We use SGD as optimizer with a learning rate of 0.1 for the token importance weight generation.

Original: Please refer to supplementary material of this paper for a more detailed discussion regarding datasets, implementation and hyperparameters.
Revised: Please refer to supplementary material of this paper for a more detailed discussion regarding datasets, implementation and hyperparameters.

Original: 3.2 Self-supervised pretraining and token-reweighted fine-tuning improve generalization
Revised: 3.2 Self-supervised pretraining and token-reweighted fine-tuning improve generalization

Original: In this section, we investigate the influence of self-supervised pretraining compared to its supervised counterpart using the provided image-level labels (de-facto standard in most current state-of-the-art methods).
Revised: In this section, we investigate the influence of self-supervised pretraining compared to its supervised counterpart using the provided image-level labels (de-facto standard in most current state-of-the-art methods).

Original: We only vary the training strategy of our backbone and use our introduced token similaritybased classifier with 15 inner-loop reweighting steps for all experiments.
Revised: We only vary the training strategy of our backbone and use our introduced token similaritybased classifier with 15 inner-loop reweighting steps for all experiments.

Original: Figure 4 illustrates that both
Revised: Figure 4 illustrates that both

Original: ViT-small and Swin-tiny with self-supervised pretraining alone ( w/o meta fine-tuning) learn more generalizable features than their respective supervised versions for both 1 -shot (left-hand side) and 5 -shot (right-hand side) scenarios.
Revised: ViT-small and Swin-tiny with self-supervised pretraining alone ( w/o meta fine-tuning) learn more generalizable features than their respective supervised versions for both 1 -shot (left-hand side) and 5 -shot (right-hand side) scenarios.

Original: In fact, across all cases w/o meta fine-tuning, the self-supervised pretrained Transformers outperform their supervised counterparts by more than 10% and up to a significant 18 .
Revised: In fact, across all cases w/o meta fine-tuning, the self-supervised pretrained Transformers outperform their supervised counterparts by more than 10% and up to a significant 18 .

Original: 19% in the case of Swin-tiny.
Revised: 19% in the case of Swin-tiny.

Original: This clearly demonstrates that our self-supervised pretraining strategy explores information that is richer and beyond the labels.
Revised: This clearly demonstrates that our self-supervised pretraining strategy explores information that is richer and beyond the labels.

Original: Another interesting insight is that the meta fine-tuning stage does not improve supervised backbones as much as their unsupervised counterparts.
Revised: Another interesting insight is that the meta fine-tuning stage does not improve supervised backbones as much as their unsupervised counterparts.

Original: Specifically after unsupervised pretraining, meta fine-tuning of the
Revised: Specifically after unsupervised pretraining, meta fine-tuning of the

Original: Transformers in conjunction with our token-reweighting strategy is able to boost the performance by
Revised: Transformers in conjunction with our token-reweighting strategy is able to boost the performance by

Original: 77% (ViT-small) and 13 .
Revised: 77% (ViT-small) and 13 .

Original: 01% (Swin-tiny) for 1 -shot, and 9 .
Revised: 01% (Swin-tiny) for 1 -shot, and 9 .

Original: 94% (ViT-small) and 10 .
Revised: 94% (ViT-small) and 10 .

Original: 10% (Swintiny) for 5 -shot.
Revised: 10% (Swintiny) for 5 -shot.

Original: While such significant improvements cannot be observed across the supervised networks, the Swin versions seem to generally start off lower after pretraining but benefit more from the fine-tuning than ViT.
Revised: While such significant improvements cannot be observed across the supervised networks, the Swin versions seem to generally start off lower after pretraining but benefit more from the fine-tuning than ViT.

Original: The observed results clearly indicate that our token-reweighted fine-tuning strategy is able to further improve the generalization of self-supervised Transformers, thus performing better on the novel tasks of the unseen test set.
Revised: The observed results clearly indicate that our token-reweighted fine-tuning strategy is able to further improve the generalization of self-supervised Transformers, thus performing better on the novel tasks of the unseen test set.

Original: Figure 5 additionally depicts projected views of the tokens of 5 instances from a novel class as well as the entire novel support set in embedding space.
Revised: Figure 5 additionally depicts projected views of the tokens of 5 instances from a novel class as well as the entire novel support set in embedding space.

Original: Representations obtained with our classifier seem to retain the instance information (‘ w/o v ’) and separation is improved when using token importance reweighting (‘ w/ v ’).
Revised: Representations obtained with our classifier seem to retain the instance information (‘ w/o v ’) and separation is improved when using token importance reweighting (‘ w/ v ’).

Original: While the projected tokens of the entire support set show partial overlap between classes as is expected due to commonalities like e.g. similar background, our reweighting clearly determines the class-characteristic tokens (displayed in their original class-respective color).
Revised: While the projected tokens of the entire support set show partial overlap between classes as is expected due to commonalities like e.g. similar background, our reweighting clearly determines the class-characteristic tokens (displayed in their original class-respective color).

Original: These results indicate that our similarity-based classifier coupled with task-specific token reweighting is able to better disentangle the embeddings of different instances from the same class as well as other classes, which prevents the network from supervision collapse and achieves the higher performance observed on the benchmarks.
Revised: These results indicate that our similarity-based classifier coupled with task-specific token reweighting is able to better disentangle the embeddings of different instances from the same class as well as other classes, which prevents the network from supervision collapse and achieves the higher performance observed on the benchmarks.

Original: They further show that self-supervised pretraining is helpful but not sufficient to achieve well-separated representations without supervision collapse that are suitable for few-shot classification.
Revised: They further show that self-supervised pretraining is helpful but not sufficient to achieve well-separated representations without supervision collapse that are suitable for few-shot classification.

Original: 3.3 Selecting helpful patches at inference time
Revised: 3.3 Selecting helpful patches at inference time

Original: Figure 6 shows a visualization of the patch importance weights v that are learned at inference time during the inner loop adaptation for the support set images.
Revised: Figure 6 shows a visualization of the patch importance weights v that are learned at inference time during the inner loop adaptation for the support set images.

Original: Brighter regions represent a higher importance weight, meaning that these patches will contribute most to the classification of query samples if matches with high similarity can be found.
Revised: Brighter regions represent a higher importance weight, meaning that these patches will contribute most to the classification of query samples if matches with high similarity can be found.

Original: Judging from the visualized weights, FewTURE seems to consistently select characteristic regions of the depicted objects, e.g., the rim of the bowls, strings of the guitar or the dogs’ facial area, and to exclude unimportant or out-of-task information.
Revised: Judging from the visualized weights, FewTURE seems to consistently select characteristic regions of the depicted objects, e.g., the rim of the bowls, strings of the guitar or the dogs’ facial area, and to exclude unimportant or out-of-task information.

Original: We further investigate the influence of the number of optimization steps in our inner loop token importance weighting method using ViT-small on mini ImageNet.
Revised: We further investigate the influence of the number of optimization steps in our inner loop token importance weighting method using ViT-small on mini ImageNet.

Original: The results in Figure 7 indicate that increasing the steps up to 20 aligns with increased performance, both during validation and testing.
Revised: The results in Figure 7 indicate that increasing the steps up to 20 aligns with increased performance, both during validation and testing.

Original: While the initial increase in test accuracy when using our token reweighting (steps > 0 ) is rather significant with 1 . 15% , the contribution of higher step numbers comes at the cost of higher
Revised: While the initial increase in test accuracy when using our token reweighting (steps > 0 ) is rather significant with 1 . 15% , the contribution of higher step numbers comes at the cost of higher

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: computational complexity
Revised: computational complexity

Original: [Table]
Revised: [Table]

Original: and we generally found anything between 5 and 15 steps to be a good performance vs .
Revised: and we generally found anything between 5 and 15 steps to be a good performance vs .

Original: inference-time trade-off (see supplementary material for further details)
Revised: inference-time trade-off (see supplementary material for further details)

Original: [Table]
Revised: [Table]

Original: 3.4 Evaluation on few-shot classification benchmarks
Revised: 3.4 Evaluation on few-shot classification benchmarks

Original: We conduct experiments using the few-shot settings established in the community, namely 5 -way 1 -shot and 5 -way 5 -shot
Revised: We conduct experiments using the few-shot settings established in the community, namely 5 -way 1 -shot and 5 -way 5 -shot

Original: meaning the network has to distinguish samples from 5 novel classes based on a provided number of 1 or 5 images per class.
Revised: meaning the network has to distinguish samples from 5 novel classes based on a provided number of 1 or 5 images per class.

Original: We evaluate our method FewTURE using two different Transformer backbones and compare our results against the current state of the art in Tablefor the mini ImageNet and tiered ImageNet
Revised: We evaluate our method FewTURE using two different Transformer backbones and compare our results against the current state of the art in Tablefor the mini ImageNet and tiered ImageNet

Original: [Table]
Revised: [Table]

Original: and in Table 2 for the CIFAR-FS and FC100 datasets.
Revised: and in Table 2 for the CIFAR-FS and FC100 datasets.

Original: It is to be noted that in contrast to previous works, we do not employ the help of any convolutional backbone but instead (and as far as we are aware for the first time) use a Transformer backbone together with our previously introduced token importance reweighting method to achieve these results.
Revised: It is to be noted that in contrast to previous works, we do not employ the help of any convolutional backbone but instead (and as far as we are aware for the first time) use a Transformer backbone together with our previously introduced token importance reweighting method to achieve these results.

Original: We are able to set new state of the art results across all four datasets in both 5 -shot and 1 -shot settings, improving particularly the 1 -shot results on mini ImageNet and tiered ImageNet by significant margins of 3 .
Revised: We are able to set new state of the art results across all four datasets in both 5 -shot and 1 -shot settings, improving particularly the 1 -shot results on mini ImageNet and tiered ImageNet by significant margins of 3 .

Original: 03% and 1 .
Revised: 03% and 1 .

Original: 74% , respectively.
Revised: 74% , respectively.

Original: 3.5 Increasing efficiency by pruning the token sequence
Revised: 3.5 Increasing efficiency by pruning the token sequence

Original: [Table]
Revised: [Table]

Original: To further improve the computational efficiency of our method, we investigate its behaviour when limiting the number of tokens that are considered to establish patch-wise correspondences to only a subset – allowing FewTURE to scale to potentially large many-way many-shot settings.
Revised: To further improve the computational efficiency of our method, we investigate its behaviour when limiting the number of tokens that are considered to establish patch-wise correspondences to only a subset – allowing FewTURE to scale to potentially large many-way many-shot settings.

Original: We use the attention maps inherent in our approach (averaged over all heads) to prune the number of patch tokens by only using the ones within the top-k attention values to compute the similarity matrix S .
Revised: We use the attention maps inherent in our approach (averaged over all heads) to prune the number of patch tokens by only using the ones within the top-k attention values to compute the similarity matrix S .

Original: The results obtained with our ViT-small backbone for pruning the number of tokens to 75% , 50% , 25% and 10% of the original token number indicate that such pruning might be an interesting avenue for future work, with our method still achieving 96 .
Revised: The results obtained with our ViT-small backbone for pruning the number of tokens to 75% , 50% , 25% and 10% of the original token number indicate that such pruning might be an interesting avenue for future work, with our method still achieving 96 .

Original: 4% of its original performance when only retaining 10% of the number of tokens.
Revised: 4% of its original performance when only retaining 10% of the number of tokens.

Original: 3.6 Ablation on the type of classifier
Revised: 3.6 Ablation on the type of classifier

Original: [Table]
Revised: [Table]

Original: We train a linear classifier as well as prototypical approach using our pre-trained ViT-small backbone for a 5-way 5-shot setting on mini Imagenet [48] to investigate the influence of the choice of classifier (Table 4).
Revised: We train a linear classifier as well as prototypical approach using our pre-trained ViT-small backbone for a 5-way 5-shot setting on mini Imagenet [49] to investigate the influence of the choice of classifier (Table 4).

Original: We were able to obtain a test accuracy of 82 .
Revised: We were able to obtain a test accuracy of 82 .

Original: 80% for the prototypical network after optimising the pre-trained backbone with metafinetuning, which is competitive to the results we obtain for our method without reweighting (’0 step’ in Figure 7(b))
Revised: 80% for the prototypical network after optimising the pre-trained backbone with metafinetuning, which is competitive to the results we obtain for our method without reweighting (’0 step’ in Figure 7(b))

Original: , but is clearly outperformed by our reweighting-based approach.
Revised: , but is clearly outperformed by our reweighting-based approach.

Original: To provide a fair comparison, we optimize the linear classifier at inference time to adapt to the support set and obtained a maximum test accuracy of 82 .
Revised: To provide a fair comparison, we optimize the linear classifier at inference time to adapt to the support set and obtained a maximum test accuracy of 82 .

Original: Both results indicate the quality of embedding our backbone is able to produce but also demonstrate the importance of our task-specific reweighting-based approach.
Revised: Both results indicate the quality of embedding our backbone is able to produce but also demonstrate the importance of our task-specific reweighting-based approach.

Original: For further ablation studies, please refer to the supplementary material.
Revised: For further ablation studies, please refer to the supplementary material.

Original: 3.7 Limitations
Revised: 3.7 Limitations

Original: Our introduced method is arguably more powerful for cases where multiple examples of the same class are provided, i.e. in N -way K -shot settings with K > 1 .
Revised: Our introduced method is arguably more powerful for cases where multiple examples of the same class are provided, i.e. in N -way K -shot settings with K > 1 .

Original: While FewTURE works well in 1 -shot settings (Table 1 and 2), the inner loop adaptation procedure still aims to exclude cross-class similarities that hurt classification performance, but has less diverse information for selecting the most helpful regions due to the lack of other in-class comparison samples – which might thus yield slightly less-refined token selections compared to multi-shot scenarios ( supplementary material ).
Revised: While FewTURE works well in 1 -shot settings (Table 1 and 2), the inner loop adaptation procedure still aims to exclude cross-class similarities that hurt classification performance, but has less diverse information for selecting the most helpful regions due to the lack of other in-class comparison samples – which might thus yield slightly less-refined token selections compared to multi-shot scenarios ( supplementary material ).

Original: While we did not face significant problems regarding the comparably very small size of our datasets (e.g. mini ImageNet with 38K compared to the usually used ImageNet with 1.28M training images), more specialized applications with highly limited training data might be negatively impacted and successfully training our method due to the reduced inductive bias present in the Transformer architecture could prove challenging.
Revised: While we did not face significant problems regarding the comparably very small size of our datasets (e.g. mini ImageNet with 38K compared to the usually used ImageNet with 1.28M training images), more specialized applications with highly limited training data might be negatively impacted and successfully training our method due to the reduced inductive bias present in the Transformer architecture could prove challenging.

Original:  Related work
Revised:  Related work

Original: Over the past few years, the family of few-shot learning (FSL) has grown diverse and broad.
Revised: Over the past few years, the family of few-shot learning (FSL) has grown diverse and broad.

Original: Those closely related to this work can be categorized into two groups: metric-based methods [30, 31, 40,
Revised: Those closely related to this work can be categorized into two groups: metric-based methods [31, 32, 41, 42,

Original: 41, 48, 53, 54] and optimisation-based methods [13, 21, 33, 39, 61].
Revised: 49, 54, 55] and optimisation-based methods [13, 18, 22, 34, 40, 62].

Original: Metric-based methods, such as
Revised: Metric-based methods, such as

Original: ProtoNet [41], DeepEMD [54], and RelationNet [43] aim to learn a class representation (prototype) by averaging the embeddings belonging to the same class and employ a predefined (ProtoNet and
Revised: ProtoNet [42], DeepEMD [55], and RelationNet [44] aim to learn a class representation (prototype) by averaging the embeddings belonging to the same class and employ a predefined (ProtoNet and

Original: DeepEMD) or learned (RelationNet) metric to perform prototype-query matching.
Revised: DeepEMD) or learned (RelationNet) metric to perform prototype-query matching.

Original: FEAT [53] and
Revised: FEAT [54] and

Original: TDM [22] take this a step further and use attention mechanisms to adapt the extracted features to the novel tasks.
Revised: TDM [23] take this a step further and use attention mechanisms to adapt the extracted features to the novel tasks.

Original: Our method instead fully utilizes the embeddings of local image regions (patch tokens), preventing loss of information and supervision collapse occurring in the aforementioned prototype-based approaches (see supplementary material).
Revised: Our method instead fully utilizes the embeddings of local image regions (patch tokens), preventing loss of information and supervision collapse occurring in the aforementioned prototype-based approaches (see supplementary material).

Original: Optimisation-based methods such as MAML [13] and Reptile [33] propose to learn a set of initial model parameters that can quickly adapt to a novel task.
Revised: Optimisation-based methods such as MAML [13] and Reptile [34] propose to learn a set of initial model parameters that can quickly adapt to a novel task.

Original: However, updating all model parameters is often not feasible given large backbones and only few labeled samples during inference.
Revised: However, updating all model parameters is often not feasible given large backbones and only few labeled samples during inference.

Original: To alleviate this so-called meta-overfitting problem, CAVIA [61] and LEO [39] propose to learn and adapt a lower dimensional representation that is mapped onto the network.
Revised: To alleviate this so-called meta-overfitting problem, CAVIA [62] and LEO [40] propose to learn and adapt a lower dimensional representation that is mapped onto the network.

Original: Our method is inspired by such lower-dimensional adaptation strategies and learns a tiny set of context-aware re-weighting factors online for each novel task without requiring higher-order gradients, resulting in a flexible and efficient framework.
Revised: Our method is inspired by such lower-dimensional adaptation strategies and learns a tiny set of context-aware re-weighting factors online for each novel task without requiring higher-order gradients, resulting in a flexible and efficient framework.

Original: Self-supervised learning for FSL .
Revised: Self-supervised learning for FSL .

Original: Although self-supervised learning is underrepresented in the context of few-shot learning, some recent works [15, 32, 42] have shown that self-supervision via pretext tasks can be beneficial when integrated as auxiliary loss.
Revised: Although self-supervised learning is underrepresented in the context of few-shot learning, some recent works [15, 33, 43] have shown that self-supervision via pretext tasks can be beneficial when integrated as auxiliary loss.

Original: S2M2 [32] employs rotation [16] and exemplars [11] along with common supervised learning during the pre-training stage.
Revised: S2M2 [33] employs rotation [16] and exemplars [11] along with common supervised learning during the pre-training stage.

Original: CTX [9] demonstrates that SimCLR [6] can be combined with supervised-learning tasks in an episodictraining manner to learn a more generalized model.
Revised: CTX [9] demonstrates that SimCLR [6] can be combined with supervised-learning tasks in an episodictraining manner to learn a more generalized model.

Original: In contrast, FewTURE demonstrates that the self-supervised pretext task (i.e., Masked Image Modelling [2, 3,24, 17, 35, 44, 59]), can be used in a standalone manner to learn more generalized features for few-shot learning on small-scale datasets.
Revised: In contrast, FewTURE demonstrates that the self-supervised pretext task (i.e., Masked Image Modelling [2, 3, 17, 25, 36, 45, 60]) can be used in a standalone manner to learn more generalized features for few-shot learning on small-scale datasets.

Original: Vision Transformers in FSL .
Revised: Vision Transformers in FSL .

Original: Transformers have been immensely successful in the field of Natural
Revised: Transformers have been immensely successful in the field of Natural

Original: Language Processing.
Revised: Language Processing.

Original: Recent studies [10, 23, 27, 47] suggest that encoding the long-range dependency of data via self-attention also yields promising results for vision tasks (e.g., image classification, joint vision-language modeling, etc.).
Revised: Recent studies [10, 24, 28, 48] suggest that encoding the long-range dependency of data via self-attention also yields promising results for vision tasks (e.g., image classification, joint vision-language modeling, etc.).

Original: However, there is an important trade-off between leveraging the rich representation capacity and the lack of inductive bias.
Revised: However, there is an important trade-off between leveraging the rich representation capacity and the lack of inductive bias.

Original: Transformers have gained a reputation to generally require significantly more training data compared to convolutional neural networks (CNNs), since properties like translation invariance, locality and hierarchical structure of visual data have to be inferred from the data [26].
Revised: Transformers have gained a reputation to generally require significantly more training data compared to convolutional neural networks (CNNs), since properties like translation invariance, locality and hierarchical structure of visual data have to be inferred from the data [27].

Original: While this data-hungry nature largely prevented the use of Transformers in problems with scarce data like few-shot learning, some recent works demonstrate that a single
Revised: While this data-hungry nature largely prevented the use of Transformers in problems with scarce data like few-shot learning, some recent works demonstrate that a single

Original: Transformer head can be successfully adopted to perform feature adaptation [9, 53].
Revised: Transformer head can be successfully adopted to perform feature adaptation [9, 54].

Original: To the best of our knowledge, FewTURE is the first approach that uses a fully Transformer-based architecture to obtain representative embeddings while being trained exclusively on training data of the respective few-shot dataset.
Revised: To the best of our knowledge, FewTURE is the first approach that uses a fully Transformer-based architecture to obtain representative embeddings while being trained exclusively on training data of the respective few-shot dataset.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we presented a novel approach to tackle the challenge of supervision collapse introduced by one-hot image-level labels in few-shot learning.
Revised: In this paper, we presented a novel approach to tackle the challenge of supervision collapse introduced by one-hot image-level labels in few-shot learning.

Original: We split the input images into smaller parts with higher probability of being dominated by only one entity and encode these local regions using a Vision
Revised: We split the input images into smaller parts with higher probability of being dominated by only one entity and encode these local regions using a Vision

Original: Transformer architecture pretrained via Masked Image Modeling (MIM) in a self-supervised way to learn a representative embedding space beyond the pure label information.
Revised: Transformer architecture pretrained via Masked Image Modeling (MIM) in a self-supervised way to learn a representative embedding space beyond the pure label information.

Original: We devise a classifier based on patch embedding similarities and propose a token importance reweighting mechanism to refine the contribution of each local patch towards the overall classification result based around intra-class similarities and inter-class differences as a function of the support set information.
Revised: We devise a classifier based on patch embedding similarities and propose a token importance reweighting mechanism to refine the contribution of each local patch towards the overall classification result based around intra-class similarities and inter-class differences as a function of the support set information.

Original: Our obtained results demonstrate that our proposed method alleviates the problem of supervision collapse by learning more generalized features, achieving new state-of-the-art results on four popular few-shot classification datasets.
Revised: Our obtained results demonstrate that our proposed method alleviates the problem of supervision collapse by learning more generalized features, achieving new state-of-the-art results on four popular few-shot classification datasets.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: The authors would like to thank Zhou et al.
Revised: The authors would like to thank Zhou et al.

Original: [59] for sharing their insights and code regarding self-supervised pretraining, as well as Dosovitskiy et al.
Revised: [60] for sharing their insights and code regarding self-supervised pretraining, as well as Dosovitskiy et al.

Original: [10], Touvron et al.
Revised: [10], Touvron et al.

Original: [46] and
Revised: [47] and

Original: Liu et al.
Revised: Liu et al.

Original: [27] for sharing details of the ViT and Swin architectures.
Revised: [28] for sharing details of the ViT and Swin architectures.

Original: Parts of this research were undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne.
Revised: Parts of this research were undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne.

Original: This Facility was established with the assistance of LIEF Grant LE170100200.
Revised: This Facility was established with the assistance of LIEF Grant LE170100200.