Original: On the Utility of Active Instance Selection for Few-Shot Learning
Revised: On the Utility of Active Instance Selection for Few-Shot Learning

Original: Abstract
Revised: Abstract

Original: Few-shot learning aims to understand new concepts by transferring knowledge and utilizing very few randomly selected labeled samples.
Revised: Few-shot learning aims to understand new concepts by transferring knowledge and utilizing very few randomly selected labeled samples.

Original: Instead of selecting these instances randomly, active learning provides a promising alternative.
Revised: Instead of selecting these instances randomly, active learning provides a promising alternative.

Original: In this work, we investigate the effectiveness of actively identifying informative samples on the performance of few-shot learning models.
Revised: In this work, we investigate the effectiveness of actively identifying informative samples on the performance of few-shot learning models.

Original: We show that despite the fact that regular classiﬁcation tasks with larger amounts of labeled data beneﬁt from active learning approaches, these beneﬁts do not reliably generalize to the fewshot learning task.
Revised: We show that despite the fact that regular classiﬁcation tasks with larger amounts of labeled data beneﬁt from active learning approaches, these beneﬁts do not reliably generalize to the fewshot learning task.

Original: We characterize the best possible active few-shot learning performance, by introducing Single-Instance-Oracle and Batch-Oracle as active methods that assume access to labels of the unlabeled pool and the test set, and show via these “upper bounds” that we do not have a signiﬁcant room for improving few-shot models through actively selecting instances.
Revised: We characterize the best possible active few-shot learning performance, by introducing Single-Instance-Oracle and Batch-Oracle as active methods that assume access to labels of the unlabeled pool and the test set, and show via these “upper bounds” that we do not have a signiﬁcant room for improving few-shot models through actively selecting instances.

Original:  Introduction
Revised:  Introduction

Original: Deep learning models are accurate when large labeled datasets are available for training, however, adapting them to novel tasks requires effort in collecting labels; a signiﬁcant problem in domains where data are expensive to label.
Revised: Deep learning models are accurate when large labeled datasets are available for training, however, adapting them to novel tasks requires effort in collecting labels; a signiﬁcant problem in domains where data are expensive to label.

Original: Few-shot learning [20, 25] addresses this challenge by transferring the knowledge and reasoning learned from relevant datasets to the target evaluation dataset via a few shots of support instances.
Revised: Few-shot learning [20, 25] addresses this challenge by transferring the knowledge and reasoning learned from relevant datasets to the target evaluation dataset via a few shots of support instances.

Original: The goal of few-shot learning is to adapt models so that they can classify samples from unseen classes during training time, provided a small number of labeled instances from those classes [16, 5, 17, 2].
Revised: The goal of few-shot learning is to adapt models so that they can classify samples from unseen classes during training time, provided a small number of labeled instances from those classes [16, 5, 17, 2].

Original: Since the number of labeled instances from these unseen classes is small the role of an individual labeled instance dramatically increases compared to the usual supervised learning setting.
Revised: Since the number of labeled instances from these unseen classes is small the role of an individual labeled instance dramatically increases compared to the usual supervised learning setting.

Original: Although previous approaches demonstrate tremendous success in transferring the knowledge, they disregard the possible impact of cautiously selecting those very few support instances.
Revised: Although previous approaches demonstrate tremendous success in transferring the knowledge, they disregard the possible impact of cautiously selecting those very few support instances.

Original: The question that remains is whether it is possible to improve few-shot models performance with fewer support instances if they are selected carefully.
Revised: The question that remains is whether it is possible to improve few-shot models performance with fewer support instances if they are selected carefully.

Original: Active
Revised: Active

Original: learning (AL) [19, 7, 8] as an approach to iteratively select the most informative instances for the model to continuously train on, improving accuracy with a very few labeled instances, shares the same goal as few-shot learning, i.e., adopting models to a new task with a few samples.
Revised: learning (AL) [19, 7, 8] as an approach to iteratively select the most informative instances for the model to continuously train on, improving accuracy with a very few labeled instances, shares the same goal as few-shot learning, i.e., adopting models to a new task with a few samples.

Original: In spite of the attention AL methods have received, they mostly provide modest beneﬁts for classiﬁcation tasks [4, 13].
Revised: In spite of the attention AL methods have received, they mostly provide modest beneﬁts for classiﬁcation tasks [4, 13].

Original: Our goal here is to explore the utility of actively selecting support instances on few-shot learning across models and datasets by studying current practices and characterizing the possible extent to which active learning can help few-shot models.
Revised: Our goal here is to explore the utility of actively selecting support instances on few-shot learning across models and datasets by studying current practices and characterizing the possible extent to which active learning can help few-shot models.

Original: In this paper, we study the effect of combining active learning algorithms with few-shot learning by focusing on active few-shot learning .
Revised: In this paper, we study the effect of combining active learning algorithms with few-shot learning by focusing on active few-shot learning .

Original: Speciﬁcally, we are interested in studying the effect of actively selecting the support set instances during the inference time of few-shot learning.
Revised: Speciﬁcally, we are interested in studying the effect of actively selecting the support set instances during the inference time of few-shot learning.

Original: Considering several existing active learning approaches, such as uncertainty based and Coreset methods [18], we evaluate the performance of two recent few-shot learning models, Prototypical Networks [20] and Simple
Revised: Considering several existing active learning approaches, such as uncertainty based and Coreset methods [18], we evaluate the performance of two recent few-shot learning models, Prototypical Networks [20] and Simple

Original: CNAPS [2] on in-domain and out-of-domain scenarios, respectively.
Revised: CNAPS [2] on in-domain and out-of-domain scenarios, respectively.

Original: Further, to characterize the possible beneﬁt of active learning impact on few-shot models, assuming full access to labels of the unlabeled pool and the test set, we introduce Single-Instance-Oracle and Batch-Oracle active learning methods, by identifying instances that maximize the test set accuracy of the model when added to the support set.
Revised: Further, to characterize the possible beneﬁt of active learning impact on few-shot models, assuming full access to labels of the unlabeled pool and the test set, we introduce Single-Instance-Oracle and Batch-Oracle active learning methods, by identifying instances that maximize the test set accuracy of the model when added to the support set.

Original: Since we only focus on the impact of active learning on few-shot models during the inference time, we can calculate the impact of selecting a new instance on the test set accuracy by having a forward pass; in contrast to regular classiﬁcation tasks which require to retrain the model.
Revised: Since we only focus on the impact of active learning on few-shot models during the inference time, we can calculate the impact of selecting a new instance on the test set accuracy by performing a forward pass; in contrast to regular classiﬁcation that requires retraining.

Original: We evaluate the performance of active few-shot learning on CIFAR10, CIFAR100, and SVHN datasets, demonstrating the incapability of current active learning techniques to effectively and consistently improve few-shot models’ performance.
Revised: We evaluate the performance of active few-shot learning on CIFAR10, CIFAR100, and SVHN datasets, demonstrating the inability of current active learning techniques to effectively and consistently improve few-shot model performance.

Original: Further, upon achieving the upper bounds of the best possible performance for active few-shot learning task through our Single-Instance-Oracle method, we show, (1) the room for improvement on active few-shot learning, depending on the benchmark and the model, may not be substantial considering the overall performance and number of labeled samples, and (2) more accurate few-shot learning models appear to beneﬁt less from actively selecting instances.
Revised: Further, upon achieving the upper bounds of the best possible performance for active few-shot learning task through our Single-Instance-Oracle, we show, (1) the room for improvement on active few-shot learning, depending on the benchmark and the model, may not be substantial considering the overall performance and number of labeled samples, and (2) more accurate few-shot learning models appear to beneﬁt less from actively selecting instances.

Original: Finally, studying active few-shot learning when multiple instances are selected, by introducing Batch-Oracle method, we shed some light on possible future directions.
Revised: Finally, studying active few-shot learning when multiple instances are selected, by introducing Batch-Oracle, we shed some light on possible future directions.

Original:  Problem Setup
Revised:  Problem Setup

Original: In this section, we introduce active and few-shot learning, setting up notations and relevant background for the remaining of the paper.
Revised: In this section, we introduce active and few-shot learning, setting up notations and relevant background for the remaining of the paper.

Original: Few-Shot Learning Inthe standard few-shot learning setup , we assume we have a large collection of instancesD
Revised: Few-Shot Learning In standard few-shot learning, we assume we have a large collection of instances

Original:  = { ( x i , y i ) } .
Revised: D = { ( x i , y i ) } .

Original: From this dataset, we build separate classiﬁcation tasks D T ⊂ D by randomly selecting data points without replacement.
Revised: From this dataset, we build separate classiﬁcation tasks D T ⊂ D by randomly selecting data points without replacement.

Original: Each D T (episode) consists of two separate sets, support set S T and query set Q T in a way that query set labels are a subset of support set labels.
Revised: Each D T (episode) consists of two separate sets, support set S T and query set Q T in a way that query set labels are a subset of support set labels.

Original: As a result, we can frame few-shot learning for each D T as correctly assigning each data point x i in the query set Q T to one of the classes appears in S T .
Revised: As a result, we can frame few-shot learning for each D T as correctly assigning each data point x i in the query set Q T to one of the classes appears in S T .

Original: Existing few-shot learning methods mostly consist of two phases, a training and a testing phase by assigning separate sets of D T to each one of them in a way that test classes never appear in the training phase.
Revised: Existing few-shot learning methods mostly consist of two phases, a training and a testing phase by assigning separate sets of D T to each one of them in a way that test classes never appear in the training phase.

Original: In this work, we focus on Prototypical
Revised: In this work, we focus on Prototypical

Original: Networks [20] and Simple CNAPS [2]as representative models for in-domain and out-of-domain few-shot scenarios.
Revised: Networks [20] and Simple CNAPS [2] for in-domain and out-of-domain few-shot scenarios.

Original: In addition to these models, there is a family of few-shot models that try to adapt the classiﬁer to the unseen classes using some form of nearest neighbors [24, 9], which we leave tothe future work.
Revised: We leave the family of few-shot models that use nearest neighbors [24, 9] to future work.

Original: Active Learning In active learning, we are interested in learning a classiﬁer that predicts a class from a set of labels y ∈ C for any an input x ⊆ R d , trained on a large set of data point pairs ( x i , y i ) .
Revised: Active Learning In active learning, we are interested in learning a classiﬁer that predicts a class from a set of labels y ∈ C for any an input x ⊆ R d , trained on a large set of data point pairs ( x i , y i ) .

Original: In this paper, we restrict our classiﬁer to be a neural network represented as ˆ y i =
Revised: In this paper, we restrict our classiﬁer to be a neural network represented as ˆ y i =

Original: σ ( φ ( x i , θ )) for input instance x i , where θ are parameters of the whole model, σ the softmax/sigmoid function, and the logits φ ( x i , θ ) come from a linear layer, i.e. φ ( x i , θ )
Revised: σ ( φ ( x i , θ )) for input instance x i , where θ are parameters of the whole model, σ the softmax/sigmoid function, and the logits φ ( x i , θ ) come from a linear layer, i.e. φ ( x i , θ )

Original: =
Revised: =

Original: θ l f i ; θ l are the parameters of ﬁnal linear layer and f i is a feature vector extracted from x i .
Revised: θ l f i ; θ l are the parameters of ﬁnal linear layer and f i is a feature vector extracted from x i .

Original: At each iteration t , active learning method has access to the current model ( θ t ), a set of labeled instances S t (contains pairs of x and y ), and a large pool of unlabeled instances U t (containing only x ).
Revised: At each iteration t , active learning method has access to the current model ( θ t ), a set of labeled instances S t (contains pairs of x and y ), and a large pool of unlabeled instances U t (containing only x ).

Original: In the batch setting, given a budget b , the goal is to identify b instances from the pool U that our active learning approach estimates, when they are labeled by an oracle, to be the most informative instances for the current classiﬁer.
Revised: In the batch setting, given a budget b , the goal is to identify b instances from the pool U that our active learning approach estimates, when they are labeled by an oracle, to be the most informative instances for the current classiﬁer.

Original: Then, after querying the oracle for the labels and adding them to S t , we retrain the classiﬁer on the new training data, and continue the next iteration.
Revised: Then, after querying the oracle for the labels and adding them to S t , we retrain the classiﬁer on the new training data, and continue the next iteration.

Original: Although in this work we focus on several uncertainty based and
Revised: Although in this work we focus on several uncertainty based and

Original: Coreset [18] AL methods, there are many other AL approaches that utlize the notion of uncertainty [21, 23, 18, 1, 27], identify instances that are not distinguishable from the pool of unlabeled samples by treating AL as a binary classiﬁer [10] and incorporate the gradient of model [1].
Revised: Coreset [18] AL methods, there are many other AL approaches that utlize the notion of uncertainty [21, 23, 18, 1, 27], identify instances that are not distinguishable from the pool of unlabeled samples by treating AL as a binary classiﬁer [10] and incorporate the gradient of model [1].

Original: Active Few-Shot Learning We introduce active few-shot learning as actively seeking the most informative unlabeled instance to add to the support set during the testing phase.
Revised: introduce active few-shot learning as actively seeking the most informative unlabeled instance to add to the support set during the testing phase.

Original: The training phase is identical to the conventional few-shot learning task.
Revised: The training phase is identical to the conventional few-shot learning task.

Original: We use ﬁve episodes in our setup during the testing phase, and use the entire test set for each class as the query set.
Revised: We use ﬁve episodes in our setup during the testing phase, and use the entire test set for each class as the query set.

Original: Randomly choosing the classes to appear in these episodes, we gather all instances with those labels in the data and divide them equally into two sets: the pool that we choose the support images actively from, denoted as U T , and the pool that we consider as the query set in the testing phase ( Q T ).
Revised: Randomly choosing the classes to appear in these episodes, we gather all instances with those labels in the data and divide them equally into two sets: the pool that we choose the support images actively from, denoted as U T , and the pool that we consider as the query set in the testing phase ( Q T ).

Original: Our goal is to build S T from U T , in the testing phase, by choosing the most informative samples given the learned few-shot model.
Revised: Our goal is to build S T from U T , in the testing phase, by choosing the most informative samples given the learned few-shot model.

Original: Although, to the best of our knowledge, we are the ﬁrst to introduce active fewshot learning as described, the notion of active few-shot learning has been used in a few previous works.
Revised: Although, to the best of our knowledge, we are the ﬁrst to introduce active few- shot learning as described, the notion of active few-shot learning has been used in a few previous works.

Original: Woodward and Finn [26] combines reinforcement learning with one-shot learning to identify potentially informative samples to label.
Revised: Woodward and Finn [26] combines reinforcement learning with one-shot learning to identify potentially informative samples to label.

Original: Moreover, the authors in [3] utilizes K-means clustering on the Prototypical Networks to better adapt few-shot models to new tasks.
Revised: Moreover, the authors in [3] utilizes K-means clustering on the Prototypical Networks to better adapt few-shot models to new tasks.

Original: Finally, [6] tries to resolve task ambiguity when dealing with small amounts of data in few-shot learning by adopting
Revised: Finally, [6] tries to resolve task ambiguity when dealing with small amounts of data in few-shot learning by adopting

Original: MAML [5] model and injecting noise into gradient descent at meta-test time to develop a probabilistic meta-learning approach.
Revised: MAML [5] model and injecting noise into gradient descent at meta-test time to develop a probabilistic meta-learning approach.

Original:  Active and Few-shot Learning models
Revised:  Active and Few-shot Learning models

Original: Here we elaborate on different active and few-shot learning methods that we consider in this work.
Revised: Here we elaborate on different active and few-shot learning methods that we consider in this work.

Original: 3.1 Few-Shot Learning Models
Revised: 3.1 Few-Shot Learning Models

Original: To study the effect of active learning on the performance of few-shot models, we consider two models: 1) Prototypical Networks [20] because of its simplicity and popularity, and 2) Simple CNAPS [2] because of its state-of-the-art results for two separate few-shot scenarios, in-domain and out-ofdomain respectively.
Revised: To study the effect of active learning on the performance of few-shot models, we consider two models: 1) Prototypical Networks [20] because of its simplicity and popularity, and 2) Simple CNAPS [2] because of its state-of-the-art results for two separate few-shot scenarios, in-domain and out-ofdomain respectively.

Original: In-domain represents scenarios in which we train and test a few-shot model on the same dataset dividing classes for training and testing purposes.
Revised: In-domain represents scenarios in which we train and test a few-shot model on the same dataset dividing classes for training and testing purposes.

Original: On the other hand, out-of-domain represents scenarios in which we consider separate datasets for training and testing respectively.
Revised: On the other hand, out-of-domain represents scenarios in which we consider separate datasets for training and testing respectively.

Original: This methodology of evaluating a few-shot model becomes more widespread upon creation of the
Revised: This methodology of evaluating a few-shot model becomes more widespread upon creation of the

Original: Meta-Dataset [22] which is a benchmark containing 10 different labeled image datasets.
Revised: Meta-Dataset [22] which is a benchmark containing 10 different labeled image datasets.

Original: Prototypical Networks Prototypical networks [20] classify each sample after calculating its feature vector through two steps: 1) computing the mean embedding or prototype for each class by averaging the embeddings of the samples of that class in the support set S T , and 2) classifying the samples in the query set Q T by calculating the similarity between each prototype and query samples’ embedding.
Revised: Prototypical Networks Prototypical networks [20] classify each sample after calculating its feature vector through two steps: 1) computing the mean embedding or prototype for each class by averaging the embeddings of the samples of that class in the support set S T , and 2) classifying the samples in the query set Q T by calculating the similarity between each prototype and query samples’ embedding.

Original: The similarity here is deﬁned as a negative of Euclidean distance.
Revised: The similarity here is deﬁned as a negative of Euclidean distance.

Original: Simple CNAPS Simple CNAPS [2] is built on top of CNAPS [17] which uses FiLM layers [15] to better adapt the representations to unseen classes during the testing phase.
Revised: Simple CNAPS Simple CNAPS [2] is built on top of CNAPS [17] which uses FiLM layers [15] to better adapt the representations to unseen classes during the testing phase.

Original: More speciﬁcally, CNAPS classify the samples by ﬁrstly extracting the feature vector using ResNet18 [11] and FiLM layers, and then calculating the class probability using a linear classiﬁer.
Revised: More speciﬁcally, CNAPS classify the samples by ﬁrstly extracting the feature vector using ResNet18 [11] and FiLM layers, and then calculating the class probability using a linear classiﬁer.

Original: The only difference between CNAPS and Simple CNAPS is incorporating the Mahalanobis distance instead of the linear classiﬁer element.
Revised: The only difference between CNAPS and Simple CNAPS is incorporating the Mahalanobis distance instead of the linear classiﬁer element.

Original: 3.2 Active Learning Models
Revised: 3.2 Active Learning Models

Original: Random: is a baseline that samples instances randomly in each iteration.
Revised: Random: is a baseline that samples instances randomly in each iteration.

Original: Uncertainty Based Sampling:
Revised: Uncertainty Based Sampling:

Original: We consider 3 uncertainty based approaches in this work: 1) entropy approach selecting samples by maximizing the entropy of class distributions based on classiﬁer output for each unlabeled sample, 2) least conﬁdence identifying instances by minimizing the highest probability class for each unlabeled sample, and 3) margin of conﬁdence which chooses the instances by minimizing the difference between ﬁrst and second highest class possibility.
Revised: We consider 3 uncertainty based approaches in this work: 1) entropy approach selecting samples by maximizing the entropy of class distributions based on classiﬁer output for each unlabeled sample, 2) least conﬁdence identifying instances by minimizing the highest probability class for each unlabeled sample, and 3) margin of conﬁdence which chooses the instances by minimizing the difference between ﬁrst and second highest class possibility.

Original: Coreset [18]: selecting instances by using their representation from the last layer in DNN to solve the k-center problem, i.e., choosing b (our budget) centers in such a way that minimized the largest distance between a data point and its nearest center.
Revised: Coreset [18]: selecting instances by using their representation from the last layer in DNN to solve the k-center problem, i.e., choosing b (our budget) centers in such a way that minimized the largest distance between a data point and its nearest center.

Original: Modiﬁed Coreset:
Revised: Modiﬁed Coreset:

Original: Considering that simple CNAPS uses Mahalanobis distance to classify instances, we introduce the Modiﬁed Coreset using Mahalanobis distance instead of Euclidean as a measure of distance in Coreset approach.[Figure]
Revised: Considering that simple CNAPS uses Mahalanobis distance to classify instances, we introduce the Modiﬁed Coreset using Mahalanobis distance instead of Euclidean as a measure of distance in Coreset approach.

Original: 3.3 Oracle Based Active learning
Revised: 3.3 Oracle Based Active learning

Original: In this section, to characterize the possible extent of active learning impact on the few-shot model, we ﬁrst introduce an oracle based single instance selection approach.
Revised: In this section, to characterize the possible extent of active learning impact on the few-shot model, we ﬁrst introduce an oracle based single instance selection approach.

Original: Then, conjecturing that single selection might impose a considerable limitation on the achievable upper bound, we provide a batch version of this method.
Revised: Then, conjecturing that single selection might impose a considerable limitation on the achievable upper bound, we provide a batch version of this method.

Original:  Single-Instance-Oracle: To better grasp the possible effect of active learning on few-shot models, we introduce a Single-Instance-Oracle method: assuming to have full access to the label of all samples and the test set, we sample the most informative instance at each iteration by identifying the sample that maximizes the model accuracy on the test set upon adding to the support set.
Revised: [Figure] Single-Instance-Oracle: To better grasp the possible effect of active learning on few-shot models, we introduce a Single-Instance-Oracle method: assuming to have full access to the label of all samples and the test set, we sample the most informative instance at each iteration by identifying the sample that maximizes the model accuracy on the test set upon adding to the support set.

Original: Batch-Oracle: We deﬁne a batch version of our Single-Instance-Oracle, by keeping top K’s most informative samples from the oracle algorithm perspective at each iteration in a beam.
Revised: Batch-Oracle: We deﬁne a batch version of our Single-Instance-Oracle, by keeping top K’s most informative samples from the oracle algorithm perspective at each iteration in a beam.

Original: These K instances result in K different models (adding each one of the samples separately to the current support set).
Revised: These K instances result in K different models (adding each one of the samples separately to the current support set).

Original: In the next iteration, we sort the informativeness of unlabeled instances according to these K models and then choose the top K most informative samples from the accumulated set.
Revised: In the next iteration, we sort the informativeness of unlabeled instances according to these K models and then choose the top K most informative samples from the accumulated set.

Original: Challenges and Utility: The biggest challenge in calculating Single-Instance-Oracle and
Revised: Challenges and Utility: The biggest challenge in calculating Single-Instance-Oracle and

Original: Batch-Oracle is the computational complexity.
Revised: Batch-Oracle is the computational complexity.

Original: Since at each iteration, we need to calculate the model accuracy on the test set upon adding each unlabeled sample separately, identifying the most informative samples even in a very limited label regime takes an order of magnitude longer than other methods.
Revised: Since at each iteration, we need to calculate the model accuracy on the test set upon adding each unlabeled sample separately, identifying the most informative samples even in a very limited label regime takes an order of magnitude longer than other methods.

Original: Just to note that, this approach is only applicable to few-shot learning since we only focus on the impact of active learning during the inference time.
Revised: Just to note that, this approach is only applicable to few-shot learning since we only focus on the impact of active learning during the inference time.

Original: We can calculate the impact of selecting a new instance on the test set accuracy by having a forward pass of the learned model; in contrast to regular classiﬁcation tasks which require to retrain the model for calculating this impact.
Revised: We can calculate the impact of selecting a new instance on the test set accuracy by having a forward pass of the learned model; in contrast to regular classiﬁcation tasks which require to retrain the model for calculating this impact.

Original:  Experiments
Revised:  Experiments

Original: Here, we ﬁrst describe our benchmarks and implementation details.
Revised: Here, we ﬁrst describe our benchmarks and implementation details.

Original: Then, we demonstrate the incapability of active learning approaches to consistently improve few-shot models in the in-domain setting.
Revised: Then, we demonstrate the incapability of active learning approaches to consistently improve few-shot models in the in-domain setting.

Original: Finally, studying the out-of-domain scenario, we shed some light on future directions.
Revised: Finally, studying the out-of-domain scenario, we shed some light on future directions.

Original: Setup To evaluate our method, we conduct several experiments on widely used image datasets
Revised: Setup To evaluate our method, we conduct several experiments on widely used image datasets

Original: CIFAR-10, CIFAR-100, SVHN.
Revised: CIFAR-10, CIFAR-100, SVHN.

Original: Both CIFAR-10 and CIFAR-100 [12] have 60 K images of size 32 × 32 with 10 and 100 classes receptively.
Revised: Both CIFAR-10 and CIFAR-100 [12] have 60 K images of size 32 × 32 with 10 and 100 classes receptively.

Original: SVHN [14] contains 32 × 32 street view house number images with 10 labels.
Revised: SVHN [14] contains 32 × 32 street view house number images with 10 labels.

Original: We trained all algorithms using the same optimization and loss, i.e., AdaGrad and the cross-entropy loss.
Revised: We trained all algorithms using the same optimization and loss, i.e., AdaGrad and the cross-entropy loss.

Original: We use validation data to tune the hyper-parameters and utilize a grid search to ﬁnd the best hyper-parameters, such as learning rate, and regularization parameter.
Revised: We use validation data to tune the hyper-parameters and utilize a grid search to ﬁnd the best hyper-parameters, such as learning rate, and regularization parameter.

Original: In all of our experiments, we report the resulted accuracy averaged over 10 runs with different random seed.[Figure] [Figure]
Revised: In all of our experiments, we report the resulted accuracy averaged over 10 runs with different random seed.

Original: 4.1 Impact of Active Selection on Few-Shot Learning
Revised: 4.1 Impact of Active Selection on Few-Shot Learning

Original: In this section we study the effect of active learning for two families of methods; 1) methods that sample instances independent of each other, i.e., estimating the effect of adding a single point to training data on the model (we consider the uncertainty based models as representative of this family), and 2) methods that adopt batch-wise strategies as their sampling procedure, i.e., selecting data points estimating the effect of a batch of samples together (we consider Coreset as an example of such). 
Revised: In this section we study the effect of active learning for two families of methods; 1) methods that sample instances independent of each other, i.e., estimating the effect of adding a single point to training data on the model (we consider the uncertainty based models as representative of this family), and 2) methods that adopt batch-wise strategies as their sampling procedure, i.e., selecting data points estimating the effect of a batch of samples together (we consider Coreset as an example of such). [Figure] [Figure]

Original: In-domain: We introduce active few-shot learning task, by actively choosing support images in the test phase.
Revised: In-domain: We introduce active few-shot learning task, by actively choosing support images in the test phase.

Original: Speciﬁcity we deﬁne support and query images for different episodes during training using previously introduced instructions [24, 22].
Revised: Speciﬁcity we deﬁne support and query images for different episodes during training using previously introduced instructions [24, 22].

Original: For the test phase query images of unseen classes, we consider all images with these labels from the test data, i.e., test set of CIFAR-10, CIFAR-100, and SVHN datasets.
Revised: For the test phase query images of unseen classes, we consider all images with these labels from the test data, i.e., test set of CIFAR-10, CIFAR-100, and SVHN datasets.

Original: As for sampling the classes for the test phase, since we only consider one episode during the test phase, for CIFAR-10 and SVHN we randomly choose two of the classes as the unseen ones, and for CIFAR-100 we randomly choose 10 classes (use the rest of classes for training the model).
Revised: As for sampling the classes for the test phase, since we only consider one episode during the test phase, for CIFAR-10 and SVHN we randomly choose two of the classes as the unseen ones, and for CIFAR-100 we randomly choose 10 classes (use the rest of classes for training the model).

Original: Moreover, for the test phase support images, we start with one random image for each label, and then increase the support images in each iteration using an active learning approach.
Revised: Moreover, for the test phase support images, we start with one random image for each label, and then increase the support images in each iteration using an active learning approach.

Original: We train CIFAR-10 and SVHN in 5 shot-5 way manner, while for the CIFAR-100 datasets we train few-shot model in a 5 shot-10 way setting.
Revised: We train CIFAR-10 and SVHN in 5 shot-5 way manner, while for the CIFAR-100 datasets we train few-shot model in a 5 shot-10 way setting.

Original: The average result of 10 runs over in-domain few-shot learning benchmarks is shown in Figures 1a, 2a and 3a.
Revised: The average result of 10 runs over in-domain few-shot learning benchmarks is shown in Figures 1a, 2a and 3a.

Original: We can derive two conclusions from these results: 1) current active learning methods do not show sufﬁcient improvements for few-shot learning, 2) even Single-Instance-Oracle does not consistently improve the performance of the model.
Revised: We can derive two conclusions from these results: 1) current active learning methods do not show sufﬁcient improvements for few-shot learning, 2) even Single-Instance-Oracle does not consistently improve the performance of the model.

Original: Finally, the Prototypical networks using all possible labels as part of the support set achieves 76 . 4 , 27 . 7 , and 71 . 1 accuracy on CIFAR-10, CIFAR-100, and SVHN benchmarks which is very close to
Revised: Finally, the Prototypical networks using all possible labels as part of the support set achieves 76 . 4 , 27 . 7 , and 71 . 1 accuracy on CIFAR-10, CIFAR-100, and SVHN benchmarks which is very close to

Original: Single-Instance-Oracle end point accuracy in each plot.
Revised: Single-Instance-Oracle end point accuracy in each plot.

Original: [Figure] Out-of-domain: As our few-shot learning model in the out-of-domain scenario we use pretrained weights of simple CNAPS [2] model.
Revised: Out-of-domain: As our few-shot learning model in the out-of-domain scenario we use pretrained weights of simple CNAPS [2] model.

Original: For the test phase query images, we adopt a similar approach to our previously described in-domain scenario with one major difference, i.e., considering all classes of CIFAR-10, CIAFR-100, and SVHN during the testing phase.
Revised: For the test phase query images, we adopt a similar approach to our previously described in-domain scenario with one major difference, i.e., considering all classes of CIFAR-10, CIAFR-100, and SVHN during the testing phase.

Original: The result of few-shot learning benchmarks in the out-of-domain scenario is shown in Figures 1b, 2b and 3b.
Revised: The result of few-shot learning benchmarks in the out-of-domain scenario is shown in Figures 1b, 2b and 3b.

Original: These results further validate our previous conclusion on the incapability of current active learning models in improving the few-shot learning task.
Revised: These results further validate our previous conclusion on the incapability of current active learning models in improving the few-shot learning task.

Original: Considering the gap between Single-Instance-Oracle and random active sampling, it appears the more accurate few-shot models, (simple CNPAS comparing to Prototypical 
Revised: Considering the gap between Single-Instance-Oracle and random active sampling, it appears the more accurate few-shot models, (simple CNPAS comparing to Prototypical [Figure]

Original:  Networks), are less promising in active few-shot task.
Revised: We Networks), are less promising in active few-shot task.

Original: Moreover, the poor performance of Coreset and Modiﬁed Coreset demonstrates the inherently different nature of active few-shot learning task in comparison to regular active learning.
Revised: Moreover, the poor performance of Coreset and Modiﬁed Coreset demonstrates the inherently different nature of active few-shot learning task in comparison to regular active learning.

Original: Finally, the Simple CNAPS using all possible labels as part of the support set achieves 73 . 2 , 47 .
Revised: Finally, the Simple CNAPS using all possible labels as part of the support set achieves 73 . 2 , 47 .

Original: 4 , and 44 .
Revised: 4 , and 44 .

Original: 2 accuracy on CIFAR-10, CIFAR-100, and SVHN benchmarks which is very close to Single-Instance-Oracle end point accuracy in each plot.
Revised: 2 accuracy on CIFAR-10, CIFAR-100, and SVHN benchmarks which is very close to Single-Instance-Oracle end point accuracy in each plot.

Original: Although current active learning methods appear poorly in both in-domain and out-of-domain scenarios,for each one of the benchmarks the batch-wise strategy shows a possible room for improvement (Coreset on in-domainSVHN, and Modiﬁed Coreset on out-of-domain CIFAR-10 and
Revised: Although current active learning methods perform poorly in both in-domain and out-of-domain scenarios, batch-wise strategies indicates there is some room for improvement (Coreset on in-domain

Original:  CIFAR-100, especially when dealing with fewer samples).
Revised: SVHN, and Modiﬁed Coreset on out-of-domain CIFAR-10 and CIFAR-100, especially when dealing with fewer samples).

Original: Now the only question that remains is if we can possibly improve the active few-shot performance upon introducing more effective batch-wise strategies.
Revised: A natural question that arises is whether we can quantify how much more can picking multiple instances in a batch can beneﬁt, over the Single-Instance-Oracle.

Original: 4.2 Effect of Batch-wise Sampling on Active Few-Shot
Revised: 4.2 Effect of Batch-wise Sampling on Active Few-Shot

Original: In this section, we compare our Single-Instance-Oracle and Batch-Oracle methods, to clarify if it is possible to further improve model performance on introducing better batch-wise strategies.
Revised: In this section, we compare our Single-Instance-Oracle and Batch-Oracle methods, to clarify if it is possible to further improve model performance on introducing better batch-wise strategies.

Original: We only consider small batches ( K = 2 ) because of the excessive computational cost in Batch-Oracle setting and evaluate our methods for the in-domain CIFAR-10 and SVHN benchmarks.
Revised: We only consider small batches ( K = 2 ) because of the excessive computational cost in Batch-Oracle setting and evaluate our methods for the in-domain CIFAR-10 and SVHN benchmarks.

Original: The average result of Batch-Oracle performance is provided in Figure 4.
Revised: The average result of Batch-Oracle performance is provided in Figure 4.

Original: As it shows, incorporating a beam search on top of the Single-Instance-Oracle method improves the model performance, showing room for improvement of active few-shot methods by adopting better batch-wise active learning algorithms.
Revised: As it shows, incorporating a beam search on top of the Single-Instance-Oracle method improves the model performance, showing room for improvement of active few-shot methods by adopting better batch-wise active learning algorithms.

Original: 4.3 Relation between Oracle Selection and Uncertainty Measures
Revised: 4.3 Relation between Oracle Selection and Uncertainty Measures

Original: To understand to what degree the uncertainty measures reﬂect the points selected by the Oracle, we compute the correlation between them on the ranking over the unlabeled samples.
Revised: To understand to what degree the uncertainty measures reﬂect the points selected by the Oracle, we compute the correlation between them on the ranking over the unlabeled samples.

Original: For this evaluation, we start with 5 random samples for each class in the support set and then rank the unlabeled samples based on each method during the inference time.
Revised: For this evaluation, we start with 5 random samples for each class in the support set and then rank the unlabeled samples based on each method during the inference time.

Original: Spearman correlation coefﬁcients between oracle selection and uncertainty measures are provided in Table 1.
Revised: Spearman correlation coefﬁcients between oracle selection and uncertainty measures are provided in Table 1.

Original: For both Simple CNAPS and Prototypical networks models, uncertainty measures do not seem to be correlated with Oracle selection, further emphasizing the need for introducing an active learning approach that goes beyond the notion of uncertainty.
Revised: For both Simple CNAPS and Prototypical networks models, uncertainty measures do not seem to be correlated with Oracle selection, further emphasizing the need for introducing an active learning approach that goes beyond the notion of uncertainty.

Original: Moreover, Prototypical networks show a weaker correlation, which is consistent with the bigger accuracy gap we observe between uncertainty methods and Single-Instance-Oracle selection, compared to Simple CNAPS.[Table]
Revised: Moreover, Prototypical networks show a weaker correlation, which is consistent with the bigger accuracy gap we observe between uncertainty methods and Single-Instance-Oracle selection, compared to Simple CNAPS.

Original:  Conclusions
Revised:  Conclusions

Original: Motivated by the need for actively selecting support set images in few-shot learning, we study the effect of active learning on few-shot models.
Revised: Motivated by the need for actively selecting support set images in few-shot learning, we study the effect of active learning on few-shot models.

Original: Establishing the upper bounds of the best possible active few-shot learning performance by introducing Single-Instance-Oracle, with the assumption of full access to the test set and the labels of unlabeled samples pool, we show: 1) current active learning methods cannot improve few-shot models substantially in comparison to random sampling, 2) even Single-Instance-Oracle method fall short in improving the few-shot models’ performance with a considerable gap.
Revised: Establishing the upper bounds of the best possible active few-shot learning performance by introducing Single-Instance-Oracle, with the assumption of full access to the test set and the labels of unlabeled samples pool, we show: 1) current active learning methods cannot improve few-shot models substantially in comparison to random sampling, [Table] even Single -Instance-Oracle

Revised: method fall short in improving the few-shot models’ performance with a considerable gap.

Original: Further, studying the effect of batch-wise active sampling by introducing
Revised: Further, studying the effect of batch-wise active sampling by introducing

Original: Batch-Oracle, we concluded a possible room for improvement upon introducing more effective batch-wise selection.
Revised: Batch Oracle, we concluded a possible room for improvement upon introducing more effective batch wise selection.

Original: In future work, for active learning to substantially improve upon simple baselines like random and uncertainty sampling, it is essential to introduce new batch-wise active learning methods that utilize the unique characteristics of the few-shot learning tasks.
Revised: In future work, for active learning to substantially improve upon simple baselines like random and uncertainty sampling, it is essential to introduce new batch-wise active learning methods that utilize the unique characteristics of the few-shot learning tasks.

Original: Although in this work we deﬁne active few-shot learning as actively selecting support set instances during the testing phase, other varieties of this problem, such as actively identifying informative instances for support and query set during the training time, remain open questions to pursue in future work.
Revised: Although in this work we deﬁne active few-shot learning as actively selecting support set instances during the testing phase, other varieties of this problem, such as actively identifying informative instances for support and query set during the training time, remain open questions to pursue in future work.

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to thank the anonymous reviewers for their detailed feedback and suggestions.
Revised: We would like to thank the anonymous reviewers for their detailed feedback and suggestions.

Original: This work is sponsored in part by the Air Force Research Laboratory (AFRL), and DARPA, and funded in part by the NSF award #IIS-1756023.
Revised: This work is sponsored in part by the Air Force Research Laboratory (AFRL), and DARPA, and funded in part by the NSF award #IIS-1756023.

Original: The views expressed are those of the authors and do not reﬂect the ofﬁcial policy or position of the funding agencies.
Revised: The views expressed are those of the authors and do not reﬂect the ofﬁcial policy or position of the funding agencies.