Original: Improving Task-Specific Generalization in Few-Shot Learning via Adaptive Vicinal Risk Minimization
Revised: Improving Task-Specific Generalization in Few-Shot Learning via Adaptive Vicinal Risk Minimization

Original: Abstract
Revised: Abstract

Original: Recent years have witnessed the rapid development of meta-learning in improving the meta generalization over tasks in few-shot learning.
Revised: Recent years have witnessed the rapid development of meta-learning in improving the meta generalization over tasks in few-shot learning.

Original: However, the task-specific level generalization is overlooked in most algorithms.
Revised: However, the task-specific level generalization is overlooked in most algorithms.

Original: For a novel few-shot learning task where the empirical distribution likely deviates from the true distribution, the model obtained via minimizing the empirical loss can hardly generalize to unseen data.
Revised: For a novel few-shot learning task where the empirical distribution likely deviates from the true distribution, the model obtained via minimizing the empirical loss can hardly generalize to unseen data.

Original: A viable solution to improving the generalization comes as a more accurate approximation of the true distribution; that is, admitting a Gaussian-like vicinal distribution for each of the limited training samples.
Revised: A viable solution to improving the generalization comes as a more accurate approximation of the true distribution; that is, admitting a Gaussian-like vicinal distribution for each of the limited training samples.

Original: Thereupon we derive the resulting vicinal loss function over vicinities of all training samples and minimize it instead of the conventional empirical loss over training samples only, favorably free from the exhaustive sampling of all vicinal samples.
Revised: Thereupon we derive the resulting vicinal loss function over vicinities of all training samples and minimize it instead of the conventional empirical loss over training samples only, favorably free from the exhaustive sampling of all vicinal samples.

Original: It remains challenging to obtain the statistical parameters of the vicinal distribution for each sample.
Revised: It remains challenging to obtain the statistical parameters of the vicinal distribution for each sample.

Original: To tackle this challenge, we further propose to estimate the statistical parameters as the weighted mean and variance of a set of unlabeled data it passed by a random walk starting from training samples.
Revised: To tackle this challenge, we further propose to estimate the statistical parameters as the weighted mean and variance of a set of unlabeled data it passed by a random walk starting from training samples.

Original: To verify the performance of the proposed method, we conduct experiments on three standard few-shot learning benchmarks and consolidate the superiority of the proposed method over state-of-the-art few-shot learning baselines.
Revised: To verify the performance of the proposed method, we conduct experiments on three standard few-shot learning benchmarks and consolidate the superiority of the proposed method over state-of-the-art few-shot learning baselines.

Original:  Introduction
Revised:  Introduction

Original: In recent years, deep learning algorithms have made tremendous progress and have been widely applied in vision [16], language [37] and speech [10] areas.
Revised: In recent years, deep learning algorithms have made tremendous progress and have been widely applied in vision [16], language [37] and speech [10] areas.

Original: Despite the prevalence, deep learning algorithms rely heavily on a sufficient amount of labeled data and often fail when only a limited number of labeled samples are available.
Revised: Despite the prevalence, deep learning algorithms rely heavily on a sufficient amount of labeled data and often fail when only a limited number of labeled samples are available.

Original: Considering the high cost of extensive data collection and annotation, it has attracted increasing attention to developing algorithms learning from a limited number of training samples, which has been widely known as few-shot learning [39, 5].
Revised: Considering the high cost of extensive data collection and annotation, it has attracted increasing attention to developing algorithms learning from a limited number of training samples, which has been widely known as few-shot learning [39, 5].

Original: The recent focus of few-shot learning is built on the meta-learning framework [9, 39, 33].
Revised: The recent focus of few-shot learning is built on the meta-learning framework [9, 39, 33].

Original: The goal of meta-learning is to train a model that can be quickly adapted to a new task with a small number of labeled samples by leveraging the knowledge from previous related tasks.
Revised: The goal of meta-learning is to train a model that can be quickly adapted to a new task with a small number of labeled samples by leveraging the knowledge from previous related tasks.

Original: Typically, a meta-learning model consists of the meta-training stage where a well-generalized feature extractor is learned from meta-training tasks and the meta-testing stage during which a classifier is built on top of the feature extractor for novel tasks.
Revised: Typically, a meta-learning model consists of the meta-training stage where a well-generalized feature extractor is learned from meta-training tasks and the meta-testing stage during which a classifier is built on top of the feature extractor for novel tasks.

Original: While the majority of meta-learning works revolved around developing better meta-training strategies to reduce the meta generalization error over all tasks, how to fully leverage the meta-trained feature extractor to build a robust classifier and improve a task-specific generalization in the meta-testing stage remains less explored.
Revised: While the majority of meta-learning works revolved around developing better meta-training strategies to reduce the meta generalization error over all tasks, how to fully leverage the meta-trained feature extractor to build a robust classifier and improve a task-specific generalization in the meta-testing stage remains less explored.

Original: [Figure]
Revised: [Figure]

Original: The challenge to building such a classifier lies in the significant deviation of the empirical distribution (i.e. the distribution formed by the training data) from the population distribution (i.e. the true distribution).
Revised: The challenge to building such a classifier lies in the significant deviation of the empirical distribution (i.e. the distribution formed by the training data) from the population distribution (i.e. the true distribution).

Original: As illustrated in Figure 1, a model well fitting the training data (support set) may generalize poorly on the testing data (query set) from the true distribution due to the bias in the empirical distribution.
Revised: As illustrated in Figure 1, a model well fitting the training data (support set) may generalize poorly on the testing data (query set) from the true distribution due to the bias in the empirical distribution.

Original: One viable solution is to approximate the true distribution more accurately – by viewing the empirical distribution as the assembling of delta functions located on each training sample, Chapelle et al.
Revised: One viable solution is to approximate the true distribution more accurately – by viewing the empirical distribution as the assembling of delta functions located on each training sample, Chapelle et al.

Original: [4] proposed the Vicinal Risk Minimization loss function where the delta functions are replaced by some density estimate in the vicinity of each training data.
Revised: [4] proposed the Vicinal Risk Minimization loss function where the delta functions are replaced by some density estimate in the vicinity of each training data.

Original: In the era of deep learning, the VRM learning principle is implemented as Mixup [47, 38], which is simple yet effective given a sufficiently large number of training data.
Revised: In the era of deep learning, the VRM learning principle is implemented as Mixup [47, 38], which is simple yet effective given a sufficiently large number of training data.

Original: However, as we have verified in
Revised: However, as we have verified in

Original: Table 5, the performance improvement by applying Mixup on the support set is limited in few-shot learning where the vicinal distributions assembled by Mixup is still significantly biased.
Revised: Table 5, the performance improvement by applying Mixup on the support set is limited in few-shot learning where the vicinal distributions assembled by Mixup is still significantly biased.

Original: To improve the generalization of a few-shot classifier during meta-testing, we are motivated to revisit the VRM principle to 1) generate adaptive vicinal distribution for each sample in support set and 2) minimize the expected vicinal loss over the vicinal distributions.
Revised: To improve the generalization of a few-shot classifier during meta-testing, we are motivated to revisit the VRM principle to 1) generate adaptive vicinal distribution for each sample in support set and 2) minimize the expected vicinal loss over the vicinal distributions.

Original: However, both are challenging.
Revised: However, both are challenging.

Original: Without loss of generality, we assume that the vicinity of each training point admits a Gaussian distribution.
Revised: Without loss of generality, we assume that the vicinity of each training point admits a Gaussian distribution.

Original: In [4], the means of the vicinal distributions are located on the original training points.
Revised: In [4], the means of the vicinal distributions are located on the original training points.

Original: As illustrated in Figure 1(b), however, setting the mean in this way is not sufficient to approximate the true distribution, as an original training point likely deviates from the mean of the ground-truth distribution of its corresponding class.
Revised: As illustrated in Figure 1(b), however, setting the mean in this way is not sufficient to approximate the true distribution, as an original training point likely deviates from the mean of the ground-truth distribution of its corresponding class.

Original: In fact, it is difficult to estimate the mean and variance based only on the labeled training samples.
Revised: In fact, it is difficult to estimate the mean and variance based only on the labeled training samples.

Original: In this paper, we assume that unlabeled samples are also available, and leverage these unlabeled samples to estimate the mean and variance of the vicinal distribution for each labeled sample.
Revised: In this paper, we assume that unlabeled samples are also available, and leverage these unlabeled samples to estimate the mean and variance of the vicinal distribution for each labeled sample.

Original: Concretely, inspired by [48], we first construct a transition probability matrix based on pairwise distances between features of all samples, and secondly perform the lazy random walk among unlabeled samples starting from each labeled sample.
Revised: Concretely, inspired by [48], we first construct a transition probability matrix based on pairwise distances between features of all samples, and secondly perform the lazy random walk among unlabeled samples starting from each labeled sample.

Original: After walking a certain number of steps, we obtain the visit probabilities of all the samples and infer the vicinal distribution whose mean and variance are the weighted mean and variance of all samples visited.
Revised: After walking a certain number of steps, we obtain the visit probabilities of all the samples and infer the vicinal distribution whose mean and variance are the weighted mean and variance of all samples visited.

Original: All that remains is the loss function over the vicinal distributions of all training samples.
Revised: All that remains is the loss function over the vicinal distributions of all training samples.

Original: An intuitive way to do this is to exploit the Monte Carlo method, which repeats random sampling to explicitly generate data points from each vicinal distribution and minimizes the loss on all the sampled data points.
Revised: An intuitive way to do this is to exploit the Monte Carlo method, which repeats random sampling to explicitly generate data points from each vicinal distribution and minimizes the loss on all the sampled data points.

Original: Unfortunately, it requires drawing a large amount of samples to minimize the Monte Carlo approximation error, which significantly increases the computation cost.
Revised: Unfortunately, it requires drawing a large amount of samples to minimize the Monte Carlo approximation error, which significantly increases the computation cost.

Original: Instead, we follow [4] to derive a closed form of the expected vicinal loss over the vicinal distributions.
Revised: Instead, we follow [4] to derive a closed form of the expected vicinal loss over the vicinal distributions.

Original: In this paper, we focus on building an optimization-based classifier in optimization-based meta-learning algorithms [9, 19], which has been proved with much more adaptability and generalization capacity in novel meta-testing tasks than a nearest-neighbour classifier in metric-based meta-learning algorithms [39, 33, 34, 24].
Revised: In this paper, we focus on building an optimization-based classifier in optimization-based meta-learning algorithms [9, 19], which has been proved with much more adaptability and generalization capacity in novel meta-testing tasks than a nearest-neighbour classifier in metric-based meta-learning algorithms [39, 33, 34, 24].

Original: Particularly, we consider both the cross-entropy loss and the SVM loss functions which are used as classifier in the state-of-the-art meta-learning algorithms [43, 19].
Revised: Particularly, we consider both the cross-entropy loss and the SVM loss functions which are used as classifier in the state-of-the-art meta-learning algorithms [43, 19].

Original: For the vicinal version of crossentropy loss, it cannot be computed in closed form.
Revised: For the vicinal version of crossentropy loss, it cannot be computed in closed form.

Original: We instead derive its quadratic approximation.
Revised: We instead derive its quadratic approximation.

Original: For SVM, we derive its closed-form vicinal loss.
Revised: For SVM, we derive its closed-form vicinal loss.

Original: In summary, our contributions are threefold:
Revised: In summary, our contributions are threefold:

Original: • We revisit vicinal risk minimization (VRM) in deep few-shot learning and derive the resulting expected vicinal loss functions for both the cross-entropy loss and SVM.
Revised: • We revisit vicinal risk minimization (VRM) in deep few-shot learning and derive the resulting expected vicinal loss functions for both the cross-entropy loss and SVM.

Original: • We propose to harness unlabeled data and a lazy random walk algorithm to generate adaptive statistical parameters for the vicinal distribution of each training sample.
Revised: • We propose to harness unlabeled data and a lazy random walk algorithm to generate adaptive statistical parameters for the vicinal distribution of each training sample.

Original: • We have validated the superiority of the proposed algorithm with extensive experiments on three benchmarks, i.e., miniImageNet, CUB, CIFAR-FS.
Revised: • We have validated the superiority of the proposed algorithm with extensive experiments on three benchmarks, i.e., miniImageNet, CUB, CIFAR-FS.

Original:  Related Works 2.1 Few-Shot Learning and Meta-learning
Revised:  Related Works 2.1 Few-Shot Learning and Meta-Learning

Original: Few-shot learning aims to recognize novel classes with insufficient labeled instances.
Revised: Few-shot learning aims to recognize novel classes with insufficient labeled instances.

Original: The recent focus of few-shot learning algorithms has been on the meta-learning framework, which trains the model with multiple episodes to enable quick learning and applies the model to improve learning in novel tasks.
Revised: The recent focus of few-shot learning algorithms has been on the meta-learning framework, which trains the model with multiple episodes to enable quick learning and applies the model to improve learning in novel tasks.

Original: The meta-learning methods fall into two major categories: metric-based methods and optimizationbased methods.
Revised: The meta-learning methods fall into two major categories: metric-based methods and optimizationbased methods.

Original: Metric-based methods [15, 39, 33, 34, 23, 45] learn a metric to compare two samples’ feature embeddings encoded by the feature extractor and classify a sample in the query set based on its distances to the samples in the support set.
Revised: Metric-based methods [15, 39, 33, 34, 23, 45] learn a metric to compare two samples’ feature embeddings encoded by the feature extractor and classify a sample in the query set based on its distances to the samples in the support set.

Original: Optimization-based approaches [9, 27, 31, 35, 41] learn a good initialization such that it can be fast adapted to novel tasks based on a few instances.
Revised: Optimization-based approaches [9, 27, 31, 35, 41] learn a good initialization such that it can be fast adapted to novel tasks based on a few instances.

Original: To tackle the data deficiency issue in few-shot learning, researchers have explored the idea of exploiting unlabeled data and developed transductive learning or semi-supervised learning techniques for few-shot learning, either by leveraging nearest neighbor graph for propagating labels [42, 14, 21] or by leveraging predicted soft or hard labels on unlabeled samples to update the class prototypes or metric [29, 12, 17, 26, 45].
Revised: To tackle the data deficiency issue in few-shot learning, researchers have explored the idea of exploiting unlabeled data and developed transductive learning or semi-supervised learning techniques for few-shot learning, either by leveraging nearest neighbor graph for propagating labels [42, 14, 21] or by leveraging predicted soft or hard labels on unlabeled samples to update the class prototypes or metric [29, 12, 17, 26, 45].

Original: Our method is not built on a metric-based method.
Revised: Our method is not built on a metric-based method.

Original: We make use of the unlabeled data to derive the vicinal distribution for each support sample and optimize the consequential vicinal loss to learn a classifier.
Revised: We make use of the unlabeled data to derive the vicinal distribution for each support sample and optimize the consequential vicinal loss to learn a classifier.

Original: Our approach shares a similar spirit of [43].
Revised: Our approach shares a similar spirit of [43].

Original: Yang et al.
Revised: Yang et al.

Original: [43] proposed to calibrate the distribution of the support samples by transferring statistics from the base classes and trained the classifier on a set of samples sampled from the calibrated distributions.
Revised: [43] proposed to calibrate the distribution of the support samples by transferring statistics from the base classes and trained the classifier on a set of samples sampled from the calibrated distributions.

Original: Unlike [43], our method obtains more accurate vicinal distributions by making use of the unlabeled data from the same task.
Revised: Unlike [43], our method obtains more accurate vicinal distributions by making use of the unlabeled data from the same task.

Original: Besides, instead of sampling extensive data from the distributions, our model directly optimizes the expected vicinal loss over the distributions, resulting in more efficient training and higher performance.
Revised: Besides, instead of sampling extensive data from the distributions, our model directly optimizes the expected vicinal loss over the distributions, resulting in more efficient training and higher performance.

Original: 2.2 Vicinal Risk Minimization
Revised: 2.2 Vicinal Risk Minimization

Original: The Vicinal Risk Minimization (VRM) principle is introduced in [4, 36].
Revised: The Vicinal Risk Minimization (VRM) principle is introduced in [4, 36].

Original: It assumes each training sample admits a vicinal distribution and optimizes the expected loss over the vicinal distributions.
Revised: It assumes each training sample admits a vicinal distribution and optimizes the expected loss over the vicinal distributions.

Original: In [4], only binary VRM classifiers are considered.
Revised: In [4], only binary VRM classifiers are considered.

Original: Our method considers multi-class classifiers and the cross-entropy loss, which is more challenging to derive the expected vicinal loss.
Revised: Our method considers multi-class classifiers and the cross-entropy loss, which is more challenging to derive the expected vicinal loss.

Original: Besides, Chapelle et al.
Revised: Besides, Chapelle et al.

Original: [4] consider Gaussian vicinities but only estimate the variances, keeping the mean the same as the original training sample.
Revised: [4] consider Gaussian vicinities but only estimate the variances, keeping the mean the same as the original training sample.

Original: Our approach further estimates the mean with the unlabeled data.
Revised: Our approach further estimates the mean with the unlabeled data.

Original: In the era of deep learning, the most popular technique under the VRM principle is Mixup [47, 38].
Revised: In the era of deep learning, the most popular technique under the VRM principle is Mixup [47, 38].

Original: Mixup constructs the vicinal distributions as the convex combination of the training data, which can be understood as a data augmentation technique that encourages the model to behave linearly in-between training examples.
Revised: Mixup constructs the vicinal distributions as the convex combination of the training data, which can be understood as a data augmentation technique that encourages the model to behave linearly in-between training examples.

Original: It has been demonstrated that Mixup is simple yet efficient and improves the generalization of deep models given sufficiently many training samples.
Revised: It has been demonstrated that Mixup is simple yet efficient and improves the generalization of deep models given sufficiently many training samples.

Original: Mixup has been introduced on the meta-learning algorithm in [44, 6], but is applied to the query set, focusing on improving the meta generalization.
Revised: Mixup has been introduced on the meta-learning algorithm in [44, 6], but is applied to the query set, focusing on improving the meta generalization.

Original: As the experiment results in [44] (Table 6) and in Table 5 shows, applying Mixup on the support set could even hurt the performance.
Revised: As the experiment results in [44] (Table 6) and in Table 5 shows, applying Mixup on the support set could even hurt the performance.

Original: Our method instead improves the performance by producing an accurate estimate of the vicinal distribution for the small support set using unlabeled data.
Revised: Our method instead improves the performance by producing an accurate estimate of the vicinal distribution for the small support set using unlabeled data.

Original:  Adaptive Vicinal Few-Shot Learning
Revised:  Adaptive Vicinal Few-Shot Learning

Original: In this section, we will introduce the proposed algorithm, AD aptive V icinal Few-Shot Learning (ADV).
Revised: In this section, we will introduce the proposed algorithm, AD aptive V icinal Few-Shot Learning (ADV).

Original: We will first introduce the VRM, then derive the expected vicinal loss for a linear classifier train with cross-entropy loss and for SVM, respectively, and finally propose the algorithm based on lazy random walks to obtain the adaptive vicinal distribution for each labeled support sample.
Revised: We will first introduce the VRM, then derive the expected vicinal loss for a linear classifier train with cross-entropy loss and for SVM, respectively, and finally propose the algorithm based on lazy random walks to obtain the adaptive vicinal distribution for each labeled support sample.

Original: We summarize the proposed method in Algorithm 1.
Revised: We summarize the proposed method in Algorithm 1.

Original: Note that our method focuses on improving the task-specific generalization in meta-testing set and is compatible with feature extractor trained by any meta-learning algorithm.
Revised: Note that our method focuses on improving the task-specific generalization in meta-testing set and is compatible with feature extractor trained by any meta-learning algorithm.

Original: 3.1 Vicinal Risk Minimization
Revised: 3.1 Vicinal Risk Minimization

Original: Given data x, y following the distribution P ( x, y ) , the loss function L , we would like to optimize a model f θ ( x ) by minimizing the expected risk
Revised: Given data x, y following the distribution P ( x, y ) , the loss function L , we would like to optimize a model f θ ( x ) by minimizing the expected risk

Original: [Equation]
Revised: [Equation]

Original: Here, x is the input data, y is its label and θ is the model parameters.
Revised: Here, x is the input data, y is its label and θ is the model parameters.

Original: In practice, P ( x, y ) is unknown and a set of training data { x i , y i } Ni =1 is available, where N is the size of training data.
Revised: In practice, P ( x, y ) is unknown and a set of training data { x i , y i } Ni =1 is available, where N is the size of training data.

Original: To learn a model, one can minimize the empirical risk, which is equivalent to minimizing the expected loss with respect to an empirical density estimate d P δ ( x, y )
Revised: To learn a model, one can minimize the empirical risk, which is equivalent to minimizing the expected loss with respect to an empirical density estimate d P δ ( x, y )

Original: = 1 N (cid:80) Ni
Revised: = 1 N (cid:80) Ni

Original: δ x i ( x ) δ y
Revised: δ x i ( x ) δ y

Original: i
Revised: i

Original: ( y ) , where δ a ( · ) is a Dirac delta function centered at a .
Revised: ( y ) , where δ a ( · ) is a Dirac delta function centered at a .

Original: Given a limited number of training sample, the empirical density estimate d P δ ( x, y ) could bias from the true density d P ( x, y ) .
Revised: Given a limited number of training sample, the empirical density estimate d P δ ( x, y ) could bias from the true density d P ( x, y ) .

Original: For a more accurate density estimate, Chapelle et al.
Revised: For a more accurate density estimate, Chapelle et al.

Original: [4] proposed to replace the delta functions δ x i ( x ) by some estimate of density in the vicinity of x i .
Revised: [4] proposed to replace the delta functions δ x i ( x ) by some estimate of density in the vicinity of x i .

Original: Given a vicinal distribution of x i as P x i ( x ) , the improved density estimate is P ν ( x, y )
Revised: Given a vicinal distribution of x i as P x i ( x ) , the improved density estimate is P ν ( x, y )

Original: = 1 N (cid:80) Ni =
Revised: = 1 N (cid:80) Ni =

Original: 1 P x i ( x ) δ y i ( y ) .
Revised: 1 P x i ( x ) δ y i ( y ) .

Original: And the corresponding vicinal risk is defined as
Revised: And the corresponding vicinal risk is defined as

Original: [Equation]
Revised: [Equation]

Original: 3.2 Vicinal Loss
Revised: 3.2 Vicinal Loss

Original: In this paper, we assume the feature extractor is trained and fixed for novel tasks.
Revised: In this paper, we assume the feature extractor is trained and fixed for novel tasks.

Original: Each task is
Revised: Each task is

Original: C -way K -shot, consisting of N = CK data support samples { ( x i , y i ) }
Revised: C -way K -shot, consisting of N = CK data support samples { ( x i , y i ) }

Original: Ni =1 .
Revised: Ni =1 .

Original: To perform VRM, we assume the feature z i of each sample x i in the support set admits a Gaussian vicinal distribution
Revised: To perform VRM, we assume the feature z i of each sample x i in the support set admits a Gaussian vicinal distribution

Original: N ( µ i , Σ i ) such that a sample ˜ z is drawn from the vicinal distribution with probability
Revised: N ( µ i , Σ i ) such that a sample ˜ z is drawn from the vicinal distribution with probability

Original: P z
Revised: P z

Original: i
Revised: i

Original: (˜ z )
Revised: (˜ z )

Original: = det(2 π Σ i ) − 12 exp( − 12 (˜ z − µ i )
Revised: = det(2 π Σ i ) − 12 exp( − 12 (˜ z − µ i )

Original: ⊤ Σ − 1 i (˜ z − µ i )) , where µ i and Σ
Revised: ⊤ Σ − 1 i (˜ z − µ i )) , where µ i and Σ

Original: i is the mean and variance of the vicinal distribution of z i .
Revised: i is the mean and variance of the vicinal distribution of z i .

Original: 3.2.1 Vicinal Cross-Entropy Loss
Revised: 3.2.1 Vicinal Cross-Entropy Loss

Original: We first focus on linear classifier with Cross-Entropy loss.
Revised: We first focus on linear classifier with Cross-Entropy loss.

Original: A linear classifier f θ ( z ) =
Revised: A linear classifier f θ ( z ) =

Original: ( W ⊤ z + b ) maps a feature z ∈ R d to its logits, where W ∈ R d × C and b ∈ R C is the parameter of the classifier.
Revised: ( W ⊤ z + b ) maps a feature z ∈ R d to its logits, where W ∈ R d × C and b ∈ R C is the parameter of the classifier.

Original: The cross-entropy loss is defined as ℓ ( h, y )
Revised: The cross-entropy loss is defined as ℓ ( h, y )

Original: = − e ⊤ y log( softmax ( f θ ( z ))) , where y ∈ 1 , 2 , ..., C is the label; e y is a one-hot vector with the y -th element being 1 .
Revised: = − e ⊤ y log( softmax ( f θ ( z ))) , where y ∈ 1 , 2 , ..., C is the label; e y is a one-hot vector with the y -th element being 1 .

Original: The expected loss of cross entropy over
Revised: The expected loss of cross entropy over

Original: Gaussian vicinal distribution of x is defined as
Revised: Gaussian vicinal distribution of x is defined as

Original: [Equation]
Revised: [Equation]

Original: Due to the softmax function, the expected loss (1) cannot be computed in closed form.
Revised: Due to the softmax function, the expected loss (1) cannot be computed in closed form.

Original: Instead, we utilize Taylor expansion to derive an approximation of Eqn.(1).
Revised: Instead, we utilize Taylor expansion to derive an approximation of Eqn.(1).

Original: By taking the second-order Taylor expansion of ℓ ( f θ (˜ z ) , y ) around the mean of the vicinal distribution
Revised: By taking the second-order Taylor expansion of ℓ ( f θ (˜ z ) , y ) around the mean of the vicinal distribution

Original: z i , i.e. µ , we obtain
Revised: z i , i.e. µ , we obtain

Original: [Equation]
Revised: [Equation]

Original: where G is the gradient and H is the Hessian w.r.t.
Revised: where G is the gradient and H is the Hessian w.r.t.

Original: input µ .
Revised: input µ .

Original: Note that H
Revised: Note that H

Original: = W ( diag ( h ) − hh ⊤ ) W ⊤ where h = softmax ( f θ ( µ )) and diag ( h ) is a diagonal matrix whose diagonal elements are equal to h [32].
Revised: = W ( diag ( h ) − hh ⊤ ) W ⊤ where h = softmax ( f θ ( µ )) and diag ( h ) is a diagonal matrix whose diagonal elements are equal to h [32].

Original: By taking the expectation over ˜ z , we have
Revised: By taking the expectation over ˜ z , we have

Original: [Equation]
Revised: [Equation]

Original: where Σ is the variance of ˜ z , h k is the k -th elements of h , W k is the k -th column of W .
Revised: where Σ is the variance of ˜ z , h k is the k -th elements of h , W k is the k -th column of W .

Original: We define
Revised: We define

Original: R θ (Σ) as the RHS of Eqn.
Revised: R θ (Σ) as the RHS of Eqn.

Original: Plugging (2), (3) and (4) into (1), we obtain the approximated expected vicinal cross-entropy loss for the linear classifier as
Revised: Plugging (2), (3) and (4) into (1), we obtain the approximated expected vicinal cross-entropy loss for the linear classifier as

Original: [Equation]
Revised: [Equation]

Original: 3.2.2 Vicinal SVM
Revised: 3.2.2 Vicinal SVM

Original: SVM is used as classifier for meta learning algorithms in [19, 43].
Revised: SVM is used as classifier for meta learning algorithms in [19, 43].

Original: We follow [19] to use the Crammer and Singer formulation of multi-class SVM [7].
Revised: We follow [19] to use the Crammer and Singer formulation of multi-class SVM [7].

Original: In [19, 7], the dual formulation of the objective for a linear multi-class SVM is obtained as:
Revised: In [19, 7], the dual formulation of the objective for a linear multi-class SVM is obtained as:

Original: [Equation]
Revised: [Equation]

Original: where α ki is the dual variable and λ is a regularization parameter.
Revised: where α ki is the dual variable and λ is a regularization parameter.

Original: The linear classifier for the k -th class is f kθ ( z ) =
Revised: The linear classifier for the k -th class is f kθ ( z ) =

Original: w ⊤ k z , where w k = (cid:80) Ni =

Revised: w ⊤ k z , where w k = (cid:80) Ni =

Original: α ki z i .
Revised: α ki z i .

Original: Note that here z is an unseen data.
Revised: Note that here z is an unseen data.

Original: We assume only labeled data admit vicinal distributions while unseen data does not admit vicinal distributions.
Revised: We assume only labeled data admit vicinal distributions while unseen data does not admit vicinal distributions.

Original: If each z i admits a vicinal distribution N ( µ i , Σ i ) , the vicinal expectation for the term z ⊤ i
Revised: If each z i admits a vicinal distribution N ( µ i , Σ i ) , the vicinal expectation for the term z ⊤ i

Original: z j in (6) becomes
Revised: z j in (6) becomes

Original: E ˜ z
Revised: E ˜ z

Original: i ∼N
Revised: i ∼N

Original: [Equation]
Revised: [Equation]

Original: µ i , Σ
Revised: µ i , Σ

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: j ∼N
Revised: j ∼N

Original: [Equation]
Revised: [Equation]

Original: and the classifier becomes
Revised: and the classifier becomes

Original: [Equation]
Revised: [Equation]

Original: If µ i =
Revised: If µ i =

Original: z i , the objective of the vicinal SVM reduces to the original SVM objective in Eqn.
Revised: z i , the objective of the vicinal SVM reduces to the original SVM objective in Eqn.

Original: Even when µ i ̸
Revised: Even when µ i ̸

Original: =
Revised: =

Original: z i , the objective of the vicinal SVM still does not take the variance of the vicinal distribution into consideration and may result in suboptimal performance.
Revised: z i , the objective of the vicinal SVM still does not take the variance of the vicinal distribution into consideration and may result in suboptimal performance.

Original: To exploit the variance information to improve the vicinal SVM classifier, we consider a nonlinear classifier, i.e. the kernel-based SVM.
Revised: To exploit the variance information to improve the vicinal SVM classifier, we consider a nonlinear classifier, i.e. the kernel-based SVM.

Original: Multi-class kernel-based SVM uses a kernel K ( · , · ) .
Revised: Multi-class kernel-based SVM uses a kernel K ( · , · ) .

Original: It admits a dual formulation similar to (6), but replaces the inner product of z i and z j in (6) by K ( z i , z j ) .
Revised: It admits a dual formulation similar to (6), but replaces the inner product of z i and z j in (6) by K ( z i , z j ) .

Original: And the classifier becomes f kθ ( z )
Revised: And the classifier becomes f kθ ( z )

Original: = (cid:80) i α ki K
Revised: = (cid:80) i α ki K

Original: ( z i , z ) .
Revised: ( z i , z ) .

Original: In this paper, we consider the widely-used RBF kernel K ( z, z ′ ) =
Revised: In this paper, we consider the widely-used RBF kernel K ( z, z ′ ) =

Original: exp( − || z − z ′
Revised: exp( − || z − z ′

Original: || 2 2 σ) , where σ is a hyper-parameter.
Revised: || 2 2 σ) , where σ is a hyper-parameter.

Original: In vicinal multi-class kernel-based SVM, we define M ( z i , z j ) =
Revised: In vicinal multi-class kernel-based SVM, we define M ( z i , z j ) =

Original: E ˜ z
Revised: E ˜ z

Original: i ∼N ( µ i , Σ i ) E ˜
Revised: i ∼N ( µ i , Σ i ) E ˜

Original: z
Revised: z

Original: j ∼N ( µ j , Σ j ) K
Revised: j ∼N ( µ j , Σ j ) K

Original: ( z i , z j ) and Q ( z i , z ) =
Revised: ( z i , z j ) and Q ( z i , z ) =

Original: E ˜ z
Revised: E ˜ z

Original: i ∼N ( µ i , Σ i ) K ( z i , z ) and obtain their computation as
Revised: i ∼N ( µ i , Σ i ) K ( z i , z ) and obtain their computation as

Original: [Equation]
Revised: [Equation]

Original: The detailed derivation is available in Appendix A. Here, A ( σ 

Revised: The detailed derivation is available in Appendix A. Here, A ( σ

Original: ) ∈ R d × d is a diagonal matrix with diagonal elements being σ 2 ; M i =
Revised: ) ∈ R d × d is a diagonal matrix with diagonal elements being σ 2 ; M i =

Original: ( A ( σ 2 ) + Σ i ) − 1 ; M ij =
Revised: ( A ( σ 2 ) + Σ i ) − 1 ; M ij =

Original: ( A ( σ 2 ) + Σ i + Σ j ) − 1 ; the operator ∥ z ∥ 2 M = z ⊤ Mz .
Revised: ( A ( σ 2 ) + Σ i + Σ j ) − 1 ; the operator ∥ z ∥ 2 M = z ⊤ Mz .

Original: Note that for computation simplification, in vicinal SVM, we assume the variance Σ
Revised: Note that for computation simplification, in vicinal SVM, we assume the variance Σ

Original: i is diagonal and set its non-diagonal elements to zeros.
Revised: i is diagonal and set its non-diagonal elements to zeros.

Original: Now we achieve the vicinal objective for multi-class kernel-based SVM as:
Revised: Now we achieve the vicinal objective for multi-class kernel-based SVM as:

Original: [Equation]
Revised: [Equation]

Original: with the classifier being f kθ ( z ) =
Revised: with the classifier being f kθ ( z ) =

Original: (cid:80) i α ki Q
Revised: (cid:80) i α ki Q

Original: ( z i , z ) .
Revised: ( z i , z ) .

Original: The objective in (9) is a quadratic program (QP) over dual variables { α ki } and can be solved by using a differentiable GPU-based QP solver [1].
Revised: The objective in (9) is a quadratic program (QP) over dual variables { α ki } and can be solved by using a differentiable GPU-based QP solver [1].

Original: 3.3 Lazy Random Walk for an Adaptive Vicinal Distribution
Revised: 3.3 Lazy Random Walk for an Adaptive Vicinal Distribution

Original: As illustrated in Figure 1, the performance of the model learned by VRM relies heavily on how close the vicinal distribution is to the true distribution.
Revised: As illustrated in Figure 1, the performance of the model learned by VRM relies heavily on how close the vicinal distribution is to the true distribution.

Original: However, it is almost impossible to estimate accurate proximate distributions from a small support set.
Revised: However, it is almost impossible to estimate accurate proximate distributions from a small support set.

Original: Therefore, we propose to exploit extra data, i.e., the unlabeled data from the same task, to reduce the estimation error of the statistical parameters for the vicinal distribution and, thus, improve the performance of the model.
Revised: Therefore, we propose to exploit extra data, i.e., the unlabeled data from the same task, to reduce the estimation error of the statistical parameters for the vicinal distribution and, thus, improve the performance of the model.

Original: We assume that a well-trained feature extractor on the meta-training tasks can preserve the similarity between the data in the feature space.
Revised: We assume that a well-trained feature extractor on the meta-training tasks can preserve the similarity between the data in the feature space.

Original: Data from the same class in the novel task would have similar features with a high probability.
Revised: Data from the same class in the novel task would have similar features with a high probability.

Original: Then, we apply the lazy random walk algorithm [48] to estimate the probability that the unlabeled data will be visited starting from a support sample, and compute the statistical parameters as the weighted mean and variance of the samples visited by the lazy random walk.
Revised: Then, we apply the lazy random walk algorithm [48] to estimate the probability that the unlabeled data will be visited starting from a support sample, and compute the statistical parameters as the weighted mean and variance of the samples visited by the lazy random walk.

Original: To perform lazy random walk, we first compute the distance D SU between the support sample
Revised: To perform lazy random walk, we first compute the distance D SU between the support sample

Original: z i and the unlabeled data { u j } N u j =1 and compute the distance D UU between the unlabeled data as:
Revised: z i and the unlabeled data { u j } N u j =1 and compute the distance D UU between the unlabeled data as:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Secondly, we transform these distances into transition probabilities by softmaxing the distance matrix over columns with a temperature τ :
Revised: Secondly, we transform these distances into transition probabilities by softmaxing the distance matrix over columns with a temperature τ :

Original: [Equation]
Revised: [Equation]

Original: Note that we set D UUjj = ∞ such that P UUjj = 0 .
Revised: Note that we set D UUjj = ∞ such that P UUjj = 0 .

Original: Thirdly, we construct the transition probabilities matrix for z i in lazy random walk by combining
Revised: Thirdly, we construct the transition probabilities matrix for z i in lazy random walk by combining

Original: P SUi : and P UU and setting the lazy stay probability β ∈ (0 , 1) as:
Revised: P SUi : and P UU and setting the lazy stay probability β ∈ (0 , 1) as:

Original: [Equation]
Revised: [Equation]

Original: where P SUi : is the i -th row of P SU , 0 n c × 1 is a all zero vector, and I N u is an identity matrix.
Revised: where P SUi : is the i -th row of P SU , 0 n c × 1 is a all zero vector, and I N u is an identity matrix.

Original: We then compute the visit probability v i at the T -th step for z i :
Revised: We then compute the visit probability v i at the T -th step for z i :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where e 1 ∈ R ( N u +1)
Revised: where e 1 ∈ R ( N u +1)

Original: × 1 is a one-hot vector with the first element being 1 .
Revised: × 1 is a one-hot vector with the first element being 1 .

Original: For computational efficiency, we keep only the top m nearest neighbors of a data visitable and set D UUij ( D SUij ) to zero
Revised: For computational efficiency, we keep only the top m nearest neighbors of a data visitable and set D UUij ( D SUij ) to zero

Original: [Table]
Revised: [Table]

Original: if u j is not in the top m nearest neighbors of u i ( z i ).
Revised: if u j is not in the top m nearest neighbors of u i ( z i ).

Original: Finally
Revised: Finally

Original: [Table]
Revised: [Table]

Original: we obtain µ i and Σ i as the weighted mean and variance of z i and { u j } N u j =1 , with the weight being v i .
Revised: we obtain µ i and Σ i as the weighted mean and variance of z i and { u j } N u j =1 , with the weight being v i .

Original: That is: µ i = m
Revised: That is: µ i = m

Original: + 1 z i + (1 − m + 1)
Revised: + 1 z i + (1 − m + 1)

Original: N u (cid:88)
Revised: N u (cid:88)

Original: j
Revised: j

Original: = v ij u j Σ
Revised: = v ij u j Σ

Original: i
Revised: i

Original: = m
Revised: = m

Original: + 1( z i − µ i )( z i − µ i )
Revised: + 1( z i − µ i )( z i − µ i )

Original: ⊤ + (1 − m + 1)
Revised: ⊤ + (1 − m + 1)

Original: N u (cid:88)
Revised: N u (cid:88)

Original: j = v ij ( u j − µ i )( u j − µ i ) ⊤ .
Revised: j = v ij ( u j − µ i )( u j − µ i ) ⊤ .

Original:  Experiments
Revised:  Experiments

Original: In this section, we verify the effectiveness of the proposed approach in improving the generalization of the few-shot learning.
Revised: In this section, we verify the effectiveness of the proposed approach in improving the generalization of the few-shot learning.

Original: We first introduce the implementation details.
Revised: We first introduce the implementation details.

Original: Then we present the results on the benchmarks, followed by a detailed study of the proposed approach.
Revised: Then we present the results on the benchmarks, followed by a detailed study of the proposed approach.

Original: We use three benchmarks for performance evaluation: mini ImageNet [39], CUB [40] and CIFARFS [2].
Revised: We use three benchmarks for performance evaluation: mini ImageNet [39], CUB [40] and CIFARFS [2].

Original: For all datasets, we construct 5-way 1-shot and 5-way 5-shot tasks by randomly samplingclasses.
Revised: For all datasets, we construct 5-way 1-shot and 5-way 5-shot tasks by randomly samplingclasses.

Original: For performance evaluation, we construct 10 , 000 meta-testing tasks and report the average classification accuracy with a 95% confidence interval.
Revised: For performance evaluation, we construct 10 , 000 meta-testing tasks and report the average classification accuracy with a 95% confidence interval.

Original: Implementation details : we focus on meta-testing phase and directly use the WideResNet-28- (WRN) [46] feature extractors pre-trained following the previous work [22].
Revised: Implementation details : we focus on meta-testing phase and directly use the WideResNet-28- (WRN) [46] feature extractors pre-trained following the previous work [22].

Original: We use ADV-CE to denote the application of ADV in the linear classifier with vicinal cross-entropy loss and ADV-SVM to denote the application of ADV in SVM.
Revised: We use ADV-CE to denote the application of ADV in the linear classifier with vicinal cross-entropy loss and ADV-SVM to denote the application of ADV in SVM.

Original: For ADV-CE, we initialize the weights of the classifier by class prototypes and optimize the vicinal loss in (5) by gradient descent for 100 steps.
Revised: For ADV-CE, we initialize the weights of the classifier by class prototypes and optimize the vicinal loss in (5) by gradient descent for 100 steps.

Original: The learning rate is determined by performing a grid search from 0 .
Revised: The learning rate is determined by performing a grid search from 0 .

Original: 001 to 1 on the tasks constructed by the meta-validation set.
Revised: 001 to 1 on the tasks constructed by the meta-validation set.

Original: For ADV-SVM, we solve the QP in (9) by using a differentiable GPU-based
Revised: For ADV-SVM, we solve the QP in (9) by using a differentiable GPU-based

Original: QP solver [1].
Revised: QP solver [1].

Original: The regularization parameter λ is set to 0 .
Revised: The regularization parameter λ is set to 0 .

Original: 1 and the parameter σ for RBF kernel is obtained via grid search from 0 .
Revised: 1 and the parameter σ for RBF kernel is obtained via grid search from 0 .

Original: 1 to 10 .
Revised: 1 to 10 .

Original: In the lazy random walk algorithm, the number of step T , the lazy stay probability β and the temperature τ and the number of visitable neighbors m are obtained via grid search in { 1 , 2 , 3 , 4 , 5 } , { 0 . 1 , 0 . 2 , 0 . 5 } , { 0 . 01 , 0 .
Revised: In the lazy random walk algorithm, the number of step T , the lazy stay probability β and the temperature τ and the number of visitable neighbors m are obtained via grid search in { 1 , 2 , 3 , 4 , 5 } , { 0 . 1 , 0 . 2 , 0 . 5 } , { 0 . 01 , 0 .

Original: 1 , 1 , 10 } , { 5 , 7 , 9 } , respectively.
Revised: 1 , 1 , 10 } , { 5 , 7 , 9 } , respectively.

Original: Notably,
Revised: Notably,

Original: ADV accesses the whole query set as unlabeled data to perform lazy random walk.
Revised: ADV accesses the whole query set as unlabeled data to perform lazy random walk.

Original: Thus, ADV is a transductive method.
Revised: Thus, ADV is a transductive method.

Original: The number of query samples per class is 15 .
Revised: The number of query samples per class is 15 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.1 Comparison with Baselines
Revised: 4.1 Comparison with Baselines

Original: We present the performance of ADV-CE, ADV-SVM with a WRN feature extractor and compare them with existing works in Table 1.
Revised: We present the performance of ADV-CE, ADV-SVM with a WRN feature extractor and compare them with existing works in Table 1.

Original: The results of baselines are from [19], [26] and [13].
Revised: The results of baselines are from [19], [26] and [13].

Original: We observe in Table 1 that, in all datasets, ADV achieves the best performance in both 1-shot and 5-shot tasks.
Revised: We observe in Table 1 that, in all datasets, ADV achieves the best performance in both 1-shot and 5-shot tasks.

Original: Notable, ADV outperforms the best competitor by at least 2% in 1-shot tasks.
Revised: Notable, ADV outperforms the best competitor by at least 2% in 1-shot tasks.

Original: The extraordinary performance of ADV in 1-shot tasks indicates its effectiveness to exploit the information of the unlabeled data from the query set to provide a more accurate distribution estimate to the true distribution of each class, addressing the issue that the empirical distribution estimated from the support set may deviate significantly from the true distribution.
Revised: The extraordinary performance of ADV in 1-shot tasks indicates its effectiveness to exploit the information of the unlabeled data from the query set to provide a more accurate distribution estimate to the true distribution of each class, addressing the issue that the empirical distribution estimated from the support set may deviate significantly from the true distribution.

Original: We also present the experimental results regarding comparison with more SOTA algorithms, including Transductive Information Maximation (TIM) [3], LaplacianShot [49] and Obliquer Manifold (OM) [25].
Revised: We also present the experimental results regarding comparison with more SOTA algorithms, including Transductive Information Maximation (TIM) [3], LaplacianShot [49] and Obliquer Manifold (OM) [25].

Original: We compare with these 3 methods because they are all independent of the pre-training of the base model and do not require training any extra module in the meta-training phase.
Revised: We compare with these 3 methods because they are all independent of the pre-training of the base model and do not require training any extra module in the meta-training phase.

Original: Note that the performance of few-shot learning is affected by both the base model obtained in the meta-training phase and the fine-tuning strategy in the meta-testing phase.
Revised: Note that the performance of few-shot learning is affected by both the base model obtained in the meta-training phase and the fine-tuning strategy in the meta-testing phase.

Original: For a fair comparison in the meta-testing phase, the base model should be the same for different methods.
Revised: For a fair comparison in the meta-testing phase, the base model should be the same for different methods.

Original: We feed all 4 methods the same fixed WRN feature extractor and the same 10,000 meta-testing tasks.
Revised: We feed all 4 methods the same fixed WRN feature extractor and the same 10,000 meta-testing tasks.

Original: As TIM and LaplacianShot use the same pre-trained weights of a feature extractor, we use these pre-trained weights for all of themethods.
Revised: As TIM and LaplacianShot use the same pre-trained weights of a feature extractor, we use these pre-trained weights for all of themethods.

Original: No data augmentation is applied to images in meta-testing tasks.
Revised: No data augmentation is applied to images in meta-testing tasks.

Original: From the results in Table 6, we observe that ADV-CE outperforms all baselines in 1-shot learning and obtains the second best performance in 5-shot learning, which is slightly lower than TIM.
Revised: From the results in Table 2, we observe that ADV-CE outperforms all baselines in 1-shot learning and obtains the second best performance in 5-shot learning, which is slightly lower than TIM.

Original: The major contributions of TIM lie in regularization on query sets, while ours can be deemed as regularization on support sets, which means its contributions are orthogonal to our contributions.
Revised: The major contributions of TIM lie in regularization on query sets, while ours can be deemed as regularization on support sets, which means its contributions are orthogonal to our contributions.

Original: We are more interested in integrating TIM to further boost the proposed method.
Revised: We are more interested in integrating TIM to further boost the proposed method.

Original: When integrating TIM with ADV-CE and applying it to the WRN feature extractor pre-trained by [22] on miniImageNet,
Revised: When integrating TIM with ADV-CE and applying it to the WRN feature extractor pre-trained by [22] on miniImageNet,

Original: ADV-CE+TIM achieves competitive performance to the SOTA methods as presented in Table 7, where the performances of the SOTA methods are reported in the original paper.
Revised: ADV-CE+TIM achieves competitive performance to the SOTA methods as presented in Table 3, where the performances of the SOTA methods are reported in the original paper.

Original: 4.2 Visualization of Vicinal Distributions
Revised: 4.2 Visualization of Vicinal Distributions

Original: The vicinal risk minimization principle in the proposed method is built on the vicinal distributions obtained by a lazy random walk.
Revised: The vicinal risk minimization principle in the proposed method is built on the vicinal distributions obtained by a lazy random walk.

Original: Therefore, the performance of ADV depends on the exactness of the vicinal distribution estimation.
Revised: Therefore, the performance of ADV depends on the exactness of the vicinal distribution estimation.

Original: Here, we visualize the vicinal distributions to evaluate their exactness.
Revised: Here, we visualize the vicinal distributions to evaluate their exactness.

Original: We run the proposed lazy random walk algorithm on two 5-way 1-shot tasks randomly constructed on the meta-testing set of miniImageNet, generate 500 samples from each vicinal distribution, and perform t-SNE algorithm on all data.
Revised: We run the proposed lazy random walk algorithm on two 5-way 1-shot tasks randomly constructed on the meta-testing set of miniImageNet, generate 500 samples from each vicinal distribution, and perform t-SNE algorithm on all data.

Original: The visualization results are presented in Figure 2.
Revised: The visualization results are presented in Figure 2.

Original: [Table]
Revised: [Table]

Original: We can observe from Figure 2 that the vicinal distribution generated by ADV covers most of the query samples of the same class.
Revised: We can observe from Figure 2 that the vicinal distribution generated by ADV covers most of the query samples of the same class.

Original: This verifies that the lazy random walk module in ADV can provide a good estimate of the vicinal distribution, which overlaps with the ground truth distribution.
Revised: This verifies that the lazy random walk module in ADV can provide a good estimate of the vicinal distribution, which overlaps with the ground truth distribution.

Original: By optimizing the expected loss over the vicinal distributions, the model trained by ADV has a smaller probability of overfitting the support samples and provides better generalization to the query set.
Revised: By optimizing the expected loss over the vicinal distributions, the model trained by ADV has a smaller probability of overfitting the support samples and provides better generalization to the query set.

Original: 4.3 Effect of the Network Architecture of the Feature Extractor
Revised: 4.3 Effect of the Network Architecture of the Feature Extractor

Original: The proposed algorithm ADV is compatible with any pre-trained feature extractor.
Revised: The proposed algorithm ADV is compatible with any pre-trained feature extractor.

Original: Here, we validate the effectiveness of ADV on three feature extractors with different backbones, which are trained by different meta-learning methods.
Revised: Here, we validate the effectiveness of ADV on three feature extractors with different backbones, which are trained by different meta-learning methods.

Original: We consider WideResNet (WRN) [46], ResNet-12 [11] and a shallow network of 4 convolutional blocks (CONV) [2].
Revised: We consider WideResNet (WRN) [46], ResNet-12 [11] and a shallow network of 4 convolutional blocks (CONV) [2].

Original: We meta-train the WRN, ResNet, CONV following [22], [19] and [2], respectively.
Revised: We meta-train the WRN, ResNet, CONV following [22], [19] and [2], respectively.

Original: We report the performance of a kernel SVM classifier and a linear classifier trained by crossentropy (denoted by CE) with and without ADV.
Revised: We report the performance of a kernel SVM classifier and a linear classifier trained by crossentropy (denoted by CE) with and without ADV.

Original: The results in Table 4 show that the performance improvement of ADV is consistent for three feature extractors.
Revised: The results in Table 4 show that the performance improvement of ADV is consistent for three feature extractors.

Original: This verifies that the effectiveness of
Revised: This verifies that the effectiveness of

Original: ADV is independent of the feature extractor.
Revised: ADV is independent of the feature extractor.

Original: It is worth noting that the larger performance gains are observed on the better feature extractors.
Revised: It is worth noting that the larger performance gains are observed on the better feature extractors.

Original: For example, the overall performance of WRN outperforms
Revised: For example, the overall performance of WRN outperforms

Original: CONV, and the performance boost of ADV-CE over CE with WRN in 1-shot tasks is 10.5%, which is larger than the boost with CONV (4.8%).
Revised: CONV, and the performance boost of ADV-CE over CE with WRN in 1-shot tasks is 10.5%, which is larger than the boost with CONV (4.8%).

Original: This observation suggests that 1) even with a superior feature extractor that well preserves the similarity between data in feature space, the classifier trained from a limited number of high-quality support data still overfits the data; 2) for a better feature extractor, ADV can exploit more information from the similarity between data to estimate the vicinal distribution of support data and alleviate the overfitting.
Revised: This observation suggests that 1) even with a superior feature extractor that well preserves the similarity between data in feature space, the classifier trained from a limited number of high-quality support data still overfits the data; 2) for a better feature extractor, ADV can exploit more information from the similarity between data to estimate the vicinal distribution of support data and alleviate the overfitting.

Original: 4.4 Ablation Study
Revised: 4.4 Ablation Study

Original: We perform an ablation study on ADV regarding the gain of the VRM principle.
Revised: We perform an ablation study on ADV regarding the gain of the VRM principle.

Original: To do this, we train a vanilla linear classifier with cross-entropy loss (denoted by CE), a linear classifier with Mixup vicinal distribution [47] (denoted by CE-Mixup), a linear classifier with Delta vicinal distribution (denoted by ADV-CE-Mean) whose mean locates on the same mean estimate by our lazy random walk module and whose variance is zero, i.e., P z i
Revised: To do this, we train a vanilla linear classifier with cross-entropy loss (denoted by CE), a linear classifier with Mixup vicinal distribution [47] (denoted by CE-Mixup), a linear classifier with Delta vicinal distribution (denoted by ADV-CE-Mean) whose mean locates on the same mean estimate by our lazy random walk module and whose variance is zero, i.e., P z i

Original: (˜ z )
Revised: (˜ z )

Original: =
Revised: =

Original: δ µ i (˜ z ) , and compare them with ADV-CE.
Revised: δ µ i (˜ z ) , and compare them with ADV-CE.

Original: We also train a multi-class SVM classifier, a multi-class kernel SVM classifier, and a kernel SVM classifier with
Revised: We also train a multi-class SVM classifier, a multi-class kernel SVM classifier, and a kernel SVM classifier with

Original: [Table]
Revised: [Table]

Original: Delta vicinal distribution (denoted by ADV-SVM-Mean).
Revised: Delta vicinal distribution (denoted by ADV-SVM-Mean).

Original: The training of multi-class SVM classifiers all follows [7].
Revised: The training of multi-class SVM classifiers all follows [7].

Original: We report the performance on three benchmarks in Table 5.
Revised: We report the performance on three benchmarks in Table 5.

Original: The results in Table 5 show that the performance of CE-Mixup only competes with CE in miniImageNet and CIFAR-FS, and is even inferior to CE on the CUB dataset.
Revised: The results in Table 5 show that the performance of CE-Mixup only competes with CE in miniImageNet and CIFAR-FS, and is even inferior to CE on the CUB dataset.

Original: This suggests that the Mixup vicinal distribution fails to improve the generalization given a limited number of data.
Revised: This suggests that the Mixup vicinal distribution fails to improve the generalization given a limited number of data.

Original: In contrast, the vicinal distribution obtained by the lazy random walk algorithm is a more accurate estimate of the true distribution, as evidenced by the fact that the performance of ADV-CE-Mean and ADV-CE is better than CE and CE-Mixup.
Revised: In contrast, the vicinal distribution obtained by the lazy random walk algorithm is a more accurate estimate of the true distribution, as evidenced by the fact that the performance of ADV-CE-Mean and ADV-CE is better than CE and CE-Mixup.

Original: To be concrete, the performance boost of ADV-CE-Mean over CE demonstrates that the mean of the vicinal distribution, which lies not on the original sample but on a weighted average of the data passed through by the lazy random walk, is closer to the ground-truth distribution.
Revised: To be concrete, the performance boost of ADV-CE-Mean over CE demonstrates that the mean of the vicinal distribution, which lies not on the original sample but on a weighted average of the data passed through by the lazy random walk, is closer to the ground-truth distribution.

Original: The performance improvement of ADV-CE over ADV-CE-Mean demonstrates that the variance of the vicinal distribution helps to alleviate the overfitting problem.
Revised: The performance improvement of ADV-CE over ADV-CE-Mean demonstrates that the variance of the vicinal distribution helps to alleviate the overfitting problem.

Original: As shown in Table 5, the performance of Kernel SVM is worse than SVM in most cases.
Revised: As shown in Table 5, the performance of Kernel SVM is worse than SVM in most cases.

Original: The reason may be that the features extracted by WRN are linearly separable and the non-linear kernel technique does not help to further improve the separability.
Revised: The reason may be that the features extracted by WRN are linearly separable and the non-linear kernel technique does not help to further improve the separability.

Original: Although the performance of Kernel SVM is inferior to SVM, we observe a performance boost of ADV-SVM, which is built on Kernel SVM.
Revised: Although the performance of Kernel SVM is inferior to SVM, we observe a performance boost of ADV-SVM, which is built on Kernel SVM.

Original: This indicates that learning from the vicinal distributions that are close to the true distribution would sufficiently improve the performance.
Revised: This indicates that learning from the vicinal distributions that are close to the true distribution would sufficiently improve the performance.

Original: We also observe the performance gain of ADV-SVM over
Revised: We also observe the performance gain of ADV-SVM over

Original: ADV-SVM-DD, again evidencing the importance of the variance of vicinal distributions in alleviating the overfitting.
Revised: ADV-SVM-DD, again evidencing the importance of the variance of vicinal distributions in alleviating the overfitting.

Original:  Conclusion and Limitations
Revised:  Conclusion and Limitations

Original: In this paper, we discuss the overfitting issue in few-shot learning, which is caused by the deviation of the empirical distribution from the ground-truth distribution.
Revised: In this paper, we discuss the overfitting issue in few-shot learning, which is caused by the deviation of the empirical distribution from the ground-truth distribution.

Original: To alleviate the overfitting, we revisit the Vicinal Risk Minimization principle in deep few-shot learning and propose to harness unlabeled data and a lazy random walk algorithm to generate adaptive statistical parameters for the vicinal distribution of each training sample.
Revised: To alleviate the overfitting, we revisit the Vicinal Risk Minimization principle in deep few-shot learning and propose to harness unlabeled data and a lazy random walk algorithm to generate adaptive statistical parameters for the vicinal distribution of each training sample.

Original: Following the VRM principle, we learn the model by optimizing the resulting expected vicinal loss.
Revised: Following the VRM principle, we learn the model by optimizing the resulting expected vicinal loss.

Original: We derive the vicinal loss for linear classifiers trained with cross-entropy loss and for SVM classifier and develop the resulting Adaptive Vicinal classifiers as ADV-CE and ADV-SVM, respectively.
Revised: We derive the vicinal loss for linear classifiers trained with cross-entropy loss and for SVM classifier and develop the resulting Adaptive Vicinal classifiers as ADV-CE and ADV-SVM, respectively.

Original: We evaluate the performance of ADV-CE and
Revised: We evaluate the performance of ADV-CE and

Original: ADV-SVM on three benchmarks and validate the effectiveness of ADV in mitigating overfitting and improving the generalization of classifiers learned from a limited number of samples.
Revised: ADV-SVM on three benchmarks and validate the effectiveness of ADV in mitigating overfitting and improving the generalization of classifiers learned from a limited number of samples.

Original: Currently, our proposed ADC is compatible with CE and SVM only.
Revised: Currently, our proposed ADC is compatible with CE and SVM only.

Original: We will extend it to other loss functions.
Revised: We will extend it to other loss functions.

Original: For the vicinal Cross-Entropy loss, we optimize an approximation obtained by second-order Taylor expansion.
Revised: For the vicinal Cross-Entropy loss, we optimize an approximation obtained by second-order Taylor expansion.

Original: However, the approximation error can be large.
Revised: However, the approximation error can be large.

Original: Thus, the performance of ADV-CE may be suboptimal and we can further improve it by deriving a better approximation for the vicinal
Revised: Thus, the performance of ADV-CE may be suboptimal and we can further improve it by deriving a better approximation for the vicinal

Original: Cross-Entropy loss.
Revised: Cross-Entropy loss.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes]
Revised: For each question, change the default [TODO] to [Yes]

Original: [No] or [N/A] .
Revised: [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] .
Revised: [Yes] .

Original: See Section(c)
Revised: See Section(c)

Original: Did you discuss any potential negative social impacts of your work?
Revised: Did you discuss any potential negative social impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] 4.
Revised: [No] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Derivation of vicinal kernel functions
Revised: A Derivation of vicinal kernel functions

Original: Here, we provide the derivation of vicinal kernel functions omitted in Section 3.2.2.
Revised: Here, we provide the derivation of vicinal kernel functions omitted in Section 3.2.2.

Original: The one-side vicinal kernel Q [Equation] z, z i ) is derived as
Revised: The one-side vicinal kernel Q ( z, z i ) is derived as

Original: [Equation]
Revised: [Equation]

Original: And the two-side vicinal kernel M ( z i , z j ) is derived as
Revised: And the two-side vicinal kernel M ( z i , z j ) is derived as

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] The derivation follows the convolution integrals of normal distribution functions and is simplified by the assumption that A ( σ 2 ) , Σ i , Σ j are diagonal.
Revised: The derivation follows the convolution integrals of normal distribution functions and is simplified by the assumption that A ( σ 2 ) , Σ i , Σ j are diagonal.

Original: B Introduction to Datasets in Experiments
Revised: B Introduction to Datasets in Experiments

Original: We evaluate the proposed method on mini ImageNet [39], CUB [40] and CIFAR-FS [2].
Revised: We evaluate the proposed method on mini ImageNet [39], CUB [40] and CIFAR-FS [2].

Original: mini ImageNet is a standard benchmark for few-shot image classification benchmark.
Revised: mini ImageNet is a standard benchmark for few-shot image classification benchmark.

Original: It consists of 100 classes from ImageNet dataset [30].
Revised: It consists of 100 classes from ImageNet dataset [30].

Original: Each class contains 600 images of size 84×84.
Revised: Each class contains 600 images of size 84×84.

Original: These classes are split into 64, 16, and 20 classes for meta-training, meta-validation, and meta-testing respectively [28].
Revised: These classes are split into 64, 16, and 20 classes for meta-training, meta-validation, and meta-testing respectively [28].

Original: CUB contains 200 classes with total 11,788 images of size 84 × 84.
Revised: CUB contains 200 classes with total 11,788 images of size 84 × 84.

Original: Following previous works [5], the base, validation, and novel split are 100, 50, and 50 classes respectively.
Revised: Following previous works [5], the base, validation, and novel split are 100, 50, and 50 classes respectively.

Original: CIFAR-FS is a variant of the CIFAR-100 dataset used for few-shot classification.
Revised: CIFAR-FS is a variant of the CIFAR-100 dataset used for few-shot classification.

Original: It containsclasses, each with 600 images of 32 × 32 pixels.
Revised: It containsclasses, each with 600 images of 32 × 32 pixels.

Original: The classes are randomly split into 64, 16, andfor meta-training, meta-validation, and meta-testing respectively.
Revised: The classes are randomly split into 64, 16, andfor meta-training, meta-validation, and meta-testing respectively.

Original: C Extra Experimental Results

Original: We provide extra experimental results regarding comparison with more SOTA algorithms, including Transductive Information Maximation (TIM) [3], LaplacianShot [49] and Obliquer Manifold (OM) [25].

Original: We compare with these 3 methods because they are all independent of the pre-training of the base model and do not require training any extra module in the meta-training phase.

Original: Note that the performance of few-shot learning is affected by both the base model obtained in the meta-training phase and the fine-tuning strategy in the meta-testing phase.

Original: For a fair comparison in the meta-testing phase, the base model should be the same for different methods.

Original: We feed all methods the same fixed WRN feature extractor and the same 10,000 meta-testing tasks.

Original: As TIM and LaplacianShot use the same pre-trained weights of a feature extractor, we use these pre-trained weights for all of the 4 methods.

Original: No data augmentation is applied to images in meta-testing tasks.

Original: We present the results in Table 6.

Original: We observe that ADV-CE outperforms all baselines in 1-shot learning and obtains the second best performance in 5-shot learning, which is slightly lower than

Original: TIM.

Original: [Table]

Original: [Table]

Original: Note that the major contributions of TIM lie in regularization on query sets, though ours can be deemed as regularization on support sets, which means its contributions are orthogonal to our contributions.

Original: We are more than interested in integrating TIM to further boost ours.

Original: When integrating TIM with

Original: ADV-CE and applying it to the WRN feature extractor

Original: pre-trained by [22] on miniImageNet, ADVCE+TIM achieves competitive performance to SOTA methods in Table 7, where the performances of

Original: SOTA methods are reported in the original paper.