Original: C ROSS - LEVEL DISTILLATION AND FEATURE DENOISING FOR CROSS - DOMAIN FEW - SHOT CLASSIFICATION
Revised: C ROSS - LEVEL DISTILLATION AND FEATURE DENOISING FOR CROSS - DOMAIN FEW - SHOT CLASSIFICATION

Original: A BSTRACT
Revised: A BSTRACT

Original: The conventional few-shot classification aims at learning a model on a large labeled base dataset and rapidly adapting to a target dataset that is from the same distribution as the base dataset.
Revised: The conventional few-shot classification aims at learning a model on a large labeled base dataset and rapidly adapting to a target dataset that is from the same distribution as the base dataset.

Original: However, in practice, the base and the target datasets of few-shot classification are usually from different domains, which is the problem of cross-domain few-shot classification.
Revised: However, in practice, the base and the target datasets of few-shot classification are usually from different domains, which is the problem of cross-domain few-shot classification.

Original: We tackle this problem by making a small proportion of unlabeled images in the target domain accessible in the training stage.
Revised: We tackle this problem by making a small proportion of unlabeled images in the target domain accessible in the training stage.

Original: In this setup, even though the base data are sufficient and labeled, the large domain shift still makes transferring the knowledge from the base dataset difficult.
Revised: In this setup, even though the base data are sufficient and labeled, the large domain shift still makes transferring the knowledge from the base dataset difficult.

Original: We meticulously design a cross-level knowledge distillation method, which can strengthen the ability of the model to extract more discriminative features in the target dataset by guiding the network’s shallow layers to learn higher-level information.
Revised: We meticulously design a cross-level knowledge distillation method, which can strengthen the ability of the model to extract more discriminative features in the target dataset by guiding the network’s shallow layers to learn higher-level information.

Original: Furthermore, in order to alleviate the overfitting in the evaluation stage, we propose a feature denoising operation which can reduce the feature redundancy and mitigate overfitting.
Revised: Furthermore, in order to alleviate the overfitting in the evaluation stage, we propose a feature denoising operation which can reduce the feature redundancy and mitigate overfitting.

Original: Our approach can surpass the previous state-of-the-art method, DynamicDistillation, by 5.44 % on 1-shot and 1.37 % on 5-shot classification tasks on average in the BSCD-FSL benchmark.
Revised: Our approach can surpass the previous state-of-the-art method, DynamicDistillation, by 5.44 % on 1-shot and 1.37 % on 5-shot classification tasks on average in the BSCD-FSL benchmark.

Original: The implementation code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/CLDFD.
Revised: The implementation code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/CLDFD.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has achieved great success on image recognition tasks with the help of a large number of labeled images.
Revised: Deep learning has achieved great success on image recognition tasks with the help of a large number of labeled images.

Original: However, it is the exact opposite of the human perception mechanism which can recognize a new category by learning only a few samples.
Revised: However, it is the exact opposite of the human perception mechanism which can recognize a new category by learning only a few samples.

Original: Besides, a large amount of annotations is costly and unavailable for some scenarios.
Revised: Besides, a large amount of annotations is costly and unavailable for some scenarios.

Original: It is more valuable to study few-shot classification which trains a classification model on a base dataset and rapidly adapts it to the target dataset.
Revised: It is more valuable to study few-shot classification which trains a classification model on a base dataset and rapidly adapts it to the target dataset.

Original: However, due to the constraint that the base data and the target data need to be consistent in their distributions, the conventional few-shot classification may not cater to the demands in some practical scenarios.
Revised: However, due to the constraint that the base data and the target data need to be consistent in their distributions, the conventional few-shot classification may not cater to the demands in some practical scenarios.

Original: For example, it may fail in scenarios where the training domain is natural images, but the evaluation domain is satellite images.
Revised: For example, it may fail in scenarios where the training domain is natural images, but the evaluation domain is satellite images.

Original: Considering this domain shift in practical applications, we focus on cross-domain few-shot classification (CD-FSC) in this paper.
Revised: Considering this domain shift in practical applications, we focus on cross-domain few-shot classification (CD-FSC) in this paper.

Original: Previous methods, such as (Mangla et al., 2020; Adler et al., 2020; Tseng et al., 2020), can handle this problem with small domain gaps.
Revised: Previous methods, such as (Mangla et al., 2020; Adler et al., 2020; Tseng et al., 2020), can handle this problem with small domain gaps.

Original: However, the CD-FSC problem with a large domain gap is still a challenge.
Revised: However, the CD-FSC problem with a large domain gap is still a challenge.

Original: BSCD-FSC (Guo et al., 2020) is a suitable benchmark for studying this problem, where the base dataset has natural images and the target datasets contain satellite images, crop disease images, skin disease images and X-ray images of sundry lung diseases.
Revised: BSCD-FSC (Guo et al., 2020) is a suitable benchmark for studying this problem, where the base dataset has natural images and the target datasets contain satellite images, crop disease images, skin disease images and X-ray images of sundry lung diseases.

Original: On this benchmark, previous methods following the traditional CD-FSC protocol train their models on the base dataset and evaluate them
Revised: On this benchmark, previous methods following the traditional CD-FSC protocol train their models on the base dataset and evaluate them

Original: [Figure]
Revised: [Figure]

Original: on the target dataset, but their performances are far from satisfactory.
Revised: on the target dataset, but their performances are far from satisfactory.

Original: STARTUP (Phoo & Hariharan, 2021) and Dynamic-Distillation (Islam et al., 2021a) introduce a more realistic setup that makes a small portion of the unlabeled target images accessible during the training phase.
Revised: STARTUP (Phoo & Hariharan, 2021) and Dynamic-Distillation (Islam et al., 2021a) introduce a more realistic setup that makes a small portion of the unlabeled target images accessible during the training phase.

Original: These target images bring a prior to the model and dramatically promote the model’s performance on the target datasets.
Revised: These target images bring a prior to the model and dramatically promote the model’s performance on the target datasets.

Original: Inspired by that, we follow their setup to explore the CD-FSC problem with a large domain shift.
Revised: Inspired by that, we follow their setup to explore the CD-FSC problem with a large domain shift.

Original: In this work, we propose a cross-level distillation (CLD), which can effectively transfer the knowledge from the base dataset and improve the performance of the student network on the target domain.
Revised: In this work, we propose a cross-level distillation (CLD), which can effectively transfer the knowledge from the base dataset and improve the performance of the student network on the target domain.

Original: Besides, we propose feature denoising (FD) to remove the noise in the features during the fine-tuning stage.
Revised: Besides, we propose feature denoising (FD) to remove the noise in the features during the fine-tuning stage.

Original: Our CD-FSC framework is given in Figure 1.
Revised: Our CD-FSC framework is given in Figure 1.

Original: The detail of CLD is shown in Figure 2, which distills a teacher’s deeper layers to a student’s shallower layers, where the student and the teacher share the same structure.
Revised: The detail of CLD is shown in Figure 2, which distills a teacher’s deeper layers to a student’s shallower layers, where the student and the teacher share the same structure.

Original: Unlike the distillation methods in STARTUP and Dynamic-Distillation, which only distill the teacher’s last layer to the student’s last layer, our CLD leads the shallow layers of the student to mimic the features generated from the deeper levels of the teacher so that the student can learn more deeper semantic information and extract more discriminative features on the target dataset.
Revised: Unlike the distillation methods in STARTUP and Dynamic-Distillation, which only distill the teacher’s last layer to the student’s last layer, our CLD leads the shallow layers of the student to mimic the features generated from the deeper levels of the teacher so that the student can learn more deeper semantic information and extract more discriminative features on the target dataset.

Original: Additionally, since the teacher networks in STARTUP and Dynamic-Distillation are pre-trained on the base dataset only, the teacher’s observation of the target data is biased.
Revised: Additionally, since the teacher networks in STARTUP and Dynamic-Distillation are pre-trained on the base dataset only, the teacher’s observation of the target data is biased.

Original: In order to calibrate the bias, we design an iterative process by building another network which shares the same structure and parameters with the historical student network, named old student network.
Revised: In order to calibrate the bias, we design an iterative process by building another network which shares the same structure and parameters with the historical student network, named old student network.

Original: In each training iteration, the features from the teacher and the old student in the same layers are dynamically fused to guide the corresponding layers of the student.
Revised: In each training iteration, the features from the teacher and the old student in the same layers are dynamically fused to guide the corresponding layers of the student.

Original: The latter the training iteration, the fewer fusion features from the teacher network, and the more from the old student network.
Revised: The latter the training iteration, the fewer fusion features from the teacher network, and the more from the old student network.

Original: Due to the target data used in training is unlabeled, the self-supervised loss is introduced to excavate the target domain information further.
Revised: Due to the target data used in training is unlabeled, the self-supervised loss is introduced to excavate the target domain information further.

Original: The self-supervised loss not only supports the network in mining valuable information on the target domain, but also brings a phenomenon where the final feature vector for classification has a small number of dominant (strongly activated) elements with the others are close to zero (Hua et al., 2021;
Revised: The self-supervised loss not only supports the network in mining valuable information on the target domain, but also brings a phenomenon where the final feature vector for classification has a small number of dominant (strongly activated) elements with the others are close to zero (Hua et al., 2021;

Original: Kalibhat et al., 2022).
Revised: Kalibhat et al., 2022).

Original: We find that during the fine-tuning phase in Figure 1, these small activated elements are redundant and considered as noise.
Revised: We find that during the fine-tuning phase in Figure 1, these small activated elements are redundant and considered as noise.

Original: Our FD operation keeps the top h largest elements and sets the others to zero.
Revised: Our FD operation keeps the top h largest elements and sets the others to zero.

Original: It is experimentally verified that FD can greatly improve the model’s performance.
Revised: It is experimentally verified that FD can greatly improve the model’s performance.

Original: Above all, our main contributions are summarized below:
Revised: Above all, our main contributions are summarized below:

Original: • We propose a cross-level distillation (CLD) framework, which can well transfer the knowledge of the teacher trained on the base dataset to the student.
Revised: • We propose a cross-level distillation (CLD) framework, which can well transfer the knowledge of the teacher trained on the base dataset to the student.

Original: We also use an old student network mechanism is also necessary to calibrate the teacher’s bias learned from the base data.
Revised: We also use an old student network mechanism is also necessary to calibrate the teacher’s bias learned from the base data.

Original: • Considering the noisy feature activations, we design a feature denoising (FD) operation that can significantly improve the performance of our model.
Revised: • Considering the noisy feature activations, we design a feature denoising (FD) operation that can significantly improve the performance of our model.

Original: • Extensive experiments are conducted to verify that our proposed CLD and FD can achieve state-of-the-art results on the BSCD-FSL benchmark with large domain gaps.
Revised: • Extensive experiments are conducted to verify that our proposed CLD and FD can achieve state-of-the-art results on the BSCD-FSL benchmark with large domain gaps.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Cross-domain few-shot classification.
Revised: Cross-domain few-shot classification.

Original: The cross-domain few-shot classification is firstly defined by (Chen et al., 2018), which trains a model on the base dataset and evaluates it on the target dataset in a different domain from the base dataset.
Revised: The cross-domain few-shot classification is firstly defined by (Chen et al., 2018), which trains a model on the base dataset and evaluates it on the target dataset in a different domain from the base dataset.

Original: LFT (Tseng et al., 2020) simulates the domain shift from the base dataset to the target dataset by meta-learning and inserts linear layers into the network to align the features from the different domains.
Revised: LFT (Tseng et al., 2020) simulates the domain shift from the base dataset to the target dataset by meta-learning and inserts linear layers into the network to align the features from the different domains.

Original: Meta-FDMixup (Fu et al., 2021) uses several labeled images from the target dataset for domain-shared feature disentanglement and feeds the domainshared features to the classifier.
Revised: Meta-FDMixup (Fu et al., 2021) uses several labeled images from the target dataset for domain-shared feature disentanglement and feeds the domainshared features to the classifier.

Original: FLUTE (Triantafillou et al., 2021) learns the universal templates of features across multi-source domains to improve the transferability of the model.
Revised: FLUTE (Triantafillou et al., 2021) learns the universal templates of features across multi-source domains to improve the transferability of the model.

Original: However, all these methods concentrate on the CD-FSC problem with small domain shifts.
Revised: However, all these methods concentrate on the CD-FSC problem with small domain shifts.

Original: Some methods handle CDFSC with large domain gaps, in which the target datasets have obvious dissimilarity from the base dataset on perspective distortion, semantics, and/or color depth.
Revised: Some methods handle CDFSC with large domain gaps, in which the target datasets have obvious dissimilarity from the base dataset on perspective distortion, semantics, and/or color depth.

Original: For example, ConfeSS (Das et al., 2021) extracts useful feature components from labeled target images.
Revised: For example, ConfeSS (Das et al., 2021) extracts useful feature components from labeled target images.

Original: ATA (Wang & Deng, 2021) does not require any prior of the target dataset and proposes a plug-and-play inductive bias-adaptive task augmentation module.
Revised: ATA (Wang & Deng, 2021) does not require any prior of the target dataset and proposes a plug-and-play inductive bias-adaptive task augmentation module.

Original: CI (Luo et al., 2022) trains an encoder on the base dataset and converts the features of the target data with a transformation function in the evaluation stage.
Revised: CI (Luo et al., 2022) trains an encoder on the base dataset and converts the features of the target data with a transformation function in the evaluation stage.

Original: UniSiam (Lu et al., 2022) adopts a self-supervised approach to address the CD-FSC problem.
Revised: UniSiam (Lu et al., 2022) adopts a self-supervised approach to address the CD-FSC problem.

Original: Among the methods dealing with large domain shifts, STARTUP (Phoo & Hariharan, 2021) is a strong baseline, which uses a few unlabeled target images in the training stage.
Revised: Among the methods dealing with large domain shifts, STARTUP (Phoo & Hariharan, 2021) is a strong baseline, which uses a few unlabeled target images in the training stage.

Original: It firstly trains a teacher network on the base dataset in a supervised fashion and transfers the teacher’s knowledge to the student by knowledge distillation (KD).
Revised: It firstly trains a teacher network on the base dataset in a supervised fashion and transfers the teacher’s knowledge to the student by knowledge distillation (KD).

Original: It jointly optimizes the cross-entropy loss with the base dataset, contrastive loss of the unlabeled target images and the KD loss to upgrade the student network.
Revised: It jointly optimizes the cross-entropy loss with the base dataset, contrastive loss of the unlabeled target images and the KD loss to upgrade the student network.

Original: Dynamic-Distillation (Islam et al., 2021a) also uses a small number of unlabeled images and KD.
Revised: Dynamic-Distillation (Islam et al., 2021a) also uses a small number of unlabeled images and KD.

Original: The main difference between Dynamic-Distillation and STARTUP is that the former upgrades the pre-trained teacher dynamically by exponential moving averages, while the latter fixes the teacher.
Revised: The main difference between Dynamic-Distillation and STARTUP is that the former upgrades the pre-trained teacher dynamically by exponential moving averages, while the latter fixes the teacher.

Original: In our work, we follow their data setup allowing a small proportion of the unlabeled target images to be seen during the training phase.
Revised: In our work, we follow their data setup allowing a small proportion of the unlabeled target images to be seen during the training phase.

Original: Different from the two methods that perform KD at the last layers of the teacher and the student, our KD is carried out at cross levels.
Revised: Different from the two methods that perform KD at the last layers of the teacher and the student, our KD is carried out at cross levels.

Original: Besides, our denoising operation further improves the performance.
Revised: Besides, our denoising operation further improves the performance.

Original: Self-supervised learning.
Revised: Self-supervised learning.

Original: Self-supervised learning is widely used in the scenarios where labels are not available for training.
Revised: Self-supervised learning is widely used in the scenarios where labels are not available for training.

Original: It defines a “pretext task” to pre-train a network.
Revised: It defines a “pretext task” to pre-train a network.

Original: For example, (Gidaris et al., 2018) pre-trains the model by predicting the rotation angle of the image.
Revised: For example, (Gidaris et al., 2018) pre-trains the model by predicting the rotation angle of the image.

Original: One popular method is contrastive learning, such as SimCLR (Chen et al., 2020) which pulls different augmented versions of the same image closer and pushes the versions from different images away.
Revised: One popular method is contrastive learning, such as SimCLR (Chen et al., 2020) which pulls different augmented versions of the same image closer and pushes the versions from different images away.

Original: Beyond contrastive learning, BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) rely only positive pairs.
Revised: Beyond contrastive learning, BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) rely only positive pairs.

Original: Knowledge distillation.
Revised: Knowledge distillation.

Original: (Hinton et al., 2015) firstly propose knowledge distillation by guiding a compact network (student) to mimic the output of a large network (teacher).
Revised: (Hinton et al., 2015) firstly propose knowledge distillation by guiding a compact network (student) to mimic the output of a large network (teacher).

Original: Since features in the intermediate layers are informative, some previous methods distill the teacher’s intermediate features (Romero et al., 2014) or attention maps of the features (Zagoruyko & Komodakis, 2017).
Revised: Since features in the intermediate layers are informative, some previous methods distill the teacher’s intermediate features (Romero et al., 2014) or attention maps of the features (Zagoruyko & Komodakis, 2017).

Original: Besides, self-distillation methods, such as BYOT (Zhang et al., 2019), distill the last layer of the network to its own shallower layers.
Revised: Besides, self-distillation methods, such as BYOT (Zhang et al., 2019), distill the last layer of the network to its own shallower layers.

Original: BYOT’s teacher and student are the same network.
Revised: BYOT’s teacher and student are the same network.

Original: In our framework (Figure 2), in addition to the KD in the intermediate layers, we design an old student that shares the same structure as the student but with different parameters.
Revised: In our framework (Figure 2), in addition to the KD in the intermediate layers, we design an old student that shares the same structure as the student but with different parameters.

Original: The introduction of this old student not only alleviates the teacher’s bias learned from the base dataset, but also has the effect of assembling multiple historic students during training.
Revised: The introduction of this old student not only alleviates the teacher’s bias learned from the base dataset, but also has the effect of assembling multiple historic students during training.

Original:  M ETHODOLOGY 3. P RELIMINARY
Revised:  M ETHODOLOGY 3. P RELIMINARY

Original: During the training period, we follow the setting in STARTUP (Phoo & Hariharan, 2021) and
Revised: During the training period, we follow the setting in STARTUP (Phoo & Hariharan, 2021) and

Original: Dynamic-Distillation (Islam et al., 2021a), where a labeled base dataset D B and a few unlabeled target images sampled from the target dataset D T are available.
Revised: Dynamic-Distillation (Islam et al., 2021a), where a labeled base dataset D B and a few unlabeled target images sampled from the target dataset D T are available.

Original: In the testing stage, the support set
Revised: In the testing stage, the support set

Original: D S comprises N classes, and K samples are randomly selected from each class in D T , which is the so-called N -way K -shot task.
Revised: D S comprises N classes, and K samples are randomly selected from each class in D T , which is the so-called N -way K -shot task.

Original: The support set D S is for fine-tuning a new classifier with the frozen encoder (the student network in this work).
Revised: The support set D S is for fine-tuning a new classifier with the frozen encoder (the student network in this work).

Original: The images in the query set D Q are randomly picked
Revised: The images in the query set D Q are randomly picked

Original: [Figure]
Revised: [Figure]

Original: supervised
Revised: supervised

Original: L ss introduced on the student from the selected N classes for evaluating the classification accuracy.
Revised: L ss introduced on the student from the selected N classes for evaluating the classification accuracy.

Original: The support set D S and the query set D Q have no overlap.
Revised: The support set D S and the query set D Q have no overlap.

Original: 3. C ROSS - LEVEL DISTILLATION
Revised: 3. C ROSS - LEVEL DISTILLATION

Original: The proposed cross-level distillation (CLD) framework is shown in Figure 2.
Revised: The proposed cross-level distillation (CLD) framework is shown in Figure 2.

Original: The teacher network f t is pre-trained on D B with the cross-entropy loss.
Revised: The teacher network f t is pre-trained on D B with the cross-entropy loss.

Original: The student network f s is expected to inherit the knowledge of the teacher and extract discriminative features of D T .
Revised: The student network f s is expected to inherit the knowledge of the teacher and extract discriminative features of D T .

Original: However, the teacher’s observation of the target data is biased since it is pre-trained on the base dataset only.
Revised: However, the teacher’s observation of the target data is biased since it is pre-trained on the base dataset only.

Original: In the i -th training iteration, if the features extracted by the student f i s directly mimic the features of the teacher, the teacher’s bias will be transferred to the student.
Revised: In the i -th training iteration, if the features extracted by the student f i s directly mimic the features of the teacher, the teacher’s bias will be transferred to the student.

Original: To reduce the bias, we introduce an old student network f i o , which is a copy of f i − τ s , where the hyper-parameter τ denotes the training iteration interval between f i − τ s and f i s .
Revised: To reduce the bias, we introduce an old student network f i o , which is a copy of f i − τ s , where the hyper-parameter τ denotes the training iteration interval between f i − τ s and f i s .

Original: To simplify the KD complexity, we divide each backbone of f s , f o , and f t into L residual blocks.
Revised: To simplify the KD complexity, we divide each backbone of f s , f o , and f t into L residual blocks.

Original: Let s il , o il , and t il be the features obtained by the student, the old student, and the teacher in the l -th block at the i -th iteration.
Revised: Let s il , o il , and t il be the features obtained by the student, the old student, and the teacher in the l -th block at the i -th iteration.

Original: The fusion between t il and o il is defined as:
Revised: The fusion between t il and o il is defined as:

Original: [Equation]
Revised: [Equation]

Original: where α i
Revised: where α i

Original: =
Revised: =

Original: iT is a dynamic weight with T being the total number of training iterations.
Revised: iT is a dynamic weight with T being the total number of training iterations.

Original: The KD loss L i dis in the i -th iteration is defined as:
Revised: The KD loss L i dis in the i -th iteration is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ω l ( · ) is a projector of the l -th student’s block for feature dimensionality alignment.
Revised: where ω l ( · ) is a projector of the l -th student’s block for feature dimensionality alignment.

Original: It is comprised of a convolutional layer, a batch normalization operator and ReLU activation.
Revised: It is comprised of a convolutional layer, a batch normalization operator and ReLU activation.

Original: Note that the KD in Equation 2 is from the ( l + 1) -th block of the teacher to the l -th block of the student.
Revised: Note that the KD in Equation 2 is from the ( l + 1) -th block of the teacher to the l -th block of the student.

Original: In our experiments, we find that this style is better than others (see Section 4.3).
Revised: In our experiments, we find that this style is better than others (see Section 4.3).

Original: The total loss L for training the student network is:
Revised: The total loss L for training the student network is:

Original: [Equation]
Revised: [Equation]

Original: where L ss is a self-supervised loss, and λ is a weight coefficient of loss function for balancing the two losses.
Revised: where L ss is a self-supervised loss, and λ is a weight coefficient of loss function for balancing the two losses.

Original: For L ss , off-the-shelf self-supervised losses like SimCLR (Chen et al., 2020) or BYOL (Grill et al., 2020) can be used.
Revised: For L ss , off-the-shelf self-supervised losses like SimCLR (Chen et al., 2020) or BYOL (Grill et al., 2020) can be used.

Original: The contrastive loss in SimCLR is:
Revised: The contrastive loss in SimCLR is:

Original: [Equation]
Revised: [Equation]

Original: where z m , z n and z q are the projected embeddings of different augmentations, ( m, n ) is a positive pair from the same image, sim( · ) is a similarity function, B is a mini-batch of unlabeled target images, and γ is a temperature coefficient.
Revised: where z m , z n and z q are the projected embeddings of different augmentations, ( m, n ) is a positive pair from the same image, sim( · ) is a similarity function, B is a mini-batch of unlabeled target images, and γ is a temperature coefficient.

Original: The self-supervised loss in BYOL is:
Revised: The self-supervised loss in BYOL is:

Original: [Equation]
Revised: [Equation]

Original: where z m and z ′ m are embeddings of the online network and the target network, respectively, and p ( · ) is a linear predictor.
Revised: where z m and z ′ m are embeddings of the online network and the target network, respectively, and p ( · ) is a linear predictor.

Original: Note that the last convolution block of the student network is not involved in KD and is trained by minimizing L ss only.
Revised: Note that the last convolution block of the student network is not involved in KD and is trained by minimizing L ss only.

Original: The reason is that the last block mainly discovers semantic information that is highly domain-specific.
Revised: The reason is that the last block mainly discovers semantic information that is highly domain-specific.

Original: Therefore, we constrain it on the target data rather than letting it learn from the teacher that is pre-trained on the base data.
Revised: Therefore, we constrain it on the target data rather than letting it learn from the teacher that is pre-trained on the base data.

Original: 3. F EATURE D ENOISING
Revised: 3. F EATURE D ENOISING

Original: The self-supervised loss brings a phenomenon where the final feature vector for classification has a small number of strongly activated elements while the others are close to zero (Hua et al., 2021;
Revised: The self-supervised loss brings a phenomenon where the final feature vector for classification has a small number of strongly activated elements while the others are close to zero (Hua et al., 2021;

Original: Kalibhat et al., 2022).
Revised: Kalibhat et al., 2022).

Original: These elements of small magnitudes are regarded as noise, which may cause overfitting.
Revised: These elements of small magnitudes are regarded as noise, which may cause overfitting.

Original: We propose a feature denoising (FD) operation to remove their contribution during the fine-tuning phase (see Figure 1).
Revised: We propose a feature denoising (FD) operation to remove their contribution during the fine-tuning phase (see Figure 1).

Original: FD keeps the largest h elements of the feature from the student network and zeros the other elements.
Revised: FD keeps the largest h elements of the feature from the student network and zeros the other elements.

Original: The FD operation is only performed on the feature s L in the last layer of the student network.
Revised: The FD operation is only performed on the feature s L in the last layer of the student network.

Original: Specifically, let s L =
Revised: Specifically, let s L =

Original: [ s ( L, 1) , s ( L, 2) , . . .
Revised: [ s ( L, 1) , s ( L, 2) , . . .

Original: , s ( L,D L ) ] and (cid:101) s L =
Revised: , s ( L,D L ) ] and (cid:101) s L =

Original: [ (cid:101) s ( L, 1) , (cid:101)
Revised: [ (cid:101) s ( L, 1) , (cid:101)

Original: s ( L, 2) , . .
Revised: s ( L, 2) , . .

Original: , (cid:101) s ( L,D L ) ] be the features before and after the FD operation, respectively, where D
Revised: , (cid:101) s ( L,D L ) ] be the features before and after the FD operation, respectively, where D

Original: L is the feature’s dimensionality.
Revised: L is the feature’s dimensionality.

Original: Then the FD operation is defined as:
Revised: Then the FD operation is defined as:

Original: [Equation]
Revised: [Equation]

Original: where β is a hyper-parameter which makes the non-zero elements more distinguishable and top h ( · ) is the operator selecting the largest h elements of the feature.
Revised: where β is a hyper-parameter which makes the non-zero elements more distinguishable and top h ( · ) is the operator selecting the largest h elements of the feature.

Original: Finally, (cid:101) s M is fed to the classifier for fine-tuning.
Revised: Finally, (cid:101) s M is fed to the classifier for fine-tuning.

Original:  E XPERIMENTS 4. D ATASETS AND I MPLEMENTATION
Revised:  E XPERIMENTS 4. D ATASETS AND I MPLEMENTATION

Original: Datasets.
Revised: Datasets.

Original: We evaluate the proposed CLD and FD for the CD-FSC problem on the BSCD-FSL benchmark (Guo et al., 2020) with large domain gaps.
Revised: We evaluate the proposed CLD and FD for the CD-FSC problem on the BSCD-FSL benchmark (Guo et al., 2020) with large domain gaps.

Original: The miniImageNet dataset (Vinyals et al., 2016) serves as the base dataset D B which has sufficient labeled images.
Revised: The miniImageNet dataset (Vinyals et al., 2016) serves as the base dataset D B which has sufficient labeled images.

Original: EuroSAT (Helber et al., 2019), CropDisease (Mohanty et al., 2016), ISIC (Codella et al., 2019) and ChestX (Wang et al., 2017) in BSCD-FSL are the unlabeled target datasets D T .
Revised: EuroSAT (Helber et al., 2019), CropDisease (Mohanty et al., 2016), ISIC (Codella et al., 2019) and ChestX (Wang et al., 2017) in BSCD-FSL are the unlabeled target datasets D T .

Original: We follow the training protocol in
Revised: We follow the training protocol in

Original: STARTUP (Phoo & Hariharan, 2021) and Dynamic-Distillation (Islam et al., 2021a), which allows the whole labeled training set of miniImageNet and a small proportion (20 % ) of unlabeled target images available during the training period.
Revised: STARTUP (Phoo & Hariharan, 2021) and Dynamic-Distillation (Islam et al., 2021a), which allows the whole labeled training set of miniImageNet and a small proportion (20 % ) of unlabeled target images available during the training period.

Original: The remaining 80 % of target images are utilized for fine-tuning and evaluating by building 5 -way K -shot tasks, K ∈ { 1 , 5 } .
Revised: The remaining 80 % of target images are utilized for fine-tuning and evaluating by building 5 -way K -shot tasks, K ∈ { 1 , 5 } .

Original: Implementation details.
Revised: Implementation details.

Original: We implement our model using the MindSpore Lite tool (Mindspore).
Revised: We implement our model using the MindSpore Lite tool (Mindspore).

Original: For a fair comparison, all the methods use ResNet-10 (Guo et al., 2020) as the backbone.
Revised: For a fair comparison, all the methods use ResNet-10 (Guo et al., 2020) as the backbone.

Original: Our
Revised: Our

Original: [Table]
Revised: [Table]

Original: BYOL
Revised: BYOL

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: model is optimized by SGD with the momentum 0.9, weight decay 1e-4, and batch size 32 forepochs.
Revised: model is optimized by SGD with the momentum 0.9, weight decay 1e-4, and batch size 32 forepochs.

Original: The rate is 0.1 at the beginning and decays by 0 .
Revised: The rate is 0.1 at the beginning and decays by 0 .

Original: 1 after the 300 th epoch and 500 th epoch.
Revised: 1 after the 300 th epoch and 500 th epoch.

Original: The hyper-parameters λ in Equation 3, and h and β in Equation 6 are set to 2 , 64 , and 0 . 4 , respectively.
Revised: The hyper-parameters λ in Equation 3, and h and β in Equation 6 are set to 2 , 64 , and 0 . 4 , respectively.

Original: For fine-tuning and evaluating the student network, we randomly sample 600 episodes of 5 -way K -shot tasks.
Revised: For fine-tuning and evaluating the student network, we randomly sample 600 episodes of 5 -way K -shot tasks.

Original: The performance is represented by the average classification accuracy over the 600 episodes within the 95% confidence interval.
Revised: The performance is represented by the average classification accuracy over the 600 episodes within the 95% confidence interval.

Original: In each episode, the parameters of the student are frozen.
Revised: In each episode, the parameters of the student are frozen.

Original: An additional linear classifier is fine-tuned by minimizing the cross-entropy loss.
Revised: An additional linear classifier is fine-tuned by minimizing the cross-entropy loss.

Original: The above mentioned hyperparameters are determined based on the EuroSAT dataset only, and then they are used for evaluation on all the datasets, showing the generalization ability of our method.
Revised: The above mentioned hyperparameters are determined based on the EuroSAT dataset only, and then they are used for evaluation on all the datasets, showing the generalization ability of our method.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: We select several basic and competitive methods for comparison in Table 1.
Revised: We select several basic and competitive methods for comparison in Table 1.

Original: The basic model is
Revised: The basic model is

Original: Transfer that trains the encoder on the labeled base dataset with the cross-entropy loss.
Revised: Transfer that trains the encoder on the labeled base dataset with the cross-entropy loss.

Original: SimCLR (Base) (Chen et al., 2020) trains the model on the base dataset (miniImageNet) with the contrastive loss of SimCLR.
Revised: SimCLR (Base) (Chen et al., 2020) trains the model on the base dataset (miniImageNet) with the contrastive loss of SimCLR.

Original: CI (Luo et al., 2022) trains the encoder on the base dataset and converts the features of the target dataset with a transformation function in the evaluation stage.
Revised: CI (Luo et al., 2022) trains the encoder on the base dataset and converts the features of the target dataset with a transformation function in the evaluation stage.

Original: Transfer+SimCLR is proposed in (Islam et al., 2021b) and exhibits good transferability, which simultaneously optimizes the cross-entropy loss on the base dataset and the contrastive loss of SimCLR on the target dataset.
Revised: Transfer+SimCLR is proposed in (Islam et al., 2021b) and exhibits good transferability, which simultaneously optimizes the cross-entropy loss on the base dataset and the contrastive loss of SimCLR on the target dataset.

Original: STARTUP (Phoo & Hariharan, 2021) trains the model with three loss functions:
Revised: STARTUP (Phoo & Hariharan, 2021) trains the model with three loss functions:

Original: cross-
Revised: cross-

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: entropy loss on the labeled base dataset, and contrastive loss and KD loss on the unlabeled target domain.
Revised: entropy loss on the labeled base dataset, and contrastive loss and KD loss on the unlabeled target domain.

Original: ATA (Wang & Deng, 2021) designs a plug-and-play inductive bias-adaptive task augmentation module.
Revised: ATA (Wang & Deng, 2021) designs a plug-and-play inductive bias-adaptive task augmentation module.

Original: BYOL (Grill et al., 2020) is trained on the target dataset.
Revised: BYOL (Grill et al., 2020) is trained on the target dataset.

Original: ConfeSS (Das et al., 2021) uses labeled target images to find out the useful components of the target image features.
Revised: ConfeSS (Das et al., 2021) uses labeled target images to find out the useful components of the target image features.

Original: Dynamic-Distillation (Islam et al., 2021a) designs a distillation loss that draws support from the dynamically updated teacher network.
Revised: Dynamic-Distillation (Islam et al., 2021a) designs a distillation loss that draws support from the dynamically updated teacher network.

Original: SimCLR trains the encoder on unlabeled target data only and shows good generalization ability.
Revised: SimCLR trains the encoder on unlabeled target data only and shows good generalization ability.

Original: For our method, we build two models BYOL+CLD+FD and
Revised: For our method, we build two models BYOL+CLD+FD and

Original: SimCLR+CLD+FD, which use Equations 5 and 4 for self-supervised losses, respectively.
Revised: SimCLR+CLD+FD, which use Equations 5 and 4 for self-supervised losses, respectively.

Original: Table 1 gives the comparisons among our models and the baselines.
Revised: Table 1 gives the comparisons among our models and the baselines.

Original: All the best results on the four datasets are obtained by either of our two models.
Revised: All the best results on the four datasets are obtained by either of our two models.

Original: In particular on EuroSAT, our SimCLR+CLD+FD can bring 9.38% and 7.42% gains over Dynamic-Distillation and SimCLR on the 5 -way 1 -shot task, respectively.
Revised: In particular on EuroSAT, our SimCLR+CLD+FD can bring 9.38% and 7.42% gains over Dynamic-Distillation and SimCLR on the 5 -way 1 -shot task, respectively.

Original: On average, on the four datasets, SimCLD+CLD+FD outperforms DynamicDistillation significantly (58.77% vs. 53.33%) for 1-shot; 66.94% vs. 65.57% for 5-shot.
Revised: On average, on the four datasets, SimCLD+CLD+FD outperforms DynamicDistillation significantly (58.77% vs. 53.33%) for 1-shot; 66.94% vs. 65.57% for 5-shot.

Original: Besides, although SimCLR+CLD+FD and BYOL+CLD+FD are based on SimCLR and BYOL, respectively, the performances of SimCLR and BYOL are improved greatly.
Revised: Besides, although SimCLR+CLD+FD and BYOL+CLD+FD are based on SimCLR and BYOL, respectively, the performances of SimCLR and BYOL are improved greatly.

Original: Finally, we visualize the feature distributions of STARTUP and our SimCLR+CLD+FD in Figure 3.
Revised: Finally, we visualize the feature distributions of STARTUP and our SimCLR+CLD+FD in Figure 3.

Original: It can be seen that
Revised: It can be seen that

Original: SimCLR+CLD+FD exhibits better clustering results at all the blocks, especially at the last one.
Revised: SimCLR+CLD+FD exhibits better clustering results at all the blocks, especially at the last one.

Original: 4. A BLATION S TUDY ON C ROSS - LEVEL D ISTILLATION
Revised: 4. A BLATION S TUDY ON C ROSS - LEVEL D ISTILLATION

Original: [Figure]
Revised: [Figure]

Original: For simplicity, the combination (according to Equation 1) of the teacher and the old student in Figure 2 is denoted as the united teacher in Figure 4 and Figure 5.
Revised: For simplicity, the combination (according to Equation 1) of the teacher and the old student in Figure 2 is denoted as the united teacher in Figure 4 and Figure 5.

Original: In these experiments, L simclr is used as
Revised: In these experiments, L simclr is used as

Original: L ss and FD is not employed.
Revised: L ss and FD is not employed.

Original: Table 2 gives the results of SimCLR, different single-block distillation structures, and CLD on EuroSAT (5-way 1-shot).
Revised: Table 2 gives the results of SimCLR, different single-block distillation structures, and CLD on EuroSAT (5-way 1-shot).

Original: We can notice that all the single-block distillation structures perform better than SimCLR, and CLD can outperform all the single-block distillation
Revised: We can notice that all the single-block distillation structures perform better than SimCLR, and CLD can outperform all the single-block distillation

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: structures.
Revised: structures.

Original: Table 3 gives the results of SimCLR, different multi-blocks distillation structures and
Revised: Table 3 gives the results of SimCLR, different multi-blocks distillation structures and

Original: CLD.
Revised: CLD.

Original: Compared with SimCLR without KD, all the multi-blocks KD structures improve the model’s overall performance.
Revised: Compared with SimCLR without KD, all the multi-blocks KD structures improve the model’s overall performance.

Original: Second, the one-to-all KD outperforms the same-level KD in most cases.
Revised: Second, the one-to-all KD outperforms the same-level KD in most cases.

Original: Finally, our CLD performs best on average among the three multi-blocks KD structures.
Revised: Finally, our CLD performs best on average among the three multi-blocks KD structures.

Original: We further explore the reason that CLD can exceed other methods.
Revised: We further explore the reason that CLD can exceed other methods.

Original: Rand Index (Rand, 1971) is a metric to reflect the quality of feature clustering.
Revised: Rand Index (Rand, 1971) is a metric to reflect the quality of feature clustering.

Original: The bigger Rand Index is, the better clustering the method provides.
Revised: The bigger Rand Index is, the better clustering the method provides.

Original: Figure 6 is comprised of the Rand Index values of the features in each residual block on the EuroSAT test dataset.
Revised: Figure 6 is comprised of the Rand Index values of the features in each residual block on the EuroSAT test dataset.

Original: Our CLD increases more than all the other methods in each block.
Revised: Our CLD increases more than all the other methods in each block.

Original: It means that our CLD can pick up more discriminative information at each level so that the model gradually hunts the useful features as the network deepens.
Revised: It means that our CLD can pick up more discriminative information at each level so that the model gradually hunts the useful features as the network deepens.

Original: Next
Revised: Next

Original: [Table]
Revised: [Table]

Original: we examine the usefulness of the old student and the effect of the training iteration interval τ (Section 3.2).
Revised: we examine the usefulness of the old student and the effect of the training iteration interval τ (Section 3.2).

Original: Table 4 shows the experimental results of different settings.
Revised: Table 4 shows the experimental results of different settings.

Original: First, CLD without the old student outperforms SimCLR.
Revised: First, CLD without the old student outperforms SimCLR.

Original: Second, using the old student is better than not using it.
Revised: Second, using the old student is better than not using it.

Original: Considering the performance and the memory requirement, we choose τ = 1 in all the other experiments on the four datasets
Revised: Considering the performance and the memory requirement, we choose τ = 1 in all the other experiments on the four datasets

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. A BLATION S TUDY ON F EATURE D ENOISING
Revised: 4. A BLATION S TUDY ON F EATURE D ENOISING

Original: On EuroSAT, the model SimCLR+CLD+FD is used to find optimal h and β on the 5-way 1-shot tasks.
Revised: On EuroSAT, the model SimCLR+CLD+FD is used to find optimal h and β on the 5-way 1-shot tasks.

Original: As shown in Figure 7, h = 64 and β = 0 .
Revised: As shown in Figure 7, h = 64 and β = 0 .

Original: 4 are chosen for the best accuracy.
Revised: 4 are chosen for the best accuracy.

Original: With h = 64 and β = 0 .
Revised: With h = 64 and β = 0 .

Original: 4 in FD, we compare using FD or not in Table 5.
Revised: 4 in FD, we compare using FD or not in Table 5.

Original: We can see that the first five models are improved with FD, while the last two are not.
Revised: We can see that the first five models are improved with FD, while the last two are not.

Original: The different positions, which the
Revised: The different positions, which the

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: h
Revised: h

Original: [Figure]
Revised: [Figure]

Original: A cc u r ac y ( % )
Revised: A cc u r ac y ( % )

Original: [Figure]
Revised: [Figure]

Original: cc u r ac y ( % )
Revised: cc u r ac y ( % )

Original: [Figure]
Revised: [Figure]

Original: FD
Revised: FD

Original: [Figure]
Revised: [Figure]

Original: +
Revised: +

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: auxiliary loss (cross-entropy and/or KD) is applied to in the two groups, are shown in Figures 8(a) and 8(b), respectively.
Revised: auxiliary loss (cross-entropy and/or KD) is applied to in the two groups, are shown in Figures 8(a) and 8(b), respectively.

Original: The reason why FD does not help in Figure 8(b) is that the loss L aux used in the final layer reduces the noisy elements of the final feature vector for classification.
Revised: The reason why FD does not help in Figure 8(b) is that the loss L aux used in the final layer reduces the noisy elements of the final feature vector for classification.

Original: [Figure]
Revised: [Figure]

Original: ℒ "#$ ℒ "#$ ℒ !!
Revised: ℒ "#$ ℒ "#$ ℒ !!

Original: ℒ !!
Revised: ℒ !!

Original: ℒ "#$ λ
Revised: ℒ "#$ λ

Original: 4. A BLATION S TUDY ON W EIGHT C OEFFICIENT OF L OSS F UNCTION
Revised: 4. A BLATION S TUDY ON W EIGHT C OEFFICIENT OF L OSS F UNCTION

Original: We give the results of SimCLR+CLD+FD with the different λ values in Equation 3 on the EuroSAT dataset (5-way 1-shot), as shown in Figure 9.
Revised: We give the results of SimCLR+CLD+FD with the different λ values in Equation 3 on the EuroSAT dataset (5-way 1-shot), as shown in Figure 9.

Original: We can see that the best result is obtained with λ = 2 .
Revised: We can see that the best result is obtained with λ = 2 .

Original: So we set λ
Revised: So we set λ

Original: = 2 in all the related experiments.
Revised: = 2 in all the related experiments.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: In this work, we handle the CD-FSC problems with large domain gaps between the base dataset and the target datasets.
Revised: In this work, we handle the CD-FSC problems with large domain gaps between the base dataset and the target datasets.

Original: We propose the cross-level distillation KD for better transferring the knowledge of the base dataset to the student.
Revised: We propose the cross-level distillation KD for better transferring the knowledge of the base dataset to the student.

Original: We also present the feature denoising operation for mitigating the overfitting.
Revised: We also present the feature denoising operation for mitigating the overfitting.

Original: Our method improves the performance of a strong baseline Dynamic-Distillation by 5.44 % on 1-shot and 1.37 % on 5-shot classification tasks on average in the BSCD-FSL benchmark, establishing new state-of-the-art results.
Revised: Our method improves the performance of a strong baseline Dynamic-Distillation by 5.44 % on 1-shot and 1.37 % on 5-shot classification tasks on average in the BSCD-FSL benchmark, establishing new state-of-the-art results.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: We gratefully acknowledge the support of MindSpore (Mindspore), CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
Revised: We gratefully acknowledge the support of MindSpore (Mindspore), CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.

Original: We would also like to express our sincere gratitude to Mr. Haichen ZHENG, Mrs. Jing LIN, and Ms. Xiao WANG for their unwavering support and confidence in our work.
Revised: We would also like to express our sincere gratitude to Mr. Haichen ZHENG, Mrs. Jing LIN, and Ms. Xiao WANG for their unwavering support and confidence in our work.

Revised: A A PPENDIX A. R ESULTS OF THE DIFFERENT BATCH SIZES

Revised: We present the results of the SimCLR+CLD+FD trained with different batch sizes on the EuroSAT dataset (5-way 1-shot) in Figure 10.

Revised: We can see that the optimal batch size is 32, so we utilize this value in all the experiments of our method.

Revised: [Figure]

Revised: A. R ESULTS OF THE DIFFERENT PERCENTAGES OF UNLABELED IMAGES

Revised: Figure 11 shows that as more unlabeled target images become available, the accuracy of the model will gradually converge to a plateau.

Revised: A. R ESULTS WITH SELF - SUPERVISED PRE - TRAINED TEACHER

Revised: [Table]

Revised: On the EuroSAT dataset (5-way 1-shot), we give the results of the SimCLR+FD+CLD where the teacher is pre-trained in a self-supervised scheme.

Revised: The performances of the SimCLR+CLD+FD are shown in Table 6.