Original: S OURCE - FREE D OMAIN A DAPTATION VIA D ISTRIBUTIONAL A LIGNMENT BY M ATCHING B ATCH N ORMALIZATION S TATISTICS
Revised: S OURCE - FREE D OMAIN A DAPTATION VIA D ISTRIBUTIONAL A LIGNMENT BY M ATCHING B ATCH N ORMALIZATION S TATISTICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we propose a novel domain adaptation method for the source-free setting.
Revised: In this paper, we propose a novel domain adaptation method for the source-free setting.

Original: In this setting, we cannot access source data during adaptation, while unlabeled target data and a model pretrained with source data are given.
Revised: In this setting, we cannot access source data during adaptation, while unlabeled target data and a model pretrained with source data are given.

Original: Due to lack of source data, we cannot directly match the data distributions between domains unlike typical domain adaptation algorithms.
Revised: Due to lack of source data, we cannot directly match the data distributions between domains unlike typical domain adaptation algorithms.

Original: To cope with this problem, we propose utilizing batch normalization statistics stored in the pretrained model to approximate the distribution of unobserved source data.
Revised: To cope with this problem, we propose utilizing batch normalization statistics stored in the pretrained model to approximate the distribution of unobserved source data.

Original: Speciﬁcally, we ﬁx the classiﬁer part of the model during adaptation and only ﬁne-tune the remaining feature encoder part so that batch normalization statistics of the features extracted by the encoder match those stored in the ﬁxed classiﬁer.
Revised: Speciﬁcally, we ﬁx the classiﬁer part of the model during adaptation and only ﬁne-tune the remaining feature encoder part so that batch normalization statistics of the features extracted by the encoder match those stored in the ﬁxed classiﬁer.

Original: Additionally, we also maximize the mutual information between the features and the classiﬁer’s outputs to further boost the classiﬁcation performance.
Revised: Additionally, we also maximize the mutual information between the features and the classiﬁer’s outputs to further boost the classiﬁcation performance.

Original: Experimental results with several benchmark datasets show that our method achieves competitive performance with state-of-the-art domain adaptation methods even though it does not require access to source data.
Revised: Experimental results with several benchmark datasets show that our method achieves competitive performance with state-of-the-art domain adaptation methods even though it does not require access to source data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In typical statistical machine learning algorithms, test data are assumed to stem from the same distribution as training data (Hastie et al., 2009).
Revised: In typical statistical machine learning algorithms, test data are assumed to stem from the same distribution as training data (Hastie et al., 2009).

Original: However, this assumption is often violated in practical situations, and the trained model results in unexpectedly poor performance (QuioneroCandela et al., 2009).
Revised: However, this assumption is often violated in practical situations, and the trained model results in unexpectedly poor performance (QuioneroCandela et al., 2009).

Original: This situation is called domain shift, and many researchers have intensely worked on domain adaptation (Csurka, 2017; Wilson & Cook, 2020) to overcome it.
Revised: This situation is called domain shift, and many researchers have intensely worked on domain adaptation (Csurka, 2017; Wilson & Cook, 2020) to overcome it.

Original: A common approach for domain adaptation is to jointly minimize a distributional discrepancy between domains in a feature space as well as the prediction error of the model (Wilson & Cook, 2020), as shown in Fig.
Revised: A common approach for domain adaptation is to jointly minimize a distributional discrepancy between domains in a feature space as well as the prediction error of the model (Wilson & Cook, 2020), as shown in Fig.

Original: 1(a).
Revised: 1(a).

Original: Deep neural networks (DNNs) are particularly popular for this joint training, and recent methods using DNNs have demonstrated excellent performance under domain shift (Wilson & Cook, 
Revised: Deep neural networks (DNNs) are particularly popular for this joint training, and recent methods using DNNs have demonstrated excellent performance under domain shift (Wilson & Cook, 2020).

Original: Many domain adaptation algorithms assume that they can access labeled source data as well as target data during adaptation.
Revised: Many domain adaptation algorithms assume that they can access labeled source data as well as target data during adaptation.

Original: This assumption is essentially required to evaluate the distributional discrepancy between domains as well as the accuracy of the model’s prediction.
Revised: This assumption is essentially required to evaluate the distributional discrepancy between domains as well as the accuracy of the model’s prediction.

Original: However, it can be unreasonable in some cases, for example, due to data privacy issues or too large-scale source datasets to be handled at the environment where the adaptation is conducted.
Revised: However, it can be unreasonable in some cases, for example, due to data privacy issues or too large-scale source datasets to be handled at the environment where the adaptation is conducted.

Original: To tackle this problem, a few recent studies (Kundu et al., 2020; Li et al., 2020; Liang et al., 2020) have proposed source-free domain adaptation methods in which they do not need to access the source data.
Revised: To tackle this problem, a few recent studies (Kundu et al., 2020; Li et al., 2020; Liang et al., 2020) have proposed source-free domain adaptation methods in which they do not need to access the source data.

Original: In source-free domain adaptation, the model trained with source data is given instead of source data themselves, and it is ﬁne-tuned through adaptation with unlabeled target data so that the ﬁne-tuned model works well in the target domain.
Revised: In source-free domain adaptation, the model trained with source data is given instead of source data themselves, and it is ﬁne-tuned through adaptation with unlabeled target data so that the ﬁne-tuned model works well in the target domain.

Original: Since it seems quite hard to evaluate the distributional discrepancy between unobservable source data and given target data, previous studies mainly focused on how to minimize the prediction error of the model with unlabeled target data, for example, by using pseudo-labeling (Liang et al., 2020) or a conditional generative model (Li et al., 2020).
Revised: Since it seems quite hard to evaluate the distributional discrepancy between unobservable source data and given target data, previous studies mainly focused on how to minimize the prediction error of the model with unlabeled target data, for example, by using pseudo-labeling (Liang et al., 2020) or a conditional generative model (Li et al., 2020).

Original: However, due to lack of the distributional alignment,the performance of those methods is not guaranteed by theories exploited in typical domain adaptation studies (Ben-David et al., 2010).
Revised: However, due to lack of the distributional alignment, those methods heavily depend on noisy target labels obtained through the adaptation, which can result in unstable performance.

Original: [Figure]
Revised: [Figure]

Original: a) General setup commonly adopted in recent typical domain adaptation methods.
Revised: a) General setup commonly adopted in recent typical domain adaptation methods.

Original: This visualization is inspired by (Wilson & Cook, 2020).
Revised: This visualization is inspired by (Wilson & Cook, 2020).

Original: [Figure]
Revised: [Figure]

Original: In this paper, we propose a novel method for source-free domain adaptation.
Revised: In this paper, we propose a novel method for source-free domain adaptation.

Original: Figure 1(b) shows our setup in comparison with that of typical domain adaptation methods shown in Fig. 1(a).
Revised: Figure 1(b) shows our setup in comparison with that of typical domain adaptation methods shown in Fig. 1(a).

Original: In our method, we explicitly minimize the distributional discrepancy between domains by utilizing batch normalization (BN) statistics stored in the pretrained model.
Revised: In our method, we explicitly minimize the distributional discrepancy between domains by utilizing batch normalization (BN) statistics stored in the pretrained model.

Original: Since we ﬁx the pretrained classiﬁer during adaptation, the BN statistics stored in the classiﬁer can be regarded as representing the distribution of source features extracted by the pretrained encoder.
Revised: Since we ﬁx the pretrained classiﬁer during adaptation, the BN statistics stored in the classiﬁer can be regarded as representing the distribution of source features extracted by the pretrained encoder.

Original: Based on this idea, to minimize the discrepancy, we train the target-speciﬁc encoder so that the BN statistics of the target features extracted by the encoder match with those stored in the classiﬁer.
Revised: Based on this idea, to minimize the discrepancy, we train the target-speciﬁc encoder so that the BN statistics of the target features extracted by the encoder match with those stored in the classiﬁer.

Original: We also adopt information maximization as in Liang et al.
Revised: We also adopt information maximization as in Liang et al.

Original: (2020) to further boost the classiﬁcation performance of the classiﬁer in the target domain.
Revised: (2020) to further boost the classiﬁcation performance of the classiﬁer in the target domain.

Original: Our method is apparently simple but effective; indeed, we will validate its advantage through extensive experiments on several benchmark datasets.
Revised: Our method is apparently simple but effective; indeed, we will validate its advantage through extensive experiments on several benchmark datasets.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In this section, we introduce existing works on domain adaptation that are related to ours and also present a formulation of batch normalization.
Revised: In this section, we introduce existing works on domain adaptation that are related to ours and also present a formulation of batch normalization.

Original: 2. D OMAIN ADAPTATION
Revised: 2. D OMAIN ADAPTATION

Original: Given source and target data, the goal of domain adaptation is to obtain a good prediction model that performs well in the target domain (Csurka, 2017; Wilson & Cook, 2020).
Revised: Given source and target data, the goal of domain adaptation is to obtain a good prediction model that performs well in the target domain (Csurka, 2017; Wilson & Cook, 2020).

Original: Importantly, the data distributions are signiﬁcantly different between the domains, which means that we cannot simply train the model with source data to maximize the performance of the model for target data.
Revised: Importantly, the data distributions are signiﬁcantly different between the domains, which means that we cannot simply train the model with source data to maximize the performance of the model for target data.

Original: Therefore, in addition to minimizing the prediction error using labeled source data, many domain adaptation algorithms try to align the data distributions between domains by adversarial training (Ganin et al., 2016; Tzeng et al., 2017; Deng et al., 2019; Xu et al., 2019) or explicitly minimizing a distributional-discrepancy measure (Long et al., 2015; Bousmalis et al., 2016; Long et al., 2017).
Revised: Therefore, in addition to minimizing the prediction error using labeled source data, many domain adaptation algorithms try to align the data distributions between domains by adversarial training (Ganin et al., 2016; Tzeng et al., 2017; Deng et al., 2019; Xu et al., 2019) or explicitly minimizing a distributionaldiscrepancy measure (Long et al., 2015; Bousmalis et al., 2016; Long et al., 2017).

Original: This approach has empirically shown excellent performance and is also validated in theory (BenDavid et al., 2010).
Revised: This approach has empirically shown excellent performance and is also closely connected to theoretical analysis (Ben-David et al., 2010).

Original: However, since this distribution alignment requires access to source data, these methods cannot be directly applied to the source-free domain adaptation setting.
Revised: However, since this distribution alignment requires access to source data, these methods cannot be directly applied to the source-free domain adaptation setting.

Original: In source-free domain adaptation, we can only access target data but not source data, and the model pretrained with the source data is given instead of the source data.
Revised: In source-free domain adaptation, we can only access target data but not source data, and the model pretrained with the source data is given instead of the source data.

Original: This challenging problem has been tackled in recent studies.
Revised: This challenging problem has been tackled in recent studies.

Original: Li et al.
Revised: Li et al.

Original: (2020) proposed joint training of the target model and the conditional GAN (Generative Adversarial Network) (Mirza & Osindero, 2014) that is to generate annotated target data.
Revised: (2020) proposed joint training of the target model and the conditional GAN (Generative Adversarial Network) (Mirza & Osindero, 2014) that is to generate annotated target data.

Original: Liang et al.
Revised: Liang et al.

Original: (2020) explicitly divided the pretrained model into two modules, called a feature encoder and a classiﬁer, and trained the target-speciﬁc feature encoder while ﬁxing the classiﬁer.
Revised: (2020) explicitly divided the pretrained model into two modules, called a feature encoder and a classiﬁer, and trained the target-speciﬁc feature encoder while ﬁxing the classiﬁer.

Original: To make the classiﬁer work well with the target features, this training jointly conducts both information maximization and self-supervised pseudo-labeling with the ﬁxed classiﬁer.
Revised: To make the classiﬁer work well with the target features, this training jointly conducts both information maximization and self-supervised pseudo-labeling with the ﬁxed classiﬁer.

Original: Kundu et al.
Revised: Kundu et al.

Original: (2020) adopted a similar architecture but it has three modules: a backbone model, a feature extractor, and a classiﬁer.
Revised: (2020) adopted a similar architecture but it has three modules: a backbone model, a feature extractor, and a classiﬁer.

Original: In the adaptation phase, only the feature extractor is tuned for the target domain by minimizing the entropy of the classiﬁer’s output.
Revised: In the adaptation phase, only the feature extractor is tuned for the target domain by minimizing the entropy of the classiﬁer’s output.

Original: Since the methods shown above do not try to align data distributions between domains, they cannot essentially avoid conﬁrmation bias of the model and also cannot beneﬁt from well-exploited theories in the studies on typical domain adaptation problems (Ben-David et al., 2010).
Revised: Since the methods shown above do not try to align data distributions between domains, they cannot essentially avoid conﬁrmation bias of the model and also cannot beneﬁt from well-exploited theories in the studies on typical domain adaptation problems (Ben-David et al., 2010).

Original: 2. B ATCH NORMALIZATION
Revised: 2. B ATCH NORMALIZATION

Original: Batch normalization (BN) (Ioffe & Szegedy, 2015) has been widely used in modern architectures of deep neural networks to make their training faster as well as being stable.
Revised: Batch normalization (BN) (Ioffe & Szegedy, 2015) has been widely used in modern architectures of deep neural networks to make their training faster as well as being stable.

Original: It normalizes each input feature within a mini-batch in a channel-wise manner so that the output has zero-mean and unit-variance.
Revised: It normalizes each input feature within a mini-batch in a channel-wise manner so that the output has zero-mean and unit-variance.

Original: Let B and { z i } Bi =1 denote the mini-batch size and the input features to the batch normalization, respectively.
Revised: Let B and { z i } Bi =1 denote the mini-batch size and the input features to the batch normalization, respectively.

Original: Here, we assume that the input features consist of C channels as z i =
Revised: Here, we assume that the input features consist of C channels as z i =

Original: [ z (1) i , ..., z ( C ) i ] and each channel contains n c features.
Revised: [ z (1) i , ..., z ( C ) i ] and each channel contains n c features.

Original: BN ﬁrst computes the means { µ c } C c =1 and variances { σ 2 c } Cc =1 of the features for each channel within the mini-batch:
Revised: BN ﬁrst computes the means { µ c } C c =1 and variances { σ 2 c } Cc =1 of the features for each channel within the mini-batch:

Original: [Equation]
Revised: [Equation]

Original: where z ( c ) i
Revised: where z ( c ) i

Original: [ j ] is the j -th feature in z ( c ) i .
Revised: [ j ] is the j -th feature in z ( c ) i .

Original: Then, it normalizes the input features by using the computed
Revised: Then, it normalizes the input features by using the computed

Original: BN statistics:
Revised: BN statistics:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is a small positive constant for numerical stability.
Revised: where (cid:15) is a small positive constant for numerical stability.

Original: In the inference phase, BN cannot always compute those statistics, because the input data do not necessarily compose a mini-batch.
Revised: In the inference phase, BN cannot always compute those statistics, because the input data do not necessarily compose a mini-batch.

Original: Instead,
Revised: Instead,

Original: BN stores the exponentially weighted averages of the BN statistics in the training phase and uses them in the inference phase to compute ˜ z in Eq.
Revised: BN stores the exponentially weighted averages of the BN statistics in the training phase and uses them in the inference phase to compute ˜ z in Eq.

Original: (2) (Ioffe & Szegedy, 2015).
Revised: (2) (Ioffe & Szegedy, 2015).

Original: Since BN renormalizes features to have zero-mean and unit-variance, several methods (Li et al., 2018; Chang et al., 2019; Wang et al., 2019) adopted domain-speciﬁc BN to explicitly align both the distribution of source features and that of target features into a common distribution.
Revised: Since BN renormalizes features to have zero-mean and unit-variance, several methods (Li et al., 2018; Chang et al., 2019; Wang et al., 2019) adopted domain-speciﬁc BN to explicitly align both the distribution of source features and that of target features into a common distribution.

Original: Since the domain-speciﬁc BN methods are jointly trained during adaptation, we cannot use these methods in the source-free setting.
Revised: Since the domain-speciﬁc BN methods are jointly trained during adaptation, we cannot use these methods in the source-free setting.

Original:  P ROPOSED METHOD
Revised:  P ROPOSED METHOD

Original: Figure 2 shows an overview of our method.
Revised: Figure 2 shows an overview of our method.

Original: We assume that the model pretrained with source data is given, and it conducts BN at least once somewhere inside the model.
Revised: We assume that the model pretrained with source data is given, and it conducts BN at least once somewhere inside the model.

Original: Before conducting domain adaptation, we divide the model in two sub-models: a feature encoder and a classiﬁer, so that BN comes at the very beginning of the classiﬁer.
Revised: Before conducting domain adaptation, we divide the model in two sub-models: a feature encoder and a classiﬁer, so that BN comes at the very beginning of the classiﬁer.

Original: Then, for domain adaptation, we ﬁne-tune the encoder with unlabeled target data with the classiﬁer ﬁxed.
Revised: Then, for domain adaptation, we ﬁne-tune the encoder with unlabeled target data with the classiﬁer ﬁxed.

Original: After adaptation, we use the ﬁne-tuned encoder and the ﬁxed classiﬁer to predict the class of test data in the target domain.
Revised: After adaptation, we use the ﬁne-tuned encoder and the ﬁxed classiﬁer to predict the class of test data in the target domain.

Original: To make the ﬁxed classiﬁer work well in the target domain after domain adaptation, we aim to obtain a ﬁne-tuned encoder that satisﬁes the following two properties:
Revised: To make the ﬁxed classiﬁer work well in the target domain after domain adaptation, we aim to obtain a ﬁne-tuned encoder that satisﬁes the following two properties:

Original: • The distribution of target features extracted by the ﬁne-tuned encoder is well aligned to that of source features extracted by the pretrained encoder.
Revised: • The distribution of target features extracted by the ﬁne-tuned encoder is well aligned to that of source features extracted by the pretrained encoder.

Original: • The features extracted by the ﬁne-tuned encoder are sufﬁciently
Revised: • The features extracted by the ﬁne-tuned encoder are sufﬁciently

Original: discriminative for the ﬁxed classiﬁer to accurately predict the class of input target data.
Revised: discriminative for the ﬁxed classiﬁer to accurately predict the class of input target data.

Original: To this end, we jointly minimize both the BN-statistics matching loss and information maximization loss to ﬁne-tune the encoder.
Revised: To this end, we jointly minimize both the BN-statistics matching loss and information maximization loss to ﬁne-tune the encoder.

Original: In the former loss, we approximate the distribution of unobservable
Revised: In the former loss, we approximate the distribution of unobservable

Original: Pretraining phase
Revised: Pretraining phase

Original: CalculateBN statistics { ො𝜇 𝑐 , ො𝜎 𝑐 }
Revised: CalculateBN statistics { ො𝜇 𝑐 , ො𝜎 𝑐 }

Original: Information maximization loss: 𝐿 𝐼𝑀
Revised: Information maximization loss: 𝐿 𝐼𝑀

Original: Target data
Revised: Target data

Original: Adaptation phase
Revised: Adaptation phase

Original: Source data
Revised: Source data

Original: E n c od e r
Revised: E n c od e r

Original: C l a ss i f
Revised: C l a ss i f

Original: i e r
Revised: i e r

Original: E n c od e r
Revised: E n c od e r

Original: C l a ss i f
Revised: C l a ss i f

Original: i e r
Revised: i e r

Original: BN
Revised: BN

Original: BN
Revised: BN

Original: BN statistics matching loss: 𝐿 𝐵𝑁𝑆 {𝜇 𝑐 , 𝜎 𝑐 }
Revised: BN statistics matching loss: 𝐿 𝐵𝑁𝑆 {𝜇 𝑐 , 𝜎 𝑐 }

Original: [Figure]
Revised: [Figure]

Original: Conv
Revised: Conv

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Softmax …
Revised: Softmax …

Original: Classification loss
Revised: Classification loss

Original: Initialize
Revised: Initialize

Original: Copy & fix source features by using the BN statistics stored in the ﬁrst BN layer of the classiﬁer, and the loss explicitly evaluates the discrepancy between source and target feature distributions based on those statistics.
Revised: Copy & fix source features by using the BN statistics stored in the ﬁrst BN layer of the classiﬁer, and the loss explicitly evaluates the discrepancy between source and target feature distributions based on those statistics.

Original: Therefore, minimizing this loss leads to satisfying the ﬁrst property shown above.
Revised: Therefore, minimizing this loss leads to satisfying the ﬁrst property shown above.

Original: On the other hand, the latter loss is to make the predictions by the ﬁxed classiﬁer certain for every target sample as well as diverse within all target data, and minimizing this loss leads to fulﬁlling the second property.
Revised: On the other hand, the latter loss is to make the predictions by the ﬁxed classiﬁer certain for every target sample as well as diverse within all target data, and minimizing this loss leads to fulﬁlling the second property.

Original: Below, we describe the details of these losses.
Revised: Below, we describe the details of these losses.

Original: 3. D ISTRIBUTION ALIGNMENT BY MATCHING BATCH NORMALIZATION STATISTICS
Revised: 3. D ISTRIBUTION ALIGNMENT BY MATCHING BATCH NORMALIZATION STATISTICS

Original: Since the whole model is pretrained with source data and we ﬁx the classiﬁer while ﬁnetuning the encoder, the BN statistics stored in the ﬁrst BN in the classiﬁer can be seen as the statistics of the source features extracted by the pretrained encoder.
Revised: Since the whole model is pretrained with source data and we ﬁx the classiﬁer while ﬁnetuning the encoder, the BN statistics stored in the ﬁrst BN in the classiﬁer can be seen as the statistics of the source features extracted by the pretrained encoder.

Original: We approximate the source-feature distribution by using these statistics.
Revised: We approximate the source-feature distribution by using these statistics.

Original: Speciﬁcally, we simply use a Gaussian distribution for each channel denoted by N (ˆ µ c , ˆ σ 2 c ) where ˆ µ c and ˆ σ 2 c are the mean and variance of the Gaussian distribution which are the stored BN statistics corresponding to the c -th channel.
Revised: Speciﬁcally, we simply use a Gaussian distribution for each channel denoted by N (ˆ µ c , ˆ σ 2 c ) where ˆ µ c and ˆ σ 2 c are the mean and variance of the Gaussian distribution which are the stored BN statistics corresponding to the c -th channel.

Original: To match the feature distributions between domains, we deﬁne the BN-statistics matching loss, which evaluates the averaged Kullback-Leibler (KL) divergence from the target-feature distribution to the approximated source-feature distribution:
Revised: To match the feature distributions between domains, we deﬁne the BN-statistics matching loss, which evaluates the averaged Kullback-Leibler (KL) divergence from the target-feature distribution to the approximated source-feature distribution:

Original: [Equation]
Revised: [Equation]

Original: where { x i } Bi =1 is a mini-batch from the target data, θ is a set of trainable parameters of the encoder, and µ c and σ c are the BN statistics of the c -th channel computed from the target mini-batch.
Revised: where { x i } Bi =1 is a mini-batch from the target data, θ is a set of trainable parameters of the encoder, and µ c and σ c are the BN statistics of the c -th channel computed from the target mini-batch.

Original: Note that, since µ c and σ c are calculated from the features extracted by the encoder, they depend on θ .
Revised: Note that, since µ c and σ c are calculated from the features extracted by the encoder, they depend on θ .

Original: Here, we also approximate the target-feature distribution with another Gaussian distribution so that the KL divergence can be efﬁciently computed in a parametric manner.
Revised: Here, we also approximate the target-feature distribution with another Gaussian distribution so that the KL divergence can be efﬁciently computed in a parametric manner.

Original: By minimizing this loss, we can explicitly reduce the discrepancy between the distribution of unobservable source features and that of target features.
Revised: By minimizing this loss, we can explicitly reduce the discrepancy between the distribution of unobservable source features and that of target features.

Original: In Eq.
Revised: In Eq.

Original: (3), we chose the KL divergence to measure the distributional discrepancy between domains.
Revised: (3), we chose the KL divergence to measure the distributional discrepancy between domains.

Original: There are two reasons for this choice.
Revised: There are two reasons for this choice.

Original: First, the KL divergence between two Gaussian distributions is easy to compute with the BN statistics as shown in Eq.
Revised: First, the KL divergence between two Gaussian distributions is easy to compute with the BN statistics as shown in Eq.

Original: Moreover, since these statistics are naturally computed in the BN layer, calculating this divergence only requires tiny calculation costs.
Revised: Moreover, since these statistics are naturally computed in the BN layer, calculating this divergence only requires tiny calculation costs.

Original: Secondly, it can be theoretically justiﬁed from the perspective of risk minimization in the target domain.
Revised: Secondly, it would be a theoretically-inspired design from the perspective of risk minimization in the target domain.

Original: When we consider a binary classiﬁcation task, the expected risk of any hypothesis h in the target domain can be upper-bounded under some mild assumptions as the following inequality (Ben-David et al., 2010):
Revised: When we consider a binary classiﬁcation task, the expected risk of any hypothesis h in the target domain can be upper-bounded under some mild assumptions as the following inequality (Ben-David et al., 2010):

Original: [Equation]
Revised: [Equation]

Original: where r S ( h ) and r T ( h ) denote the expected risk of h under the source-data distribution p S and target-data distribution p T , respectively, d 1 ( p, q ) represents the total variation distance between p and q , and β is a constant value that is expected to be sufﬁciently small.
Revised: where r S ( h ) and r T ( h ) denote the expected risk of h under the source-data distribution p S and target-data distribution p T , respectively, d 1 ( p, q ) represents the total variation distance between p and q , and β is a constant value that is expected to be sufﬁciently small.

Original: This inequality roughly gives a theoretical justiﬁcation to recent domain adaptation algorithms, that is, joint minimization of both the distributional discrepancy between domains (corresponding to the second term of the bound in Eq.
Revised: This inequality roughly gives a theoretical justiﬁcation to recent domain adaptation algorithms, that is, joint minimization of both the distributional discrepancy between domains (corresponding to the second term of the bound in Eq.

Original: (4)) and the prediction error of the model (corresponding to the ﬁrst term of the bound in Eq.
Revised: (4)) and the prediction error of the model (corresponding to the ﬁrst term of the bound in Eq.

Original: Here, the total variation distance can be related to the KL divergence by Pinsker’s inequality (Csiszar & K¨orner, 2011):
Revised: Here, the total variation distance can be related to the KL divergence by Pinsker’s inequality (Csiszar & K¨orner, 2011):

Original: [Equation]
Revised: [Equation]

Original: Consequently, we can guarantee that minimizing the KL divergence between domains minimizes the bound of the target risk.
Revised: Consequently, we can guarantee that minimizing the KL divergence between domains minimizes the bound of the target risk.

Original: 3. D OMAIN ADAPTATION WITHOUT ACCESS TO THE SOURCE DATA
Revised: 3. D OMAIN ADAPTATION WITHOUT ACCESS TO THE SOURCE DATA

Original: Only aligning the marginal feature-distributions between domains does not guarantee that the ﬁxed classiﬁer works well in the target domain, because the features extracted by the encoder are not necessarily discriminative.
Revised: Only aligning the marginal feature-distributions between domains does not guarantee that the ﬁxed classiﬁer works well in the target domain, because the features extracted by the encoder are not necessarily discriminative.

Original: If the features are sufﬁciently discriminative for the classiﬁer, we can expect that the output of the classiﬁer is almost always a one-hot vector but is diverse within the target data.
Revised: If the features are sufﬁciently discriminative for the classiﬁer, we can expect that the output of the classiﬁer is almost always a one-hot vector but is diverse within the target data.

Original: Therefore, following the approach presented in Liang et al.
Revised: Therefore, following the approach presented in Liang et al.

Original: (2020), we also adopt the information maximization loss to make the classiﬁer work accurately.
Revised: (2020), we also adopt the information maximization loss to make the classiﬁer work accurately.

Original: [Equation]
Revised: [Equation]

Original: where H ( p ( y ))
Revised: where H ( p ( y ))

Original: = − (cid:80) y (cid:48)
Revised: = − (cid:80) y (cid:48)

Original: p ( y (cid:48) ) log p ( y (cid:48) ) is the entropy function, and f θ ( x ) denotes the output of the classiﬁer.
Revised: p ( y (cid:48) ) log p ( y (cid:48) ) is the entropy function, and f θ ( x ) denotes the output of the classiﬁer.

Original: The ﬁrst term in the right-hand side of Eq.
Revised: The ﬁrst term in the right-hand side of Eq.

Original: (6) is the negative entropy of the averaged output of the classiﬁer, and minimizing it leads to large diversity of the output within the minibatch.
Revised: (6) is the negative entropy of the averaged output of the classiﬁer, and minimizing it leads to large diversity of the output within the minibatch.

Original: The second term is the averaged entropy of the classiﬁer’s output, and minimizing it makes the outputs close to one-hot vectors.
Revised: The second term is the averaged entropy of the classiﬁer’s output, and minimizing it makes the outputs close to one-hot vectors.

Original: Therefore, the features extracted by the target encoder are induced to be discriminative by minimizing the information maximization loss.
Revised: Therefore, the features extracted by the target encoder are induced to be discriminative by minimizing the information maximization loss.

Original: Finally, our source-free domain adaptation method is formulated as joint minimization of both the
Revised: Finally, our source-free domain adaptation method is formulated as joint minimization of both the

Original: BN-statistics matching loss in Eq.
Revised: BN-statistics matching loss in Eq.

Original: (3) and the information maximization loss in Eq.
Revised: (3) and the information maximization loss in Eq.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where D t is the target dataset from which the mini-batch is sampled, and a hyper-parameter λ controls the balance between the two terms.
Revised: where D t is the target dataset from which the mini-batch is sampled, and a hyper-parameter λ controls the balance between the two terms.

Original: Note that this optimization can be conducted without the source data, which means that we do not need to access to the source data during adaptation.
Revised: Note that this optimization can be conducted without the source data, which means that we do not need to access to the source data during adaptation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted experiments with several datasets that are commonly used in existing works on domain adaptation.
Revised: We conducted experiments with several datasets that are commonly used in existing works on domain adaptation.

Original: Speciﬁcally, we used digit recognition datasets (MNIST (LeCun et al., 1998),
Revised: Speciﬁcally, we used digit recognition datasets (MNIST (LeCun et al., 1998),

Original: USPS (LeCun et al., 1990), and SVHN (Netzer et al., 2011)) and an object recognition dataset (Ofﬁce-31 dataset (Saenko et al., 2010)).
Revised: USPS (LeCun et al., 1990), and SVHN (Netzer et al., 2011)) and an object recognition dataset (Ofﬁce-31 dataset (Saenko et al., 2010)).

Original: In the experiment, we ﬁrst pretrained the model with the source training data.
Revised: In the experiment, we ﬁrst pretrained the model with the source training data.

Original: Following the setup in Liang et al.
Revised: Following the setup in Liang et al.

Original: (2020), we used standard cross-entropy loss with label smoothing for this pretraining.
Revised: (2020), we used standard cross-entropy loss with label smoothing for this pretraining.

Original: Then, we apply our source-free domain adaptation
Revised: Then, we apply our source-free domain adaptation

Original: [Table]
Revised: [Table]

Original: method to ﬁne-tune the pretrained model with the target training data.
Revised: method to ﬁne-tune the pretrained model with the target training data.

Original: We used Adam optimizer for both pretraining and adaptation.
Revised: We used Adam optimizer for both pretraining and adaptation.

Original: The number of iterations in the optimization was set to 30,
Revised: The number of iterations in the optimization was set to 30,

Original: [Table]
Revised: [Table]

Original: and the batch size was set to 64.
Revised: and the batch size was set to 64.

Original: The hyper-parameter λ in Eq.
Revised: The hyper-parameter λ in Eq.

Original: (7) is set to 10 in all experiments except for those in section 4.3.
Revised: (7) is set to 10 in all experiments except for those in section 4.3.

Original: The performance of the domain adaptation is evaluated by test accuracy of the ﬁne-tuned model on the target test data.
Revised: The performance of the domain adaptation is evaluated by test accuracy of the ﬁne-tuned model on the target test data.

Original: We report the averaged accuracy as well as the standard deviation over ﬁve runs with random initialization of the model at the pretraining phase.
Revised: We report the averaged accuracy as well as the standard deviation over ﬁve runs with random initialization of the model at the pretraining phase.

Original: We compared the performance of our method with those of the state-of-the-art methods for sourcefree domain adaptation (Li et al., 2020; Liang et al., 2020), which are most related to our work.
Revised: We compared the performance of our method with those of the state-of-the-art methods for sourcefree domain adaptation (Li et al., 2020; Liang et al., 2020), which are most related to our work.

Original: We did not include the work by Kundu et al.
Revised: We did not include the work by Kundu et al.

Original: (2020) in this comparison, because it is designed for more difﬁcult setting, called universal domain adaptation.
Revised: (2020) in this comparison, because it is designed for more difﬁcult setting, called universal domain adaptation.

Original: For reference, we also show the performance of the recent methods for typical domain adaptation (Tzeng et al., 2017; Deng et al., 2019; Xu et al., 2019), though they require access to the source data during adaptation.
Revised: For reference, we also show the performance of the recent methods for typical domain adaptation (Tzeng et al., 2017; Deng et al., 2019; Xu et al., 2019), though they require access to the source data during adaptation.

Original: 4. O BJECT RECOGNITION DATASET
Revised: 4. O BJECT RECOGNITION DATASET

Original: The Ofﬁce-31 dataset comprises three domains: Amazon (A), DSLR (D), and Webcam (W).
Revised: The Ofﬁce-31 dataset comprises three domains: Amazon (A), DSLR (D), and Webcam (W).

Original: We examined all possible combinations for the adaptation, which results in six scenarios.
Revised: We examined all possible combinations for the adaptation, which results in six scenarios.

Original: Following the setup of Ganin et al.
Revised: Following the setup of Ganin et al.

Original: ; Liang et al.
Revised: ; Liang et al.

Original: (2020), we used ResNet-50 pretrained with the ImageNet classiﬁcation dataset as a backbone model.
Revised: (2020), we used ResNet-50 pretrained with the ImageNet classiﬁcation dataset as a backbone model.

Original: We removed the original FC layer from the pretrained
Revised: We removed the original FC layer from the pretrained

Original: ResNet-50 and added a bottleneck FC layer (256 units) and a classiﬁcation FC layer (31 units).
Revised: ResNet-50 and added a bottleneck FC layer (256 units) and a classiﬁcation FC layer (31 units).

Original: A BN layer is put before and after the bottleneck layer, and we used the last one to calculate our
Revised: A BN layer is put before and after the bottleneck layer, and we used the last one to calculate our

Original: BN-statistics matching loss.
Revised: BN-statistics matching loss.

Original: Note that the backbone part is ﬁxed in our experiments.
Revised: Note that the backbone part is ﬁxed in our experiments.

Original: Table 1 shows the test accuracy of the adapted models at the target data.
Revised: Table 1 shows the test accuracy of the adapted models at the target data.

Original: The results shown above the double line are those of the source-free domain adaptation methods, while the remaining ones are those of the other typical domain adaptation methods.
Revised: The results shown above the double line are those of the source-free domain adaptation methods, while the remaining ones are those of the other typical domain adaptation methods.

Original: Our method achieved the best accuracy at three out of six scenarios, and, surprisingly, its performance reached or exceeded the performance of the state-of-the-art typical domain adaptation methods in those cases.
Revised: Our method achieved the best accuracy at three out of six scenarios, and, surprisingly, its performance reached or exceeded the performance of the state-of-the-art typical domain adaptation methods in those cases.

Original: Moreover, our method also shows competitive performance in the other scenarios except for A → D. Since SHOT (Liang et al., 2020) also adopts the information maximization loss, these results indicate that our BN-statistics matching loss substantially improves the performance of the adaptation by successfully reducing the distributional discrepancy between domains.
Revised: Moreover, our method also shows competitive performance in the other scenarios except for A → D. Since SHOT (Liang et al., 2020) also adopts the information maximization loss, these results indicate that our BN-statistics matching loss substantially improves the performance of the adaptation by successfully reducing the distributional discrepancy between domains.

Original: The model adaptation (Li et al., 2020) also works well through the all scenarios.
Revised: The model adaptation (Li et al., 2020) also works well through the all scenarios.

Original: However, considering that it requires training of a conditional GAN while adaptation, our method is quite appealing due to simplicity of its training procedure as well as its high performance.
Revised: However, considering that it requires training of a conditional GAN while adaptation, our method is quite appealing due to simplicity of its training procedure as well as its high performance.

Original: 4. D IGIT RECOGNITION DATASETS
Revised: 4. D IGIT RECOGNITION DATASETS

Original: We examined USPS ↔ MNIST and SVHN → MNIST scenarios.
Revised: We examined USPS ↔ MNIST and SVHN → MNIST scenarios.

Original: Following the previous studies (Long et al., 2018; Liang et al., 2020), we used the classical LeNet-5 network for the former scenario,
Revised: Following the previous studies (Long et al., 2018; Liang et al., 2020), we used the classical LeNet-5 network for the former scenario,

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: while a variant of LeNet, called DTN, is used for the latter one.
Revised: while a variant of LeNet, called DTN, is used for the latter one.

Original: For both models, we used the last
Revised: For both models, we used the last

Original: BN layer in the model to calculate the BN statistics matching loss in our method.
Revised: BN layer in the model to calculate the BN statistics matching loss in our method.

Original: Table 2 shows the experimental results with the digit recognition datasets.
Revised: Table 2 shows the experimental results with the digit recognition datasets.

Original: Although our method did not achieve the best performance among the source-free methods, it stably achieved the second highest accuracy in all scenarios.
Revised: Although our method did not achieve the best performance among the source-free methods, it stably achieved the second highest accuracy in all scenarios.

Original: Similarly in the results with Ofﬁce-31 dataset, our method exceeds the performance of the typical domain adaptation methods in two scenarios, namely USPS → MNIST and SVHN → MNIST.
Revised: Similarly in the results with Ofﬁce-31 dataset, our method exceeds the performance of the typical domain adaptation methods in two scenarios, namely USPS → MNIST and SVHN → MNIST.

Original: 4. P ERFORMANCE SENSITIVITY TO THE HYPER - PARAMETER AND DATASET SIZE
Revised: 4. P ERFORMANCE SENSITIVITY TO THE HYPER - PARAMETER AND DATASET SIZE

Original: We investigated the performance sensitivity of our method to the hyper-parameter setting and that to the size of the target dataset.
Revised: We investigated the performance sensitivity of our method to the hyper-parameter setting and that to the size of the target dataset.

Original: In this experiment, we used SVHN → MNIST scenario, and the experimental settings are same with those in the previous experiment unless otherwise noted.
Revised: The experimental settings are same with those in the previous experiment unless otherwise noted.

Original: Our method introduces single hyper-parameter, which is λ in Eq.
Revised: Our method introduces single hyper-parameter, which is λ in Eq.

Original: We ﬁrst investigated the performance sensitivity to the value of λ .
Revised: We ﬁrst investigated the performance sensitivity to the value of λ .

Original: Since we can only access the unlabeled target data during adaptation, it is essentially hard to appropriately tune the hyper parameter.
Revised: Since we can only access the unlabeled target data during adaptation, it is essentially hard to appropriately tune the hyper parameter.

Original: Therefore, high stability of the performance under a suboptimal setting of the hyper-parameter is required in the source-free domain adaptation.
Revised: Therefore, high stability of the performance under a suboptimal setting of the hyper-parameter is required in the source-free domain adaptation.

Original: In the experiment, we varied the value of λ from 0 .
Revised: In the experiment, we varied the value of λ from 0 .

Original: 01 to 100 and used it in our method to conduct the adaptation.
Revised: 01 to 100 and used it in our method to conduct the adaptation with the digit recognition datasets.

Original: Figure 3(a) shows how the test accuracy of the adapted model changes according to the value of λ .
Revised: Figure 3 shows how the test accuracy of the adapted model changes according to the value of λ .

Original: The performance of our method is quite stable against the change of the value of λ .
Revised: In all adaptation scenarios, the performance of our method is quite stable against the change of the value of λ .

Original: It keeps almost same within the wide range of the value of λ , speciﬁcally 0 .
Revised: It keeps almost same within the wide range of the value of λ , speciﬁcally 0 .

Original: 1 ≤ λ ≤ 50 .
Revised: 2 ≤ λ ≤ 50 .

Original: We also investigated the performance of our method in case of small-scale target data.
Revised: We also investigated the performance of our method in case of small-scale target data.

Original: This investigation is crucial, because, considering the motivation of domain adaptation, we cannot always expect sufﬁciently large amount of the target data.
Revised: This investigation is crucial, because, considering the motivation of domain adaptation, we cannot always expect sufﬁciently large amount of the target data.

Revised: Since MNIST is the largest target dataset used in our experiments, we conducted this investigation with SVHN → MNIST adaptation.

Original: To make the small-scale target data, we randomly selected a subset of the original target training data whilekeeping the class prior same with that in
Revised: To make the small-scale target data, we randomly selected a subset of the original target training data while

Original: [Figure]
Revised: [Figure]

Original:  the original dataset.
Revised: keeping the class prior same with that in the original dataset.

Original: Figure 3(b) shows how the test accuracy after the adaptation changes according to the size of the target dataset.
Revised: Figure 4 shows how the test accuracy after the adaptation changes according to the size of the target dataset.

Original: As decreasing the number of the target data, the performance of our method becomes deteriorated to some extent.
Revised: As decreasing the number of the target data, the performance of our method becomes deteriorated to some extent.

Original: However, even when there are only 600 samples in the target dataset, our method still achieved 98 . 0% , which is comparable performance with those of the typical domain adaptation methods using full target dataset as well as source dataset.
Revised: However, even when there are only 600 samples in the target dataset, our method still achieved 98 . 0% , which is comparable performance with those of the typical domain adaptation methods using full target dataset as well as source dataset.

Original: Figure 4 shows how the test accuracy by the model changes during adaptation.
Revised: Figure 5 shows how the test accuracy by the model changes during adaptation.

Original: The accuracy is stably and monotonically improved even when the number of the target data is small.
Revised: The accuracy is stably and monotonically improved even when the number of the target data is small.

Original: It means that our method can effectively avoid overﬁtting to the small-scale target dataset.
Revised: It means that our method can effectively avoid overﬁtting to the small-scale target dataset.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a novel domain adaptation method for source-free setting.
Revised: We proposed a novel domain adaptation method for source-free setting.

Original: To match the distributions between unobservable source data and given target data, we utilize the BN statistics stored in the pretrained model to explicitly estimate and minimize the distributional discrepancy between domains.
Revised: To match the distributions between unobservable source data and given target data, we utilize the BN statistics stored in the pretrained model to explicitly estimate and minimize the distributional discrepancy between domains.

Original: This approach is quite efﬁcient in terms of  computational cost and is also theoretically validated.
Revised: This approach is quite efﬁcient in terms of the computational cost and can be justiﬁed from the perspective of risk minimization in the target domain.

Original: Experimental results with several benchmark datasets have shown that our method performs well even though it does not require the access to the source data.
Revised: Experimental results with several benchmark datasets have shown that our method performs well even though it does not require the access to the source data.

Original: Moreover, its performance was empirically quite stable against suboptimal hyper-parameter setting or limited size of the target dataset.
Revised: Moreover, its performance was empirically quite stable against suboptimal hyper-parameter setting or limited size of the target dataset.

Original: In concolusion, we argue that our method is quite promising to tackle many real-world problems that are hard to solve with existing domain adaptation methods.
Revised: In conclusion, we argue that our method is quite promising to tackle many real-world problems that are hard to solve with existing domain adaptation methods.