Paper: SF(DA)$^2$: Source-free Domain Adaptation Through the Lens of Data Augmentation (ICLR-2024)

Reviewer comment (Novelty):
The proposal of SNC is not new. SFDA method AaD already introduced quite similar method (in both motivation and final objective form). More specifically, the proposed SNC just takes [1] into the SFDA task.

Author response:
AaD and our method have distinct motivations. We define an augmentation graph based on the given target features. Performing spectral clustering on this augmentation graph identifies meaningful partitions or clusters in the prediction space, which is equivalent to minimizing the SNC loss. While our method and AaD share similar forms for their final loss functions, as you pointed out, our research introduces an innovative perspective on SFDA and significantly expands the scope of research in this area, as acknowledged by Reviewer 44EQ.

Additionally, while we were inspired by the spectral contrastive loss (Spectral CL) to find partitions of our augmentation graph, there are distinct differences between Spectral CL and our SNC loss. Spectral CL utilizes augmented data as positive pairs, and data augmentation requires prior knowledge about the dataset. For instance, transforming the color of a lemon to green would turn it into a lime, thus changing the class of the data. Such augmentation that fails to preserve class information can introduce bias to the model, potentially bringing harm to the model's performance [1]. Hence, while Spectral CL requires class-preserving augmentations based on prior knowledge, our SNC loss determines positive pairs without needing explicit augmentation and prior knowledge.

Reviewer comment (Experiments):
The introductions of IFA and FD are the main contribution of the paper. In my understanding, IFA loss plays similar role as the first term in SNC loss, while FD loss plays the similar role as the second term of SNC loss (here FD loss may be better as it quantifies the similarity degrees between different categories). I am just wondering what if only using FD and IFA losses for adaptation.

Author response:
The IFA loss is derived as the upper bound of the expectation of the first term of the SNC loss. However, the FD loss is not directly related to the second term of the SNC loss. Therefore, we can expect that using only IFA and FD losses may not correctly identify the partition of the augmentation graph, leading to inadequate adaptation. This is confirmed through experiments on the VisDA dataset, which demonstrate an accuracy of 51.4%, in line with our expectations.

Reviewer comment (Evaluation):
In Fig. 2, the results show that the proposed method can not improve SHOT, while in AaD it indicates that SHOT is actually conducting similar operation as NRC and AaD. Do authors have clear explanation about the phenomenon, is it due to the pseudo labeling used in SHOT?

Author response:
Thank you for the good question. IFA and FD losses utilize local neighbor information in the feature space, while NRC and AaD share a similar philosophy. In contrast, SHOT aligns the target features with global source hypotheses for each class. In the AaD paper, the following is mentioned: "In a nutshell, compared to SHOT and BNM, our method first considers the local feature structure to cluster target features." Moreover, IFA and FD losses make use of class-wise covariance matrices, while SHOT does not. This difference in approach between SHOT and IFA can contribute to a decrease in performance.

Reviewer comment (Evaluation):
In Fig. 3, with only FD or IFA on SNC will deteriorate the performance. The authors posit some hypothesis for explanation, while I think there may need more results on several other datasets to prove the assumption. By the way, I think there is no information we could get from Fig. 3(b), since it is neither not consistent with the corresponding accuracy, nor cable to guide the hyperparameter tuning.

Author response:
Thanks for your insightful question! IFA identifies directions in the feature space that encodes class semantic information and augments features along those directions. If the feature space is entangled (meaning different directions in the space do not clearly correspond to distinct class semantics), performing feature augmentation in such an entangled feature space may not preserve class information, bringing harm to performance. Applying only FD loss introduces additional constraints to the model by disentangling the feature space. Therefore, the absence of a term utilizing the disentangled space could result in a performance drop.


The VisDA-RSUT dataset used for the ablation study has highly imbalanced classes, resulting in an extremely entangled feature space where using only FD or IFA degrades performance. In contrast, in the VisDA dataset with balanced classes, there was no significant performance drop, with results showing SNC: 87.9%, SNC+IFA: 88.0%, and SNC+FD: 87.9%.

In our analysis presented in Section 3.4, we found that the denominator of the IFA loss encourages the kernel space of the model's Jacobian matrix to align with the principal direction of the tangent space of the augmented data manifold, which is in accordance with a finding on data augmentation [2]. Therefore, by comparing SNC (red) and SNC+IFA (yellow) in Fig. 3(b), we can observe a decrease in cosine similarity due to implicit feature augmentation. Moreover, comparing SNC+IFA (yellow) with SNC+IFA+FD (blue) shows that the addition of the FD loss further reduces cosine similarity, indicating that the FD loss facilitates more effective feature augmentation. While SNC+FD (green) might seem redundant for the explanation, it was included to align with Fig. 3(a).

Reviewer comment (Experiments):
There are two extra hyperparameters for FD and IFA losses, and there seems no ablation study for those hyperparameters. In the paper, it mentioned that $\alpha_1$ and $\alpha_2$ are set to different values for different dataset. I conjecture they may be sensitive on some datasets and hard to tune in the unsupervised way.

Author response:
Thanks for the suggestion! To report the optimal performance for each dataset, we conducted slight hyperparameter tuning. However, in six datasets, we experimented with only two settings for each hyperparameter ($\alpha_1$=1e-4, 1e-3) ($\alpha_2$=1, 10), achieving state-of-the-art performance. The model's performance is not sensitive to hyperparameters $\alpha_1$ and $\alpha_2$. As you suggested, we newly conducted an ablation study on $\alpha_1$ and $\alpha_2$, and the results on the VisDA dataset are presented in the following tables:

<Table S5.> Ablation study on $\alpha_1$

|$\alpha_1$|Per-class (%)|
|:---:|:---:|
|1e-3|87.3|
|1e-4|88.1|
|1e-5|88.0|

<Table S6.>  Ablation study on $\alpha_2$

|$\alpha_2$|Per-class (%)|
|:---:|:---:|
|1|88.0|
|10|88.1|
|20|87.4|

We added the additional ablation studies in Tables 10 and 11 in the revision.

We hope that our response has addressed your concerns. Thank you very much!

Reviewer comment (Novelty):
What are the differences between the proposed SNC loss and the loss function in the existing AaD [1]? As far as I know, the purpose of the loss in AaD is to encourage similar neighbors to be close to each other and dissimilar neighbors to be far apart within a batch. This is similar to the SNC loss proposed in this paper. The author needs to clarify the differences between these two loss functions.

Author response:
**Response:** AaD and our method have distinct motivations. We define an augmentation graph based on the given target features. Performing spectral clustering on this augmentation graph identifies meaningful partitions or clusters in the prediction space, which is equivalent to minimizing the SNC loss. While our method and AaD share similar forms for their final loss functions, as you pointed out, our research introduces an innovative perspective on SFDA and significantly expands the scope of research in this area, as acknowledged by Reviewer 44EQ.

Reviewer comment (Theory):
Are the estimated covariance matrices accurate in the IFA and FD losses, given the domain shift between the source and target domains? The estimation of these matrices relies on the output probabilities of the target data. However, this estimation process may introduce biases, especially when there are significant dissimilarities between the classes in the target domain and those in the source domain. As a result, the probabilities associated with these classes in the target domain, as generated by the source domain model, may be greatly reduced. Consequently, there is a potential risk of losing the capability to effectively learn these specific classes during the subsequent learning process, presenting a challenge to rectify the situation.

Author response:
**Response:** Thank you for the good question. The IFA loss estimates the covariance matrix using the given target features and pseudo-labels for implicit feature augmentation. However, during the initial adaptation phase, the feature space tends to become entangled due to domain shift, which may lead to an inaccurate covariance matrix and potentially degrade model performance. Therefore, we designed the influence of the estimated covariance matrix to be small during early training and gradually increase as training progresses (as indicated by \lambda in Equation 5). Additionally, to further disentangle the feature space, we introduced the FD loss and regularized the features to distribute in different directions for each class.

 If a pretrained model has been trained well, each direction in the feature space is likely to contain valuable class semantic information, and one might expect the cosine distances between groups of features belonging to the same ground-truth class to be close. Even in cases where domain shift may lead to incorrect assignments, the FD loss can be expected to reduce the cosine distance between groups of features from the same class during the process of disentangling the feature space.

However, for two groups of features belonging to the same ground-truth category to be assigned to different partitions, one group of features must possess features that are more similar to those of other classes than to its ground-truth class. In such scenarios, it is challenging to achieve unsupervised domain adaptation without supervision.

Reviewer comment (Experiments):
IFA operations have shown promise as a universally applicable data augmentation technique in unsupervised tasks. Can IFA be applied to other existing unsupervised tasks? Can you provide examples to validate its effectiveness?

Author response:
**Response:** We appreciate your suggestion. The IFA loss is derived from an augmentation graph in the feature space, taking into account a pretrained model and domain shift. Applying this to different unsupervised tasks would require a redefinition of the augmentation graph and the derivation of a new loss tailored to the specific task. Therefore, we consider this to fall beyond the scope of this paper. Nonetheless, we appreciate your suggestion and believe it could be an interesting avenue for future work.

Reviewer comment (Experiments):
In VisDA, Domain-Net, and PointDA-10 datasets, IFA loss has a weight of 1e-4, while FD loss has a weight of 10, resulting in a limited impact of IFA loss on the training process compared to FD loss. Can the inclusion of the other two datasets, apart from the VisDA-Rust task, demonstrate the significance of this new data augmentation technique?

Author response:
**Response:** Since the two losses serve as different types of regularizers, it's difficult to claim that IFA's impact is small solely due to the difference in weight. (For instance, the hyperparameter for weight decay commonly used in neural networks typically has a scale around 1e-3, whereas the hyperparameter for gradient penalty used in GANs often uses a scale of about 10. However, employing both doesn't necessarily mean that the impact of the gradient penalty is substantially stronger.) As depicted in Figure 3(a), even adding just 1e-3 of IFA results in a performance increase of over 5%p. While consistent performance improvements were observed in other datasets (from 87.9% to 88.1% in VisDA and from 68.0% to 68.3% in DomainNet), the enhancement is particularly significant in the highly imbalanced dataset. This notable performance boost in imbalanced datasets is believed to be due to the FD loss enabling distinct covariance matrices for each class and utilizing this for feature augmentation, thereby allowing minority classes to learn in distinct directions within the feature space.

Reviewer comment (Experiments):
Why are there no comparisons of NRC and AaD methods on the DomainNet dataset? Similarly, why are the results of AaD not compared on the PointDA-10 dataset?

Author response:
**Response:** We appreciate your suggestion. Following your suggestion, we newly compared our method with NRC and AaD on the DomainNet and PointDA-10 datasets and the results are presented in the following tables:

<Table S3.> Accuracy on the DomainNet dataset.

|Method|S$\rightarrow$P|C$\rightarrow$S|P$\rightarrow$C|P$\rightarrow$R|R$\rightarrow$S|R$\rightarrow$C|R$\rightarrow$P|Avg.|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|NRC|65.7|58.6|64.5|82.3|58.4|65.2|68.2|66.1|
|AaD|65.4|54.2|59.8|81.8|54.6|60.3|68.5|63.5|
|**SF(DA)$^2$**|67.7|59.6|67.8|83.5|60.2|68.8|70.5|**68.3**|

<Table S4.> Accuracy on the PointDA-10 dataset.

|Method|Model$\rightarrow$Shape|Model$\rightarrow$Scan|Shape$\rightarrow$Model|Shape$\rightarrow$Scan|Scan$\rightarrow$Model|Scan$\rightarrow$Shape|Avg.|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|AaD|69.6|34.6|67.7|28.8|68.0|66.6|55.9|
|**SF(DA)$^2$**|70.3|35.5|68.3|29.0|70.4|69.2|**57.1**|
 
We added the additional ablation studies in Tables 2 and 4 in the revision.

Reviewer comment (Novelty):
In the Introduction, the authors mention that recent studies have used data augmentation techniques to improve adaptation performance. However, there is a lack of more relevant work cited, such as [2] and [3].

Author response:
**Response:** Thank you for suggesting additional related work. In the revision, we included these related works in Appendix D.3 (highlighted in red).

Reviewer comment (Experiments):
In the ablation study, why do two regularizations need to be applied simultaneously?

Author response:
Thanks for your insightful question! Data augmentation that fails to preserve class information can introduce bias to the model. For instance, transforming the color of a lemon to green would turn it into a lime, potentially bringing harm to the model's performance [1].  

IFA identifies directions in the feature space that encodes class semantic information and augments features along those directions. If the feature space is entangled (meaning different directions in the space do not clearly correspond to distinct class semantics), performing feature augmentation in such an entangled feature space may not preserve class information, bringing harm to performance. Applying only FD loss introduces additional constraints to the model by disentangling the feature space. Therefore, the absence of a term utilizing the disentangled space could result in a performance drop.  

However, a synergistic effect arises when IFA loss and FD loss are jointly employed. The disentangled feature space obtained via FD loss allows the estimated covariance matrix to provide directions for class-preserving feature augmentation. Subsequently, the IFA loss performs the feature augmentation, leading to a notable performance improvement.

Reviewer comment (Theory):
The confusion matrix is estimated in an online way. How to guarantee its quality/accuracy?

Author response:
The IFA loss estimates the covariance matrix using the given target features and pseudo-labels for implicit feature augmentation. However, during the initial adaptation phase, the feature space tends to become entangled due to domain shift, which may lead to an inaccurate covariance matrix and potentially degrade model performance. Therefore, we designed the influence of the estimated covariance matrix to be small during early training and gradually increase as training progresses (as indicated by \lambda in Equation 5). Additionally, to further disentangle the feature space, we introduced the FD loss and regularized the features to distribute in different directions for each class.