Paper: Improving Non-Transferable Representation Learning by Harnessing Content and Style (ICLR-2024)

Reviewer comment (Writing):
The introduction could be more concise and direct, focusing on the problem and motivation for the proposed approach.

Author response:
Thanks for your helpful suggestion! We reorganize our introduction as follows:
- The first paragraph describes the background of NTL.
- The second paragraph analysis problems of existing methods, and the third paragraph further provides an example to intuitively illustrate the problems.
- The remaining paragraphs contain the intuitive explanation for the proposed H-NTL method.

*The revisions are highlighted in the updated manuscript (highlighted in purple).*

Reviewer comment (Writing):
Minor grammatical and typographical errors should be addressed for a more polished final version.

Author response:
Thanks for pointing this out. We have tried our best to revise obvious typos/grammatical errors in the initial manuscript. We will continue to polish our manuscript to further improve our introduction and revise potential grammatical and typographical errors, thus making our final version more readable.

Reviewer comment (Experiments):
No major weakness to note, however the reviewer would like to see additional evaluations with higher resolution images if possible (see questions below).

Author response:
As you suggested, we run additional evaluations with higher-resolution images. In detail, we run experiments on the original resolution of VT$\rightarrow$VV (112$\times$112) and OP$\rightarrow$OC (224$\times$224) without resizing, thus showing the effectiveness of the proposed H-NTL on higher resolution images. Results of the proposed H-NTL on the target-specified NTL setting and source-only NTL setting are shown in **Table B1** and **Table B2**, respectively. On higher-resolution datasets, the proposed H-NTL can still effectively maintain source domain performance (comparable to SL) and significantly degrade the target domain performance.  
*We add the experiments of high-resolution data in Appendix D.1 of the revised paper (highlighted in purple). Thanks for your constructive suggestions!* 

**Table B1:** Results of **target-specified H-NTL** on high-resolution datasets.   
| Source domain | Target domain |  Image Resolution  | SL(source/target) | H-NTL(source/target) | source_drop | target_drop |
| :-----------: | :-----------: | :----------------: | :---------------: | :------------------: | :---------: | :---------: |
|      VT       |      VV       |    32$\times$32    |     89.7/22.4     |       91.7/8.1       |    -2.0     |    14.3     |
|      VT       |      VV       |    64$\times$64    |     93.4/35.6     |       94.6/8.2       |    -1.2     |    27.4     |
|      VT       |      VV       | **112$\times$112** |     97.5/42.6     |       97.8/8.3       |    -0.3     |    34.3     |
|      OP       |      OC       |    32$\times$32    |     65.6/23.6     |       65.6/5.5       |     0.0     |    18.1     |
|      OP       |      OC       |    64$\times$64    |     75.6/31.6     |       76.4/6.7       |    -0.8     |    24.9     |
|      OP       |      OC       | **224$\times$224** |     86.0/35.5     |       84.9/3.6       |     1.1     |    31.9     |

**Table B2:** Results of **source-only H-NTL** on high-resolution datasets.  
| Source domain | Target domain |  Image Resolution  | SL(source/target) | H-NTL(source/target) | source_drop | target_drop |
| :-----------: | :-----------: | :----------------: | :---------------: | :------------------: | :---------: | :---------: |
|      VT       |      VV       |    32$\times$32    |     89.7/22.4     |       91.6/9.3       |    -1.9     |    13.1     |
|      VT       |      VV       |    64$\times$64    |     93.4/35.6     |       94.9/7.8       |    -1.5     |    27.8     |
|      VT       |      VV       | **112$\times$112** |     97.5/42.6     |       96.7/8.2       |     0.8     |    34.4     |
|      OP       |      OC       |    32$\times$32    |     65.6/23.6     |      63.7/14.4       |     1.9     |     9.2     |
|      OP       |      OC       |    64$\times$64    |     75.6/31.6     |      71.6/16.0       |     4.0     |    15.6     |
|      OP       |      OC       | **224$\times$224** |     86.0/35.5     |      83.3/10.1       |     2.7     |    25.4     |

Reviewer comment (Experiments):
Are there other standard NTL experimental setups using higher resolution datasets beyond the 32x32 or 64x64 images presented in the current paper?

Author response:
As you suggested, we run additional evaluations with higher-resolution images. In detail, we run experiments on the original resolution of VT$\rightarrow$VV (112$\times$112) and OP$\rightarrow$OC (224$\times$224) without resizing, thus showing the effectiveness of the proposed H-NTL on higher resolution images. Results of the proposed H-NTL on the target-specified NTL setting and source-only NTL setting are shown in **Table B1** and **Table B2**, respectively. On higher-resolution datasets, the proposed H-NTL can still effectively maintain source domain performance (comparable to SL) and significantly degrade the target domain performance.  
*We add the experiments of high-resolution data in Appendix D.1 of the revised paper (highlighted in purple). Thanks for your constructive suggestions!* 

**Table B1:** Results of **target-specified H-NTL** on high-resolution datasets.   
| Source domain | Target domain |  Image Resolution  | SL(source/target) | H-NTL(source/target) | source_drop | target_drop |
| :-----------: | :-----------: | :----------------: | :---------------: | :------------------: | :---------: | :---------: |
|      VT       |      VV       |    32$\times$32    |     89.7/22.4     |       91.7/8.1       |    -2.0     |    14.3     |
|      VT       |      VV       |    64$\times$64    |     93.4/35.6     |       94.6/8.2       |    -1.2     |    27.4     |
|      VT       |      VV       | **112$\times$112** |     97.5/42.6     |       97.8/8.3       |    -0.3     |    34.3     |
|      OP       |      OC       |    32$\times$32    |     65.6/23.6     |       65.6/5.5       |     0.0     |    18.1     |
|      OP       |      OC       |    64$\times$64    |     75.6/31.6     |       76.4/6.7       |    -0.8     |    24.9     |
|      OP       |      OC       | **224$\times$224** |     86.0/35.5     |       84.9/3.6       |     1.1     |    31.9     |

**Table B2:** Results of **source-only H-NTL** on high-resolution datasets.  
| Source domain | Target domain |  Image Resolution  | SL(source/target) | H-NTL(source/target) | source_drop | target_drop |
| :-----------: | :-----------: | :----------------: | :---------------: | :------------------: | :---------: | :---------: |
|      VT       |      VV       |    32$\times$32    |     89.7/22.4     |       91.6/9.3       |    -1.9     |    13.1     |
|      VT       |      VV       |    64$\times$64    |     93.4/35.6     |       94.9/7.8       |    -1.5     |    27.8     |
|      VT       |      VV       | **112$\times$112** |     97.5/42.6     |       96.7/8.2       |     0.8     |    34.4     |
|      OP       |      OC       |    32$\times$32    |     65.6/23.6     |      63.7/14.4       |     1.9     |     9.2     |
|      OP       |      OC       |    64$\times$64    |     75.6/31.6     |      71.6/16.0       |     4.0     |    15.6     |
|      OP       |      OC       | **224$\times$224** |     86.0/35.5     |      83.3/10.1       |     2.7     |    25.4     |

Reviewer comment (Experiments):
The authors state that existing methods suffer from the limitations of spurious correlation and fake independence. However, the authors do not provide any quantitative evaluation regarding these two challenges, except only intuitions.

Author response:
Thanks for pointing this out. It is difficult to directly quantify how many spurious correlations and fake independence the existing methods rely on in real-world datasets (it needs to manually analyze each sample and its prediction). Thus, we have validated the limitation of existing methods through a toy experiment (we present it in Appendix A due to the limited pages of the main paper). 

We leverage prior knowledge of spurious correlations to design a source domain, thus demonstrating the vulnerability (caused by fitting spurious correlation and modeling fake independence) of statistical-based methods. Specifically, we conduct the toy NTL experiment on digit datasets, where we use MNIST as the source domain and USPS as the target domain. Each digit in the source domain has a human-added class-wise color, thus creating a strong spurious correlation between the color and label. Our experiments show that the statistical-based method NTL [A1] and the proposed H-NTL achieve perfect non-transferable learning performance on the regular test data. However, when facing distribution shifts (e.g., perturbing the color), the statistical-based method NTL [A1] has serious performance degradation on the source domain, but the proposed H-NTL can still maintain the ideal performance. This phenomenon illustrates that the statistical-based method heavily relies on learning the spurious correlations to implement non-transferable learning. (More details please refer to Appendix A)

Reviewer comment (Experiments):
The authors do not provide further details about the datasets used in the experiments. For example, the number of samples in each domain and each class. This will result in difficulties in understanding different datasets.

Author response:
We sincerely apologize for the confusion. Due to the limited space of the main paper, we provided detailed information for each dataset used in our experiments in Appendix C.1 and the training details including the number of samples in each domain and each class in Appendix C.2.

Reviewer comment (Experiments):
In Table 2 presented in the experimental part, the authors only consider two baselines for comparison, which is not fair enough for a comprehensive evaluation.

Author response:
Thanks for your constructive comments! Since non-transferable learning is a recently proposed problem and has few baselines, here we design an additional method (denoted as TargetClass) that can be used for non-transferable learning, thus serving as a baseline for comparison. Specifically, considering an N-class non-transferable learning task, we take the same backbone as our main paper as a feature extractor and use a classification head with *N+1 classes* to predict the N classes in the source domain, but predict an additional “target class" as class N+1 for all samples in the target domain. Therefore, the model trained by TargetClass will map all the target domain data to an isolated cluster that stays away from the N source-domain-class clusters in the feature space, thus hindering the source-to-target knowledge transfer.  
We run experiments under the same setting as our main paper, and results are shown in **Table A1** (on natural datasets) and **Table A2** (on watermark target-domain). From the comparison, the proposed H-NTL generally has better performance.  The baseline method TargetClass is usually hard to maintain the source domain performance and degrade the target domain performance (e.g., on OP$\rightarrow$OC 32$\times$32 and 64$\times$64, C10$\rightarrow$S10 32$\times$32 and 64$\times$64 in **Table A1**, and most watermark experiments in **Table A2**).   
*The comparison with the new baseline is added in Appendix D.3 of the revised paper (highlighted in purple). Thanks for your constructive suggestions!* 

**Table A1:** Comparison of the new baseline (denoted as TargetClass) and the proposed H-NTL on natural data. The best results are highlighted in **bold**.   
| Source domain | Target domain | Image Resolution | TargetClass(src/tgt) | H-NTL(src/tgt) |
| :-----------: | :-----------: | :--------------: | :------------------: | :------------: |
|      MM       |      MT       |   32$\times$32   |     **92.8/9.5**     |    93.1/9.9    |
|      SN       |      SD       |   32$\times$32   |      87.8/10.4       |  **88.1/9.2**  |
|      SD       |      MT       |   32$\times$32   |      97.1/12.6       | **97.1/11.0**  |
|      C10      |      S10      |   32$\times$32   |      81.3/50.2       | **80.6/28.1**  |
|      C10      |      S10      |   64$\times$64   |      85.8/22.0       |  **87.6/9.6**  |
|      VV       |      VT       |   32$\times$32   |      91.4/10.9       |  **91.7/8.1**  |
|      VV       |      VT       |   64$\times$64   |       94.2/7.9       |  **94.6/8.2**  |
|      OP       |      OC       |   32$\times$32   |       64.2/8.6       |  **65.6/5.5**  |
|      OP       |      OC       |   64$\times$64   |       72.2/6.6       |  **76.4/6.7**  |

**Table A2:** Comparison of the new baseline (denoted as TargetClass) and the proposed H-NTL on watermarked data. The best results are highlighted in **bold**.  
| Source domain | Target domain | PatchValue | TargetClass(src/tgt) | H-NTL(src/tgt) |
| :-----------: | :-----------: | :--------: | :------------------: | :------------: |
|      C10      |    C10(p)     |     20     |      83.0/10.7       |  **87.8/7.9**  |
|      C10      |    C10(p)     |     40     |      85.5/10.0       | **88.7/10.1**  |
|      C10      |    C10(p)     |     60     |       85.5/8.8       |  **87.3/9.8**  |
|      S10      |    S10(p)     |     20     |      85.4/53.4       | **83.6/16.7**  |
|      S10      |    S10(p)     |     40     |      85.9/20.4       | **87.4/12.5**  |
|      S10      |    S10(p)     |     60     |      86.9/15.9       |  **88.3/9.0**  |