Paper: Domain Bridge: Generative Model-based Domain Forensic for Black-box Models (ICLR-2024)

Reviewer comment (Evaluation):
For the experiments, Table 1 and 2 showcase that the proposed method produces the description with extra properties. For example, "bird" is changed to "bird sitting on a branch". Does the black-box model predict "bird" based on both "bird" and "branch"? The authors should provide more analyses into why the proposed method generates this redundant description.

Author response:
Dear Reviewer 6Wte,

Thank you very much for your thorough and insightful review.

In your feedback regarding weaknesses, you pointed out some bias-related issues concerning both the black-box target model and the stable diffusion generator employed in our investigation. We acknowledge and appreciate your observations. 

Indeed, despite the use of state-of-the-art models as the target models of our investigation, it has come to our attention that they still exhibit inherent biases. For instance, the classifier's classification of birds based on both the presence of branches and the actual bird itself, or another classifier associating hair with gender, can be indicative of such biases. 

Our primary objective is to identify the data domain of an unknown target model. However, it is worth noting that the training dataset itself may have possessed biases, such as an abundance of bird images depicting birds on branches. Consequently, our approach may provide a more accurate description of the underlying data distribution than the original class name, thereby aiding in the exposure of bias within the target model.

Furthermore, our approach also uncovered biases within the pre-trained diffusion model, although this was not our primary intention. For instance, when prompted to generate images of females, it tended to produce more images with long hair than short hair. While this may reflect real-life distributions, we remain uncertain whether this should be the case for a generative model. It stands to reason that a better diffusion model could enhance the effectiveness of our investigative process.

Reviewer comment (Experiments):
Table 4 indicates that the proposed method is affected by the biases of diffusion models. Thus, will adopting more diffusion models improve performance?

Author response:
Dear Reviewer 6Wte,

Thank you very much for your thorough and insightful review.

In your feedback regarding weaknesses, you pointed out some bias-related issues concerning both the black-box target model and the stable diffusion generator employed in our investigation. We acknowledge and appreciate your observations. 

Indeed, despite the use of state-of-the-art models as the target models of our investigation, it has come to our attention that they still exhibit inherent biases. For instance, the classifier's classification of birds based on both the presence of branches and the actual bird itself, or another classifier associating hair with gender, can be indicative of such biases. 

Our primary objective is to identify the data domain of an unknown target model. However, it is worth noting that the training dataset itself may have possessed biases, such as an abundance of bird images depicting birds on branches. Consequently, our approach may provide a more accurate description of the underlying data distribution than the original class name, thereby aiding in the exposure of bias within the target model.

Furthermore, our approach also uncovered biases within the pre-trained diffusion model, although this was not our primary intention. For instance, when prompted to generate images of females, it tended to produce more images with long hair than short hair. While this may reflect real-life distributions, we remain uncertain whether this should be the case for a generative model. It stands to reason that a better diffusion model could enhance the effectiveness of our investigative process.

Reviewer comment (Evaluation):
Why did Scenario 4 achieve better performance than the original accuracy? Does it suggest that the black-box models are overfitting to the crafted description?

Author response:
To address your two specific questions:
2. In the typical scenarios of cloning, the process is often primarily limited by the number of samples available for querying the target, and acquiring these samples is often a challenging task. In our experiment, we utilized 5,000 samples. Scenario 4 introduces a new setting. In this scenario, the investigator initially employs our method to identify the data domain of the target model. Subsequently, a new model is trained from scratch, using the images generated by the generative model within the identified class domain. Given that the generative model itself was trained on a very large dataset, it is reasonable to expect that the newly trained model may demonstrate improved performance.

Reviewer comment (Experiments):
The experiments in this paper exclusively employ the corpus-based method as the sole point of comparison. This limited range of comparison experiments may not provide a comprehensive assessment. Furthermore, the corpus-based method is not extensively introduced or detailed.

Author response:
We acknowledge your point on using the corpus-based method as the primary comparison. However, note that the corpus-based method is the most closely related work for this new problem, making it the most relevant and appropriate for comparison at this stage. We agree that future work could benefit from a broader range of comparative studies to further validate and expand our findings.

Reviewer comment (Reproducibility):
description of the algorithm, discussed in 4.1.2, is too vague. It is not enough for reproducibility, as there are many things that should be defined, e.g. Description Grouper, clustering algorithm, how to select depth of the tree, etc. The authors did not provide any code

Author response:
In response to the concerns about the algorithm's description and reproducibility, we acknowledge that the initial manuscript may not have provided exhaustive details. However, to address this, we plan to release the code upon publication.

Reviewer comment (Experiments):
the authors consider very old classifiers, e.g. GoogLeNet, etc. to demonstrate efficiency of the proposed approach. At the same time, on recent models from hugging face the approach did not demonstrate significant efficiency

Author response:
Regarding the choice or target classifier, we have conducted additional experiments on newer backbones, such as VIT-based classifiers. The performance on these modern architectures is consistent with our findings on older models like GoogLeNet. This shows that the architecture of the black-box model does not significantly impact the performance of our approach, as our setting is black-box and relies solely on hard labels.

Reviewer comment (Writing):
The algorithm part is not very clear.

Author response:
We understand that the algorithm's description may have appeared unclear in our initial submission. To remedy this, we intend to include a more detailed and step-by-step description of the algorithm in our revised manuscript. We also plan to release the code upon publication.

Reviewer comment (Experiments):
The experimental data set is relatively small.

Author response:
The choice of datasets was driven by the need to demonstrate the effectiveness of our approach in identifying fine-grained classes. We can expand our evaluation to include larger and more diverse datasets.

Reviewer comment (Evaluation):
In Sec 6.3, the concept of model cloning is introduced without proper context. If the purpose of model cloning is to create a generative process that replicates the original dataset, the evaluation in Table 3 appears insufficient, as it only measures generation quality but not diversity. Metrics for distribution shift should be included, and the choice of the four scenarios should also be justified (e.g., it's unclear how scenario 1 serves in evaluating the proposed method).

Author response:
Dear Reviewer kDtS,

We appreciate your insights regarding the potential use of generative models. However, we would like to clarify that our work **is not** about fine-grained training or improving dataset understanding. Our primary objective is to **infer the input data domain** of an unknown model, especially in contexts where an investigator has access to an undocumented API endpoint or a single checkpoint file. This capability enables the investigator to determine the specific function of the model, such as discerning whether it is a face classifier or a medical image classifier.

Tables 1 and 2 in our paper demonstrate the efficacy of our method in accurately identifying the correct domain of a model. The ideal outcome, is when the identified domain exactly matches the original domain of the model.

Regarding model cloning in Section 6.3, our intention is not to replicate any dataset. Instead, model cloning in our context refers to creating a white-box version of a black-box model, aiming to match its performance as closely as possible.

The tree structure depicted in Figure 1 is illustrative of the iterative search process our method undergoes. This process, often goes through many iterations.

We acknowledge your suggestion regarding the alignment of the bullet points under Table 3. We will rearrange these points to enhance clarity and facilitate better understanding of the data presented in the table.

We hope this clarification sheds more light on the objectives of our work. If there are any further questions or aspects of our paper that you would like to discuss, please feel free to reach out. We are more than willing to provide additional explanations or engage in further discussions to ensure a thorough understanding of our research. Thank you again for your valuable feedback and insights.

Best regards,  
Author(s)

Reviewer comment (Presentation):
It appears that the bullet points under Table 3 might not align correctly with the table. Should the third and fourth points be switched?

Author response:
Dear Reviewer kDtS,

We appreciate your insights regarding the potential use of generative models. However, we would like to clarify that our work **is not** about fine-grained training or improving dataset understanding. Our primary objective is to **infer the input data domain** of an unknown model, especially in contexts where an investigator has access to an undocumented API endpoint or a single checkpoint file. This capability enables the investigator to determine the specific function of the model, such as discerning whether it is a face classifier or a medical image classifier.

Tables 1 and 2 in our paper demonstrate the efficacy of our method in accurately identifying the correct domain of a model. The ideal outcome, is when the identified domain exactly matches the original domain of the model.

Regarding model cloning in Section 6.3, our intention is not to replicate any dataset. Instead, model cloning in our context refers to creating a white-box version of a black-box model, aiming to match its performance as closely as possible.

The tree structure depicted in Figure 1 is illustrative of the iterative search process our method undergoes. This process, often goes through many iterations.

We acknowledge your suggestion regarding the alignment of the bullet points under Table 3. We will rearrange these points to enhance clarity and facilitate better understanding of the data presented in the table.

We hope this clarification sheds more light on the objectives of our work. If there are any further questions or aspects of our paper that you would like to discuss, please feel free to reach out. We are more than willing to provide additional explanations or engage in further discussions to ensure a thorough understanding of our research. Thank you again for your valuable feedback and insights.

Best regards,  
Author(s)