Paper: Counterfactual Image Generation for adversarially robust and interpretable Classifiers (ICLR-2024)

Reviewer comment (Experiments):
**Limited to Binary Tasks**: A major limitation of the paper is that it only addresses binary classification tasks. It would be interesting to expand its applicability to multiclass problems to demonstrate broader utility, as mentioned in the discussion section.

Author response:
We acknowledge that being limited to binary classification may seem like a limitation. However, this setting covers a wide range of problems in medical health diagnosis, structural health monitoring, and other tasks involving anomaly detection. Furthermore, we argue that visual explanations of class predictions are most informative when compared against a “background” class, as, for example, directly comparing an image of a table against an image of a chair may not yield the best visual explanation of what a chair is. As such, our framework would be applicable to multi-class problems when formulating them as several, separate binary classification tasks where each class is either present or absent from the image in question.

Reviewer comment (Experiments):
**Single Seed Experiments**: The experiments in the paper are limited to training on a single seed, making it difficult to assess the significance of performance differences and the true impact of the proposed cycle consistency loss on convergence. Multiple seed experiments would provide a more robust evaluation.

Author response:
We acknowledge the importance of this point. In our experiments, although there was some variability in the perceived quality of counterfactual image generations across different runs, the classification performance remained mostly consistent. Our primary focus was on demonstrating the utility of counterfactual image generation for image classifiers, rather than on classification performance per se. Hence, we prioritized showcasing other metrics in the limited space available. Additionally, given the short timeframe for rebuttal and the substantial training time required (12 to 24 hours for larger models), systematically re-training all models to collect comprehensive multi-seed data was not feasible.

Reviewer comment (Presentation):
**Experiment Clarity**: The presentation of experiments can be confusing and should be more detailed. For instance, the "Hybrid D" model is never introduced in the paper. The explanation of the computation of performance when using D is also presented *after* showing results. The description of Table 2 is also unclear, making it challenging for readers to understand the methodology and the comparison.

Author response:
The Hybrid discriminator is introduced in Section 3.1.2. We added a sentence explicitly stating the name that will be used for the hybrid architecture in the revised version.

Reviewer comment (Theory):
**Misleading Introduction**: The paper introduces the approach as "combining classifier and discriminator in a single model" (in the abstract), which is incorrect since the generator and discriminator are fundamentally different.

Author response:
We indeed combined the classifier and discriminator into a single model, as it has outputs for estimating the “fakeness” of an image as well as one for the class prediction. The confusion may come from the fact that we added the auxiliary task of classifying the images to the generator as well. In our framework, both the generator and discriminator make predictions for the class.

Reviewer comment (Experiments):
**Lack of Comparative Analysis**: The paper lacks a comparison with other counterfactual approaches, which could provide insights into the quality of the counterfactuals produced and help position the proposed method within the broader context of counterfactual research.

Author response:
We acknowledge the absence of a comparative analysis with other counterfactual approaches in our paper. Our challenge was the scarcity of comparable counterfactual image generation methods that reported performance metrics aligning with our focus, such as saliency map evaluation against segmentation labels. Additionally, re-implementing existing methods with sufficient fidelity for an accurate performance comparison proved to be complex. As such, we focused on demonstrating the efficacy of our method within the constraints of available comparative data and reproducibility standards.

Reviewer comment (Experiments):
It could be interesting to have a rule of thumb in which model to use, G or D ? Both seems to be strong for classification, but do they have their own advantages ?

Author response:
They do indeed both have good classification performances. If efficiency is important, then the discriminator is the better choice due to its smaller architecture size. If the saliency maps are always necessary, then we would suggest using only the generator. However, the one would obtain the best performing model by combining the generator's and discriminator's prediction as kind of ensemble.

Reviewer comment (Experiments):
Can we do more than 1 cycle in the "cycle consistency loss" ? The loss is described with $c \geq 1$, but experiments only show $c=1$ or $c=0$.

Author response:
There is no limit to the number of cycles that could be used until hardware limitations (memory) are reached. We experimented with values higher than 1, but the minor improvements did not justify the longer training times.

Reviewer comment (Experiments):
In Section 4.5, the authors mention "...adding perturbations to the input images by plotting the strength of the attack (**step size** for PGD over 10 iterations..." This setup is questionable. For a regular PGD attack, the **perturbation bound** is a more direct reflection of the attack strength. However, in this section, the authors plot the F1 score w.r.t. the attack step size (Figure 3a), which is less informative for showing attack strength. Also, the authors should state clearly in the PGD experiment setup what value the perturbation bound takes.

Author response:
There was indeed important information missing. We added the values used for the perturbation bound, norm, and codebase used for the attack. We selected a rather small L_inf bound of 0.1 because we are interested in small, imperceptible modifications of the image.

Reviewer comment (Experiments):
The framework is a modification (changing the discriminator objective) derived from the training process of an image-to-image GAN. The claim of simply using absolute pixel differences of image translation as counterfactual explainability is not grounded. It only visualizes the changing semantics and is not always sufficient as a counterfactual explanation to the target classifier. Recent years have witnessed more effective and solid approaches to generating semantic counterfactuals/adversaries [1,2,3,4,5,6]. The user can perform adversarial training with these methods to fine-tune and improve the target classifier. The paper should compare with more baselines from this line of research.

Author response:
The generator’s objective is to create an image belonging to the opposite class compared to the original image and is as such trained to produce counterfactuals. The information about the semantics of the two classes is provided by the gradients flowing through the discriminator. Indeed, this information may be imperfect when the discriminator focuses on non-class-relevant features, like radiologist annotations in medical images. In such scenarios, our generator might include these annotations in healthy samples, creating images that are not necessarily counterfactuals in real-world terms, but still counterfactuals relative to the dataset and the discriminator’s predictions. The evaluation on segmentation masks yields a quantitative metric for the produced counterfactual samples and can reveal such “Clever Hans” situations.

Reviewer comment (Experiments):
There are other generative paradigms like diffusion models and VAEs, which have shown the capability to perform image-to-image translation. It can also be feasible that we jointly train an image-to-image diffusion model with the target classifier. The paper should show sufficient validity in adopting the GAN paradigms (e.g., convergence, efficiency) in the experiments.

Author response:
Indeed, other models could be excellent at generating counterfactual images as well. However, the counterfactual image generation per se is only a subtask in our study, which aims at merging two lines of work: one focusing on using GANs for creating counterfactual images and one using GANs for creating adversarially robust classifiers. As such, our models are not comparable to diffusion models or VAEs.

Concerning the question on other GAN paradigms: our method could be used in combination with other GAN variants that aim at improving training stability and output quality, such as StyleGAN. We adopted the CycleGAN approach because it is an intuitive addition and has already been proposed for counterfactual image generation: Charachon et al., “Leveraging conditional generative models in a general explanation framework of classifier decisions”.

Reviewer comment (Writing):
Writing is unclear. (page 2) “to minimally alter the images such that they are classified into the opposing class” => What is the opposing class?

Author response:
Our method is designed for binary image classification tasks, which involve two distinct classes. In this context, the 'opposing class' refers to the alternative category that an image does not currently belong to. The aim is to achieve minimal alterations to the original image to transition it convincingly into this opposite class.

Reviewer comment (Writing):
Writing is unclear. (page 1) “We argue that by fixing the classifier’s parameters, current attribution methods using GANs forfeit the opportunity to train a more robust classifier simultaneously, even though it has been previously observed that adversarial attacks could also be employed as tools for interpreting the model’s decision-making process“ => What is the “even though" sentence for?

Author response:
Our critique of current explainability methods using GANs is that they miss an opportunity for dual improvement. By fixing the classifier's weights during the adversarial generation phase, these methods manage to highlight a classifier’s proneness to small perturbations but do not offer a solution for how to overcome this limitation. Our proposed approach addresses this gap. \
We acknowledge that this paragraph lacked some clarity and revised it in the new version.

Reviewer comment (Writing):
Writing is unclear. (page 2, page 4) “This methodology has the benefits of (i) creating adversarial examples that augment the dataset, making the classification more robust against subtle perturbations“ => augmenting dataset can be done after training Generative Models and making the classifier more robust can be done during the training (under the method in this paper).

Author response:
Unlike pre-trained generative models, our methodology creates targeted augmentations in an online fashion during training. The generator in our framework is designed to produce minimally altered images that are misclassified by the discriminator in terms of both realism and class attribution. As the generator and discriminator are trained together in a min-max game, these augmented samples progressively improve. This leads to increasingly refined counterfactual samples.

Reviewer comment (Writing):
Writing is unclear. (Figure 1, page 4) “Conversely, G must deceive D by producing realistic samples attributed to the opposite class by D.” => What does this mean?

Author response:
This is probably related to a previous question: in this work, we focused on binary image classification. The second important aspect of our work is the fact that we combine the classifier and discriminator in a single network. It must predict the class of the image and whether the image was generated or not. The generator, on the other hand, must create images that the discriminator (i) misclassifies (attributes to the wrong class) and (ii) fails to detect that it was generated (outputs a high probability that the image belongs to the real distribution although it was generated).

Reviewer comment (Writing):
The terminology “Counterfactual” is used in an unreasonable way; how is damage/no-damage related to factual/counterfactual?

Author response:
We use the terminology of a of research leveraging counterfactual images as explainability for neural image classifiers:\
CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Nets, Daniel Nemirovsky et al.\
GANterfactual—Counterfactual Explanations for Medical Non-experts Using Generative Adversarial Learning, Silvan Mertes et al. \
EXPLAINING IMAGE CLASSIFIERS BY COUNTERFACTUAL GENERATION, Chang et al.  
We included additional citations of related work in the area of counterfactual image generation as additional information for the readers.

Reviewer comment (Novelty):
Novelty is limited; it seems like the proposed method is a simple variant of ACGAN [1,2].

Author response:
ACGANs indeed share similarities with our method in integrating the discriminator and classifier. However, our approach differs significantly, primarily in our generator's design, which is unconditioned, as opposed to the class-conditioned generator in ACGANs. This distinction is crucial for generating explanations in our framework, where we compare the original image directly with its counterfactual generated by G. To achieve something similar with ACGANs, one would have to obtain the latent representation z of the original image, generate a new image conditioned on class 0, and another on class 1, and then compare these two generated images. In this scenario, one would be comparing two generated images, where important features from the original image may have been lost or modified. ACGANs were not designed to solve the same problem that we are targeting.

Reviewer comment (Novelty):
Novelty is limited; only binary classification is discussed.

Author response:
We acknowledge that being limited to binary classification may seem like a limitation. However, this setting covers a large range of problems in medical health diagnosis, structural health monitoring, and other tasks involving anomaly detection. Furthermore, we argue that visual explanations of class predictions are most informative when compared against a “background” class, as, for example, directly comparing an image of a table against an image of a chair may not yield the best visual explanation of what a chair is. As such, our framework would be applicable to multi-class problems when formulating them as several, separate binary classification tasks where each class is either present or absent from the image in question.

Reviewer comment (Writing):
The framework is not that easy to understand. In particular, it is not clear why the classification head is needed in the U-net of the generator. Overall, there seems to be many kinds of classifications happening at different stages. There is one happening in the generator, and then also in the discriminator. While the reader *can* follow along, the overall framework lacks a bit of intuition. This is also because the authors have claimed certain things in the text for which there is not much justification. For example, at the end of Section 3.2.2, the authors claim that the objective “bolsters both the training stability and the expressive capability of the generator”. It is not clear what expressivity exactly means, and how exactly are the authors measuring the stability of the generator training.

Author response:
We acknowledge two main points of complexity: (i) the generator’s classification head, and (ii) the role of cycle consistency loss. \
(i) The generator's classification head is crucial as our generator is unconditioned, a notable deviation from similar models. This classification head infuses direct class information into the generator, enhancing its performance under the multi-task learning paradigm. While we were not able to include detailed ablation studies due to the page limit,  we plan to publish the codebase for reproducing our results, which may shed more light on the impact of different hyperparameters on the final model performance. \
(ii) The cycle consistency loss, as discussed by Charachon et al. in “Leveraging conditional generative models in a general explanation framework of classifier decisions” (c.f. citation in Section 3.2.2), is proven effective in related contexts. It contributes to the overall stability and expressive capability of the generator, ensuring more reliable and coherent outputs.

Reviewer comment (Presentation):
There is some confusion in the way results are presented. In Table 2, what is the difference between the top and bottom sections of the table? What do non-adversarially trained equivalents mean; i.e. how exactly were those generators and discriminators trained? Furthermore, the nomenclature used in the paper to refer to different models is a bit confusing across the paper. For example, in Section 4.5, the authors use the phrase “D is more robust compared to its non-adversarial counterpart.” But the figure that they are referring to, Fig. 3, does not have any “D” in it. There is a “Hybrid D” and “D_fake”. I would strongly recommend the authors to be consistent with the naming scheme.

Author response:
The non-adversarially trained models in our study were developed using standard supervised learning techniques. Due to space constraints, we couldn't include all training details in the paper, but we intend to release our complete codebase post-review, which will offer comprehensive insights into our implementation. \
The “D” in Section 4.5. refers to the “Hybrid D”, where “Hybrid” specifies the architecture of the discriminator used in this experiment. Following your suggestion, we also mentioned the discriminator’s architecture in the Section referring to the Figure in question. Also, we appreciate your feedback on consistency in naming. There was a wrong notation in the legend of Figure 3, where we noted D_fake instead of p_fake.  We have revised Figure 3 and its legend to align with the notation established in Section 3.2.1.

Reviewer comment (Evaluation):
About the attribution maps: If I understand correctly, in Fig. 4, the way the authors are computing the saliency maps under the “Segmentation” column, which is their primary method, is through a difference between an input image ‘x’ and its transformed image G(x). However, the region highlighted by this difference does not mean it is the same region used by the discriminator to classify them as damaged or undamaged. In other words, just because we can see the difference between two kinds of images does not mean that the neural network is looking at the same kind of difference as well. Therefore, there is not much point of comparison to the methods in “Attribution methods”.

Author response:
Given that our generator is unconditioned and receives updates through gradients from the discriminator, it's reasonable to infer that it minimally alters features critical to the discriminator's classification. The generator's primary guidance, apart from an auxiliary objective indicating the original image's class, comes from its goal to change the discriminator’s class and “fakeness” predictions. Therefore, the differences highlighted between 'x' and G(x) are indeed indicative of the regions the discriminator focuses on for its decision-making.

Reviewer comment (Writing):
The word damaged — in real-damaged vs real-undamaged — is confusing. Maybe replace with a different word because damage might also mean adversarial example.

Author response:
We have also adapted the word damaged - in real-damaged vs real-undamaged - to a less ambiguous formulation.