Paper: Deep concept removal (ICLR-2024)

Reviewer comment (Experiments):
Lack of comparative methods. Many strategies for deep concept removal have already been proposed[1], such as adversarial concept removal mentioned by the authors. However, they did not compare their method with these classic approaches during the experimental phase.

Author response:
We want to emphasize that existing concept removal methods are all (to the best of our knowledge) post-hoc. For post-hoc methods, there is an issue stemming from the entanglement of the features in the last layer of representation, and removing the concept unavoidably removes correlated fatures. In fact, [1] theoretically proves that. We also note that existing methods are only designed to sensor the information, rather generalizing to non-concept instances. Our method is in-processing (and yes, more expensive), but it addresses a more difficult problem. For the sake of it, we implemented LEACE on Striped MNIST, results shown below. We use model M7 trained for 100 epochs with ERM, then apply LEACE, then train a logistic regression classifier.

|                  | EMNIST concept dataset |                   | MNIST concept dataset |                   |
|--------------------------|------------------------|-------------------|------------------------|-------------------|
|                          | acc test               | acc train         | acc test               | acc train         |
| LEACE                     | 0.59(0.01)             | 1 (0)             | 0.59(0.01)             | 1 (0)             |

[1] Abhinav Kumar, Chenhao Tan, Amit Sharma. Probing Classifiers are Unreliable for Concept Removal and Detection

Reviewer comment (Evaluation):
The performance of the method presented in the paper is also lacking. Table 2 shows a comparison between the authors' method and others, but it seems their method is not optimal, performing worse in some cases than methods without concept annotation.

Author response:
We want to bring some clarity why we do the experiments in Table 2. Unlike in the controlled MNIST experiment, while concept removal assessment is not possible in dataset like Celeb-A.  The DRO problem hence can serve as a verification of concept removal. The other baselines in Table 2 address the problem of DRO directly. They are not concerned with concept removal.

Reviewer comment (Experiments):
From the objective function of the method, it can be applicable to different downstream tasks. However, the paper only validated it on classification tasks. Is this method equally applicable to object detection or image generation tasks?

Author response:
Unfortunatelly, we do not have convincing experiments at the moment. We have tried applying the method to VAE, but there is no quantitative way to verify the results. Furthermore, it is not that trivial how to apply the method to the generative tasks. For instance, in the VAE case, the encoder itself can potentially add the concept back, even if the encoder successfully removed it. For instance, if we train on data that has some dependencies between the concept, it can use that information to infer whether the concept was in the input or not. In other words, both the decoder and encoder suffer from detrimental correlations, while our method is only suitable for removing concept from encoder.

Reviewer comment (Experiments):
lack of comparison with other concept removal baseline methods in Table 2

Author response:
The problem with existing concept removal methods is they are only designed to sensor the information, rather generalizing to non-concept instances. For instance, the paper [1] theoretically proves that it cannot remove a concept without harming features that are correlated with it. For the sake of it, we implemented LEACE on Striped MNIST, results shown below. We use model M7 trained for 100 epochs with ERM, then apply LEACE, then train a logistic regression classifier:

| EMNIST concept dataset|  |   MNIST concept dataset | |
|---------------|----------------|----------------|--------------|
|acc test        |  acc train      |    acc test        |  acc train      |
|    0.59(0.01)|      1 (0)            |    0.59(0.01)   |  1 (0)|

[1] Abhinav Kumar, Chenhao Tan, Amit Sharma. Probing Classifiers are Unreliable for Concept Removal and Detection

Reviewer comment (Experiments):
lack of comprehensive results on Section 6, this paper does not show the advantage of their method on practical celebrity dataset.

Author response:
We include Section 6 for sake of demonstration, in order to highlight the difference between fair representation and concept removal. In fair representation, we are concerned with statistical independence. In concept removal, we want to remove information associated with detrimental concepts, while keeping the useful information that correlates with it. While statistical dependence is subject to certain demographics (training set), a concept must be defined independently of the demographics.

Reviewer comment (Experiments):
lack of ablation study on their training loss modules, for example, the effect of Penalty term of Eq. 3.2, and different values of \lambda of Eq. 3.1.

Author response:
We include some ablation studies in Secion C.1, see end of section for choice of $\lambda$. We included Figure 9 incorrectly, fixed after revision. We apologize for this oversight.

Reviewer comment (Experiments):
can the method be adapted to generative model where the concept removal task is of more importance?

Author response:
We tried applying it to VAE. It is hard to assess the results it quantitatively. Furthermore, in VAE, even if the concept is removed by the encoder, it can be added by the decoder using correlated features.

Reviewer comment (Experiments):
The main experiment in S4 only demonstrates RQ1 and RQ2 on MNIST. The MNIST dataset is a good sanity check or starting point, but it is not enough to properly demonstrate the usefulness of this approach. A simple linear model can give 90% accuracy on the MNIST task and it is not representative of modern computer vision tasks. The experiments would be more convincing if the results hold on (a) “harder” tasks such as CIFAR-10 or CIFAR-100 and larger models (e.g., ResNet50 and ViTs)

Author response:
This is a very good point. Unfortunately, we did not manage to make it work on problems where we are required to overfit. On all our MNIST experiments, in the cases where it works well, the train accuracy is usually smaller than test accuracy. Similarly, for the DRO we typically need to regularize in order to be robust. For both CIFAR-10 and CIFAR-100, we are required to overfit on train set in order to achieve competitive performance on test set.

Reviewer comment (Experiments):
The experiments in S4 do not adequately support RQ2 (concept datasets can be OOD). The OOD concept dataset considered (EMNIST) is in fact quite similar to the MNIST dataset. It would be useful to provide a more nuanced understanding of when OOD concept datasets fail to remove concepts for example.

Author response:
It is hard to say whether it is OOD enough, but there is not many good alternatives. We also want to emphasize that in this particular example, we require quite limited data. Learning invariant representations, to the best of our knowledge, requires access to the transformation [1, 2]. On the contrary, we only require some observations of the transformation on a relevant but not the same dataset. We also note that linear adversarial classifiers play a crucial role in this. If we use an MLP (equivalent to adversarial domain adaptation [3]), and no implicit gradients, the performance significantly drops. In both cases, whether the concept is based on EMNIST or MNIST. But there is also a bigger discrepancy between the two for the MLP:

|         | EMNIST Concept Dataset |                | MNIST Concept Dataset |                |
|---------|------------------------|----------------|-----------------------|----------------|
|         | acc test               | acc train      | acc test              | acc train      |
| MLP (DA)    | 18 (0.04)            | 1 (0)          | 0.44 (0.17)           | 1 (0)          |

[1] Mitrovic et al. Representation learning via invariant causal mechanisims

[2] von Kuegelgen et al. Self-supervised learning with augmentations provably isolates content from style

[3] Ganin, Lempitsky. Unsupervised domain adaptation by backpropagation

Reviewer comment (Evaluation):
The deep concept removal method does not work on the Waterbirds dataset. In particular, it performs *significantly* worse than ERM. The authors do acknowledge this and hypothesize that their method may be more effective at removing high-level features than low-level features. However, this explanation is not convincing at all. What makes a concept high-level? Does this definition of high-level concept distinguish CelebA and “striped MNIST” concept from waterbirds? I am not sure if subpopulation robustness is the right task to evaluate concept removal. The images in the concept dataset with and without the concept may differ in many ways other than the presence of the concept, and this may lead to a spurious CAV.

Author response:
This is a good question and we do not have a clear answer to it. However, let us give the following simple example. If we use a randomly initialized network, the penultimate layer most likely will contain information about the color, but not about the stripes (which can in fact be filtered out with random network [5]). We believe the same is true for Waterbirds, and often the algorithms for DRO utilize this information, by calling it a short cut feature. In that sense, it is very difficult to steer away the network from encoding that low-level information.

[5] Ulyanov, Vedaldi, Lempitsky. Deep image prior.

Reviewer comment (Evaluation):
Results from Fig 7 are not satisfactorily parsed. Why does the concept accuracy for “young” and “glasses” not deteriorate, whereas the concept accuracy for “gender” is reduced?

Author response:
This example was handcrafted exactly to demonstrate the ability to remove one specific concept (gender), rather than everything that is correlated with it (glasses, young).

Reviewer comment (Evaluation):
In Figure 3, for the images which do not have stripes as a concept, a reconstruction of those representations introduces stripes. This seems counterproductive.

Author response:
The goal of this Figure is to show that the reconstruction is the same for whether the stripes are present in the input or not. That is, we have learned invariant representation with respect to the presence of stripes. We do not claim that the decoder removes the stripes, but we claim that the encoder removes information of whether stripes are present or not.