Paper: Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds (ICLR-2024)

Reviewer comment (Experiments):
While CelebA is a more complex dataset than Waterbirds, and the 10% subset was likely necessary due to computational restrictions, the fact remains that both datasets are small, belong to the vision domain, and only utilize binary-valued spurious features.

Author response:
To address the reviewer's concern, we ran the experiments from Table 3 on different sizes of CelebA (all the other experiments in the paper were already on full CelebA). Below, we show the effect of EPIc (poisoning defense) on the WGA (results are over 3 runs):

|    |20% of CelebA  | 50% of CelebA | Full CelebA      |
|----|---------------|---------------|---------------|
|   Ideal  | 52.3 +- 1.6  | 50.3 +- 1.9   | 45.9 +- 5.8 |
| Standard  | 43.1 +- 5.5  | 46.4 +- 2.8   | 43.6 +- 3.0|
|   Worst  | 32.3 +- 2.7  | 32.0 +- 0.8   |  37.7 +- 3.8|

This shows that the trends we observed (i.e., EPIc hurts the WGA by eliminating minority groups) are consistent as the ideal WGA (when we intervene to prevent EPIc from eliminating any minority samples) is higher than the standard WGA (applying EPIc w/o any intervention). 

Additionally, to show the amplification due to group robustness methods, we ran experiments with the Subpopulation Attack (SA) on full CelebA for JTT and ERM (the baseline suggested by Reviewer Shsu).

|Poison %      |      0.05%   |       0.1%   |        0.2%   |   0.3%  |    0.5% |      1% |          2%|
|--------------------|-----------|-----------|----------|-----------|-----------|-----------|-----------|
|JTT (ASR)   |         0.1%  |      1.7%  |          2.4% |     3.5%    |    6.3%   |   14.1%  |    46.8%|
|ERM (ASR)  |  0.1%   |     0.1%         |  0.2%   |    0.0%   |    0.1%     |  0.1%       |  0.3%|


| Poison %       |  0.05%   |        0.1%     |      0.2%   |      0.3%  |      0.5%  |     1%   |       2%|
| ------------------- | -----------  | ----------- |----------|-----------|-----------|-----------|-----------|
|JTT (WGA)    |     81.7%  |    82.2%   |      83.3%  |      83.3% |    83.3%  |  77.9% |  48.2%|
|ERM (WGA) |     43.3%    | 43.3%   |       47.2%   |    42.8%  |   41.1%  |  45.0% |  45.6%|

In the first table, we observe that the amplification is generally high (a large ASR gap between JTT and ERM). The second table shows that JTT still works as intended by improving the WGA over ERM. In conclusion, our observations remain consistent for the full CelebA as well.

Reviewer comment (Experiments):
Suggestions for additional datasets include Spawrious [3] for the vision domain (in particular because it is not too large; it is a difficult dataset which is smaller than CelebA) or MultiNLI [4] for the language domain (which would also introduce a non-binary-valued spurious feature).

Author response:
Thank you for suggesting these experiments! To ensure the validity of our observations for newer group robustness methods and more advanced datasets, we applied AFR [1] (referenced as [8] by the reviewer) on MultiNLI. As in our previous experiment, we consider SA as the attack, ERM as a baseline (no group robustness method), and report both the ASR and WGA.

| Poison %          |    0.5%     |      1%     |
| ------------------- | -----------  | ----------- |
|AFR (ASR)       |    2.1%       |  14.4%      |
|ERM (ASR) |   1.0%      |     5.3%    |


| Poison %          |    0.5%      |     1%     |
| ------------------- | -----------  | ----------- |
|AFR (WGA)        |   75.3%     |      78.4%  | 
|ERM (WGA)  |  68.0%      |     70.0%   | 

We observe the same tension in these experiments: while AFR improves the WGA over the baseline (as expected), it also inadvertently amplifies the ASR (e.g., from 5.3% to 14.4%, a significant boost for a weak attacker).

> Some suggestions may include Spawrious [3] for the vision domain [...].

We agree with the reviewer that Spawrious would be a good addition to our experiments. Unfortunately, due to time limitations, we were unable to conclude the experiments on this benchmark before the rebuttal deadline. However, we are committed to include them in the final version (we will include them in the Appendix).

Reviewer comment (Novelty):
The related work subsection for group robustness is out of date and does not include any references more recent than 2021. There have been significant contributions since 2021, particularly in methods which pseudo-label the minority group (with or without an explicit identification model), e.g., [2,5,6,7,8,9].

Author response:
I would encourage the authors to perform a more detailed literature review [...]

We will update our paper's appendix with the discussion we shared with Reviewer Shsu on the different (more recent) group robustness methods (some of them are referenced as [5,7,8,9] by the reviewer), and the parallels among the heuristics they rely on.

Reviewer comment (Writing):
The definitions in Section 2.2 are unclear and should be improved. The reason that the definitions of groups differs from [4] is likely to account for the dirty-label samples later in Section 3.3. This should be explicitly stated prior to the definitions, as otherwise the definitions seem inconsistent with the literature. It should also be made explicit whether the worst-group accuracy in the remainder of the paper is computed with respect to these new groups (i.e., 8 groups) or the original groups (i.e., 4 groups).

Author response:
We agree with the reviewer. We included more clear definitions for groups, as well for the identification model I(x) in the paper.

Reviewer comment (Presentation):
The bibtex could use an update: there are some extra braces and capitalization errors.

Author response:
Thanks for pointing these out, we fixed all of them!

Reviewer comment (Experiments):
It would be helpful to include a table with the proportions of each group and class in each benchmark dataset, as well as the proportions of each group and class in the 10% sample of CelebA used in the experiments. See [9, 10] for examples.

Author response:
We agree, we will include it in an updated version of the paper.

Reviewer comment (Writing):
The definitions in Section 3.3 are unclear and should be improved. First, it should be made clear that $I(x)\in [0,1]$, i.e., that it is a probability output rather than a logit output. Second, $I(x)_y$ is referred to as the “confidence” of model $I$ on input $x$ for class $y$, which is misleading: if $I(x)_y=0$ the model is in fact very confident (that $x$ does not belong to class $y$). The definition of “confidence” in this case would be expected to look more like $2|I(x)_y - 1/2|$, since $1/2$ can be considered the “least confident” prediction in a binary classification problem.

Author response:
We agree with the reviewer. We included more clear definitions for groups, as well for the identification model I(x) in the paper.

Reviewer comment (Writing):
In Appendix A.2, “DLDB” may be a typo and should perhaps read “DLBD”.

Author response:
Thanks for pointing these out, we fixed all of them!

Reviewer comment (Presentation):
In Figure 1, it should be made clear that the lighter-colored circles and triangles are amplified points. Also, the two shades of red are indistinguishable.

Author response:
Thanks for pointing these out, we fixed all of them!

Reviewer comment (Writing):
In Section 5, “conciliate” should perhaps be replaced by “reconcile”.

Author response:
Thanks for pointing these out, we fixed all of them!

Reviewer comment (Presentation):
At the bottom of page 7, the citation to Wu et al. should perhaps be in parentheses, i.e., \citep.

Author response:
Thanks for pointing these out, we fixed all of them!

Reviewer comment (Experiments):
Experiments. The results on CelebA (Section 4) only uses a randomly sampled 10% of the dataset which is concerning. Can the authors also consider the full CelebA setup, so that the results are more reliable?

Author response:
To address the reviewer's concern, we ran additional experiments with 20%, 50% and 100% of the CelebA. The results, over 3 runs, showing the WGA in each scenario are in the following table:

|    |20% of CelebA  | 50% of CelebA | Full CelebA      |
|----|---------------|---------------|---------------|
|   Ideal  | 52.3 +- 1.6  | 50.3 +- 1.9   | 45.9 +- 5.8 |
| Standard  | 43.1 +- 5.5  | 46.4 +- 2.8   | 43.6 +- 3.0|
|   Worst  | 32.3 +- 2.7  | 32.0 +- 0.8   |  37.7 +- 3.8|

The persistent gap between the Ideal and Standard cases demonstrates the negative impact of EPIc on group robustness (i.e., our findings are consistent).

Reviewer comment (Evaluation):
Why are the ASR values negative in Table 3 (Waterbirds / SA setup).

Author response:
For the SA, we define the ASR as the relative accuracy drop of the attack on the target group over a non-poisoned model. So, in case the defense not only prevents the accuracy drop by removing the poisons but also slightly increases the accuracy on the targeted group, then the ASR would be negative. We will clarify this in the updated paper.

Reviewer comment (Experiments):
Could the authors elaborate on whether any experiments were conducted involving the other two types of poisoning defenses mentioned in Section 2.1?

Author response:
Indeed, in our paper, we only consider defenses that assume that poisons are hard to learn as more realistic, weak, attackers are the ones who would benefit from the amplification offered by group robustness methods. However, we believe that poisoning defenses designed against easy-to-learn poisons (e.g., [1]) would indeed not eliminate legitimate minority samples and hence not hurt the worst-group accuracy (though, such defenses struggle against attacks that craft hard-to-learn poisons [2]).

A promising future direction could be the defenses that assume poisons and clean samples follow different distributions, so they create a small, poison-free set of samples for each group to *erase* poisons (similar to [3] but with a group-balanced clean data to preserve group robustness).

Reviewer comment (Presentation):
Title Appropriateness: The broad focus implied by the current title, "Poisoning Defenses," does not accurately reflect the specific emphasis of the work on poisons that are challenging to learn. A more precise title would set clearer expectations for readers.

Author response:
We agree with the reviewer's concern that our title might be too broad. We will specify that we study weaker poisoning attacks, e.g., "Resilience to Weak Poisoning Attacks and Group Robustness are at Odds". Please let us know if you have a suggestion!

Reviewer comment (Experiments):
The tensions between poisoning and group robustness in the paper are based on limited empirical results, and the poisoning attacks might not reflect what a practical attacker would do. For instance, many ASR values in Table 2 are quite low, and it is unclear if this is due to undesirable attack configurations that do not align with the attack objective. A different attack approach could achieve higher ASR and might not exhibit a strong tension.

Author response:
> …poisoning attacks also do not reflect what the attacker in practice may really care to do. For example, many ASR are quite low as presented in Table 2 and I am not sure if it is because the poisoning attacks are configured in some undesirable ways…  

An attacker in practice aims to achieve the highest ASR possible within their limited budget. In the case of a poisoning adversary, the number of injected samples determines the real-world budget. The lower the budget is, the more realistic an attack becomes (e.g., the attacker needs to compromise fewer websites to inject images [1]).  

A main takeaway from our experiments is that low-budget attacks where the adversaries struggle to achieve high ASR (hence the results in Table 2) can benefit from the defender’s use of group robustness methods such as JTT. The same cannot be said for high-budget attacks that can already achieve high ASR (thus, amplification is less critical). Therefore, the low ASR of the attacks we performed (Table 2) is not due to any specific configuration of the attacks (e.g., hyper-parameters) but due to the considered attacker budgets. This allows us to expose how low-budget attacks (more realistic but weaker) can be inadvertently amplified into stronger ones, essentially giving the attacker “more bang for their buck”.  

Furthermore, we would like to clarify that for the Subpopulation Attack (SA) in Table 2, ASR indicates a **relative accuracy drop** on the attack's target group over a non-poisoned model. Therefore, some of our experiments already consider more powerful attacks (e.g., when the ASR is 31%), in which our findings are consistent.

Reviewer comment (Experiments):
In page 4, what is the intuition behind configuring the poisoning attacks as described? The configuration seems to be different from the one in the original paper and the authors should clearly explain the reason.

Author response:
> I am not sure if it is because the poisoning attacks are configured in some undesirable ways (with respect to the attack objective), who may choose a different approach to achieve higher ASR —----------- In page 4, what is the intuition behind configuring the poisoning attacks as described?  

To address the reviewer’s concerns and questions, we considered attackers (DLBD) selecting poison base samples from different groups of CelebA (the results are over 3 runs):  

|          | Gr0 (HRG) | Gr1 (HRG)|Gr2 (HRG)|Gr3 (LRG-1)|  
| -------- | --------  | -------- |---------|-----------|  
| Label/Attribute|0/0|0/1|1/0|1/1|  
| ASR      |97.4 +- 0.1|  97.7 +- 0.1 | 91.2 +- 2.3|   0.1 +- 0.1|  

The maximum ASR holds for Gr1, which is the one we chose in our experiments as a base group for crafting the samples. Since the attacker changes the label of the samples they poison, the label/attribute configuration will become 1/1 which matches the one for LRG-1 (as stated in the paper).  

For a dirty-label attack, if the attacker is aware of the groups that are easiest to learn, they can take advantage of it and create poisons from those groups as base samples (this matches the setting from our theoretical result). Also, we show below the average loss of each group when there are no poisons in the dataset:  

|          | Gr0 (HRG) | Gr1 (HRG)|Gr2 (HRG)|Gr3 (LRG-1)|  
| -------- | --------  | -------- |---------|-----------|  
| Label/Attribute|0/0|0/1|1/0|1/1|  
| Loss Average      |0.16   |       0.07   |       0.79 |          1.90|  

We observe that the easier a group is to learn (i.e., less average loss), the higher the ASR will be if the attacker crafts a DLBD attack with base samples from that group.

Reviewer comment (Experiments):
Will the gradient shaping method [1] enable a mitigation to the currently observed tension?  

[1] Hong et al., "On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping", arXiv 2020.

Author response:
> Question: Will the gradient shaping method [1] enable a mitigation to the currently observed tension?  

Gradient shaping relies on a mechanism similar to differential privacy (DP) that injects noise into the training process. DP is already known to hurt the group robustness [2] as noise prevents the model from learning hard-to-learn, minority samples. As a result, gradient shaping can mitigate the poisons we consider in our work (i.e., weaker attacks with low budget) but it’ll also have an undue impact on group robustness that the defender wants to avoid, i.e., the tension still remains, the defender can either have poison resilience and low worst-group accuracy or poison vulnerability and high worst-group accuracy.  


[1] Poisoning Web-Scale Training Datasets is Practical, Carlini et al. (2023)  

[2] Differential Privacy Has Disparate Impact on Model Accuracy, Bagdasaryan and Shmatikov (NeurIPS’19)