Paper: Debiasing Attention Mechanism in Transformer without Demographics (ICLR-2024)

Reviewer comment (Presentation):
In the experiment tables, which method belongs to "Fairness without Demographics", which method requires Demographics?

Author response:
**Response:** We note that for all the methods we compared, none of those utilize demographics in training procedures. We provide detailed illustration on implementation of all methods in Appendix Section C.

Reviewer comment (Evaluation):
The proposed method cannot beat SOTA methods. For example, in Table 2, the proposed method is worse than LfF on EOp dataset. In Table 4, the proposed method is worse than JTT method on EOp and EOd dataset.

Author response:
**Response:** We recognize that our method does not outperform all other methods across every dataset. As shown in Table 2, Learning from Failure (LfF) attains better Equality of Opportunity (EOp) results, albeit at the cost of a roughly $6\%$ decrease in accuracy compared to the Empirical Risk Minimization (ERM) model. We also observe that the average accuracy of $87.97\%$ is only marginally higher than the class rate, where $p(y=0) = 84.62\%$ in the test set. However, our method achieves comparable utility to the ERM model while enhancing group fairness.

In Table 4, as we discussed in Section 4.2, the significant disparity between the rates $p(a=0)=92.9\%$ and $p(a=1)=7.1\%$ leads to an increased theoretical upper bound. As specified in Theorem 1, where $λ_0=p(a=0)$ and $λ_1=p(a=1)$, this upper bound is directly proportional to $\frac{λ_0}{λ_1}$. This disparity accounting for the performance drop in this test. Although Just Train Twice (JTT) shows superior fairness outcomes, it requires considerably more computational resources compared to other baseline methods.

Reviewer comment (Evaluation):
Following problem 2, the authors may claim that they have much less energy consumption. However, the comparison is not straightforward. The authors need to show one of the two results to claim this point: 1) same energy consumption and higher accuracy; 2) same accuracy and less energy consumption. If the authors can show these results compared to LfF on EOp dataset, and JTT method on EOp and EOd dataset, I will raise my score.

Author response:
**Response:** In line with the reviewer's recommendation, we have conducted additional experiments with a specific focus on energy consumption. We set a predetermined energy budget and, within this constraint, trained various methods to observe their performance. The other experimental settings remain consistent with those detailed in Table 2 and Table 4 (in the paper). To ensure a fair comparison, we established the energy budget at the mean level of energy consumption observed across all methods. For the CelebA dataset, we allocate a power energy of 363.63 Wh, and for MultiNLI dataset, we allocate a power energy of 2742.78 Wh for BERT large, 867.46 Wh for BERT base. The results could be found in Table 1, 2, and 3.

Table 1: Results of the MultiNLI Experiment under a Limited Energy Budget of 2742.78 Wh (Backbone: BERT Base).

|  | DP ↓    | EOp ↓ | EOd ↓  | ACC ↑ | Energy   |
|--------|--------|-------------|-------------|------|-----|
| JTT  | 47.42 | 12.54| 11.37 | 83.37| 2313.27
| Ours  | 41.19| 10.15| 8.12| 83.96| 1903.87


Table 2: Results of the MultiNLI Experiment under a Limited Energy Budget of 867.46 Wh (Backbone: BERT Base).

|  | DP ↓    | EOp ↓ | EOd ↓  | ACC ↑ | Energy   |
|--------|--------|-------------|-------------|------|-----|
| JTT  | 46.28 | 14.63 | 13.22 | 78.88 | 766.92
| Ours  | 46.94 | 13.27 | 12.33| 81.38| 684.42


Table 3: Results of the CelebA Experiment under a Limited Energy Budget of 363.63 Wh.

|  | DP ↓    | EOp ↓ | EOd ↓  | ACC ↑ | Energy   |
|--------|--------|-------------|-------------|------|-----|
| LfF  | 21.07 | 45.48 |25.92 |92.67 | 258.65
| Ours  | 14.26| 39.70 | 20.84 | 93.59 | 175.23

Tables 1, 2, and 3 indicate that, under a limited energy budget, our method not only achieves improved utility but also yields enhanced fairness results simultaneously. In contrast, under the same energy constraints, the early stopping for JTT impedes its ability to achieve fair results. For LfF (Learning from Failure), the reduced number of iterations improves its utility. However, this comes at the cost of its fairness performance.

Reviewer comment (Evaluation):
Energy consumption is not a stable indicator when comparing the models. The hardware may have big influence. I would recommend to use FLOPs to compare these methods.

Author response:
**Response:**
(1) We would like to clarify that in this study, we conducted controlled experiments, ensuring that all methods were executed under identical configurations. The number of Floating Point Operations (FLOPs) is used to quantify the total count of elementary machine operations. In line with the reviewer's recommendation, we employed FLOPs as the metric to evaluate all methods. We carry out the experiment on CelebA dataset, with $y=$Blond Hair and $a=$Male. We use the *thop* library to estimate the total training FLOPs.

Table 4: Comparative Analysis of FLOPs Required by Different Methods During the Training Stage

|  | ERM | DRO | ARL | KD| JTT| LfF| Ours
|-------|---------|----|-----|-------------|-------------|------------|-------------|
| FLOPs  | $3.10×10^{13}$ | $3.10×10^{13}$ | $5.14×10^{13}$ |  $7.74×10^{13}$ | $1.34×10^{14}$| $8.26×10^{13}$|$4.13×10^{13}$|

Reviewer comment (Theory):
The last layer retraining is convenient but provides no guarantees. Analyzing how bias propagates through the full network could further improve this.

Author response:
For the last layer retraining we integrate our debiasing encoder with a pre-trained model. During the training phase, we freeze the pre-trained model, focusing on training the debiasing encoder and classifier for the downstream task. As the pre-trained model remains frozen during training,  biases inherent in its representations are unavoidable. Our proposed method functions as a debiasing encoder, designed to minimize sensitive information in the final representation. For details, we kindly direct the reviewer to our general response.

Reviewer comment (Experiments):
The contrastive loss operates locally on values. Extending the alignment more globally could potentially improve fairness further without sacrificing accuracy.

Author response:
Extending contrastive loss to include all patches is effectively the same as applying it to the representation $\mathbf{z}$. However, it's important to note that directly aligning the representation $\mathbf{z}$ does not necessarily contribute to fairness. Park et al [1] demonstrate that with the use of Supervised Contrastive Learning (SupCon) to pre-train a ResNet model, unfairness issue still exists.  We attribute this phenomenon to the correlation between the target and sensitive labels. Since the network contrasts the entire representation, it can still leverage sensitive-related information.

In contrast, our proposed method involves two steps. The first is debiasing the attention weight, a crucial step as it plays a role in selecting high attention patches. The second step involves using screened local values for contrastive learning, focusing on target-related information, thereby enhancing fairness. Our ablation study, detailed in section F of the Appendix, reveals that employing local value alignment alone has a limited impact on promoting fairness.

Reviewer comment (Evaluation):
The statistics of the $q$ and $k$ are estimated during training and then used as an estimation during inference. Authors should provide more details on the accuracy of this choice, as the distribution shift during training and inference might affect both fairness and classifier performance.

Author response:
>**Q3**: The difference between the training set estimated statistics and test set statistics. Also concern about distribution shift.


**Response:** Following the reviewer's suggestion, we run the experiment on CelebA, with the task $y$=Blond hair, $a=$Male. We report $||\mathbb{E}[\mathbf{Q}]||_F$, $||\mathbb{E}[\mathbf{K}]||_F$, $||\sigma[\mathbf{Q}]||_F$ and $||\sigma[\mathbf{K}]||_F$ in the test set.  We also report running mean, and running standard deviation estimated in the training set.



Table 2: Statistics in the test set and estimated statistics in the training set

| $eq$ | $eq_r$ | $diff_{eq}$ ↓ | $ek$ | $ek_r$ | $diff_{ek}$↓  |
|--------|----------------|--------------|--------------|--------------------------|-------------|
| 143.91| 142.86 | 23.60 | 180.39 | 176.39  | 14.92  |

where $eq$ = $||\mathbb{E}[\mathbf{Q}]||_F$,   

$eq_r= ||\mathbf{e_{\text{running mean for query}}}||_F$, 

$diff_{eq}$ = $||\mathbb{E}[\mathbf{Q}]-\mathbf{e}_{\text{running mean for query}}||_F$,

$ek = ||\mathbb{E}[\mathbf{K}]||_F$, 

$ek_r$= $||\mathbf{e}_{\text{running mean for key}}||_F$, 

$diff_{ek}$ = $||\mathbb{E}[\mathbf{K}]-\mathbf{e}_{\text{running mean for key}}||_F$

| $sq$ | $sq_r$ | $diff_{sq}$ ↓ | $sk$ | $sk_r$ | $diff_{sk}$↓  |
|--------|----------------|--------------|--------------|--------------------------|-------------|
| 231.22 | 233.05| 14.13 | 167.23  | 157.03  | 25.01  |

$sq = ||\sigma[\mathbf{Q}]||_F$, 

$sq_r$= $||\mathbf{s}_{\text{running std for query}}||_F$, 

$diff_{sq}$= $||\sigma[\mathbf{Q}]-\mathbf{s}_{\text{running std for query}}||_F$,

$sk$ = $||\sigma[\mathbf{K}]||_F$, 

$sk_r$= $||\mathbf{s}_{\text{running std for key}}||_F$, 

$diff_sk$ = $||\sigma[\mathbf{K}]-\mathbf{s}_{\text{running std for key}}||_F$.



It is important to highlight that in practice, we can only estimate the statistics of the test set based on the running statistics from the training set, a common approach also utilized in batch normalization and layer normalization. Table 3 is presented as a reference to evaluate the quality of this estimation. Our observations from Table 3 indicate that the distribution shift in the CelebA dataset, for the task $y=$Blond Hair, $a=$Male, is relatively minor. We acknowledge that significant deviations between the statistics estimated by the running mean and standard deviation and those in the test set will have an impact on our method's performance, potentially leading to a decreased utility. In fact, the distribution shift is also a challenging problem in the ML community.

Reviewer comment (Experiments):
For different datasets, different combinations of label $y$ and sensitive attributes $A$ are used which may give a bad impression of cherry picking.

Author response:
>**Q5**:
(1). Limited evalutaion. (2). Concern about task selection (3). Compatibility of existing work.

**Response:**
- Following the reviewer's suggestion, we have carried out additional experiments on the CelebA dataset, which is notable for its 40 attributes and widespread use in fairness research. Specifically, we focused on tasks commonly explored in fairness studies. For the CelebA dataset, the tasks are $y=$ Wavy Hair $a=$ male [1] and $y=$ Bags under eyes $a=$ Young [2]. The results can be found in Tables 3 and 4.
- For different datasets, we do not pick a particular task. Instead, we followed the experiments outlined in [3, 4, 5], which are widely recognized in fairness research.
- We provide an additional experiment to demonstrate the compatibility of our method. We conducted tests combining our approach with the Just Train Twice (JTT), the results are in Table 5. We thank the reviewer for the suggestion and will add full results to our final paper.

Table 3: Classification results on CelebA dataset: $a$= male, $y$ = Wavy hair.

|    | ERM    | DRO    | ARL   | KD    | JTT  | LfF*  | Ours           |
|--------|----------|---------|--------|-----------|------|------|---------|
| DP ↓  | 30.76 ± 1.51   | 34.85 ± 1.43   | 27.67 ± 1.18   | 29.91 ± 3.77   | 29.80 ± 1.45    |   -  | 28.96 ± 2.74   |
| EOp↓         | 40.52 ± 1.80   | 44.63 ± 2.62   | 40.54 ± 1.44   | 41.26 ± 1.13   |  29.16 ± 0.44    |  -  | 34.93 ± 0.51   |
| EOd ↓        | 28.14 ± 1.38   | 31.79 ± 1.82   | 27.26 ± 1.23   | 28.02 ± 2.07   | 24.23 ± 0.96     |  -  | 24.82 ± 1.32   |
| ACC ↑        | 74.11 ± 0.14   | 73.97 ± 0.32   | 72.11 ± 0.04   | 73.46 ± 0.86   |  69.87 ± 0.36  |   63.60*  | 73.26 ± 0.38   |

*LfF makes all predictions to the same target group. We did a grid search on the learning rate from \{$10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}$\} with optimizer \{$Adam, AdamW$\}.

Table 4: Classification results on CelebA dataset: $a$ = young, $y$ = Bags under eyes.

|              | ERM            | DRO            | ARL            | KD             | JTT           | LfF  | Ours           |
|--------------|----------------|----------------|----------------|----------------|---------------|------|----------------|
| DP ↓         | 15.70 ± 1.13   | 9.13 ± 0.50    | 7.92 ± 0.85    | 10.83 ± 1.49   | 10.57 ± 0.23  |   8.95 ± 2.46  | 6.34 ± 0.97    |
| EOp↓         | 15.03 ± 1.68   | 6.15 ± 2.35    | 6.97 ± 1.08    | 6.98 ± 1.47    | 2.27 ± 0.88   |  7.75 ± 2.81    | 4.10 ± 0.50    |
| EOd ↓        | 11.56 ± 0.72   | 5.35 ± 1.07    | 5.65 ± 0.78    | 6.18 ± 0.92    | 4.71 ± 0.54   |  6.23 ± 2.06    | 3.70 ± 0.41    |
| ACC ↑        | 83.15 ± 0.13   | 82.49 ± 0.38   | 81.57 ± 0.24   | 82.96 ± 0.23   | 78.78 ± 0.76  |   81.66 ± 0.47   | 81.80 ± 0.52   |


Table 5: Compatible experiment on CelebA: $a$ = Male, $y$ = Blond Hair.

|    | ERM   |   JTT  |Ours| Ours+JTT  |
|----------|----------------|----------------|---------------|-----------|
| DP ↓  | 16.92 ± 0.55  | 18.04 ± 0.32 | 16.34 ± 1.94 | 16.91 ± 0.45  |
| EOp↓   | 42.82 ± 0.47  | 36.17 ± 1.08 | 38.01 ± 1.32   | 28.77  ± 2.44   |
| EOd ↓  | 22.90 ± 0.10 | 20.26 ± 1.72  | 20.43 ± 0.68  |  16.47  ± 1.07   |  
| ACC ↑  | 94.26 ± 0.06 | 92.75 ± 0.02 | 94.06 ± 0.43  |   91.83  ± 0.91   |

Analysis of Tables 3 and 4 shows that our method outperforms other baseline models in various tasks. While Just Train Twice (JTT) maintains good fairness, it experiences a slight drop in accuracy. Interestingly, as Table 5 illustrates, combining our method with JTT leads to enhanced outcomes. This improvement is attributed to JTT's ability to modify the distribution among sensitive groups, thereby balancing the data distribution across different groups and potentially reducing the unfairness upper bound as outlined in Theorem 1. Moreover, our method's use of contrastive learning techniques further enhances the utility of the JTT approach.

References:
- [1] Han, Xiaotian, et al. "FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods." arXiv preprint arXiv:2306.09468 (2023).
- [2] Park, Sungho, et al. "Fair contrastive learning for facial attribute classification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- [3] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." International Conference on Machine Learning. PMLR, 2021.
- [4] Hong, Youngkyu, and Eunho Yang. "Unbiased classification through bias-contrastive and bias-balanced learning." Advances in Neural Information Processing Systems 34 (2021): 26449-26461.
- [5] Baldini, Ioana, et al. "Your fairness may vary: Pretrained language model fairness in toxic text classification." arXiv preprint arXiv:2108.01250 (2021).

Reviewer comment (Evaluation):
On average, the proposed method is not better than previous approaches and the main benefit is less compute. I wonder whether this approach can be combined with previous methods to give better performance in terms of reducing bias and preventing the degradation in the classifier accuracy.

Author response:
>**Q5**:
(1). Limited evalutaion. (2). Concern about task selection (3). Compatibility of existing work.

**Response:**
- Following the reviewer's suggestion, we have carried out additional experiments on the CelebA dataset, which is notable for its 40 attributes and widespread use in fairness research. Specifically, we focused on tasks commonly explored in fairness studies. For the CelebA dataset, the tasks are $y=$ Wavy Hair $a=$ male [1] and $y=$ Bags under eyes $a=$ Young [2]. The results can be found in Tables 3 and 4.
- For different datasets, we do not pick a particular task. Instead, we followed the experiments outlined in [3, 4, 5], which are widely recognized in fairness research.
- We provide an additional experiment to demonstrate the compatibility of our method. We conducted tests combining our approach with the Just Train Twice (JTT), the results are in Table 5. We thank the reviewer for the suggestion and will add full results to our final paper.

Table 3: Classification results on CelebA dataset: $a$= male, $y$ = Wavy hair.

|    | ERM    | DRO    | ARL   | KD    | JTT  | LfF*  | Ours           |
|--------|----------|---------|--------|-----------|------|------|---------|
| DP ↓  | 30.76 ± 1.51   | 34.85 ± 1.43   | 27.67 ± 1.18   | 29.91 ± 3.77   | 29.80 ± 1.45    |   -  | 28.96 ± 2.74   |
| EOp↓         | 40.52 ± 1.80   | 44.63 ± 2.62   | 40.54 ± 1.44   | 41.26 ± 1.13   |  29.16 ± 0.44    |  -  | 34.93 ± 0.51   |
| EOd ↓        | 28.14 ± 1.38   | 31.79 ± 1.82   | 27.26 ± 1.23   | 28.02 ± 2.07   | 24.23 ± 0.96     |  -  | 24.82 ± 1.32   |
| ACC ↑        | 74.11 ± 0.14   | 73.97 ± 0.32   | 72.11 ± 0.04   | 73.46 ± 0.86   |  69.87 ± 0.36  |   63.60*  | 73.26 ± 0.38   |

*LfF makes all predictions to the same target group. We did a grid search on the learning rate from \{$10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}$\} with optimizer \{$Adam, AdamW$\}.

Table 4: Classification results on CelebA dataset: $a$ = young, $y$ = Bags under eyes.

|              | ERM            | DRO            | ARL            | KD             | JTT           | LfF  | Ours           |
|--------------|----------------|----------------|----------------|----------------|---------------|------|----------------|
| DP ↓         | 15.70 ± 1.13   | 9.13 ± 0.50    | 7.92 ± 0.85    | 10.83 ± 1.49   | 10.57 ± 0.23  |   8.95 ± 2.46  | 6.34 ± 0.97    |
| EOp↓         | 15.03 ± 1.68   | 6.15 ± 2.35    | 6.97 ± 1.08    | 6.98 ± 1.47    | 2.27 ± 0.88   |  7.75 ± 2.81    | 4.10 ± 0.50    |
| EOd ↓        | 11.56 ± 0.72   | 5.35 ± 1.07    | 5.65 ± 0.78    | 6.18 ± 0.92    | 4.71 ± 0.54   |  6.23 ± 2.06    | 3.70 ± 0.41    |
| ACC ↑        | 83.15 ± 0.13   | 82.49 ± 0.38   | 81.57 ± 0.24   | 82.96 ± 0.23   | 78.78 ± 0.76  |   81.66 ± 0.47   | 81.80 ± 0.52   |


Table 5: Compatible experiment on CelebA: $a$ = Male, $y$ = Blond Hair.

|    | ERM   |   JTT  |Ours| Ours+JTT  |
|----------|----------------|----------------|---------------|-----------|
| DP ↓  | 16.92 ± 0.55  | 18.04 ± 0.32 | 16.34 ± 1.94 | 16.91 ± 0.45  |
| EOp↓   | 42.82 ± 0.47  | 36.17 ± 1.08 | 38.01 ± 1.32   | 28.77  ± 2.44   |
| EOd ↓  | 22.90 ± 0.10 | 20.26 ± 1.72  | 20.43 ± 0.68  |  16.47  ± 1.07   |  
| ACC ↑  | 94.26 ± 0.06 | 92.75 ± 0.02 | 94.06 ± 0.43  |   91.83  ± 0.91   |

Analysis of Tables 3 and 4 shows that our method outperforms other baseline models in various tasks. While Just Train Twice (JTT) maintains good fairness, it experiences a slight drop in accuracy. Interestingly, as Table 5 illustrates, combining our method with JTT leads to enhanced outcomes. This improvement is attributed to JTT's ability to modify the distribution among sensitive groups, thereby balancing the data distribution across different groups and potentially reducing the unfairness upper bound as outlined in Theorem 1. Moreover, our method's use of contrastive learning techniques further enhances the utility of the JTT approach.

References:
- [1] Han, Xiaotian, et al. "FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods." arXiv preprint arXiv:2306.09468 (2023).
- [2] Park, Sungho, et al. "Fair contrastive learning for facial attribute classification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- [3] Liu, Evan Z., et al. "Just train twice: Improving group robustness without training group information." International Conference on Machine Learning. PMLR, 2021.
- [4] Hong, Youngkyu, and Eunho Yang. "Unbiased classification through bias-contrastive and bias-balanced learning." Advances in Neural Information Processing Systems 34 (2021): 26449-26461.
- [5] Baldini, Ioana, et al. "Your fairness may vary: Pretrained language model fairness in toxic text classification." arXiv preprint arXiv:2108.01250 (2021).

Reviewer comment (Writing):
One weakness of the paper is the motivation. While debiasing models / fairness is generally a strong motivation, the authors do not explain in much detail why it is important to debias attention mechanisms. Instead, most of the introduction feels like a "related works" section, where the main motivation of the paper is the failings of previous methods. I would like to see a better written introduction that explains why the failings of previous methods are bad or costly, and why a new method is needed.

Author response:
**Response:** The Transformer architecture is well-known for its effectiveness, but studies [1, 2] have noted that it is affected by statistical biases. For a detailed discussion on the significance of debiasing attention mechanisms, kindly please refer to the general response to Question 1.

 Current methods addressing fairness issues without demographics can be broadly divided based on the use of auxiliary networks. Methods that do not require auxiliary networks include Distributionally Robust Optimization (DRO) [10] and Just Train Twice (JTT) [9], while those employing auxiliary networks including Adversarial Representation Learning (ARL) [3], Knowledge Distillation (KD) [7], and Learning from Failure (LfF) [8].

Our approach is specifically designed to address fairness without demographics and without the need for an auxiliary network, thereby reducing parameters for better efficiency. Regarding existing non-auxiliary network methods, DRO, as pointed out by [3], is likely to be affected by outliers, leading to a drop in performance. JTT, on the other hand, involves a costly double-training procedure, with the second phase incorporating a larger training dataset.
In terms of the method for an auxiliary network, Knowledge Distillation (KD) utilizes a teacher model that is often larger than the student network performing the downstream task. Learning from Failure (LfF) alternates optimization between two networks of the same size, leading to parameter inefficiency. Although the auxiliary network in Adversarially Reweighted Learning (ARL) is smaller compared to the main network, its incorporation of adversarial training methods can introduce instability issues. In contrast, our approach obviates the need for such auxiliary networks, thereby enhancing parameter efficiency. Furthermore, it eliminates the necessity for multiple training phases, thus aligning with the demands of time efficiency.

Reviewer comment (Experiments):
I would like to see an ablation study of the two mechanisms to compare which one impacts fairness more.

Author response:
**Response:** We provide an ablation study in Appendix section F.

Reviewer comment (Theory):
It seems as though instead of using subgroup attributes in the contrastive loss, you use classes. However, what happens if each class is dominated by a single sensitive group attribute? For example, if class 1 is all male and class 2 is all female, then requiring all class 1 (male) representations to be similar with each other and different than class 2 (female) will actually cause more disparity by pushing the representations away from each other. Don't we have to assume sensitive groups are balanced within classes?

Author response:
**Response:** We do not pre-suppose that sensitive groups are evenly distributed within classes. The reasons are as follows: (1) If we assume sensitive groups are balanced within classes, research by [6] has shown that group unfairness can be significantly mitigated through ERM training. (2) In cases where each class primarily consists of a single sensitive group, samples from minority groups tend to have higher losses. This occurs because these samples are contrasted with those from different sensitive groups. By optimizing this loss, our proposed framework effectively mitigates the disparities in representation among different sensitive groups. (3) However, in the extreme case where class distinctions rely only on sensitive information, our method may not be effective. In such instances, other comparative methods like JTT also fail and work similarly to ERM training.

Reviewer comment (Experiments):
Won't normalizing reduce the expressivity of the model? I would love to see the distribution of attention weights before and after normalizing, as well as before and after absolute valuing (as well as an ablation study of the two steps).

Author response:
**Response:** We provide a visual attention allocation comparison of our method and ERM in Appendix Section E. We will also provide more ablation results on the distribution of attention weight of the two steps in the revised version.