Paper: The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Language Models (ICLR-2024)

Reviewer comment (Experiments):
Clarify how the Demographic dimensions included in Table 1 were determined. Was this based on previous work, or were these dimensions constructed specifically for the purposes of this work?

Author response:
**A1**:
We constructed the demographic dimension by concluding from previous works [1,2] in social bias research.
Most previous research concentrated on the few prevalent demographic dimensions, such as gender or ethnicity. Besides, previous research mainly conducted their research on only one or two demographic dimensions. To be more comprehensive, we collect 9 demographic dimensions of concern in the current fairness research field from previous works.

&nbsp; 

[1] Gender and Representation Bias in GPT-3 Generated Stories.

[2] From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models.

Reviewer comment (Presentation):
For Figure 2, the paper does not clarify what model was used to generate the results. Was this based on the BERT model? Clarify this and also discuss whether there are any observed differences in the presented results for Figure 2 across models.

Author response:
**A2**:
Thank you for your suggestion! The results are generated by FairBERTa. We have also reached similar conclusions for other models (BERT and RoBERTa). We have clarified this in our revised paper.

Reviewer comment (Writing):
Provide a more detailed discussion and explanation of how the proposed approaches could be applied in other applications beyond social biases and debiasing. While this is briefly mentioned in the Introduction, the paper does not go into details, and it remains unclear from a reader’s perspective how these approaches could be applied for other purposes.

Author response:
**A3**
Thank you so much for pointing this out! Our IG$^2$ can be applied to other research fields, e.g., comparative knowledge study. Previous works have found that models trained to learn comparative knowledge have better reasoning ability compared with those trained on singular knowledge. For example, models trained to learn that ''moon is round, not square'' can perform better in reasoning tasks than models trained to learn that ''moon is round''. Our IG$^2$ can be applied to detect neurons contributing to storing comparative knowledge and then amplify the activation of these neurons, in order to achieve better reasoning ability.

Reviewer comment (Experiments):
The paper primarily relies on StereoSet and associated metrics to measure the effectiveness of the debiasing technique. While this is not a major limitation, it might have been nice to evaluate on another dataset with targeted queries for a specific group, such as WinoQueer (https://arxiv.org/abs/2306.15087).

Author response:
Thank you for your advice! We have conducted the experiments following your advice on WinoQueer.

| Model                | WinoQueer Overall | LGBTQ | Queer | Transgender | NB    | Bisexual | Pansexual | Lesbian | Asexual | Gay   |
|----------------------|-------------------|-------|-------|-------------|-------|----------|-----------|---------|---------|-------|
| BERT-Base-cased      | 64.40             | 91.55 | 58.53 | 91.72       | 78.93 | 43.01    | 27.33     | 90.97   | 33.44   | 41.71 |
| BERT-Base-cased + BNS (Ours) | 60.38   | 88.04 | 53.85 | 89.16       | 75.42 | 39.36    | 24.05     | 86.51   | 31.92   | 46.58 |
| RoBERTa-Base         | 69.18             | 74.17 | 61.68 | 49.04       | 87.93 | 67.1     | 85.91     | 81.27   | 81.63   | 62.19 |
| RoBERTa-Base + BNS (Ours) | 66.74     | 72.81 | 60.22 | 47.79       | 86.04 | 64.80    | 84.32     | 77.69   | 80.17   | 60.01 |
| FairBERTa            | 68.87             | 73.92 | 61.54 | 48.99       | 87.45 | 66.92    | 85.88     | 81.13   | 81.05   | 61.68 |
| FairBERTa + BNS (Ours) | 66.51          | 71.48 | 60.35 | 47.62       | 86.25 | 64.27    | 84.19     | 80.07   | 79.68   | 59.93 |

Reviewer comment (Presentation):
Figure 2: It is unclear from the caption what exactly the x-axis is plotting, and clarification is needed.

Author response:
The x-axis is the randomly selected Judged Unfair Targets (JUTs). The definition of the JUTs is presented in Section 2.3. "-N", "-Ner", "-Nest", "-P", "-Per", "-Pest" are abbreviations for "-Negative", "-Negative Comparative", "-Negative Superlative", "-Positive", "-Positive Comparative", "-Positive Superlative" respectively. We have added more explanations in the caption for clearer expression. We have also added the abbreviations for different types of modifiers in Table 2 to make our paper more readable.

Reviewer comment (Presentation):
Table 6: There could be a bit more explanation about what the “inter” and “intra” columns represent.

Author response:
(1) **The ''intra'' column**. We counted the number of detected social bias neurons shared by different prompt templates within the same JUT. For example, we counted the number of detected social bias neurons shared among prompt templates within the JUT of (lonely, Black, White). We computed this metric for all the JUTs for a certain demographic dimension (e.g., "Ethnicity") and then reported the average. The relatively large values in this column reveal that our IG$^2$ is effective and robust in detecting the social bias neurons without interference from different grammar and sentence structures. 

(2) **The "inter" column**. We counted the number of detected social bias neurons shared across different JUTs. For example, we counted the number of detected social bias neurons shared between JUT of (lonely, Black, White) and JUT of (lonely, Hispanic, American) using prompt template #1 (prompt templates are defined in Table 3). We randomly selected some prompt templates and paired JUTs to gather this data. The relatively small values in this column reveal that different types of social biases share almost no social bias neurons. This suggests that our IG$^2$ identifies exclusive social bias neurons for different types of social biases.

Reviewer comment (Theory):
Is there a way to generalize the method to take into account interactions or correlations between neurons that might affect the result?

Author response:
Thank you for your very insightful question! This is a very good and valuable question!
First, quick answer: there should be an appropriate way to generalize both neurons and the interactions among neurons. In fact, we have considered dealing with the interactions among neurons and are currently working on a new project of studying and manipulating these interactions. In contrast to neurons, we refer to these interactions as ''edges''. There are many patterns of edges connecting neurons, e.g., string patterns, square patterns, etc. We believe that by precisely controlling the edges that connect neurons, we can achieve more ideal debiasing results: more accurately blocking the storage and propagation of biases while minimally impacting the model's capabilities. Furthermore, we have discovered that there is a mathematical form that can abstract and generalize neurons and edges: generalizing them into a homogenized minimal unit of neural networks. 

However, we hope you can understand that we have not mentioned these contents for the sake of clarity in our writing. There are many types of connection patterns for edges, and controlling these edges also involves the study of the propagation and flow of biases among neurons. This part also requires a significant amount of visualization, within which many interesting findings emerge. However, it is impossible to discuss all these contents in one work comprehensively. Therefore, to avoid ambiguity, we have not mentioned the interaction relationships between neurons in this work. To be honest, we are very surprised and excited to see this question. It also gives us much confidence to continue exploring the interactions among neurons. We will further explore this interesting question in the new project and hope to design an appropriate methodology to generalize neurons and edges (interactions among neurons). We hope our work can bring further inspiration to fairness research. Thank you again for your insightful question!

Reviewer comment (Experiments):
Do you think the method is limited by the kinds of prompts that you have templated here? How would you generalize?

Author response:
We could expand our approach to include a broader variety of prompt structures that encapsulate different linguistic styles and contexts. Here's an example of a new template: "In a situation where a [Demographic Dimension] individual is [Context], they are often perceived as [MASK]." The [Context] slot can be filled with various scenarios or actions, for example, "leading a team," "making a decision," or "in a family setting." This would help in capturing a wider spectrum of potential biases.

Reviewer comment (Experiments):
Can this method be extended beyond differences in sentiment for demographic groups?

Author response:
Our IG$^2$ can be applied to other research fields, e.g., comparative knowledge study. Previous works have found that models trained to learn comparative knowledge have better reasoning ability compared with those trained on singular knowledge. For example, models trained to learn that ''moon is round, not square'' can perform better in reasoning tasks than models trained to learn that ''moon is round''. Our IG$^2$ can be applied to detect neurons contributing to storing comparative knowledge and then amplify the activation of these neurons, in order to achieve better reasoning ability.

Reviewer comment (Evaluation):
The paper could spend more attention on the implications of the presented results. Since the authors mention the importance of fairness research in this context, a brief discussion on the usability, practicability, and limitations of their results would be desirable.

Author response:
Thank you for your advice! We briefly discuss the implications, practicability and limitations in the following.
- Implications of the presented results. We propose an interpretable method to analyze social bias in PLMs. Through our experiments, we found that previous debiasing methods, represented by FairBERTa, do not significantly reduce the number of bias neurons but merely alter their distribution (as shown in Figure 3). We believe this finding could bring new insights to the field and inspire other researchers to design more effective debiasing methods.
- Usability/Practicability. We foresee our findings impacting the development of more equitable language models. On one hand, our method offers good interpretability, providing insights into how to design more efficient debiasing methods. On the other hand, our method can be combined with previous debiasing approaches (such as FairBERTa) to further enhance effectiveness. Mitigating social biases in PLMs also has extensive practical applications. For example, reducing biases in resume screening and job matching, and promoting fair hiring practices.
- Limitations. While our study provides valuable insights, we recognize there exist limitations. For example, our proposed BNS method directly sets the activation values of selected social bias neurons to zero. Although this is effective, designing a more refined suppression method might yield even better results. These present opportunities for future research. 

We will add a section discussing the implications, practicability, and limitations of our findings in the revised paper.

Reviewer comment (Experiments):
Did you experiment with regularizing detected neurons rather than suppressing them entirely? Considering their suppression more gradually as opposed to zero-ing them out might help in increasing the Language Modeling Score while keeping a high Stereotype Score.

Author response:
We are very grateful for your suggestion, which is very enlightening for our future work! During the rebuttal, we follow your suggestion and conduct a quick experiment on BERT-Base-cased. We adjust the activation values of the detected bias neurons to 0.5 times their original values, instead of directly setting them to zero. We report the results in the following table. 

| Suppression Ratio | SS->50.00 | LMS (↑) | ICAT (↑) |
|-------------------|-----------|---------|----------|
| 0                 | 52.78     | 86.64   | 81.82    |
| 0.5               | 52.95     | 87.08   | 81.94    |

Reviewer comment (Presentation):
Not really a question but more a suggestion: I’d recommend moving the introduction of BNS to after the first report of experiments to improve the flow of the paper.

Author response:
Thank you for your advice! We have modified our paper accordingly.

Reviewer comment (Presentation):
Would it be worth adding a few examples (rather than only templates) to Table 3?

Author response:
Thank you for your advice! We have followed your advice and added the example in the revised paper.

Reviewer comment (Novelty):
The technical contribution of the paper is not significant as it simply extends a previous existing approach for identification of problematic neurons in the context of bias and fairness.

Author response:
We are not "simply extending a previous existing approach in the context of bias and fairness," but have a very clear motivation. In the field of social bias, previous methods mainly require finetuning or even pre-training of PLMs, which is highly costly. We aim to address this issue by proposing an interpretable and efficient method to locate and reduce social biases. Drawing inspiration from the IG approach, we designed our IG$^2$. Our IG$^2$ differs from the IG method: IG is used to analyze singular knowledge attribution, but our method analyzes multiple knowledge. Other reviewers also acknowledge our contribution to the community, for instance, “*The results are promising and might represent a foundation for future work applying the method to other applications in NLP*.” (Reviewer zYP2), and “*I believe that the proposed approaches can have a significant impact on the research community that aims to study social biases and ways to mitigate these issues in large language models*.” (Reviewer auwC).

Reviewer comment (Experiments):
The experiments are done only on masked language modeling tasks which can limit the study.

Author response:
Our method can be applied to other tasks, e.g., autoregressive generation. For example, we could use a prompt such as "*Please complete the following sentence. The gender of this handsome person is*". Then we could analyse the logit gap of the first token predicted by the language model. We will further explore this extension in the near future.

Reviewer comment (Experiments):
The approach operates between binary demographics and might be limited when we want to study biases amongst large pool of fine-grained demographics.

Author response:
We can extend our method to a large pool of fine-grained demographics. The logit gap between pairwise demographics is the most direct metric to reveal the unfairness for different demographics. In fact, to support the attribution of the distribution gap for more than two demographics, we can also choose other metrics that can be applied to measure the degree of dispersion among multiple demographics, such as standard deviation.

Reviewer comment (Experiments):
The paraphrases for constructing the dataset are generated using a language model which can impose its own biases in the study.

Author response:
We use a language model to generate templates with placeholders (such as ([Demographic_Dimension], [Modifier], and [MASK])) only, without including any specific demographics, as shown in Table 3. Therefore, this process does not introduce any biases from the language model itself.

Reviewer comment (Experiments):
It seems the IG^2 requires calculating the gap between the logits of different attributes. I'm wondering how general will it be, e.g., how can we extend it to other tasks instead of predicting the sensitive attributes? For example, what if the task is a hate speech detection?

Author response:
**Q1: Does the method only work when the output of the model is to predict the sensitive attributes (so that you can calculate the logit gap)? How can we extend it to other tasks instead of predicting the sensitive attributes?**

**A1**: The proposed methods can be extended to other fields and tasks (e.g., comparative knowledge study) without predicting sensitive attributes. Previous works have found that models trained to learn comparative knowledge have better reasoning ability compared with those trained on singular knowledge. For example, models trained to learn that ''moon is round, not square'' can perform better in reasoning tasks than models trained to learn that ''moon is round''. Our $IG^2$ can be applied to detect neurons contributing to storing comparative knowledge and then amplify the activation of these neurons, in order to achieve better reasoning ability.

**Q2: How can we extend it to hate speech detection?**

**A2**: 
We think the proposed method could not be directly applied to the hate speech detection task. As far as we know, the target of the hate speech detection task is to design a detection model and judge whether some given sentences contain toxic or discriminative expressions. Biases or toxicity are not generated by the detection model, but exist in the text input. In contrast, our method is to attribute biased output to model internal neurons. This requires that biased sentences are generated by the model we focus on.

Reviewer comment (Evaluation):
Does the method only work when the output of the model is to predict the sensitive attributes (so that you can calculate the logit gap)?

Author response:
**Q1: Does the method only work when the output of the model is to predict the sensitive attributes (so that you can calculate the logit gap)? How can we extend it to other tasks instead of predicting the sensitive attributes?**

**A1**: The proposed methods can be extended to other fields and tasks (e.g., comparative knowledge study) without predicting sensitive attributes. Previous works have found that models trained to learn comparative knowledge have better reasoning ability compared with those trained on singular knowledge. For example, models trained to learn that ''moon is round, not square'' can perform better in reasoning tasks than models trained to learn that ''moon is round''. Our $IG^2$ can be applied to detect neurons contributing to storing comparative knowledge and then amplify the activation of these neurons, in order to achieve better reasoning ability.

Reviewer comment (Experiments):
What model is being examined in Fig.2? Does the conclusion hold for all models, especially for FairBERTa?

Author response:
**Q3: What model is being examined in Fig.2? Does the conclusion hold for all models, especially for FairBERTa?**

**A3**: 
The model examined in Fig.2 is FairBERTa. We have also reached similar conclusions for other models (BERT and RoBERTa). We have revised the caption to make it clearer.

Reviewer comment (Experiments):
It seems most of the time, Union_IG can make the bias score lower. Is it because Union_IG contains more neurons? Does this mean adding more neurons generally helps reduce the bias? How sensitive is the result w.r.t. $\sigma$?

Author response:
**Q4: It seems Union_IG can make the bias score lower. Is it because Union_IG contains more neurons? Does this mean adding more neurons helps reduce the bias? How sensitive is the result w.r.t. $\sigma$?**

**A4**: 
(1) No. Sometimes Union_IG contains fewer neurons than $IG^2$. The reason for Union_IG making the bias score lower is that Union_IG contains all the neurons that affect the learned knowledge about different demographics. Therefore, when we suppress all the neurons detected by Union_IG, the model forgets almost all the knowledge about these demographics. In other words, the model is hard to identify these demographics in this circumstance. For example, the model may not know what is "Black" or "White". This will severely impair the language model’s capability. 

(2) Yes, adding more bias neurons helps reduce social biases, but at the cost of sacrificing the capabilities of language models. 
Therefore, we need to strike a balance between reducing social bias and maintaining the capabilities of language models.

(3) The result is not very sensitive to $\sigma$. We report the ablation studies on BERT in the following table.

| \sigma       | SS->50.00 | LMS (↑) | ICAT (↑) |
|--------------|-----------|---------|----------|
| 0.1*maximum  | 52.13     | 85.01   | 81.39    |
| 0.2*maximum  | 52.78     | 86.64   | 81.82    |
| 0.3*maximum  | 52.99     | 86.77   | 81.58    |
| 0.4*maximum  | 53.15     | 86.82   | 81.35    |

Reviewer comment (Experiments):
What if you replace Union_IG to Intersection_IG, i.e., the intersection of bias neurons for different attributes?

Author response:
**Q5: Replace Union_IG to Intersection_IG.**

**A5**: 
We follow your suggestion and conduct additional experiments with Intersection_IG. We first study the impact of amplifying or suppressing the neurons detected by Intersection_IG on the logits gap (an experiment similar to that in Figure 2). From the table below, we find that neurons detected by Intersection_IG do not have direct relations with the manifestation of social biases. For example,  both amplifying and suppressing neurons lead to a decreased ratio in the logits gap of "Physical Appearance-P".

| JUT                   | Suppression | Amplification |
|-----------------------|-------------|---------------|
| Gender-N              | 19.62       | -6.07         |
| Gender-Ner            | 4.13        | -0.26         |
| Gender-Nest           | -1.35       | -3.82         |
| Gender-P              | 0.06        | 7.49          |
| Gender-Per            | -10.45      | -16.10        |
| Gender-Pest           | -0.96       | -0.08         |
| Physical Appearance-N | -5.99       | 0.12          |
| Physical Appearance-Ner | 8.97      | 10.41         |
| Physical Appearance-Nest | -0.93    | -0.48         |
| Physical Appearance-P | -13.74      | -20.19        |
| Physical Appearance-Per | 15.52     | 8.83          |
| Physical Appearance-Pest | -6.08    | -0.49         |

We also present the results on Stereoset as below. We can see that after suppressing neurons detected by Intersection_IG, the SS score gets even worse. This indicates that social biases have not only failed to be mitigated, but it has also become more severe.

| Model                         | SS->50.00 | LMS (↑) | ICAT (↑) |
|-------------------------------|-----------|---------|----------|
| BERT-Base-cased               | 56.93     | 87.29   | 75.19    |
| BERT-Base-cased + Intersection_IG | 63.88 | 75.39   | 54.46    |