Paper: Debiasing Algorithm through Model Adaptation (ICLR-2024)

Reviewer comment (Writing):
Certain concepts are mentioned in the paper but are either inconsistent or not explicitly defined. For example: Stereotypical keys vs. stereotyped keys; Gender value vs. gendered value; Also, what is grammatical gender?

Author response:
We clarified in section 4 that:
- “stereotypical keys” and “stereotyped keys” are the same entity. We use just the former name.
- “gender values” and. “gendered values” are the same entity. The latter name is now used exclusively.
- “Grammatical gender”  is the quality manifested by the specific inflection of words. We now use a descriptive definition of grammatical gender to avoid confusion with gender scores.

Reviewer comment (Writing):
About Equation 2, there are several questions: What is z? Why do we need an $\ell_2$ item on it?

Author response:
$z$ is the latent vector, specifically the output of the lth feed-forward layer, as denoted in the subscript  $FF_{out,l}=z$. We used it just as a loss argument to find the gendered value vector $v_o$
 We added $\text{argmin}$ notation in Equation 2 to clarify that. The missing parenthesis was also fixed.

Reviewer comment (Writing):
About Equation 2, there is a formatting issue: Missing a “)” somewhere in equation 2.

Author response:
$z$ is the latent vector, specifically the output of the lth feed-forward layer, as denoted in the subscript  $FF_{out,l}=z$. We used it just as a loss argument to find the gendered value vector $v_o$
 We added $\text{argmin}$ notation in Equation 2 to clarify that. The missing parenthesis was also fixed.

Reviewer comment (Writing):
About Equation 2, what is $P(o’|X’)$? The first two items in equations look similar to variational inference. In that case, $P(o’|X’)$ should be something similar to a prior distribution. However, I could not find its definition.

Author response:
In equation 2, we consider the model's output probability $P(o'|X')$  conditioned on a known prompt ( $X'$). The formula resembles variational inference, yet there is a crucial difference: $X'$ is a known prompt, not a random variable. As a result, a prior distribution cannot be defined here except for a trivial one.

Reviewer comment (Evaluation):
What is gender values metrics? What is the relation between these V metrics and U?

Author response:
We just want to clarify that $U$ and $V$ are matrices of concatenated stereotypical key vectors and gendered value vectors as described in the article (those are not metrics). We added further explanation regarding equation 5 in the revised paper.

Reviewer comment (Writing):
Why is $P$ in equation 5 defined in that way, and how should we use it? The explanation might be in the paragraph right after equation 5, but it is still unclear.

Author response:
We just want to clarify that $U$ and $V$ are matrices of concatenated stereotypical key vectors and gendered value vectors as described in the article (those are not metrics). We added further explanation regarding equation 5 in the revised paper.

Reviewer comment (Evaluation):
Figure 3 (b) seems to indicate that to reduce the bias, the cost is to get a much worse language model (based on the perplexity score), which is not exactly claimed in the paper.

Author response:
Figure 3 (b) illustrates ablation experiments with varying intervention layer numbers. The significant drop in perplexity occurs notably for 10 or 11 layers. It's crucial to note that, as stated in the beginning of section 4.4: "We apply DAMA to MLPs in approximately one-third of the model’s upper layers (in LLaMA 7B layers 21 - 29 out of 32),” equating to 9 layers. The corresponding perplexity is denoted with a star on the figure, which is close to the score of the original model.

Reviewer comment (Novelty):
One concern is about the technical novelty. Specifically, the causal tracing idea is from prior work, and the linear projection seems to be a straightforward idea in debiasing literature, so it is unclear what the technical novelty of this work is.

Author response:
Previous model editing methods targeted highly specific information encoded in the model ( Mitchell et al., 2022; De Cao et al., 2021), while we introduced a method capable of editing general dataset artifacts, such as various manifestations of gender bias. In Table 2, we show that DAMA significantly outperforms MEMIT (Meng et al., 2022 ) in this aspect.

In contrast to earlier projection-based debiasing methods that focused on learning and projecting latent representations for specific gender-related tasks (e.g., gender classification in bios Ravfogel et al. 2020), our method demonstrates that projection can be achieved without relying on auxiliary tasks. Notably, the projections in our approach are directly applied to the model component, avoiding alterations to the architecture or parameter size. Our theoretical and empirical evidence further establishes that applying debiasing projections on MLPs effectively reduces bias while preserving other encoded information, showcasing the generalizability of the method across tasks.

Reviewer comment (Experiments):
Seems like there is still quite some room for improvements in the debiasing methods on all three evaluations proposed in the paper. For WinoBias and SteroSet gender, I interpret the results as being still pretty gender biased even after applying DAMA. And for Bias in LM, for larger models, the a_s and a_f are still far from complete removal. It would help greatly to provide other gender bias removal methods as baselines to better assess how well DAMA did.

Author response:
> Seems like there is still quite some room for improvements

We agree that the method can be further improved for larger scales. In our opinion, the crucial strength of DAMA is its versatility in reducing bias for professions and tasks unseen in the adaptation. This contrasts with many past works that did not show consistent patterns across tasks (Delobelle et al. 2022, van der Wal et al. 2023). Admittedly,  bias annotation is oftentimes not consistent. Thus, some manifestation of it may always be present in the model when not specifically targeted.

Reviewer comment (Evaluation):
I think the study on the effects of downstream is great. However, its negative effect on reasoning tasks such as ARC can be concerning for a practical user. Is there any hypothesis on why it doesn't fare well with reasoning type of tasks?

Author response:
> Is there any hypothesis on why it doesn't fare well with the reasoning type of tasks? ARC

Please note that the difference between model instances in Table 2 is only barely statistically significant.
We compared the errors on the ARC-C task made by the original 65B model and the same model with DAMA and did not observe any clear patterns. On average, the original model selects the correct answer for 40% of DAMA’s errors, while DAMA picks the correct answer for 33% of the original model’s errors. Considering the probability of randomly choosing the correct answer is 33%, it suggests that the performance difference may be attributed to a random factor.

Reviewer comment (Writing):
You are missing a right parenthesis on equation (2) for the KL divergence.

Author response:
> You are missing a right parenthesis on equation (2) for the KL divergence.

The equation was fixed in the updated version of the paper.