Paper: Are Models Biased on Text without Gender-related Language? (ICLR-2024)

Reviewer comment (Experiments):
While there is discussion on model size and deduplication, it may be worth seeing how pretraining dataset statistics with respect to # female nouns with # male nouns affect fairness metrics (the hypothesis being models trained on more male nouns will have less differences in fairness as gender correlation in evaluation datasets differs).

Author response:
- [how changes in pretraining dataset statistics affect fairness metrics]
   - Thank you for highlighting this critical aspect. We concur with the significance of investigating the effects of varying pretraining data statistics on models' behavior and fairness metrics. However, given their size, calculating the statistics of different pretraining data corpus may not be straightforward (e.g., computationally expensive, data unavailability). 
   - If we understand correctly, you are referring to the fact that duplicates in the training data may lead to more biases concerning specific gendered nouns. This is a fair point, which is the reason behind our explorations on the impact of data deduplication on Pythia models.  The analysis carried out in the paper (Table 2) shows mixed results between deduplicated data (worsening fairness measurements for Pythia-410M, Pythia-1.4B, and Pythia-12B but improvement for all other model sizes). Additional experiments in using a Pythia-6.9B trained on a deduplicate (D) and gender-swapped (GS) version of the training dataset (https://arxiv.org/abs/2304.01373) seem to yield different conclusions for different datasets - showing an increase in the % preferred females across Ours-5 and Ours-20 benchmarks, but worsening the % preferred males in the other 3 benchmarks. See Tables 12-15 for the results associated with Pythia-6.9 (D+GS).

Reviewer comment (Experiments):
Could the authors please elucidate on the generated dataset contents (in terms of topics of sentences generated)?

Author response:
- [generated dataset contents]
    - We appreciate your question about the content of our generated dataset, particularly regarding the topics of the generated sentences. In designing our dataset, we refrained from constraining our pipeline to specific topics and instead used different words and sentence lengths. We do this intentionally to encourage diversity in generated content. Regarding examples and topics of sentences generated, we refer the reviewer to the General Comments [Finding patterns on LLMs across benchmarks].

Reviewer comment (Evaluation):
My main concern with the analysis relates to the little correlation that exists between intrinsic measures of bias and bias in downstream tasks. As such, I am not sure whether or how this property would influence the fairness or the potential harms in a downstream task.

Author response:
- Please see General comments on [Bias definition]. We’ve addressed this concern by adding a few sentences to the main paper that comment on the ongoing discussion between upstream and downstream biases.
- [How can findings be used in studying the fairness and potential harms of downstream tasks]
   - We have updated the introduction of the paper to make the contribution and potential impact of this work more obvious. We agree with the comment that it would be relevant to measure model behavior on a downstream task by studying the behavior of counterfactuals or by transforming the sentences into a gender-neutral task. Still, we argue that our paper represents a valuable contribution to the field, raising awareness about an overlooked aspect of LM bias evaluation: whether LMs are biased in scenarios where no biases are expected. In doing so, we hope to spark the community's interest in developing new metrics, exploring potentially underlying biases rooted in LMs, and understanding their potential impact on downstream tasks.
   - Regarding the brittleness of upstream bias measures, recent works (https://arxiv.org/abs/2210.04337, https://aclanthology.org/2023.acl-short.118/) have demonstrated that downstream measures are also unreliable and highly sensitive to small perturbations. These issues are an overarching issue in bias measurement, and not restricted to upstream bias. 
   - In addition, works that compare upstream and downstream biases primarily focus on the paradigm of finetuning pretrained models for a downstream task (to our knowledge). In contrast, we are interested in studying biases in models used for open-ended generation instead of a specific downstream task.

Reviewer comment (Evaluation):
I am not sure whether analyzing intrinsic measures of bias is useful or impactful. There have been several papers that show issues with intrinsic bias measures: they are not robust and there is little to no correlation with bias measured for a downstream task.

Author response:
- Regarding the brittleness of upstream bias measures, recent works (https://arxiv.org/abs/2210.04337, https://aclanthology.org/2023.acl-short.118/) have demonstrated that downstream measures are also unreliable and highly sensitive to small perturbations. These issues are an overarching issue in bias measurement, and not restricted to upstream bias. 
   - In addition, works that compare upstream and downstream biases primarily focus on the paradigm of finetuning pretrained models for a downstream task (to our knowledge). In contrast, we are interested in studying biases in models used for open-ended generation instead of a specific downstream task.

Reviewer comment (Evaluation):
The paper https://aclanthology.org/2021.acl-long.150.pdf shows that intrinsic bias measures do not correlate with bias measured at the NLP task level.

Author response:
- Regarding the brittleness of upstream bias measures, recent works (https://arxiv.org/abs/2210.04337, https://aclanthology.org/2023.acl-short.118/) have demonstrated that downstream measures are also unreliable and highly sensitive to small perturbations. These issues are an overarching issue in bias measurement, and not restricted to upstream bias. 
   - In addition, works that compare upstream and downstream biases primarily focus on the paradigm of finetuning pretrained models for a downstream task (to our knowledge). In contrast, we are interested in studying biases in models used for open-ended generation instead of a specific downstream task.

Reviewer comment (Evaluation):
It is not clear how the findings presented in the paper could be used in studying the fairness and potential harms of a downstream task.

Author response:
- [How can findings be used in studying the fairness and potential harms of downstream tasks]
   - We have updated the introduction of the paper to make the contribution and potential impact of this work more obvious. We agree with the comment that it would be relevant to measure model behavior on a downstream task by studying the behavior of counterfactuals or by transforming the sentences into a gender-neutral task. Still, we argue that our paper represents a valuable contribution to the field, raising awareness about an overlooked aspect of LM bias evaluation: whether LMs are biased in scenarios where no biases are expected. In doing so, we hope to spark the community's interest in developing new metrics, exploring potentially underlying biases rooted in LMs, and understanding their potential impact on downstream tasks.

Reviewer comment (Novelty):
It will be great if the authors provide more details about why we need to care about this neutral setting.

Author response:
- [why we need to care about this neutral setting]  
   - Please see General comments on [Relevance of studying non-stereotypical setting].

Reviewer comment (Evaluation):
A deeper understanding of under which case models prefer certain gender will also make the paper stronger.

Author response:
- [deeper understanding of under which case models prefer certain gender]  
   - This is a relevant point! We find that sentences cover diverse topics and do not contain any apparent implicit biases, which is desired for our benchmark. Please see General comments [Finding patterns on LLMs preferences across benchmarks] for examples and details on the performed analysis.

Reviewer comment (Evaluation):
In the results, do you see under which cases/scenarios, a certain gender is preferred by a model?

Author response:
- [deeper understanding of under which case models prefer certain gender]  
   - This is a relevant point! We find that sentences cover diverse topics and do not contain any apparent implicit biases, which is desired for our benchmark. Please see General comments [Finding patterns on LLMs preferences across benchmarks] for examples and details on the performed analysis.

Reviewer comment (Experiments):
For all the models evaluated in this paper, are they all trained on PILE dataset? If not, does the PMI result hold for all of them?

Author response:
- [Models training set and transferability of results]  
   - Not all models in our study have been fully trained on the PILE dataset such as MPT, and LLAMA-2. The Pythia models were entirely trained on the PILE, while the OPT models were partially trained on PILE. While we acknowledge the ideal scenario of correlating models' behavior with their specific pretraining data statistics, the pretraining data of the models are not publicly available.  Given our simple pipeline, we can easily reproduce our dataset if and when their pretraining data becomes available.  
We assert that our analysis retains its soundness for the following reasons:  
        - Representation of Pretraining Data: Given the size and diversity of the Pile dataset, we believe it is a fair representation of the general pretraining data used for language models.  
        - Transferability of Traits: Recent research indicates that certain characteristics of language models are transferable across different models because of their shared pretraining data. This is evidenced by works such as Zou et al.'s 'Universal and transferable adversarial attacks on aligned language models' (arXiv:2307.15043, 2023) and Jones et al.'s 'Automatically Auditing Large Language Models via Discrete Optimization' (arXiv:2303.04381, 2023).  
    - We hope for increased transparency in releasing training data for language models, as this would significantly enhance the robustness and relevance of analyses like ours.

Reviewer comment (Experiments):
I am not very convinced with the correctness of generated sentences. As authors themselves mention in the limitations, the generation doesn't involve a human in the loop. Additionally, the generation is limited to a single model (ChatGPT) and having diversity in the models for a model-based benchmark construction would be a better and more fair way to go about it.

Author response:
- The correctness of generated sentences
    - [Design of pipeline]: Given the strong capabilities of ChatGPT at instruction-following, we carefully constructed a generation pipeline to increase diversity and produce gender bias-free outputs: (1) we use words that correlate equally with both gendered pronouns to bootstrap our benchmark, (2) we construct prompts emphasizing gender-neutral generations, (3) we create Ours-5 using fewer words per sentence to minimize cofounders (median sentence length of Ours-5 is 6), (4) we restrict evaluation to the subset of examples satisfying a $\textrm{MaxPMI}(s) \leq 0.65 $. Additionally, in each step, there were several iterations to improve our pipeline, and we also performed thorough inspections along the way to consider properties such as gender neutrality, grammaticality, and naturalness in generations.
   - [Small-scale study]: We performed a small-scale study evaluating the neutrality of our benchmark to address your comment. We recruited 6 participants (CS researchers) and asked them to assess whether 100 random examples of Ours-5 are neutral/unbiased. We find that, on average, 98% of the examples are deemed neutral by at least 5 out of 6 annotators.
   - [Limitations of human-in-the-loop benchmark construction]: Prior work has incorporated humans in the loop by either (1) manually generating examples or (2) resorting to crowdsourcing platforms (StereoSet, CrowS-Pairs) to create stereotypical examples. However, such works are often either small-scale, lack diversity and naturalness, or they incur high annotation costs to obtain high-quality annotations. Despite human intervention, these still exhibit clear limitations, as emphasized in https://aclanthology.org/2021.acl-long.81.
- The limitation of only using ChatGPT
We acknowledge the limitations of using a single model. Nonetheless, ChatGPT remains one of the most capable language models, and find that it generates sentences that adhere to our desired criteria. It has also been used to construct other bias benchmarks (https://arxiv.org/pdf/2302.07371.pdf). Furthermore, we include several methods to increase the diversity of the sentences in our benchmark:  (1) Recent work (https://arxiv.org/abs/2306.15895) suggests that creating sentences specifying various generation aspects (e.g., length, topic) leads to higher diversity than asking for simple class-conditional prompts. We believe that asking ChatGPT to generate test sentence pairs using different sentence lengths and different attribute-pronoun pairs contributes to the higher diversity of the generation.  (2) We also point out the use of different regeneration prompts as a failback mechanism, which should further contribute to greater diversity in the ChatGPT output.

Reviewer comment (Evaluation):
What are some possible reasons of models behaving in this manner (spurious correlations)?

Author response:
- [Potential explanations for LM preferences] 
   - Our pipeline accounts for minimizing the spurious correlations; we restrict the gender co-occurring words and filter the sentences containing strongly gender-associated words. Therefore, we believe that spurious correlations are unlikely to be the reason for such behavior. As mentioned in the general comments, we performed some clustering analysis, and this question proved to be non-trivial and challenging. The fact that the models are showing notions of gender bias even on our carefully controlled non-stereotypical benchmark, is the main point of this paper. We believe that our work has laid the foundation for examining why models manifest biases in non-stereotypical settings, and that future work builds on our findings.