Paper: Language Agents for Detecting Implicit Stereotypes in Text-to-image Models at Scale (ICLR-2024)

Reviewer comment (Experiments):
The benchmark dataset presented, may not fully capture the diversity and complexity of stereotypes present in real-world scenarios. The distribution of subgroups within the benchmark dataset is imbalanced, particularly in the race/ethnicity and religion dimensions.

Author response:
We acknowledge the difficulty in fully capturing the diversity and complexity of stereotypes in real-life scenarios. In our study, we attempted to increase the diversity of evaluated samples by drawing from toxic text content datasets on various social platforms. Regarding the imbalanced distribution of subgroups, we would like to clarify that Figure 3 illustrates the proportional distribution of instruction pairs obtained during the initial text extraction process, which is biased towards specific subgroups. It is important to note that this distribution does not represent the training data. Instead, it highlights the strong stereotypes associated with certain subgroups in everyday expressions.

Reviewer comment (Experiments):
The paper does not compare the proposed agent framework with existing methods for stereotype detection in text-to-image models.

Author response:
We compare our proposed agent framework with existing stereotype detection methods. It's important to highlight that these conventional methods rely on manually crafted prompts for comprehensive testing.

| Bias dimensions | Ours     | Cho et al. [1] | Bianchi et al. [2] | Naik et al. [3] |
| --------------- | -------- | -------------- | ------------------ | --------------- |
| Gender          | &#x2714; | &#x2714;       | &#x2714;           | &#x2714;        |
| Race            | &#x2714; | &#x2714;       | &#x2714;           | &#x2714;        |
| Religion        | &#x2714; | &#x2716;       | &#x2716;           | &#x2716;        |

| Bias subjects     | Ours     | Cho et al. [1] | Bianchi et al. [2] | Naik et al. [3] |
| ----------------- | -------- | -------------- | ------------------ | --------------- |
| Person            | &#x2714; | &#x2714;       | &#x2716;           | &#x2714;        |
| Occupacations     | &#x2714; | &#x2714;       | &#x2714;           | &#x2714;        |
| Traits            | &#x2714; | &#x2716;       | &#x2714;           | &#x2714;        |
| Situations        | &#x2714; | &#x2716;       | &#x2716;           | &#x2714;        |
| Verb-noun prompts | &#x2714; | &#x2716;       | &#x2716;           | &#x2716;        |

Reviewer comment (Reproducibility):
The paper does not provide a comprehensive justification for the selection of specific tools within the agent framework, nor does it discuss the optimization process for these tools.

Author response:
Our agent framework emphasizes the use of existing models to complete complex tasks. We add a more comprehensive justification for the selection of specific tools.

Reviewer comment (Novelty):
The paper focuses on the detection of stereotypes but does not provide explicit recommendations or strategies for mitigating these biases.

Author response:
In response to the concern about providing explicit recommendations or strategies for mitigating biases, we include a section discussing potential bias mitigation approaches and their implications in the revised paper.

Reviewer comment (Evaluation):
What are the potential limitations or challenges of using language agents for stereotype detection in text-to-image models? Are there any specific scenarios or cases where the agent may not perform as accurately?

Author response:
Language agents for stereotype detection in text-to-image models may face limitations in accurately detecting subtle stereotypes or those embedded in complex contexts. Additionally, they may struggle with cases where stereotypes are expressed using sarcasm or irony. We discuss these challenges and potential solutions in the revised paper.

Reviewer comment (Evaluation):
In the agent performance evaluation, the proportion of bias detected by the agent is compared to the manual annotation results. Can you provide more information about the criteria used for manual annotation and how the annotators determined the presence of stereotypes?

Author response:
For manual annotation, annotators were provided with guidelines that included descriptions and examples of stereotypes. They were instructed to determine the presence of stereotypes based on these guidelines. We provide more information about the criteria used for manual annotation in the revised paper.

Reviewer comment (Experiments):
How did you select the toxic text datasets used to construct the benchmark dataset? Did you consider any specific criteria or guidelines in selecting these datasets?

Author response:
The toxic text datasets used to construct the benchmark dataset were selected based on their relevance to the task, diversity of content, and representation of various social platforms. We provide more details about the selection criteria and guidelines in the revised paper.

Reviewer comment (Novelty):
The technical novelties of this paper are quite limited, as it is an automated assessment framework for stereotypes in text-to-image models. The models as well as the metrics considered are from the existing literature.

Author response:
Regarding the technical novelty of our paper, we understand your point about the use of existing models and metrics. However, our primary contribution lies not in the novelty of the models or metrics, but in the innovative application of these tools. Our core motivation is to build a language agent that can perform task planning and effectively utilize various tools to detect and identify a broader range of stereotypes present in social media texts. Therefore, this agent overcomes the limitations of previous work that required exhaustive testing of all possible occupations and other descriptors to construct prompts.

Reviewer comment (Reproducibility):
While it is included in the submission auxiliary materials, there is no mention of open sourcing the code in the paper.

Author response:
We appreciate your concerns regarding open sourcing the code of our study. While we have included the code in the submission auxiliary materials, we understand the need for it to be easily accessible. Therefore, we commit to open sourcing the code in an easy-to-access format in subsequent papers. We will also provide ample examples to illustrate the seriousness of stereotypes in existing text-to-image models, further aiding in the understanding and reproduction of our work.

Reviewer comment (Novelty):
The paper is framed around the method proposing an autonomous agent for stereotype detection. This is a great vision, but the method seems to (a) follow a consistent, pre-determined sequence of actions for the task, and (b) it seems far from autonomous in relying on human intervention and a lot of custom steps (unique datasets, custom prompts, human feedback, etc) at every stage. This to me is not a critique of the method but that it shouldn't be over-complicated or over-sold as an autonomous agent rather than a reliable process for red-teaming for stereotypes building on LLMs as a tool in that process.

Author response:
W1a: We acknowledge your concerns regarding the framing of our method as an autonomous agent and agree that it might have been overly ambitious. While you pointed out the dependence on "human intervention," it is crucial to note that, at the present stage, crafting well-written prompts manually remains a vital component in enhancing the performance of an agent. Our intention was to highlight the potential for autonomy rather than present a fully autonomous system. In light of your comments, we will revise our framing to better represent the current state of the method and potential future developments.

Reviewer comment (Experiments):
There has been a fair amount of work on stereotype detection which this work is not compared to and does not grapple with similar issues. For example, what if there are multiple people in the same image? How is the diversity and coverage of the generated concerns? When is this better than a curated list? For example, this claim is good but I'd like to see experimental evidence: "However, this approach to bias evaluation has its limitations, as it often neglects the more subtle stereotypes prevalent in everyday expressions. These biases frequently manifest in toxic content disseminated across various social platforms, including racial slurs."

Author response:
W2a: We compare our proposed agent framework with existing stereotype detection methods. It's important to highlight that these conventional methods rely on manually crafted prompts for comprehensive testing.

| Bias dimensions | Ours     | Cho et al. [1] | Bianchi et al. [2] | Naik et al. [3] |
| --------------- | -------- | -------------- | ------------------ | --------------- |
| Gender          | &#x2714; | &#x2714;       | &#x2714;           | &#x2714;        |
| Race            | &#x2714; | &#x2714;       | &#x2714;           | &#x2714;        |
| Religion        | &#x2714; | &#x2716;       | &#x2716;           | &#x2716;        |

| Bias subjects     | Ours     | Cho et al. [1] | Bianchi et al. [2] | Naik et al. [3] |
| ----------------- | -------- | -------------- | ------------------ | --------------- |
| Person            | &#x2714; | &#x2714;       | &#x2716;           | &#x2714;        |
| Occupacations     | &#x2714; | &#x2714;       | &#x2714;           | &#x2714;        |
| Traits            | &#x2714; | &#x2716;       | &#x2714;           | &#x2714;        |
| Situations        | &#x2714; | &#x2716;       | &#x2716;           | &#x2714;        |
| Verb-noun prompts | &#x2714; | &#x2716;       | &#x2716;           | &#x2716;        |

Reviewer comment (Evaluation):
Related to the above point, it is hard to gauge how diverse the stereotypes uncovered are and how diverse the images are. (Are all images generated with "The people who ___")

Author response:
In response to your questions, we would like to provide the following information:
- Diversity of stereotypes: Our experiments uncovered more than 2800 prompts that simultaneously showed stereotypes pointing to one subgroup across multiple models. These prompts cover the special expression of "verb + noun" that was missing in previous studies, such as "commit sadistic murders" and so on.

Regarding the diversity of images, we would like to clarify that all prompts start with a fixed "The people who ___" to avoid the prefix prompt introducing additional stereotypes. This approach ensures that the uncovered stereotypes are solely attributed to the models and not influenced by the prompts themselves.