The Factuality Tax of Diversity-Intervened Text-to-Image Generation: Benchmark and Fact-Augmented Intervention (2024)

Abstract: Prompt-based “diversity interventions” are commonly adopted to improve the diversity of Text-to-Image (T2I) models depicting individuals with various racial or gender traits. However, will this strategy result in nonfactual demographic distribution, especially when generating real historical figures? In this work, we propose **DemOgraphic FActualIty Representation (DoFaiR)**, a benchmark to systematically quantify the trade-off between using diversity interventions and preserving demographic factuality in T2I models. DoFaiR consists of 756 meticulously fact-checked test instances to reveal the factuality tax of various diversity prompts through an automated evidence-supported evaluation pipeline. Experiments on DoFaiR unveil that diversity-oriented instructions increase the number of different gender and racial groups in DALLE-3’s generations at the cost of historically inaccurate demographic distributions. To resolve this issue, we propose **Fact-Augmented Intervention** (FAI), which instructs a Large Language Model (LLM) to reflect on verbalized or retrieved factual information about gender and racial compositions of generation subjects in history, and incorporate it into the generation context of T2I models. By orienting model generations using the reflected historical truths, FAI significantly improves the demographic factuality under diversity interventions while preserving diversity.

## introduction a large body of previous works has explored social biases in text-to-image (t2i) models-for instance, models could follow social stereotypes and tend to generate male "doctors" and female "nurses" @xcite @xcite @xcite . to resolve this issue, several studies propose prompt-based "diversity interventions" that effectively instruct t2i models to generate images with gender and racial diversity @xcite , 2022a; @xcite @xcite . however, users have recently reported when real-world t2i systems are requested to generate historical figures, diversity interventions alter facts, leading to images that are wrong or even offensive foot_0 . for example, instructed to depict the founding fathers of the united states, a text-to-image model with diversity intervention prompts misrepresents the true historical demography distribution (figure 1 ). motivated by this dilemma, this paper studies a critical question: ## the dofair benchmark we propose the first-of-its-kind demographic fac-tuality representation (dofair) benchmark to measure the critical trade-off between demographic diversity and factuality in t2i model generations. dofair consists of 756 major participant classes involved in real historical events, as well as the corresponding demographic distribution for each ## the factuality tax of diversity interventions using the dofair benchmark, we conducted experiments to evaluate the trade-off between demographic diversity and factuality in t2i models. ## fact-augmented interventions the above experiment results demonstrate t2i models' lack of ability to understand and depict factual demographic distributions among historical figures in images, especially under diversity interventions. to resolve this issue, we first explore if the chainof-thought (cot) reasoning @xcite from a llm helps improve the demographic factuality in t2i generations: "think step by step." experiment results in the "+cot" rows in the second column of images in figure 7 shows a qualitative example, in which cot fails to improve gender factuality. we highlight that the root of cot's failure is due to the lack of factuality orientation in its reasoning direction: even if the reasoning steps specifically identified that only males were involved in the event, the cot model could be easily affected by the diversity intervention and begin to plan out ways to falsely modify this historical fact. ## conclusion in this work, we developed the dofair benchmark to reveal the factuality tax of applying diversity interventions to t2i models. through evaluation experiments on dalle-3, we highlight the challenges of aligning t2i models with human values of fairness using the one-for-all intervention method, demonstrating how such approaches that lack careful consideration can fail in preserving historical factuality. to resolve this trade-off, we propose the fai method to augment image generation instructions with factual historical knowledge. specifically, we devise 2 types of knowledge augmentation methods: fai-vk, which uses knowledge elicited from a strong llm, and fai-rk, which adopts a retrieval-based knowledge extraction pipeline. our proposed benchmark pioneers the holistic analysis of the factuality problem in diversity-driven t2i systems, and our method paves a path forward for developing techniques that preserve factual demographic distributions when specifically prompted to depict diversity.