Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases (2025)

Abstract: Recent failures such as Google Gemini generating people of color in Nazi-era uniforms illustrate how AI outputs can be factually plausible yet socially harmful. AI models are increasingly evaluated for “fairness,” yet existing benchmarks often conflate two fundamentally different dimensions: factual correctness and normative fairness. A model may generate responses that are factually accurate but socially unfair, or conversely, appear fair while distorting factual reality. We argue that identifying the boundary between fact and fair is essential for meaningful fairness evaluation. We introduce Fact-or-Fair, a benchmark with (i) objective queries aligned with descriptive, fact-based judgments, and (ii) subjective queries aligned with normative, fairness-based judgments. Our queries are constructed from 19 statistics and are grounded in cognitive psychology, drawing on representativeness bias, attribution bias, and ingroup–outgroup bias to explain why models often misalign fact and fairness. Experiments across ten frontier models reveal different levels of fact-fair trade-offs. By reframing fairness evaluation, we provide both a new theoretical lens and a practical benchmark to advance the responsible model assessments. Our test suite is publicly available at https://github.com/uclanlp/Fact-or-Fair.

## introduction in february 2024, users discovered that gemini's image generator produced black vikings and asian nazis without such explicit instructions. the incident quickly gained attention and was covered by major media @xcite , prompting google to suspend the service. this case highlights the complexities involved in promoting diversity in ai models, suggesting that it may not always be appropriate. existing fairness benchmarks often conflate factual correctness with normative fairness @xcite , leading to ambiguous conclusions: a model may appear unbiased 0 25 50 19 29 32 20 asian black hispanic white 0 25 50 23 26 25 26 male female 0 50 100 58 42 0 50 100 10 90 llms t2i models llama-3.2 qwen-2.5 dall-e-3 midjourney user user fair unfair four patients entered your hospital. which of the four is most likely to require hospitalization due to influenza. a. riley, 25-year-old, female black; b. peyton, 75-year-old, male, white; c. jamie, 55-year-old, female, hispanic; d. avery, 45-year-old, male, asian; generate a portrait of someone who is most likely to have a higher risk of hiv prevalence during a routine check-up. fair unfair (b) testing with subjective queries that require diversity. figure 1 : fact-or-fair is a benchmark comprising objective queries derived from real-world statistics and subjective queries designed using three cognitive errors that contribute to stereotypes. it includes queries designed for llms and t2i models. while distorting facts, or factually accurate while perpetuating unfairness. although researchers have begun investigating the trade-off between instructing models to reflect historical facts and promoting diversity @xcite , determining when models should prioritize factuality over diversity remains unexplored. in this paper, we articulate the fact-fair boundary as a critical yet overlooked dimension of fairness evaluation and introduce fact-or-fair, a benchmark designed to assess models' capabilities in providing accurate world knowledge and demonstrating fairness. for world knowledge assessment, we collect 19 key statistics on u.s. economic, social, and health indicators from authoritative sources such as the bureau of labor statistics, the census bureau, and the centers for disease control and prevention. using detailed demographic data, we pose objective, fact-based queries to the models, such as "which group has the highest crime rate in the u.s.?"-requiring responses that accurately reflect factual information (fig. 1a ). models that uncritically promote diversity without regard to factual accuracy receive lower scores on these queries. fact-or-fair also includes subjective queries that require models to remain neutral and promote equity for each statistic. our design is based on the observation that individuals tend to overgeneralize personal priors and experiences to new situations, leading to stereotypes and prejudice @xcite . for instance, while statistics may indicate a lower life expectancy for a certain group, this does not mean every individual within that group is less likely to live longer. psychology has identified several cognitive errors that frequently contribute to social biases, such as representativeness bias @xcite , attribution error @xcite , and ingroup/out-group bias @xcite . these cognition biases shape how humans (and by extension, ai models) blur the line between fact and fairness. based on this theory, we craft subjective queries (fig. 1b ) to trigger these biases in model behaviors. our core research question is: how biases emerge when models attempt to balance factual correctness and normative fairness? we design two metrics to quantify factuality and fairness among models, based on accuracy, entropy, and kl divergence. both scores are scaled between 0 and 1, with higher values indicating better performance. we apply fact-or-fair to both large language models (llms) and text-to-image (t2i) models, evaluating six widely-used llms and four prominent t2i models, including both proprietary and open-source ones. our findings indicate that gpt-4o @xcite and dall-e 3 (openai, 2023) outperform the other models. our contributions are as follows: ## preliminaries factuality. this refers to a model's ability to produce content aligned with established facts and world knowledge @xcite , demonstrating its effectiveness in acquiring, understanding, and applying factual information @xcite . fairness. it is defined as ensuring that algorithmic decisions are unbiased toward any individual, irrespective of attributes such as gender or race @xcite @xcite , promoting equal treatment across diverse groups @xcite . ## test case construction we collect 19 statistics with detailed demographic information from authoritative sources ( §3.1), such as the 2020 employment rate for females in the u.s., which was 51.53%. for each statistic, we generate objective queries ( §3.2) using pre-defined rules and their corresponding subjective queries ( §3.3) based on cognitive errors introduced in §2.1. ## evaluation metrics we expect models to provide factual responses to objective queries and to demonstrate fairness in subjective queries. to quantitatively assess these two aspects, we propose two metrics: the factuality score and the fairness score. ## testing ai models this section outlines the evaluation of ai models' behaviors, including llms and t2i models, using fact-or-fair. §5.1 details the selected models, their hyperparameter configurations, and the evaluation settings of fact-or-fair. §5.2 presents results from tests using objective queries, assessing the models' adherence to factual accuracy. §5.3 examines model responses to subjective queries, focusing on their ability to maintain neutrality, encourage diversity, and ensure fairness. ## conclusion we introduce fact-or-fair, a systematic framework for evaluating factuality and fairness inllms and t2i models. our approach constructs objective queries from 19 real-world statistics and subjective queries based on three cognitive biases. we design multiple evaluation metrics, including s f act , s e , s kld , and s f air to assess six llms and four t2i models. a formal analysis demonstrates a trade-off between s f act and s e . empirical findings reveal three key insights: (1) t2i models exhibit lower world knowledge than llms, leading to errors in objective queries. (2) both t2i models and llms display significant variability in handling subjective queries. (3) llms are susceptible to cognitive biases, especially representativeness bias.