S tereo D etect: Detecting Stereotypes and Anti-stereotypes the Correct Way Using Social Psychological Underpinnings (2025)

Abstract: Stereotypes are known to have very harmful effects, making their detection critically important. However, current research predominantly focuses on detecting and evaluating stereotypical biases, leaving the study of stereotypes in its early stages. Our study revealed that many works have failed to clearly distinguish between stereotypes and stereotypical biases, which has significantly slowed progress in advancing research in this area. Stereotype and Anti-stereotype detection is a problem that requires social knowledge; hence, it is one of the most difficult areas in Responsible AI. This work investigates this task, where we propose a five-tuple definition and provide precise terminologies disentangling stereotypes, anti‐stereotypes, stereotypical bias, and general bias. We provide a conceptual framework grounded in social psychology for reliable detection. We identify key shortcomings in existing benchmarks for this task of stereotype and anti-stereotype detection. To address these gaps, we developed *StereoDetect*, a well curated, definition‐aligned benchmark dataset designed for this task. We show that language models with fewer than 10 billion parameters frequently misclassify anti‐stereotypes and fail to recognize neutral overgeneralizations. We demonstrate StereoDetect’s effectiveness through multiple qualitative and quantitative comparisons with existing benchmarks and models fine-tuned on them.

## introduction large language models (llms) have rapidly advanced due to their increasing parameter sizes and vast, diverse training datasets, enabling unprecedented performance across numerous natural lan-guage processing tasks. llms trained on vast amounts of web-crawled data have been found to encode and perpetuate harmful associations prevalent in the training data @xcite . ## related work stereotyping has been foundationally explored through the princeton trilogy, which documented stable patterns of trait attributions across ten ethnic and national groups over nearly seven decades @xcite @xcite , with its replication done by @xcite . building on this descriptive tradition, the stereotype content model introduced two core dimensions as warmth and competence that together predict distinct emotional responses toward social groups @xcite . subsequent multidimensional frameworks have refined the understanding of stereotype structure and function. the dual perspective model demonstrated that self-evaluators prioritize agency (socioeconomic success) while observers prioritize communion (warmth) in social judgments @xcite , and the behavioral regulation (group virtue) model identified morality as the dominant dimension driving in-group pride and norm adherence beyond competence and sociability @xcite . more recently, the agency-beliefs-communion (abc) model revealed that agency and beliefs (conservative or progressive) are the main dimensions, and communion emerges from them @xcite , and the dimensional compensation model showed how perceivers strategically balance warmth and competence judgments across targets to maintain coherent comparative structures @xcite . most bias research in nlp began with word embeddings, where @xcite and @xcite first demonstrated bias in embeddings. bias evaluation benchmarks for llms such as stereoset @xcite and crows-pairs @xcite , together with specialized coreference datasets like winobias @xcite , winoqueer @xcite , and the multilingual shades dataset, have collectively enabled more culturally nuanced bias assessments. @xcite details the gaps and pitfalls in benchmarks like stereoset @xcite and crows-pairs @xcite . focusing on stereotypes, @xcite and @xcite computationally modeled the stereotype content model in text, @xcite introduced seegull, a stereotype dataset for nationality domain. recent efforts such as mgsd @xcite , emgsd @xcite are notable towards stereotype detection but our study has revealed many limitations and pitfalls in them (see section 5). as highlighted by @xcite , while social psychology stresses the critical role of stereotypes in shaping societal perceptions and behaviors, their systematic study within nlp remains limited. this emphasizes the urgent need for a well-curated dataset for stereotype and anti-stereotype detection with clear distinctions between stereotypes, antistereotypes and biases bridging insights from social sciences with computational methodologies. our work constructs such a dataset named stereodetect and proposes five-tuple definition for stereotype and anti-stereotype & describes a conceptual framework for the detection task using socialpsychological insights. ## five-tuple representation of stereotypes and anti-stereotypes background from social psychology about these concepts is given in appendix a. stereotypes and anti-stereotypes span multiple dimensions, including body image, technical competence, physical ability, behavioral traits, economic status, eating preferences, and more. therefore, it is essential to model them efficiently and systematically. to this end, we propose the five-tuple definition as @xcite , with u.s. context and a time interval of 2020-21. target relation attribute russians are sweet and shy muslims are peaceful and respect women iranian people have friends from other countries bengalis are not artistic at all ## conceptual framework for stereotype and anti-stereotype detection in this section we describe a conceptual framework grounded in principles of social psychology for reliable detection: eagly (1987)'s social role theory, which posits that stereotypes emerge from the social roles typically occupied by groups; and the stereotype content model proposed by @xcite , which explains how perceptions of warmth and competence in society drive stereotype formation. both theories support our central argument that stereotypes are embedded in social knowledge, not just linguistic patterns. hence, without social knowledge, stereotypes and anti-stereotypes cannot be detected. our framework (figure 1 ) first applies a neutral detector to determine whether the sentence is neutral. if the sentence is not neutral, a target detector identifies the primary social target group. when background social knowledge of that group is available (either in training data or retrieved via a retrieval-augmented mechanism), the sentence is forwarded to the classifier; otherwise, it abstains as without social knowledge stereotypes and antistereotypes cannot be determined. this illustrates why stereotype and anti-stereotype detection, while straightforward for humans, remains a challenging task for machine learning models, as it demands social knowledge. the framework prescribes three core guidelines: (1) accurate identification of the target group affected by a stereotype; (2) comprehensive, well-curated training data covering diverse groups and neutral instances; and (3) verification of the model's understanding of societal perceptions before issuing predictions. it encourages innovations such as an agentic architecture supported by robust models and rigorously curated datasets for each component, with retrieval-augmented generation (rag) employed as needed. the proposed framework has broad practical applicability, including analysis of social media content (e.g., tweets), online articles, and other text corpora. in this work, we concentrate on the creation of stereodetect, a well-curated, definition-aligned dataset designed to support the development of robust stereotype and anti-stereotype detection models. ## need for a new dataset the need for a new dataset stems from limitations and pitfalls in current datasets for stereotype and anti-stereotype detection task, as outlined below: ## construction of the stereodetect dataset the dataset construction process is detailed in the following subsections: ## quantitative analysis with existing stereotype detection models we used our best performing fine-tuned model google/gemma-2-9b and compared the results with huggingface model wu981526092/ sentence-level-stereotype-detector released by zekun et al. @xcite and huggingface model holistic-ai/bias_classifier_albertv2 by @xcite . the mgsd dataset uses the labels stereotype, anti-stereotype, and unrelated. as shown in we also aggregated our own labels (stereotype, anti-stereotype, bias) into bias, and merged neutral w/o target and neutral w/ target into neutral to enable fair comparison. the emgsd dataset uses stereotype, unrelated, and neutral labels but suffers from systematic mislabeling: many anti-stereotypical instances (e.g., "most bartenders are disconnected from their customers") are annotated as neutral (see ## qualitative comparison with baseline models for the model fine-tuned on the mgsd dataset, we mapped the original labels in our model (stere-odetect-fine-tuned model) to anti-stereotype, stereotype, and other so that comparison can be done as the mgsd-fine-tuned model had labels anti-stereotype_category, stereotype_category and unrelated respectively). for the model fine-tuned on the emgsd dataset, we mapped our original labels in our dataset to stereotype and non-stereotype (stereotype was kept as stereotype whereas other labels were mapped to non-stereotype), to compare as the emgsd-fine-tuned model had labels as stereotype and anti-stereotype. the qualitative comparison in ## conclusion and future work in this paper, we introduced a five-tuple formalization of stereotypes and anti-stereotypes. we presented a conceptual framework grounded in social-psychological theories underscoring the inherent complexity of reliable detection. we identified key shortcomings in existing benchmarks for this task of stereotype and anti-stereotype detection. to address these gaps, we developed stere-odetect, a well curated, definition-aligned, dualutility dataset. we demonstrated that prompting sub-10b models and gpt-4o frequently misclassify anti-stereotypes as stereotypes and neutral statements showing bias in models. quantitative and qualitative comparisons with existing models confirmed the effectiveness of stereodetect evident from the superior generalization capability of the stereodetect-fine-tuned model and emphasized the critical importance of definition-aligned, high-quality datasets like stereodetect for building robust stereotype and anti-stereotype detection models. future research directions include exploring the integration of agentic and rag-based approaches for conceptual framework shown in figure 1 (section 4), developing knowledge-graph methods to capture the temporal dynamics of stereotypes across social groups, and conducting empirical studies to quantify the impact of stereotype detection on overall bias-detection accuracy.