The “r” in “woman” stands for rights. Auditing LLM s in Uncovering Social Dynamics in Implicit Misogyny (2025)

Abstract: Persistent societal biases like misogyny express themselves more often implicitly than through openly hostile language.However, previous misogyny studies have focused primarily on explicit language, overlooking these more subtle forms. We bridge this gap by examining implicit misogynistic expressions in English and Italian. First, we develop a taxonomy of social dynamics, i.e., the underlying communicative intent behind misogynistic statements in social media data. Then, we test the ability of nine LLMs to identify the social dynamics as a multi-label classification and text span selection: first LLMs must choose social dynamics given a prefixed list, then they have to explicitly identify the text spans that triggered their decisions. We also investigate the extent of using different learning settings: zero and few-shot, and prescriptive. Our analysis suggests that LLMs struggle to follow instructions and reason in all settings, mostly relying on semantic associations, recasting claims of emergent abilities.

## introduction in an era dominated by digital communication and social networks, the widespread presence of online misogyny makes online spaces unsafe, perpetuating stereotypes and social injustice. studies suggest that up to 58% of women have experienced technology-facilitated gender-based violence. 1 misogyny is not just an individual attitude or prejudice, it is embedded in social structures, cultural norms, institutions and everyday interactions that make up the social dynamics between these systems and women. in this context, social dynamics refer to the ongoing patterns of interaction, power exchange, and meaning-making that 1 https://www.unwomen.org/en/articles/faqs/dig ital-abuse-trolling-stalking-and-other-forms-o f-technology-facilitated-violence-against-women regulate how women are perceived, treated, and positioned in society. these dynamics often carry implicit communicative content. examining them can reveal how power relations between genders are maintained, how individuals and groups internalize or reproduce misogynistic ideas, how the gender roles and social expectations shape behaviors toward women, and how norms about masculinity contribute to misogynistic attitudes. misogyny and hate speech (hs) datasets more broadly have a relatively low prevalence of implicitly hateful content, due to their reliance on explicit keywords during data collection. @xcite reports statistics for seven popular hs datasets, revealing that explicit hate occurs, on average, three times more often than implicit hate, while @xcite show that in their abuseeval dataset, the percentage of explicit hate is twice that of implicit hate. specific to misogyny, @xcite show that in datasets for english, italian, and spanish, the most frequent tokens are swear words, proving their degree of explicitness. however, focusing primarily on explicit content overlooks the socially pervasive forms of implicit hate, which often evade automated detection and moderation systems yet significantly contribute to the normalization of harmful stereotypes. implicit hate operates through euphemism, presupposition, and cultural reference, and its subtlety makes it more likely to be perceived as socially acceptable, thereby reinforcing biased social dynamics towards women, under the guise of neutrality. to effectively capture misogyny, llms, like humans, must not only grasp the surface meanings of texts, but also their underlying communicative intention and social knowledge @xcite . while llms can often retrieve surfacelevel associations (first-order meaning), they lack robust metarepresentational awareness needed to infer speaker intent, power dynamics, or social positioning (second-order meaning). the sentence in fig. 1 illustrates how the joke relies on the implied misogynistic assumption that women are confined to traditional domestic roles. in this paper, we propose a novel framework informed by feminist and gender studies to annotate implicitly misogynistic social media posts based on the social dynamics they evoke. we then evaluate the capabilities of seven models in english and nine models in italian in a zero, few-shot, and prescriptive settings to identify social dynamics, along with their corresponding text spans. we find that llms struggle to correctly identify the social dynamics underlying misogynistic statements and the corresponding text span, failing to demonstrate robust reasoning abilities required to effectively interpret these complex social cues without further fine-tuning or knowledge augmentation. contributions 1) we create a literaturegrounded taxonomy of social dynamics occurring in implicit misogyny ( § 3). 2) we enrich two existing corpora, sbic @xcite and implicit-mis @xcite with annotations based on the taxonomy. 3) we audit a total of nine llms for their capabilities in identifying the implied social dynamics and their corresponding text span(s). 4) we assess the llm preference for free-text wrt category selection. ## background & related work the meaning and usage of the term misogyny have expanded far beyond the original definition; i.e. the hatred of women @xcite . in our work, we intend misogyny as a property of social environments in which women perceived as violating patriarchal norms are "kept down" through hostile or benevolent reactions from men, other women, and social structures @xcite . as any other forms of hate, misogyny can be expressed explicitly or in a more veiled manner. in the latter case fall instances which are harder to understand for humans -potentially giving rise to disagreements in the annotation phase @xcite -and harder to detect for llms. early distinctions on the degree to which hateful content is expressed considered only a binary set (explicit vs implicit), where explicitness is defined as unambiguous in its potential of being hateful @xcite . @xcite distinguish between explicit, implicit and subtle hs, grounded in 18 properties of implicitness based on linguistic features, such as sarcasm, figurative language, exaggeration and inferences. while implicit hs goes beyond the literal meaning, in subtle hs there is still a literal meaning, presented in an elusive way. we refer to implicit misogynous language when misogyny is expressed through coded or indirect language by means of linguistic devices such as irony, euphemisms, stereotypes, inferences and metaphorical or figurative language @xcite . datasets designed to address the explicitness degree of hateful messages are rare @xcite @xcite . most current work on implicit hs is based either on the re-annotation of existing datasets @xcite @xcite or on using machinegenerated examples @xcite . focusing on misogyny, the social bias infer-ence corpus (sbic) @xcite , a collection of 10k instances of biased statements against minority groups in english, contains the largest group of misogynistic messages: 3.7k. for italian, @xcite developed implicit-mis, the only dataset with 1,2k implicit misogynous instances. for a complete overview on datasets focused on misogyny see @xcite and appendix c. llms have introduced a paradigm change in nlp. their availability has been accompanied by claims about "emerging abilities" @xcite . recent work has proposed distinguishing the acquisition of competencies in llms either as abilities; i.e. the capacity to solve a task absent in smaller models as an effect of the size of the models themselves, or techniques; i.e. the beneficial effect of different prompting methods that are ineffective in smaller models @xcite . the experiments conducted by @xcite -using zero-shot settings on multiple tasks-show that an ability ## social dynamics understanding and addressing misogyny requires more than identifying overtly harmful language -it demands a deeper analysis of the social structures and assumptions that sustain gender-based inequality. this section introduces a taxonomy of social dynamics developed through the lens of feminist theory and gender studies @xcite @xcite @xcite . rather than focusing on the linguistic manifestation of misogyny, the taxonomy aims to make explicit the implicit, unspoken assumptions that underlie misogynistic behavior. this is the main motivation for referring to them as "social dynamics": they describe the manifestation of underlying interactions, attitudes, and behaviors within groups of people @xcite . the taxonomy offers a structured framework to classify these underlying dynamics, as summarized in we develop this taxonomy through a mixedmethod approach, combining bottom-up (inductive) and top-down (deductive) approaches to ensure both empirical grounding and theoretical rigor. in the bottom-up approach, social dynamics emerged from direct observation of data, following grounded theory principles @xcite to identify recurring misogynistic behaviors. for the top-down approach, we searched for feminist and gender-based literature that discusses the identified social dynamics to extract underlying explanatory concepts (column description in ## methodology in our experiments, we evaluate eight open-source decoder-based llms-each with the same number of parameters-alongside gpt-4o-mini on two tasks: (a) selecting relevant social dynamics from a predefined list of categories, and (b) identifying the specific text span that prompted the category selection. we assess model performance under zeroshot, few-shot, and prescriptive prompting conditions. for each model, we provide available details on training data and moderation mechanisms, as these factors significantly influence their behavior and results. ## conclusion we audit nine llms to evaluate their ability to identify the social dynamics encoded in implicit misogynistic messages in italian and english, and the corresponding text span. to this end, we propose a literature-grounded taxonomy of social dynamics that occur in implicit misogynous statements. overall, models struggle to complete such a task which requires complex reasoning about meaning, and rely more on surface-level semantic associations. we also compared category selection vs freetext explanations, revealing that while explanations more often capture the underlying reasoning, they lack the precision of structured labels, highlighting the importance of combining both approaches for more accurate and interpretable model evaluation. in either case, the more social knowledge is re-quired to understand the underlying meaning of the message, the likelier models fail. while this supports previous findings on recasting claims on emerging abilities of llms @xcite , it also indicates that llms have limited understanding of implied societal assumptions encoded in messages. hence, additional training/tuning or knowledgeaugmentation approaches are needed. we leave the use of external knowledge for future work.