Do *they* mean ‘us’? Interpreting Referring Expression variation under Intergroup Bias (2024)

Abstract: The variations between in-group and out-group speech (intergroup bias) are subtle and could underlie many social phenomena like stereotype perpetuation and implicit bias. In this paper, we model intergroup bias as a tagging task on English sports comments from forums dedicated to fandom for NFL teams. We curate a dataset of over 6 million game-time comments from opposing perspectives (the teams in the game), each comment grounded in a non-linguistic description of the events that precipitated these comments (live win probabilities for each team). Expert and crowd annotations justify modeling the bias through tagging of implicit and explicit referring expressions and reveal the rich, contextual understanding of language and the world required for this task. For large-scale analysis of intergroup variation, we use LLMs for automated tagging, and discover that LLMs occasionally perform better when prompted with linguistic descriptions of the win probability at the time of the comment, rather than numerical probability. Further, large-scale tagging of comments using LLMs uncovers linear variations in the form of referent across win probabilities that distinguish in-group and out-group utterances.

## introduction social bias in language is generally studied by identifying undesirable language use towards a specific demographic group @xcite @xcite @xcite ; however, we can enrich our understanding of bias in communication by understanding it as differences in behavior situated in social relationships. intergroup bias is the social bias stemming from the intergroup relationship between the speaker and target reference of an utterance. @xcite . @xcite modeled intergroup relationships (in-group and out-25% : 75% @xcite calling offensive plays like @xcite 're fucking scared. got some pressure on mahomes finally 45% : 55% our pass blocking has been very good so far. figure 1 : we construct a parallel language corpus of comments from nfl team subreddits, grounding each comment in the live win probabilities. we then tag relevant entities in each comment with intergroup tags using annotators and train llms to tag intergroup references. how does language systematically change with respect to state-of-the-world when referring to an individual in one's in-group versus out-group? in this work, we take major steps towards answering this question, using a task architecture in the classical nlp pipeline @xcite . earlier work in the linguistic intergroup bias (lib) hypothesis @xcite focused exclusively on the predicate, and the bias was described using an ad-hoc lexical categorization system @xcite ) over predicates. however, the form of referencing the in-group or out-group can reveal subtle biases as well. consider the comments in figure 1 , sampled from two fan forums discussing a live nfl game and part of the dataset described in this paper. commenters refer to their in-group (team they support) and out-group (opponent team in that game) by name, sub-groups, pronouns as well as implicitly -sometimes they choose not to refer to either group at all. by tagging these references with appropriate labels which denote the relationship of the referent to the speaker, we can ask: how does the intergroup bias manifest in referent forms? to answer this question, we introduce a new dataset of interpersonal language -comments from game threads on online forums dedicated to fandoms for teams in the national football league (nfl). through careful data curation, we construct a parallel corpus of sports comments, with comments from fans of both teams in a game, aligned in time and grounded in win probabilities (wp). by focusing on referring expressions, we can formulate investigating the intergroup bias as a tagging task: given a comment, the group affiliation of the writer, and the state-of-the-world, return a tagged comment with appropriate referring expressions tagged as @xcite , @xcite or @xcite (see figure 1 ). annotation and preliminary analysis reveal that the form of the referent that speakers use when referring may have systematic intergroup variations. we train large language models (llms) to automate tagging on our dataset, and examine their performance on our task. few-shot performance on gpt-4o improves when using linguistic descriptions of wps; fine-tuned llama-3 models performed better, although incorporating wp had little effect. using our best performing model to tag 100,000 comments from our dataset, we discover two striking linguistic behaviors at scale: these findings add much needed color to the lib hypothesis -natural language is productive, and commenters can express their (implicit) intergroup bias in different ways. this work also lays the foundation for future explorations of other intergroup variations (in event descriptions, for example) in sports-talk and other domains. we share all our code and data online foot_0 . intergroup bias linguistic intergroup bias (lib) theory @xcite hypothesizes that stereotypes are transmitted and persist in communication through systematic linguistic asymmetry -socially desirable in-group behaviors and socially undesirable out-group behaviors are encoded at a higher level of abstraction. the lib has been reproduced in psychological experiments and analyses @xcite ; it has also been used as an indicator for a speaker's prejudicial attitudes @xcite , and racism (schnake and ruscher, 1998). govindarajan et al. (2023a,b) take inspiration from the lib at large to study intergroup bias as a general phenomenon in online language use. while they find regularities in its variation with emotion that neural models can 'learn' to identify in-group and out-group utterances more accurately than humans, probing experiments fail to describe humanobservable intergroup variations in language. this work studies a much larger dataset than in their work, and by modeling the bias as a tagging task to referents, we discover characteristic lexical variations at scale that complement lib findings. sports language language use in the domain of sports has been a rich source of analyses and studies within computational linguistics, including from the perspective of quantifying social biases. @xcite studied commentator racial biases in descriptions of football players, reaffirming previous findings illustrating clear differences in terms of sentiment descriptions (white players were more likely to be described as intelligent), and name itself (white players were more likely to be referred to by their first name). @xcite focused on one aspect of language usage among (and between) fans of nba teams: intra-group behavior with and without social contact with the out-group. they find that fans with intergroup con-tact are more likely to use negative languagethey were more polarized than before. our work differs from previous work in two major ways. firstly, we focus on the intergroup bias -how do fans talk about their team (in-group), versus the opponent (out-group)? secondly, this paper grounds the analysis of intergroup bias in numerical descriptions of the state-of-the-world. the state-of-the-world in a sports game at any moment can be described using the scoreboard, thus providing grounding for utterances follow. nonlinguistic, numerical descriptions of the events that precipitate an utterance overcome the drawbacks of using ad-hoc, proximally derived metrics like social desirability (in lib) or affect (in govindarajan et al. ( 2023b )) as an axis to study linguistic variation. as we shall describe in §3, sports games, and in particular nfl games, are rich with statistical information amenable to describe the state-of-theworld on a well-calibrated numerical scale. ## modeling intergroup bias with llms large language models (llms) have shown remarkable abilities in various domains over the last few years @xcite . our novel tagging framework to model intergroup bias requires linguistic understanding, knowledge of the nfl and its teams, as well as complex reasoning over why a commenter might choose certain word forms compatible with the state-of-the-world -making llms well suited to this task. in this section, we design modeling experiments to tag comments from our dataset with intergroup labels with 2 goals: ## conclusion we enhance our understanding of intergroup bias by building a parallel corpus of sports comments grounded in win probabilities from live games. annotation experiments show that modeling the bias as a tagging problem over referring words can reveal unobserved variations, as well as make it amenable to large-scale modeling. through fewshot and finetuning experiments, we find that llms can out-match human baseline performance at this task, but struggle to reason over win probabilities, or use it meaningfully towards tagging. tagging a large sample of our dataset reveals linear trends between various referring expressions and wp, showing that intergroup bias can manifest in commenter's choice of who to refer to when commenting on a game and how. careful data curation and understanding, combined with focused usage of llms as statistical information processing tools can thus uncover linguistic variations in social language use online at scale. in future work we plan to exploit the parallel nature of our corpus further to understand team differences in language variation, as well as how wp can be effectively incorporated into a model of social meaning.