How people talk about each other: Modeling Generalized Intergroup Bias and Emotion (2023)

Abstract: Current studies of bias in NLP rely mainly on identifying (unwanted or negative) bias towards a specific demographic group. While this has led to progress recognizing and mitigating negative bias, and having a clear notion of the targeted group is necessary, it is not always practical. In this work we extrapolate to a broader notion of bias, rooted in social science and psychology literature. We move towards predicting interpersonal group relationship (IGR) - modeling the relationship between the speaker and the target in an utterance - using fine-grained interpersonal emotions as an anchor. We build and release a dataset of English tweets by US Congress members annotated for interpersonal emotion - the first of its kind, and ‘found supervision’ for IGR labels; our analyses show that subtle emotional signals are indicative of different biases. While humans can perform better than chance at identifying IGR given an utterance, we show that neural models perform much better; furthermore, a shared encoding between IGR and interpersonal perceived emotion enabled performance gains in both tasks.

## introduction currently, most work studying bias in nlp situates bias as negative or pejorative language use towards an individual or group based on traits like race, gender, etc @xcite @xcite @xcite . while these approaches greatly advance our understanding of bias in language and its impact and mitigation in nlp, focusing on specific demographic dimensions or an individual's intent is limiting and not always practical. research in psychology and social science suggests a different perspective. bias can be seen as a relationship between people and groups, situated in context @xcite ; as such, bias refers to differences in behavior (in this case language use) as a result of differences in the relationship between speaker and target. the language we produce is biased in one way or another, whether we intend to or not, and whether that bias is positive, negative, or not clearly associated with any valuation @xcite . in psychological work on linguistic intergroup bias @xcite , bias originates from the relationship between the speaker and target of an utterance, i.e. their interpersonal dynamics, and manifests later in subtle ways. consider the utterances (tweets) in (1), drawn from our collected data in which the identity of the speaker and target are masked: both express support and admiration towards the target referent doe -however, the second example uses words indicative that the speaker and target do not share a relevant social identity (in this case, their political party), expressed by words like bipartisan. the intensity of admiration expressed is also greater in (1-a) than (1-b). thus, these two seemingly similar statements differ along interpersonal dimensions that are instructive as to how the bias of the speaker seeps into the utterance. we now introduce two new tasks that directly model language use in terms of two interpersonal dimensions: (i) interpersonal group relationship (igr) prediction, where we seek to understand how people talk about others who they consider to be in their same social group (in-group), versus those they consider outside their social group (outgroup), and (ii) perceived interpersonal emotion detection, where we situate these differences in terms of the emotion expressed in text towards or in connection with a target individual described in the utterance. note that interpersonal emotion is different from a more standard, utterance level emotion detection task, as illustrated in row 2 of we present a first-of-its-kind, annotated dataset for fine-grained interpersonal emotion detection, consisting of 3,033 tweets from members of the us congress; all of these tweets mention another congress member, hence providing us with 'found supervision' for igr prediction (whether the speaker and the target belong to the same political party). our analyses show that while positive interpersonal emotions appear in both in-and outgroup situations, negative emotions like anger and disgust are overwhelmingly present in the latter. meanwhile, human judgments for in vs. out-group membership on this dataset are overly reliant on the polarity of emotion; specifically, human judges are much less likely to attribute positive emotions towards out-group targets. baseline performances for perceived interpersonal emotion detection shows that this is a challenging task, as is consistent with existing work in emotion detection in general @xcite . in particular, emotions in this dataset are often expressed with considerable subtlety, likely a characteristic of official political speech. to investigate whether igr and emotions are intertwined and useful towards each other, we further developed a multi-task model for the prediction of both. we found compelling evidence that multi-tasking igr and interpersonal emotion improves performance on both tasks with over 10% improvement in detection of disgust in out-group contexts, and 3% improvement in igr prediction. to summarize the contributions of this paper, we tackle generalized intergroup bias, a notion of bias rooted in social psychology that applies to all the various differences in the ways that people talk about others in their in-group or out-group. standard bias tasks in nlp, and the broader goal of debiasing models could thus be set in a more general context. we present the first dataset to study both interpersonal group membership and emotion, which allows us to analyze both human and model behavior in terms of how the two interact with each other. we release our code and data online at github.com/venkatasg/interpersonal-bias. ## interpersonal contexts & emotions our aim is to build a generalized, data-driven approach towards studying bias situated in interpersonal utterances, which we define as any utterance where there is a target individual being talked about or referred to. our goal is to model two novel tasks described below; examples are shown in interpersonal group relationship igr is defined by the relationship between the speaker and target of an utterance. people belong to multiple social groups as part of their identity, however usually only some identities are salient in an utterance in context. we define in-group utterances as ones where the speaker and target are in the same social group, and out-group utterances as one where they are in different social groups. given an utterance u written by an individual s with target t, the igr prediction task classifies whether s and t belong to the same social group within the context of u. interpersonal emotion we define perceived interpersonal emotion as the emotion expressed by a speaker s towards, or in connection with the target t of the utterance u, as perceived by a reader. we use the plutchik wheel of emotions, which is widely adopted in the community, as the basis of our emotion taxionomy @xcite ; we use the 8 fundamental emotions (admiration, anger, disgust, fear, interest, joy, sadness, surprise) instead of the full 24 emotions in the wheel due to data sparsity. interpersonal emotions may be different, or a subset of, emotion for the whole of an utterance, as illustrated in rows 2, 3 and 4 of ## data collection in our area of focus, we require natural language data which satisfies the following criteria: (1) each utterance must have at least one target about whom the utterance mainly concerns. (2) the relationship between the speaker and the target must be inferred based on metadata or other information. specifically, we are interested in aspects of their social identity that they share or differ on. the dataset we collect comes from tweets by members of us congress where other members are mentioned in the same tweet. we use this as a convenient testbed: each member's group affiliation (i.e., their party identity) is public, thus we can easily know whether the speaker is tweeting to a target in their own party or not. 1 in other words, this dataset gives us "found supervision" for our first task of igr prediction. for our second task, we annotate a subset of these tweets for perceived interpersonal emotion; this is, to our knowledge, the first dataset dedicated to interpersonal emotion. ## preliminary analysis how are emotions distributed? when observing the distribution of aggregated emotion labels themselves, a clear pattern emerges as seen in figure 2 shows the co-occurrence of interpersonal emotions in our dataset. we can see that emotions that are farther apart and more dissimilar, such as admiration and disgust, joy and sadness, co-occur infrequently. emotions that are closer such as anger and disgust, admiration and joy, cooccur much more often. the only outlier is the higher than normal co-occurrence of admiration with sadness -after a closer examination, this can be attributed to tweets expressing admiration and sadness at the passing, or end of the career, of a fellow congressperson. who were the targets of negative emotions? on further analysis, it appears that most of the out-group disgust and anger is directed at 3 handles -@speakerryan, @speakerpelosi, and @speakerboehner who were all speakers of the house of representatives over most of the time period of our dataset. 63.7% of disgust and 64.3% of anger is directed towards these three twitter handles. 11.9% of all tweets in our dataset are directed at these handles, indicating the preponderance of negative interpersonal emotion directed at the speaker of the house. however, we note that negative emotions like anger and disgust were still expressed towards 51 and 45 different individuals in our dataset, respectively. can humans predict in/out-group? while our data naturally comes with "gold" igr labels, what is unexplored is whether the distinction between in-group and out-group speech is prominent and noticeable by humans. additionally, it is also unclear if humans might have their own expectation of how in/out-group speech should be characterized. concretely, we investigate if human annotators were capable of accurately performing the igr prediction task when the speaker and target are masked. two authors of this paper, one a social science graduate student, and the other a computational linguistics graduate student, annotated 50 random tweets from our validation data which they had not been exposed to earlier for in/out group labels. their fleiss κ agreement score was 0.64, indicating moderate agreement. to check how accurate their judgements were, we calculate for each annotator their f1 score against our "gold" in/out group labels. their f1 scores on these 50 tweets were 0.67 and 0.63, which as we will discuss in section 6, only match simple baselines of supervised systems. annotators comments indicate that they overly relied on the sentiment of tweets to make the classification -positive sentiment means in-group and negative sentiment means out-group. while negative emotions are over-represented in out-group situations as do pre-trained representations capture interpersonal emotions? pre-trained language models have been found to learn sentence representations that cluster by domain without supervision (aharoni and goldberg, 2020). we wished to investigate if any of our annotated properties cluster inherently in reduced representations of the tweets in our data. to obtain unsupervised representations, we use bertweet (nguyen et al., 2020), a language model pre-trained on 850m english tweets. we take the 768 dimensional embeddings from the final layer of @xmath0, and dimensionally reduce them to 2 dimensions using umap @xcite . figure 3 shows the distribution of tweets, color coded for interpersonal emotions. while there is a lot of overlap between representations when stratified by emotion, we can see that some emotions that are intuitively opposite, like admiration & disgust, joy & sadness are moderately separable. this indicates that interpersonal emotions do define some topic or domain level properties of a tweet. ## experiments we detail our experiments for the two novel tasks discussed in section 2: predicting the igr (ingroup or out-group) given a tweet, and predicting the interpersonal emotion given a tweet. we present baselines for the two tasks separately, and also present a multi-task model to gauge the extent to which knowledge of igr may help in predicting interpersonal emotion, and vice versa. ## results and analysis interpersonal group relationship in modeling igr, we find that sentiment-rule performs not much better than chance ( interpersonal emotion we find that the emolex baseline, which relies purely on lexical cues, performs dismally on our data, with poor performance in both in-group and out-group settings( ## related work emotion and stance detection a wealth of work has looked at corpora and models for the detection of perceived emotion in social media text @xcite @xcite @xcite . however existing work doesn't distinguish between emotion of a sentence as a whole, versus interpersonal emotion towards a target. the task closest to our study of interpersonal emotions is stance detection: whether the author has a favourable, neutral, or negative position towards a proposition or target. @xcite looked at stance in five target domains are given: abortion, atheism, climate change, feminism and hillary clinton. while stance detection focuses on a collection of utterances with the same topic, our interest is in modeling interpersonal emotion towards a target individual which is more finegrained and can vary in each utterance.