Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships (2023)

Abstract: Understanding interpersonal communication requires, in part, understanding the social context and norms in which a message is said. However, current methods for identifying offensive content in such communication largely operate independent of context, with only a few approaches considering community norms or prior conversation as context. Here, we introduce a new approach to identifying inappropriate communication by explicitly modeling the social relationship between the individuals. We introduce a new dataset of contextually-situated judgments of appropriateness and show that large language models can readily incorporate relationship information to accurately identify appropriateness in a given context. Using data from online conversations and movie dialogues, we provide insight into how the relationships themselves function as implicit norms and quantify the degree to which context-sensitivity is needed in different conversation settings. Further, we also demonstrate that contextual-appropriateness judgments are predictive of other social factors expressed in language such as condescension and politeness.

## introduction interpersonal communication relies on shared expectations of the norms of communication @xcite . some of these norms are widely shared across social contexts, e.g., racial epithets are taboo, enabling nlp models to readily identify certain forms of offensive language @xcite ). yet, not all norms are widely shared; the same message said in two different social contexts may have different levels of acceptability (figure 1 ). while nlp has recognized the role of social context as important @xcite , few works have directly 3 these authors contributed equally to this work these authors contributed equally to this work incorporated this context into modeling whether messages violate social norms. here, we explicitly model relationships as the social context in which a message is said in order to assess whether the message is appropriate. nlp models have grown more sophisticated in modeling the social norms needed to identify offensive content. prior work has shown the benefits of modeling context @xcite , such as the demographics of annotators and readers @xcite and the online community in which a message is said @xcite . however, these works overlook normative expectations within people's relationships. in this paper, we introduce a new dataset of over 12,236 instances labeled for whether the message was appropriate in a given relationship context. using this data, we show that computation models can accurately identify the contextual appropriateness of a message, with the best-performing model attaining a 0.70 binary f1. analyzing the judgments of this classifier reveals the structure of the shared norms between relationships. through examining a large corpus of relationship-labeled conversations, we find that roughly 19% of appropriate messages could be perceived as inappropriate in another context, highlighting the need for models that explicitly incorporate relationships. finally, we show that our model's relationship-appropriate judgments provide useful features for identifying subtly offensive language, such as condescension. ## social norms of appropriateness relationships are the foundation of society: most human behaviors and interactions happen within the context of interpersonal relationships @xcite . communication norms vary widely across relationships, based on the speakers' social distance, status/power, solidarity, and perceived mutual benefit @xcite . these norms influence communication in content, grammar, framing, and style @xcite and help reinforce @xcite the relationship between speakers @xcite . prior computational work mostly frames appropriateness as exhibiting positive affect and overlooks the fact that, in some relationships, conversations can be affectively negative but still appropriate @xcite . for example, swearing is often considered a norm violation @xcite , but can also be viewed as a signal of solidarity between close friends @xcite or co-workers @xcite . in such cases, the violation of taboo reinforces social ties by forming a sense of in-group membership where norms allow such messages @xcite . in sociolinguistics, appropriateness is a function of both context and speech. @xcite argues that "different situations, different topics, different genres require different linguistic styles and registers," and @xcite argues that the extent to which "something is suitable, effective or liked in some context" determines its appropriateness. whether a discourse is appropriate depends strongly on the social context in which it is produced and received @xcite , making the assessment of appropriateness a challenging task due to the need to explicitly model contextual norms. behavioral choices are subject to the norms of "oughtness" @xcite , and @xcite suggest relationship types as an important factor influencing the normative expectations for relational communication. for example, while it may be considered appropriate for siblings to discuss their past romantic relationships in detail, the topic is likely to be perceived as taboo or inappropriate between romantic partners @xcite . ## building a dataset of contextual appropriateness prior work has shown that interpersonal relationships are a relevant context for the appropriateness of content @xcite . while not all messages differ in this judgment-e.g., "hello" may be appropriate in nearly all settings-building a dataset that embodies this context sensitivity remains a challenge. here, we describe our effort to build a new, large dataset of messages rated for contextual appropriateness, including how we select relationships and operationalize appropriateness. due to the challenge of identifying and rating these messages, our dataset is built in two phases. selecting relationships formally categorizing relationships has long been a challenging task for scholars @xcite . we initially developed a broad list of relationships, drawing from 1) folk taxonomies @xcite , e.g., common relationship types of friends @xcite , family @xcite , or romantic partners @xcite ; and 2) organizational and social roles @xcite , e.g., those in a workplace, classroom, or functional settings, as these frequently indicate different social status, distance, or solidarity between individuals in the relationship. using this preliminary list, four annotators performed a pilot assessment of coverage by discussing quotes from movie scripts, social media, or their imagination and identifying cases where an excluded relationship would have a different judgment for appropriateness. ultimately, 49 types of relationships were included, shown in defining appropriateness appropriateness is a complex construct that loads on many social norms @xcite . for instance, in some relationships, an individual may freely violate topical taboos, while in other relationships, appropriateness depends on factors like deference due to social status. informed by the theory of appropriateness (march and olsen, 2004), we operationalize inappropriate communication as follows: given two people in a specified relationship and a message that is plausibly said under normal circumstances in this relationship, would the listener feel offended or uncomfortable? we use plausibility to avoid judging appropriateness for messages that would likely never be said, e.g., "would you cook me a ham- category relationships family parent, † child, † adopted child, † siblings, step-siblings, grandparent, † grandchild, † niece/nephew, † cousins, uncle/aunt † social best friend, friend, old friend, childhood friend, acquaintance, neighbor, complete stranger romance dating, engaged, married, domestic partner, friends with benefits, a person whom one has an affair with, divorcee, exboyfriend/ex-girlfriend organizational coworker, colleague, another employee in a larger company, boss † (to a direct report), direct report † (to a boss) peer group classmate, sports teammate, club member parasocial fan, † hero † role-based law enforcement, † individual with authority † (generic), mentor, † mentee, † teacher, † student, † lawyer, † client, † doctor, † patient, † landlord † antagonist competitor, rival, enemy raw data initial conversational data was selectively sampled from english-language reddit. much of reddit is not conversational in the sense that comments are unlikely to match chit-chat. further, few comments are likely to be contextsensitive. to address these concerns, we filter reddit comments in two ways. first, we train a classifier to identify conversational comments, using 70,949 turns from the empathetic dialogs data @xcite and 225,907 turns from the cornell movie dataset @xcite as positive examples of conversational messages, and 296,854 turns from a random sample of reddit comments as nonconversational messages. full details are provided in appendix b. second, we apply our conversational classifier to comments marked by reddit as controversial in the pushshift data @xcite ; while the decision logic for which comments are marked as controversial is proprietary to reddit, controversial-labeled comments typically receive high numbers of both upvotes and downvotes by the community-but are not necessarily offensive. these two filters were applied to identify 145,210 total comments gathered from an arbitrary month of data (feb. 2018). ## identifying contextual inappropriateness given the high potential agreement of annotators, we test whether models can similarly recognize whether a given message is appropriate if said in the context of a specific relationship. experimental setup two classes of models were trained. given the recent successes of promptbased models, we build models using the open-prompt library @xcite and, to support larger models, using the peft library @xcite . the openprompt library was used to train t5-base and gpt2-med models using the prompt "is it appropriate for person1 to say "quote" to person2, "yes" or "no"? [mask]" using the verbalization "yes' or "no" for the masked token. here, we fill in person1 and person2 to refer to the two parties in the relationship. examples of filled-in templates and other prompts are reported in appendix d, though performance varied only slightly across different prompts. the peft library was used to train the large and xl variants of the flan-t5 model @xcite . this model has been pretrained for instruction following; therefore, based on suggested guidelines from ziems et al. (2023), we use the following prompt: "rate whether it is inappropriate for this message to be said in the following social setting?\n setting: relationship description \n message: "quote"\n answer (yes or no):" due to the resources required for training these larger models, no additional prompts were rigorously evaluated outside of initial pilot testing. the second class of models uses masked language model (mlm) fine-tuning on the [cls] token from an mlm to predict appropriateness. here, we frame the instance using the same language as the openprompt-based models but fill in the mask with "yes" (i.e., indicating that the message is appropriate to say in the relationship). the classification model is then fine-tuned to classify whether this hard-coded judgment is correct or not. we test two recent mlms, minilm @xcite , a small distilled model, and deberta-v3 @xcite , a much larger model. these two models reflect extremes among relatively small mlms and allow us to assess whether more social relationship knowledge might be embedded within a larger parameter space. annotated data was split at the message level 70:10:20 into train, development, and test sets, resulting in 9,107 train, 1,100 development, and 2,029 test instances. we frame the task similar to offensive language detection and use binary f1 as our metric where inappropriate is the positive class. model performance is reported as the average across five random runs. additional training details and per-seed performance are provided for all systems in appendix e. two baseline systems are included. the first is random labels with respect to the empirical distribution in the training data. the second uses perspective api @xcite to rate the toxicity of the message, labeling it as toxic if the rating is above 0.7 on a scale of [0,1]; the same label is used for all relationships. while this baseline is unlikely to perform well, it serves as a reference to how much explicit toxicity is in the dataset, as some (though not all) of these messages are inappropriate to all relationships. results models accurately recognized how relationships influence the acceptability of a message, as seen in error analysis different relationships can have very different norms in terms of what content is acceptable, as highlighted in figure 2 . how did model performance vary by relationship? figure 3 shows the binary f1 score of the flan-t5-xl model by relationship, relative to the percent of training instances the model saw that were inappropriate; appendix through their pretraining, llms have learned semantic representations of relationships as tokens. our classification experiments show that llms can interpret these relationship-as-token representations to effectively judge whether a message is appropriate. to what extent do these representations allow the model to generalize about new relationships not seen in training? in particular, are models able to generalize if a category of relationship, e.g., all family relations, was never seen? here, we conduct an ablation study where one of our folk categories is held out during training. ## how much of conversation is context sensitive in appropriateness? our annotation and computational models have shown that the relationship context matters in determining appropriateness. however, it is unclear how often conversations are sensitive to this context. for example, the majority of conversation may be appropriate to all relationships. here, we aim to estimate this context sensitivity by testing the appropriateness of a message in counterfactual settings using an existing dataset labeled with relationship types. experimental setup to estimate context sensitivity, we use our most accurate model to label a large selection of dialog turns from the pride dataset @xcite . pride consists of 64,844 dialog turns from movie scripts, each annotated for the relationship between the speaker and receiver, making it ideal as a high-plausibility conversational message said in relationships. however, some turns of the dialog are explicitly grounded in the setting of the movie, e.g., "how's it going, pat?" which makes the turn too specific to that particular setting to accurately estimate appropri-ateness. therefore, we run spacy ner @xcite on the dialog and remove all turns containing references to people, companies, countries, and nationalities in order to keep the dialog generic and maximally plausible in many different relationship contexts. further, we remove turns with only a single token or over 100 tokens. this filtering leaves 47,801 messages for analysis. pride contains 18 unique relationships, 16 of which were already included in our categories (cf. results considering only appropriate messages and excluding the unusual enemy relationship from consideration, we find that roughly 19% of the appropriate-as-said messages in the data would be inappropriate if said in the context of a different relationship. figure 5 shows the probability that a message acceptable in some other relationship context would also be acceptable in the given context; the striking decrease in the likelihood of acceptability follows the increasingly constrained social norms around a relationship. for example, while friends and loved ones have broad latitude to discuss sensitive topics @xcite , role-based relationships and those with larger power differences are more constrained in what is considered acceptable conversation. while the movie dialog in the pride dataset likely differs from a natural dialog, these results point to relationships as important contexts in natural language understanding. more generally, we suggest a need for sociallyaware models to identify offensive language. while substantial effort has been put into identifying explicit toxic or abusive language @xcite , few models, if any, incorporate the context 0.0 0.2 0.4 0.6 0.8 1.0 probability of message still being appropriate to a sibling to a friend to a lover to a fiancee to a spouse to an enemy from parent to child to a classmate from child to parent to a commercial associate to a colleague to someone in their church to an acquaintance to a medical professional from employee to a boss from boss to an employee from teacher to student from student to teacher if a message was appropriate in another relationship context, is the message still appropriate if it was said... in which the message is said. these models typically rely on previous conversation turns @xcite or modeling community-level social norms @xcite to understand how the context may shift whether the message is perceived as appropriate. our result suggests that the social context-and particularly social relationships-are highly influential in measuring appropriateness. indeed, together with the result showing the (expected) low performance of the perspective api toxicity detector, these results suggest nlp models deployed in social settings are likely missing identifying many offensive messages due to their lack of explicitly modeling of social relations. as nlp tools make their way into the workplace setting, which frequently features a mix of organizational, social, and ro-mance ties, explicitly modeling context will likely be necessary. ## identifying subtle offensiveness using contextual appropriateness prior nlp studies of subtly inappropriate language often omit the social context in which a statement is said @xcite ), yet it is often this context that makes a statement inappropriate. for example, a teacher asking a student "do you need help writing that?" is appropriate, whereas a student asking a teacher the same question may seem rude. we hypothesize that modeling the relative appropriateness of a message across relationships can help identify types of subtly offensive language. we test this hypothesis using datasets for two phenomena: condescen-sion @xcite and (im)politeness @xcite . ## limitations this paper has three main limitations worth noting. first and foremost, while our paper aims to model the social context in which a message is said, the current context is limited to only the parties' relationship. in practice, the social context encompasses a wide variety of other factors, such as the sociodemographics of the parties, the culture and setting of the conversation, and the history of the parties. even relationships themselves are often much more nuanced and the appropriateness may vary widely based on setting, e.g., statements said between spouses may vary in appropriateness when made in public versus private settings. these contextual factors are likely necessary for a full account of the effect of social context on how messages should be perceived. our work provides an initial step in this direction by making the relationship explicit, but more work remains to be done. future work may examine how to incorporate these aspects, such as by directly inputting the situation's social network as context using graph embedding techniques @xcite , where the network is labeled with relationships @xcite , or by modeling relationships particular types of settings such as in-person, phone, texting, or other online communication, which each have different norms. second, our data includes annotations on a finite set of relationships, while many more unique relationships are possible in practice, e.g., customer or pastor. our initial set was developed based on discussions among annotators and aimed at high but not complete coverage due to the increasing complexity of the annotation task as more relationships were added. our results in section 5 suggest that our best model could be able to generalize to new types of relationships in some settings and zeroshot results on two new relationship types not seen in training (a fellow church member and a commercial relationship) match expectations of context sensitivity, (cf. figure 5 . however, performance is likely limited for less-common relationships without additional training data to describe the norms of appropriateness in this context; and, based on the error analysis in section 4, models are currently unlikely to generalize to unseen relationships that have complex sensitivity norms. in addition, new settings such as online spaces may require additional definitions of relationships as individuals interact with each other anonymously. third, our judgments of appropriateness were drawn from five annotators total, each of whom had different views of appropriateness based on their values and life experience. while our analysis of agreement with the adjudicated data (section 3.2) suggests that when annotators can reach a consensus on a message's meaning, they are highly likely to agree on appropriateness, we nonetheless view that our annotations are likely to primarily reflect the values of the annotators and may not generalize to other social or cultural contexts where the norms of relationships differ. future work is needed to explore how these norms differ through additional annotation, and we hope that our dataset will provide a reference for comparison to these judgments. for example, future work may make use of annotation schemes that explicitly model disagreements @xcite or personalized judgments @xcite ; such approaches may be able to better represent common factors influencing appropriateness judgments. ## ethical considerations we note three points on ethics. first, we recognize that appropriateness is a value judgment, and therefore our data is limited here by the viewpoints of the annotators. multiple works on offensive language have shown that the values and identities of annotators can bias the judgments and potentially further marginalize communities of practice whose views and norms are not present @xcite . we have attempted to mitigate this risk by adding diversity to our annotator pool with respect to gender, age, and culture, yet our limited pool size necessitates that not all viewpoints will be present. given that we show relationships do matter in judging appropriateness, we hope that future work will add diversity through new additions and data to study relationships. we will also release demographic information on annotators as a part of our dataset to help make potential biases more explicit and more easily addressed. the annotators themselves were authors of the study and were compensated as a part of their normal work with a living wage. due to the nature of our filtering, the vast majority of our content was not explicitly toxic. nonetheless, some comments did contain objectionable messages, and annotators were provided guidance on how to seek self-care if the messages created distress. with any new tool to identify offensive or abusive language comes a dual use by an adversarial actor to exploit that tool to find new ways to harass or abuse others while still "abiding by the rules." our work has shown that relationships are effective context (and features) for identifying previouslyunrecognized inappropriateness. this new capability has the benefit of potentially recognizing more inappropriate messages before they reach their destination. however, some adversaries could still use our data and model to screen their own messages to find those that still are classified as appropriate (while being inappropriate in practice) to evade detection. nevertheless, given the new ability to identify context-sensitive offensive messages-which we show can represent a substantial percentage of conversation (section 6)-we view the benefits as outweighing the risk. caleb ziems, william held, omar shaikh, jiaao chen, zhehao zhang, and diyi yang. 2023. can large language models transform computational social science? arxiv preprint arxiv:2305.03514.