Social Commonsense for Explanation and Cultural Bias Discovery (2023)

Abstract: Social commonsense contains many human biases due to social and cultural influence (Sap et al., 2020; Emelin et al., 2020). We focus on identifying cultural biases in data, specifically causal assumptions and commonsense implications, that strongly influence model decisions for a variety of tasks designed for social impact. This enables us to examine data for bias by making explicit the causal (if-then, inferential) relations in social commonsense knowledge used for decision making, furthering interpretable commonsense reasoning from a dataset perspective. We apply our methods on 2 social tasks: emotion detection and perceived value detection. We identify influential social commonsense knowledge to explain model behavior in the following ways. First, we augment large-scale language models with social knowledge and show improvements for the tasks, indicating the implicit assumptions a model requires to be successful on each dataset. Second, we identify influential events in the datasets by using social knowledge to cluster data and demonstrate the influence that these events have on model behavior via leave-K-out experiments. This allows us to gain a dataset-level understanding of the events and causal commonsense relationships that strongly influence predictions. We then analyze these relationships to detect influential cultural bias in each dataset. Finally, we use our influential event identification for detecting mislabeled examples and improve training and performance through their removal. We support our findings with manual analysis.

## introduction social commonsense knowledge helps humans maneuver through everyday life, aiding in situations that may require social nuance or cultural knowledge. commonsense knowledge acquisition has been an important goal in nlp @xcite @xcite and in the past few years there has been a surge of research focusing specifically on improving social commonsense understanding for neural models @xcite @xcite . however, social commonsense may contain many human biases due to social and cultural influence @xcite . we aim to discover influential cultural biases in social applications datasets via the social and causal commonsense knowledge @xcite @xcite present in each dataset, to identify bias in causeeffect commonsense relationships. we define cultural bias as a positive or negative cultural attitude toward a social structure (see section 3.1) and define causal commonsense following @xcite , as if-then, inferential relations. specifically, we are interested in exploring biases towards the following social structures: religion, economy, family, government, education and technology. to the best of our knowledge, we are the first to empirically discover influential cultural biases in social tasks, using causal commonsense knowledge about social interactions, emotional reactions, and human needs to explain underlying cultural trends in social applications datasets. we focus on two social tasks: emotion detection in social media and community value detection from interviews. we define a social task as any task that is intended to have social impact and requires social knowledge to correctly resolve. it is particularly important to identify influential biases in datasets and applications which are designed for social impact. hurricaneemo @xcite is an emotion detection task that focuses on perceived emotional reactions to events that occur during natural disasters. for example, events that exist in this dataset like "thanks god" may elicit emotions like awe, and events like "personx sends -to congress" may elicit emotions like con-tempt, showing a positive cultural bias for religion and a negative cultural bias for government. stories2insights @xcite is a value detection task that focuses on interviews conducted to identify community needs in developing countries, aiming to capture perceived values based on certain events. for example, events like "reading the bible" indicate indigenous values (defined by social norms and religion) whereas events like "providing for children" indicate intrinsic hu-man needs (defined by health and quality of life). this illustrates a positive culture bias for christianity and family. we aim to discover causal social knowledge that indicates cultural biases in each dataset, by analyzing model behavior on both tasks to gain a dataset-level understanding of events that influence performance. we identify these biases via casual social knowledge, which encodes the relationships between events and the triggered reactions. we derived our knowledge from the social knowledge graph atomic @xcite . consider the hurrica-neemo example in fig. 1 , in which the atomic knowledge makes explicit the event-emotion causal relationships in the tweet that bring about the perceived emotion remorse. identifying that the event "says prayers" causes perceived traits like sorrow and stressed, allows us to understand which events in the tweet cause the perceived emotion and contribute to the specific cultural biases. next, consider the stories2insights example, in which the knowledge demonstrates the causal relationships that support the perceived value social signifi-cance (defined by identity and status). identifying that the event "sells -to the public" causes perceived traits like influential and professional, sheds light on how the event's associated cultural biases contribute to the perceived value. in this paper, we gain a dataset-level understanding of influential causal social commonsense relationships, allowing an exploration of the underlying social and cultural biases that explain model behav-ior on social tasks. to this end, we first extract atomic knowledge for each datapoint, utilizing a combination of tf-idf and bertscore @xcite . we then append different components of this knowledge to the input, to focus strongly on either the cause, the effect, or the causal relation between the two. we use bert-aug, which ingests knowledge augmented data as input to bert @xcite , to yield improvements on both tasks. these improvements illustrate that the underlying causal social assumptions made explicit by the integrated knowledge do indeed increase model accuracy on the task and thus augment a model's understanding of the task. to discover underlying cultural biases, we investigate the influence of underlying events in each task. we first identify underlying events in the data by clustering datapoints using k nearest neighbors (knn) around commonsense events, using fine-tuned contextual embeddings. we then identify which of these events are influential by performing leave-kout experiments, which detect the influence of train clusters on task performance. understanding cultural biases in social applications data via the implicit commonsense knowledge present in the data is important for analyzing the limitations of the dataset and the respective tasks. by making explicit the underlying cultural assumptions and causal relationships, we are able to identify the biases that data from a certain source may have, which is paramount when using this data in the development of technology for other applications @xcite . finally, we use our methods to identify mislabeling in event clusters. datasets used for training deep learning models are large and often contain noisy labels, even if the data was collected via crowdsourcing @xcite . we identify events whose train clusters cause performance improvement when removed in leave-k-out experiments, indicating mis- tf-idf bertscore data extraction dataset texts atomic dataset texts + top 3 events knowledge integration +% knowledge influence event selection knn score clusters underlying event clusters leave-k-out δ% +% word match remove event clusters from train mislabeled cluster & removal heuristic mislabeled removal bert contextual embedding task classification layer labeling. overall, our contributions are: ## models & knowledge augmentation we aim to 1) extract knowledge that makes explicit the underlying causal social commonsense relationships in each datapoint and 2) propose a simple integration method to show that this knowledge is able to increase model accuracy, demonstrating the role of these relationships in improving a model's understanding of the task. ## event influence & bias discovery we aim to (1) identify influential social and causal events in each dataset, and (2) analyze their respective inferential relationships to discover attitudes toward target social structures. to identify influential events in the datasets, we first extract underlying atomic events from each dataset via event selection and then measure how influential these events are via leave-k-out experiments (illustrated in event selection and leave-k-out in the knowledge influence section of fig. 2 ). we then explore the following types of social and cultural structures in our datasets: religion, economy, family, government, education and technology. we consider a dataset to be biased if there exists a positive or negative attitude toward one of these structures in the influential underlying events. ## conclusion we used causal social commonsense knowledge to discover influential events and relationships that explain model behavior and pinpointed instances of cultural bias. first, we found that using large-scale language models augmented with causal social knowledge improved our social classification tasks, illustrating that the knowledge made the underlying social assumptions in the dataset explicit. then, we identified underlying events in each dataset by clustering data around atomic knowledge, to pinpoint cultural biases that the dataset may exhibit. we found that some of this knowledge strongly influenced model behavior through leave-k-out experiments, providing a dataset-level understanding of influential events and causal social commonsense relationships and allowing an analysis of the datasets' implicit influential cultural biases. finally, we used these underlying and influential events to identify mislabeled train examples and thus improve training and performance.