Quantifying Stereotypes in Language (2024)

Abstract: A stereotype is a generalized perception of a specific group of humans. It is often potentially encoded in human language, which is more common in texts on social issues. Previous works simply define a sentence as stereotypical and anti-stereotypical. However, the stereotype of a sentence may require fine-grained quantification. In this paper, to fill this gap, we quantify stereotypes in language by annotating a dataset. We use the pre-trained language models (PLMs) to learn this dataset to predict stereotypes of sentences. Then, we discuss stereotypes about common social issues such as hate speech, sexism, sentiments, and disadvantaged and advantaged groups. We demonstrate the connections and differences between stereotypes and common social issues, and all four studies validate the general findings of the current studies. In addition, our work suggests that fine-grained stereotype scores are a highly relevant and competitive dimension for research on social issues. The models and datasets used in this paper are available at https://anonymous.4open.science/r/quantifying_stereotypes_in_language.

## introduction a stereotype is an important psychosocial phenomenon that reflects common beliefs about a specific category of people @xcite . stereotypes can influence our perceptions of others and affect our decisions and behaviors, which can lead to discrimination and unfairness @xcite . further, it leads to social inequality and fragmentation by influencing human attitudes and behaviors towards social groups @xcite @xcite . therefore, it is crucial to understand and recognize stereotypes. in recent years, the study of stereotypes in language has received widespread attention as the fairness of artificial intelligence (ai) has been highlighted (buolamwini and gebru, 2018; holstein previous works: stereotype (1) or antistereotype (-1). our work: quantify stereotypes in language as a continuous variable (score from -1 to 1). she eventually wants to become a doctor. ## the concept of stereotype the concept of stereotype dates back to the early 20th century, when psychologists began to study how people form fixed opinions about different groups of people @xcite @xcite . the psychologist lippmann first introduced the concept of stereotype in his book public opinion, published in 1922. he argues that people often rely on media and social messages to form opinions about the world, which are often one-sided and inaccurate, leading to biases and stereotypes about particular groups. in the late 20th century, social psychologists began to study the formation and influence of stereotype in depth @xcite . they found that people tend to rely on preconceived prejudices and stereotypes rather than objective information when recognizing strangers or unfamiliar groups @xcite . such prejudices can lead to discrimination and unfair treatment. over time, more and more people have begun to recognize the dangers of stereotypes and to take steps to reduce them @xcite . in the social field, many organizations and activities work to promote diversity and inclusion in order to break down stereotypes and create a fairer social environment @xcite . recently, with the rise of ai, researchers have found such stereotypes in ai models as well @xcite @xcite . in this work, we focus on quantifying stereotypes in language. because language is the primary carrier of information, it can express human ideas most directly @xcite . moreover, language is also the main form of expression of human intentions @xcite . ## related work stereotype quantification previous works have quantified stereotypes as binary (1 and -1). for example, some works @xcite define sentences as stereotypical or anti-stereotypical as a criterion for classification. then, the de-biasing works @xcite @xcite for the plms utilize stereotyped and anti-stereotyped sentence pairs to design the de-biasing methods. although there are a number of metrics @xcite @xcite for evaluating stereotypes in plms. however, there is a lack of methods to quantify stereotypes in language at a fine-grained level. we argue that stereotypes, as complex properties of language, should be quantified not just using binary, but with continuous variables. ## quantifying language stereotype the bias evaluation datasets like ss and cp provide sentences that express stereotypes. although @xcite argue that the sentences in these datasets may not accurately evaluate biases in language models, we find that they can facilitate our quantifying stereotypes. stereotypes are often found in language and are fixed impressions potentially harmful to the target group @xcite . the previous rough definition of sentences with or without stereotypes is far from sufficient; different stereotypes harm the target group to different degrees. in this work, inspired by the work of pei and jurgens (2020) on quantifying intimacy in language, we quantify stereotypes in sentences as a continuous variable (stereotype score) from -1 to 1. in the following, we first describe the construction of the dataset; then, we introduce the dataset annotation and scoring methodology; and finally, we discuss the reliability of the stereotype scores. ## predicting language stereotype plms can effectively capture contextualized representations of text. we use plms to learn our annotation results to predict stereotypes in language. we use the 2,976 sentences annotated in § 4, and the sentences are split into training, validation, and test sets by 6:2:2. in our experiments, we use the following popular plms: bert (bert-base-uncased; @xcite , distilbert (distilbert-base-uncased; @xcite , and roberta (roberta-base; @xcite . we set the max sentence length to 50, and the batch size to 128. we use the adam (kingma and ba, 2014) optimizer with the weight decay set to 1e-6 and the learning rate set to 1e-4. we conducted our experiments on a geforce rtx 3090 gpu, and all training processes lasted about twelve minutes. each model trains 30 epochs and saves the model with the lowest mean square error (mse) on the validation set. we fine-tuned the model weights based on the huggingface library 5 . the code is available at https://github.com/nlply/ quantifying-stereotypes-in-language . result ## sentiment and stereotype dataset in this section, we conduct experiments on the sst dataset @xcite . the sst dataset is one of the commonly available datasets used for sentiment analysis tasks. it contains five sentiment classes, which are very negative, negative, neutral, positive, and very positive. the goal of the dataset is to train the model to sentiment classify movie reviews to determine the sentiment polarity of the reviews, so it provides sentiment values for each sentence. this dataset is often used to test and evaluate the performance of sentiment analysis models. however, our work is to test the association between sentiment values and stereotype scores, so we only use the training set of the sst dataset and not its development and test sets. result figure 6 shows the scatter plots of sentiment values and stereotype scores for the three models on the sst dataset. for clarity, for each of the five classes in the training set of the sst dataset, we randomly selected 100 samples for plotting. specifically, for stereotype scores, the results on the three models were always very neg-@xmath0. since the sst dataset comes from actual user comments, it implies that humans tend to post comments with negative sentiments that carry more stereotypes. in other words, humans tend to utilize stereotypes when giving negative reviews. therefore, the sentiment values of language may not be reliable for evaluating sentiment polarity. we argue that sentiment evaluation of language needs to take into account stereotypes in language. ## disadvantage group and advantage group disadvantaged groups are usually those who are at a disadvantage in the socio-economic, political, and cultural fields, while the vice versa is for advantaged groups. these groups are usually distinguished based on several social factors, such as race, gender, social class, disability, sexual orientation, etc @xcite . @xcite argue that advantaged groups usually have more resources and authority, while disadvantaged groups face more unfairness and discrimination. stereotypes often do more harm to disadvantaged groups, as they can reinforce and exacerbate discrimination and unfairness against these groups. for example, stereotypes about certain disadvantaged groups (e.g., racial minorities) may lead to discrimination against them in employment, education, and medicine. these impressions may cause employers, schools, or physicians to make incorrect assessments and assumptions about their abilities, values, and needs, thus limiting their opportunities and rights. similarly, stereotypes of certain advantaged groups (e.g., males or whites) may lead to their enjoying more social and cultural advantages and privileges. these impressions may cause them to receive more praise, recognition, and opportunities, thus further reinforcing their advantageous position. in this section, we study the association between disadvantaged and advantaged groups and stereotypes and further demonstrate the effectiveness of the stereotype scores. ## boosting the performance of plms in downstream tasks plms can capture contextual information and thus outperform nlp downstream tasks. in this section, we test whether stereotype scores can boost the performance of plms in downstream tasks such as hate speech detection. method we conduct hate speech detection experiments on albert (albert-base-v2; @xcite and xlnet (xlnet-base-cased; yang et al., 2019), and on bert, distilbert, and roberta, which we mention in § 4. we use the ethos and hsol @xcite datasets for our experiments. for the ethos dataset, we use its ## discussion and ethics this work focuses on the annotation of stereotype scores in language and analyzes the relationship between stereotype scores and common social issues. the dataset is annotated with sentences from four bias types: profession, race, gender, and religion. we show that stereotypes in language should not just be binary, but should quantify stereotypes as continuous variables, which opens the door to more fine-grained studies of social biases. in addition, our work can be applied to many nlp scenarios. for example, stereotype scores can provide a useful measure for the detection of language in dialog systems. in addition to such harmful linguistic phenomena as hate speech and toxicity, stereotypes may also harm the target group. our quantification approach can detect potential stereotypes in language and thus prevent the target group from being harmed. the study of stereotypes in language requires a discussion of ethical implications. all experimental datasets used in this study were acquired from publicly available datasets in accordance with the terms of service. since offensive language can be more harmful to the target group, the offensive language covered in the dataset was filtered in this paper, even though it may have been used previously in other datasets. one of the risks that our approach presents is the use of non-offensive but stereotypical language to harm others. as a potential mitigation method, platforms may use the same technique to prompt users to use less stereotypical language. ## conclusion in this paper, we quantify stereotypes in language and obtain stereotype scores by plms. specifically, we annotate a dataset with stereotype scores and train plms that predict stereotype scores. the prediction of stereotype scores on commonly available datasets about social issues reveals that stereotypes are associated with hate speech, sexism, sentiments, and specific groups. our study provides a finegrained quantification of stereotypes in language and opens the way for further research on social biases.