V al N orm Quantifies Semantics to Reveal Consistent Valence Biases Across Languages and Over Centuries (2021)

Abstract: Word embeddings learn implicit biases from linguistic regularities captured by word co-occurrence statistics. By extending methods that quantify human-like biases in word embeddings, we introduce ValNorm, a novel intrinsic evaluation task and method to quantify the valence dimension of affect in human-rated word sets from social psychology. We apply ValNorm on static word embeddings from seven languages (Chinese, English, German, Polish, Portuguese, Spanish, and Turkish) and from historical English text spanning 200 years. ValNorm achieves consistently high accuracy in quantifying the valence of non-discriminatory, non-social group word sets. Specifically, ValNorm achieves a Pearson correlation of r=0.88 for human judgment scores of valence for 399 words collected to establish pleasantness norms in English. In contrast, we measure gender stereotypes using the same set of word embeddings and find that social biases vary across languages. Our results indicate that valence associations of non-discriminatory, non-social group words represent widely-shared associations, in seven languages and over 200 years.

## introduction new transparency-enhancing methods for static word embedding evaluation incorporate crossdisciplinary techniques that can quantify widelyaccepted, intrinsic characteristics of words @xcite @xcite . a promising approach for developing transparent intrinsic evaluation tasks is to evaluate word embeddings through the lens of cognitive lexical semantics, which captures the social and psychological responses of humans to words and language @xcite @xcite . such an approach could provide a representativeness evaluation method for word embeddings used in quantifying and studying biases. this paper presents the computational approach valnorm, that accurately quantifies the valence dimension of biases and affective meaning in word embeddings, to analyze widely shared associations of non-social group words. implicit biases, as well as the intrinsic pleasantness or goodness of things, namely valence, have been well researched with human subjects @xcite @xcite . valence is one of the principal dimensions of affect and cognitive heuristics that shape attitudes and biases in humans @xcite . valence is described as the affective quality referring to the intrinsic attractiveness/goodness or averseness/badness of an event, object, or situation @xcite . for word embeddings, we define valence bias as the semantic evaluation of pleasantness or unpleasantness that is associated with words (e.g., kindness is associated with pleasantness and torture is associated with unpleasantness). word embedding evaluation tasks are methods to measure the quality and accuracy of learned word vector representations from a given text corpus. the two main types of evaluation tasks are intrinsic evaluation, which analyzes and interprets the semantic or syntactic characteristics of word embeddings (e.g., word similarity), and extrinsic evaluation, which measures how well word embeddings perform on downstream tasks (e.g., partof-speech tagging, sentiment classification) @xcite . we focus on intrinsic evaluation, specifically the semantic quality of word embeddings that have been shown to learn human-like biases, such as gender and racial stereotypes @xcite . our intrinsic evaluation task, valnorm, accurately quantifies the valence dimension of biases and affective meaning in word embeddings. simply, valnorm provides a statistical measure of the pleasant/unpleasant connotation of a word. we validate valnorm's ability to quantify the semantic quality of words by implementing the task on word embeddings from seven different languages (chinese, english, german, polish, portuguese, spanish, and turkish) and over a time span of 200 years (english only). our results showcase that non-discriminatory non-social group biases introduced by @xcite are consistent across cultures and over time; people agree that loyalty is pleasant and that hatred is unpleasant. additionally, we use the word embedding association test (weat) @xcite to measure the difference between social group biases and non-discriminatory biases in seven languages. we compare valnorm to six widely used, traditional intrinsic evaluation tasks that measure how semantically similar two words are to each other and how words relate to each other. all intrinsic evaluation tasks, including valnorm, measure the correlation of the computed scores to humanannotated scores. we implement all intrinsic evaluation tasks on seven word embedding sets (english only), which were trained using four different embedding algorithms and five different training text corpora (see figure 1 ) to ensure that the valnorm results are not model or corpus specific. valnorm achieves pearson correlation coefficients (ρ) in the range [0.82, 0.88] for the seven english word embedding sets, outperforming the six traditional intrinsic evaluation tasks we compare our results to. we summarize our three main contributions: 1) we quantify semantics, specifically the valence dimension of affect (pleasantness/unpleasantness) to study the valence norms of words and present a permutation test to measure the statistical significance of our valence quantification, 2) we introduce valnorm, a new intrinsic evaluation task that measures the semantic quality of word embeddings (validated on seven languages), and 3) we establish widely-shared associations of valence across languages and over time. extended methodology, results, dataset details are in the appendices; the open source repository will be made public. ## datasets we use two main sources of data: 1) word embeddings and 2) human-annotated validation datasets. ## experiments we conduct three main experiments to 1) quantify the valence statistics of words in text corpora, 2) evaluate our intrinsic evaluation task, valnorm and 3) investigate widely-shared non-social group valence associations across languages and over time. ## results quantifying valence norms. we implement the sc-weat using valence evaluative attributes and target word sets, that are hypothesized to represent valence norms, from bellezza's lexicon, anew, and warriner's lexicon. our initial experiments signalled widely shared associations of valence scores with ρ ∈ [0.82, 0.88] for all seven english word embeddings using bellezza's lexicon (vocabulary size of 399 6 ) as the target word set. the corresponding p-values have a spearman's correlation coefficient greater than ρ ≥ 0.99 to the effect sizes, indicating statistically significant results. valnorm performance. figure 1 compares the performance of valnorm using three valence lexica to five word similarity tasks and one analogy task. valnorm using bellezza's lexicon overperforms all other intrinsic evaluation tasks on word embeddings trained on five corpora via four algorithms. widely shared associations. we compute the variance (σ 2 ) of the effect sizes for the flowersinsects-attitude, instruments-weapons-attitude, and gender-science weat bias tests across all seven language word embeddings. in ## discussion valnorm as a new intrinsic evaluation task. figure 1 compares our three implementations of valnorm to six traditional intrinsic evaluation tasks, with bellezza's lexicon performing the highest, most likely because it is designed specifically to measure valence norms and it is smaller than the other valence lexica. valnorm computes an effect size rather than just the cosine similarity, the metric for word similarity and word analogy tasks. notably, valnorm using bellezza's lexicon (399 valence tasks) outperforms wordsim (353 similarity tasks). using anew (1,035 valence tasks) and warriner's lexicon (13,915 valence tasks), valnorm consistently outperforms simlex (999 similarity tasks), rw (2,034 similarity tasks), and simverb (3,5000 similarity tasks). these results suggest that valnorm measures va-lence accurately and consistently, regardless of the task size, whereas results of all other intrinsic evaluation tasks have high variance and lower accuracy. valnorm's performance supports our hypothesis that valence norms are captured by word co-occurrence statistics and that we can precisely quantify valence in word embeddings. ## conclusion valence norms reflect widely-shared associations across languages and time, offering a distinction between non-social group biases and social group biases (gender, race, etc.). these valence associations are captured in word embeddings trained on historical text corpora and from various languages. we document widely-shared non-social group associations as well as culture-specific associations via word embeddings. while the social group biases we measure vary, we find that non-social group valence norms are widely-shared across languages and cultures and stable over 200 years. we present valnorm as a new intrinsic evaluation task which measures the quality of word embeddings by quantifying the preservation of valence norms in a word embedding set. valnorm, which has three implementations with increasing vocabulary sizes, outperforms traditional intrinsic evaluation tasks and provides a more sensitive evaluation metric based on effect size, as opposed to the cosine similarity metric of other evaluation tasks. computationally quantifying valence of words produce a high correlation to human-rated valence scores, indicating that word embeddings can measure semantics, particularly valence, with high accuracy. the results of valence norms as statistical regularities in text corpora provides another layer of transparency into what word embeddings are learning during their training process.