SWEAT : Scoring Polarization of Topics across Different Corpora (2021)

Abstract: Understanding differences of viewpoints across corpora is a fundamental task for computational social sciences. In this paper, we propose the Sliced Word Embedding Association Test (SWEAT), a novel statistical measure to compute the relative polarization of a topical wordset across two distributional representations. To this end, SWEAT uses two additional wordsets, deemed to have opposite valence, to represent two different poles. We validate our approach and illustrate a case study to show the usefulness of the introduced measure.

## introduction in this short paper, we introduce a method to score polarization of different corpora with respect to a given topic. the method is intended to support studies where two different corpora are compared (e.g., news sources inspired by different political positions or social communities characterized by different viewpoints) to investigate whether they convey implicit attitudes towards a given topic. this corpus-wise comparison -its main peculiarity with respect to the large body of work proposed to study and correct bias in nlp models (we refer to @xcite for a detailed survey on bias in nlp models) -is based on a new measure that we introduce, the sliced word embedding association test @xcite . sweat is an extension of the word embedding association test (weat) proposed by @xcite , which measures the comparative polarization for a pair of topical wordsets (e.g., insects and flowers) against a pair of attribute wordsets (e.g., pleasant and unpleasant) in a single-corpus distributional model (e.g. 1950 american newspaper articles). in this context, with polarization we refer to the phenomenon for which two communities have opposite attitudes against some topic. with sweat we extend this approach by measuring the relative polarization for a single topical wordset -the topic, using a pair of stable attribute wordsets deemed to have opposite valence -the two poles, in a pair of aligned distributional models representing the semantics of two different corpora. we explain the rationale behind sweat with an example. suppose that we want to investigate whether two different italian news sources, e.g., la repubblica (known to be closer to center-left political positions) and il giornale (known to be closer to center-right political positions) hold different and opposite viewpoints about a topic, e.g., "berlusconi" (a reference center-right italian politician in the recent past). we can collect a news corpus from la repubblica and one from il giornale to train two different distributional models in such a way that they are aligned @xcite @xcite . we expect that some words have stable meanings while other change across corpora reflecting the different viewpoints. we can then select a set of words describing the "berlusconi" topic, whose representations are expected to differ across corpora, and two wordsets having respectively positive and negative valence (the two poles), whose representations are expected to be stable across corpora. the main idea behind sweat is the following: if the two corpora hold polarized views about the topic, the "berlusconi" wordset will be associated more strongly with the positive rather than with the negative pole in one corpus (il giornale), while the opposite association will hold in the other corpora (la repubblica). sweat measures this difference and reports effect size and significance. contributions. we introduce the sweat, a novel statistical measure to study relative polarization in distributional representations. we additionally introduce a lexicon selection pipeline and an easy-to-use code to create visualizations. we believe our measure can be useful for different use cases in the computational social science field. we share a repository with an easy to use implementation of our measure. foot_0 2 background: weat @xcite introduce the word embedding association test (weat) to test whether distributional representations exhibited the same implicit biases detected in social sciences studies through behaviorally-measured word associations @xcite . the weat compares the relative associations of a pair of target concepts x and y (e.g., science and arts) to a pair of attribute concepts a and b (e.g., male and female) in a distributional vector space e; x, y , a, and b are all sets that contain representative words for the concept. the statistical measure is based on the following formula: the value s(w, a, b) is instead computed as: where the effect size is defined as: significance is computed through a permutation test @xcite over the possible partition of equal size for the union of target-wordsets p [x ∪ y ] = {(x i , y i )} i . the p-value is computed as the rate of scores, from all possible permutations, that are higher than the tested one: depending on the sign of the score the association could be either x ∼ a, y ∼ b for positive scores and x ∼ b, y ∼ a for negative ones. where the ∼ indicates semantic association. ## sweat pipeline our sweat pipeline is composed of two main components: a statistical test to evaluate relative polarization and a lexicon refinement pipeline -based on aligned word embeddings -to help researchers select the pole wordsets. ## validation on the reddit corpus we validated the sweat measures using subcorpora from the reddit corpus. 2 the corpus, previously used by @xcite , contains all the posts on the social media platform reddit for the year 2014. we selected for this experiment the english boards anarchocapitalism-frugal and baby-bumps-childfree. the first pair is centered around topics of finance and capitalism, with anarchocapitalism (ac) expected to have positive polarization, due to its pro-capitalism beliefs, and frugal (fr) negative. the second pair is focused on parenting (i.e. having children), with babybumps (bb) expressing positive opinions about this matter and child-free (cf) being negatively polarized. we do pairwise comparisons using the sweat measure. the embeddings are generated and aligned using twec @xcite . lexicon and wordset selection. for the polewordsets, we used the inquirer sentiment lexicon @xcite that was selected for its large size and generality. we used the pipeline described in the above section as a filter. the topical wordsets for each pair, ac-fr and bb-cf, were selected in a semiautomatic fashion (see appendix) in order to manually remove ambiguous terms which could bias the final result (e.g., the term fine is polysemous denoting both a financial penalty and good, and thus had to be removed from both pole and topical wordsets to prevent undue associations). the two derived topic wordset for capitalism and parenting are presented below: capitalism: market, value, economic, capital, price, wealth, profit, companies, interest, cost, competition, trade parenting: baby, birth, child, daughter, family, father, kid, mother, pregnant, son, parent, children results and visualizations figure 1 presents the four cumulative visualizations illustrating the four explored relations. the figure shows two centered horizontal stacked barcharts, one for each aligned model e 1 , e 2 . the two significant polarizations can be seen on the main diagonal: in the top left for the capitalism topic the fr corpus can be seen being strongly negative while the ac one has an overall positive but more nuanced position; in the bottom right, the polarization difference over the parenting topic is much more pronounced, with cf carrying a strong negative polarization in stark contrast with the positive one expressed by bb, thus confirming our hypotheses of polarization of the subreddits. ## case study on italian newspapers we provide an analysis on silvio berlusconi that focuses on the homonymous politician who, among other things, is the indirect owner of the italian il giornale (gio) newspaper and is, as an exponent of the center-right coalition, seldom praised by more left-leaning newspaper such as la repubblica (rep). the dataset (see appendix) contains a collection of roughly 40 thousand articles in italian from march 2018 to june 2019 for rep and gio. as a lexicon we used sentix @xcite . only the elements meeting all the following empirical criteria were considered in the final lexicon: pos tagging as either adjective or verb; intensity equal or greater to 0.75; either positive or negative score equal or greater to 0.75; polarity equal @xmath0. we then applied the lexicon filtering pipeline. we conducted the analysis on the topic silvio berlusconi manually selecting the following topical-wordset: berlusconi: cavaliere, berlusconi, arcore, mediaset, fininvest, silvio, rete4, fi, pdl, iene, vespa, tg5 the polarization detected by the sweat analysis confirms a significantly more positive polarization in the gio corpus and a more negative one in rep (@xmath1.795, @xmath2.3276, p-@xmath3.0038). figure 2 shows the results of this association and confirms our general intuition. ## limitations the sweat is able to capture relative semantic polarizations of topic wordsets. however, the measure is not suitable for fully unsupervised datadriven inference on the polarization of a corpora collection, similarly to the weat measure. the measure results are dependent on the choice of topic wordsets, and it is possible to find that some wordset is significantly polarized even though its elements are not semantically related to each other. for this reason, it is advised that researchers exercise caution in selecting which words to include in the topic wordset, and which to exclude. ## related work there has been great attention to the problem of polarization and bias in word embeddings in the latest years. we refer the reader to @xcite for relevant works. sweat and the applications we have shown have also a strong connection with the body of work related to the detection of hyper-partisan media @xcite , where extreme left-wing or right-wing positions of the news have to be detected. we want to mention that the introduction of the weat measure has have been of great influence on the community. indeed recently, @xcite extended weat to have specific support for gendered languages (where different terms can have male and female counterparts). @xcite use the weat measure to show how different domains contain different biases. instead, @xcite propose xweat as a cross-lingual version of the weat measure. ## conclusions we have described a new measure to compute relative polarization on corpus-specific distributional representations. measuring implicit attitudes is a crucial endeavor in social sciences, and in particular social psychology @xcite . recent developments have shown that such attitudes are not only observable in human behavior, but can also be captured via text analyses @xcite . sweat brings a further advancement in this research line by allowing to validate hypotheses on the polarization of content from different text sources.