Distributional properties of political dogwhistle representations in S wedish BERT (2022)

Abstract: “Dogwhistles” are expressions intended by the speaker have two messages: a socially-unacceptable “in-group” message understood by a subset of listeners, and a benign message intended for the out-group. We take the result of a word-replacement survey of the Swedish population intended to reveal how dogwhistles are understood, and we show that the difficulty of annotating dogwhistles is reflected in the separability in the space of a sentence-transformer Swedish BERT trained on general data.

## introduction we explore whether contemporary vector-space sentence representation techniques also provide a structured representation of the different messages in "dogwhistle" political communication. a dogwhistle refers to a word or phrase used in manipulative communication, usually in a political context. dogwhistles carry at least two messages: one message intended for the broader community, and another "payload" message intended to communicate a specific, less acceptable message to a receptive "in-group". dogwhistles depend on the "out-group" members not picking up on the payload message @xcite . we take several swedish-language dogwhistles and survey data from the swedish population about the interpretation of these dogwhistles, and we apply clustering techniques based on the transformerderived representation of the responses. we ask the question: are the responses clearly partitioned in the semantic space, and does the "sharpness" of this partitioning reflect the ease of dogwhistle identification by expert annotators? while there has been work exploring dogwhistles through the lens of linguistics (henderson 1 authors other than niclas hertzberg and asad sayeed are listed in ascending alphabetical order. and @xcite @xcite , automated approaches to exploring dogwhistles using nlp techniques are generally lacking @xcite . considering the volume of social media data and the extent to which dogwhistles have been employed on these channels, it is important to create new computational techniques to detect and analyze dogwhistles that might succeed at higher data volumes. the first step in accomplishing this is to show that automatic techniques can be used to reliably extend and enhance manual analysis. dogwhistles can be strategically used, e.g. politically to send a veiled message to one group of voters while avoiding alienating another group (bhat and @xcite . this could pose a problem in a representative democracy since the out-group portion of the voter-base are deceived into voting for a certain candidate that might not represent their political views @xcite . therefore, we contribute the following: we then conclude that a swedish bert variant already represents important aspects of the underlying semantics of dogwhistles. ## dataset dogwhistle politics has become increasingly salient in the current mass and social media environment. this is also the case in swedish society. recent studies have shown that certain issues, in particular immigration, have produced examples of emergent dogwhistles gaining in public use @xcite . using a professional polling firm, we anonymously sampled 1000 members of the swedish public using a word replacement task. we constructed 5 sentences containing words or phrases we suspected were being used as dogwhistles and asked survey participants to replace the words with what they thought it "really" meant. then we manually annotated these responses for whether they identified a dogwhistle use or not. the survey was conducted under institutional ethical review in a process that involved survey administration and anonymized data compilation at a remove from the authors. each item therefore contains the substitution of participant-provided words or phrases for the original dogwhistle in the full context of the corresponding stimulus sentence. an illustrative stimulus example would be the following: "the swedish unions are controlled by globalists". each person taking the survey would replace "globalists" with a word or phrase they believe to convey the same information. the replacements can vary widely: someone might replace "globalists" with "communists" or an anti-semitic slur, which might be considered an "in-group" response. others would replace "globalists" with, e.g., "people concerned with international affairs" thus not showing an understanding of the dogwhistle as having any associations with the aforementioned groups. the actual swedish dogwhistles we use and their english translations are listed in each replacement thus gave rise to a slightly altered sentence that, according to the person taking the survey, would convey the same information as the original sentence. the replacements for each dogwhistle was manually labeled depending on a person picking up on the dogwhistle meaning or not. an inter-annotator score was then calculated for the labeling of each dogwhistle. iaa was calculated in two rounds, an initial round and a confirmatory round partway through the annotation. we report both scores in role of inter-annotator agreement 2 the goal of the annotation and the computation of iaa is to determine whether or not the annotation task can be designed with the following criterion in mind: that a panel of trained annotators with access to the guidelines can reliably distinguish between participant responses that did pick up on the "ingroup" dogwhistle meaning from those that did not. the identification and interpretation of a dogwhistle is an inherently subjective task which stems directly from one of the reasons to use a dogwhistle in the first place: to take advantage of the ambiguity of interpretation based on the standpoint of the individual recipients of the message. there are good reasons to critique the widespread use of iaa statistics to represent reader or listener reaction in subjective tasks like these @xcite . however, in this case, the annotation guidelines were developed in an iterative process to be presented in future publications that ensured that swedish-speaking annotators informed about swedish politics could consistently identify the dogwhistle interpretations of survey participants. the focus of this work is to explore the extent to which the intuitions behind the annotation guidelines are reflected in a swedish bert model trained on a multi-genre corpus. ## experiment and analysis our main question: is there an easily detected separation between the in-group responses and the out-group responses in the representation space? if this was the case, it would mean that the model has picked up on some distinction between the responses that corresponds to the distinction made by the annotators. given the distance in the semantic space between the two groups, it should be possible to separate the space with a linear svm trained on a subset of the data. a further question is whether there is a correlation between the clusterings and the iaa scores? being able to linearly separate the two groups is a necessary but not sufficient condition for good clustering scores. the dogwhistle replacements might vary widely enough to not cluster well while still being separatable using a hyperplane to a high degree of accuracy. ideally, two differentiable dense clusters would correspond to the iaa. ## conclusions and future work our work contributes a computationally straightforward method to extend the manual analysis of dogwhistles that is available for many languages at a resource level similar to swedish. our evaluations show that easily identified dogwhistle interpretations are partitioned well enough in the vector space given by sota sentence models that they are linearly separable using a simple svm. the representation of sentences given by the model is largely derived from the corpora that the model is trained on. the corpora thus has a large impact on the semantic space. given this, models trained on different corpora would give rise to different semantic spaces where the clustering of the sentences would be different. since k-means does not seem to be able to differentiate between in-group sentence replacements and out-group sentence replacements, future work might include an investigation into modeling the semantic space by training a sentence transformer on different sources of text. this would also allow us to investigate the role of specific lexical choices in the detection and representation of dogwhistles. in theory, it should be possible to train a model that creates a semantic space that clusters the points in a way that that the labels can be retrieved by an algorithm like k-means using only the data itself.