One Word, Two Sides: Traces of Stance in Contextualized Word Representations (2022)

Abstract: The way we use words is influenced by our opinion. We investigate whether this is reflected in contextualized word embeddings. For example, is the representation of “animal” different between people who would abolish zoos and those who would not? We explore this question from a Lexical Semantic Change standpoint. Our experiments with BERT embeddings derived from datasets with stance annotations reveal small but significant differences in word representations between opposing stances.

## introduction our opinions are reflected in the way we talk. people with opposing stances on a particular topic may use different words when talking about it. for example, only people against the use of face masks during the covid-19 pandemic would sometimes refer to them as "muzzles". in this paper, however, we do not investigate what words are used by each side. instead, we compare how speakers who disagree on a subject use the same words. specifically, we want to know whether contextual models capture a difference between the representation of a word (e.g., "mask") when it is used by people who are in favor vs. against a certain target (e.g., the compulsory use of face masks). we address this question from the perspective of lexical semantic change (lsc). work on lsc typically tries to detect word meaning changes across two or more periods of time @xcite , but its techniques have also been employed to identify synchronic differences in word usage, for instance across different ages, genders, professions @xcite , domains @xcite , or cultures @xcite . as opposed to related studies that investigate lsc between different viewpoints @xcite figure 1 : example instances of "mask" from the covid19 stance dataset @xcite . we compare the within-and the between-stance usage similarity. 2021), our goal is not to explore the usage of specific words, and we do not evaluate our method based on the ranking of words by meaning stability. we rather want to determine whether vector representations reflect a higher similarity in word usage within a stance than between different stances (see example in figure 1 ). we explore this question relying on datasets annotated with stance information. before that, we test different context-sensitive embedding models on a simulated scarce-data setting. this allows us to select a robust representation type that can identify the words that are used most differently between stances. our long-term goal is to detect differences in word usage between speakers in a conversation, which could point to their level of conceptual alignment @xcite ; that is, the extent to which dialog participants "mean the same things when using the same words" @xcite . in this study we present a first step in this direction. representations that are sensitive to opinion differences could be useful to identify disagreements and misalignment in dialog. ## methodology in this section we introduce the data and the models used in our experiments. we also describe our similarity measure and the criteria for evaluation. foot_0 ## conclusion and future work we have shown that bert word representations are sensitive to the opinion expressed in the sentences they are derived from. differences in similarity found between concurring and conflicting stances are small, but significant; and words with the highest differences tend to be central to the topic. our approach can serve to identify points of discrepancy with regard to a target, and it can be useful for stance detection and debate analysis. our experiments on semcor provide valuable insight on the sufficient amount of word instances needed to obtain quality representations. this is relevant for low-resource lsc and, more generally, for inferring word vectors from little data. in future work, we plan to apply this methodology to dialog. sets p and q would each correspond to the utterances of one speaker in a conversation. the similarity measure would act as an approximation of the conceptual or stance alignment between the two participants, indicating whether speakers share opinions and use words in a similar way.