What Sounds “Right” to Me? Experiential Factors in the Perception of Political Ideology (2021)

Abstract: In this paper, we challenge the assumption that political ideology is inherently built into text by presenting an investigation into the impact of experiential factors on annotator perceptions of political ideology. We construct an annotated corpus of U.S. political discussion, where in addition to ideology labels for texts, annotators provide information about their political affiliation, exposure to political news, and familiarity with the source domain of discussion, Reddit. We investigate the variability in ideology judgments across annotators, finding evidence that these experiential factors may influence the consistency of how political ideologies are perceived. Finally, we present evidence that understanding how humans perceive and interpret ideology from texts remains a challenging task for state-of-the-art language models, pointing towards potential issues when modeling user experiences that may require more contextual knowledge.

## introduction social media companies, like twitter, facebook, and reddit, play an important role in political discourse by providing a space for users to interact with different viewpoints. understanding political discussion on these platforms often requires one to identify the ideologies behind texts, as understanding the viewpoints reflected in a text can provide insight into the partisanship of beliefs @xcite or the persuasive strategies used by different ideological groups @xcite . prior research on political discussion often relies on a "ground-truth" to aid in obtaining ideology labels for social media data. for example, due to the scale of political content on social media, a common paradigm is to obtain some ground-truth labels that are propagated to a larger set of texts using semi-supervised learning @xcite . the relationship between a social media artifact and various forms of established political knowledge can also be used to ground or validate ideology labels. some examples of this include using author interactions with politicians with known party affiliations @xcite , ideological communities @xcite , and central users @xcite as a starting heuristic, or evaluating a labeling approach by comparing geolocation tags attached to posts with historical voting patterns @xcite . a limitation of these approaches, however, is that behavior on social media does not evenly or uniformly reflect the held political beliefs of participants. while there is evidence that people tend to engage with others who share similar beliefs @xcite , people also commonly interact with or even seek out communities and users they do not agree with @xcite . additionally, the practice of displaying one's political beliefs, which many grounding techniques rely on, varies in prevalence across online communities @xcite @xcite . the concept of linguistic agency @xcite also challenges the idea that individual factors, such as ideology, are predictably presented in text. based on an author's social goals for participating in political discussion, it may not be contextually relevant to project a strong impression of their political ideology. people engaged in interactive political discussion, however, still form perceptions about the alignments of others based on how they sound, often relying on their own conceptions of ideology in the process. the issue of perceiving ideology also plays a role when ideology labels are obtained using crowdsourced annotators. while making judgments, the annotator plays a similar role to a user participating in the discussion when perceiving the ideology of the speaker behind a text. however, annotators are expected to assign an explicit ideology label to a text with less contextual knowledge about how the text was produced. thus, annotators may rely heavily on their own experiential factors, such as one's own beliefs or level of political engagement, when considering ideology. as a result, this process may introduce inconsistencies and biases in ideological labels used for political analysis. in this paper, we present an exploration of how experiential factors play a role in how annotators perceive ideology in text. building upon prior work investigating annotation bias @xcite @xcite @xcite , we construct an annotated corpus of posts from political subcommunities on reddit but incorporate additional contextual information about the annotators making ideology judgments. 1 while previous work @xcite has shown that source-side contextual features, such as user profiles and previous tweets, can influence label quality in stance annotation, we focus our analyses on contextual factors on the side of annotators. most similar to our work, @xcite and @xcite examine the impact of an annotator's identity and openness on their ability to accurately assess author attributes, including political orientation. in our work, however, we examine the impact of an annotator's political beliefs, knowledge, and reddit familiarity, on their judgments, using factors more specific to political participation on reddit. we additionally consider the issue of annotator bias in ideology labeling not as an issue of accuracy but rather an issue of social variability. under this view, we evaluate the performance of a state-of-the-art language model on its capacity to mirror different human perceptions of ideology to examine whether extralinguistic factors introduced through annotation may degrade model performance compared to other labels. ## dataset construction our dataset is drawn from the popular content aggregation and discussion platform reddit. political discussion on reddit is centered on subreddits, subcommunities centered on support for specific political candidates, organizations, and issues. for our analyses, we aim to label political distinctions on reddit along the left-right political spectrum in u.s. politics. using the monthly dumps from may to september 2019 from the reddit pushshift api @xcite , we collect all submissions and comments from the top political subreddits foot_1 by subscriber count. the collected subreddits were manually labeled as left or right, based on the subreddit description and top posts. we then select the top 12 left and top 12 right subreddits from the monthly dumps where discussion is primarily focused on u.s. politics. 3 the selected subreddits are shown in ## perceptions vs. heuristic labels prior work @xcite suggests that deep text classification models perform poorly when labels are influenced by extralinguistic contextual factors. while the semi-supervised labels that we generated are based on a behavioral heuristic outside of the text, our analyses of human judgments suggest that the annotation process introduced additional interactional factors into ideological labeling. we investigate whether these factors influence model performance by evaluating a bert-based @xcite model on its ability to match human judgments on the paired ideology ranking task. for our evaluation model, we finetune bertmask on the 24 subreddit corpus. next, for each text, we average its contextual embeddings in two ways: over (a) all tokens in the text and (b) all entity-related tokens in the text. we then concatenate the averaged embeddings, then use the resulting vector as input to a pairwise logistic regression model. for each annotator group, we use the majority answer for each question as the group label. ## conclusion and future work in this paper, we reconsider the idea of groundtruth labels of political ideology and investigate the impact of experiential factors on human perception of ideology in text. we construct and analyze an annotated corpus that incorporates experiential information about annotators, finding evidence that annotator backgrounds influence the consistency of political ideology judgments and that current classification models struggle to match human perceptions of ideology across different groups. from our analyses on factors contributing to variations in judgments, there is a greater need for targeted recruiting of annotators that are familiar with and contextualized to the domain being annotated. in future work, we aim to extend our investigation to examine how stylistic elements of text contribute to people's perception of political ideologies in interaction. these analyses may provide further insight into the effectiveness of political communication strategies or the differences in how political groups interact with in-group and out-group members.