POLITICS : Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection (2022)

Abstract: Ideology is at the core of political science research. Yet, there still does not exist general-purpose tools to characterize and predict ideology across different genres of text. To this end, we study Pretrained Language Models using novel ideology-driven pretraining objectives that rely on the comparison of articles on the same story written by media of different ideologies. We further collect a large-scale dataset, consisting of more than 3.6M political news articles, for pretraining. Our model POLITICS outperforms strong baselines and the previous state-of-the-art models on ideology prediction and stance detection tasks. Further analyses show that POLITICS is especially good at understanding long or formally written texts, and is also robust in few-shot learning scenarios.

## introduction ideology is an ubiquitous factor in political science, journalism, and media studies @xcite @xcite . decades of work has gone into measuring ideology based on voting data @xcite , survey results @xcite @xcite , social networks @xcite , campaign donation records @xcite , and textual data @xcite @xcite . each of those approaches has its strengths and weaknesses. for instance, many political figures do not have voting records; surveys are expensive and politicians are often unwilling to disclose ideology. by contrast, political text is abundant, ubiquitous, yet challenging to work with since language is complex in nature, often domainspecific, and generally unlabeled. there thus remains a strong need for general-purpose tools for measuring ideology using text that can be applied across multiple genres. news story: donald trump tests positive for covid-19. daily kos (left): it's now clear that donald trump lied to the nation about when he received a positive test for covid-19. . . . they're continuing to act as if nothing has changed-and that disregarding science and lying to the public are the only possible strategies. the washington times (right): trump says he's "doing very well" . . . president trump thanked the nation for supporting him friday night as he left the white house to be hospitalized for covid-19. "i want to thank everybody for the tremendous support. . . ." mr. trump said in a video recorded at the white house. breitbart (right): president donald trump thanked americans for their support on friday as he traveled to walter reed military hospital for further care after he was diagnosed with coronavirus. "i think i'm doing very well. . ." trump said in a video filmed at the white house and posted to social media. using text as data, computational models for ideology measurement have rapidly expanded and diversified, including classical machine learning methods such as ideal point estimation @xcite , naive bayes @xcite , support vector machines @xcite , latent variable models @xcite , and regression @xcite ; and more recent neural architectures like recurrent neural networks @xcite and transformers @xcite . nonetheless, most of those models leverage datasets with ideology labels drawn from a single domain, and it is unclear if any of them can be generalized to diverse genres of text. trained on massive quantities of data, pretrained language models (plms) have achieved state-ofthe-art performance on many text classification problems, with an additional fine-tuning stage on labeled task-specific samples @xcite . though plms suggest the promise of generalizable solutions, their ability to acquire the knowledge needed to detect complex features such as ideology from text across genres remains an open question. plms have been shown to capture linguistic structures with a local focus, such as task-specific words, syntactic agreement, and semantic compositionality @xcite . although word choice is indicative of ideology, ideological leaning and stance are often revealed by which entities and events are selected for presentation @xcite @xcite , with the most notable strand of work in framing theory @xcite . one such example is demonstrated in figure 1 , where daily kos criticizes trump's dishonesty while the washington times and breitbart emphasize the good condition of his health. in this work, we propose to train plms for a wide range of ideology-related downstream tasks. we argue that it is critical for plms to consider the global context of a given article. for instance, as pointed out by @xcite , one way to acquire such context is through comparison of news articles on the same story but reported by media of different ideologies. given the lack of suitable datasets, we first collect a new large-scale dataset, bignews. foot_0 it contains 3,689,229 english news articles on politics, gathered from 11 united states (us) media outlets covering a broad ideological spectrum. we further downsample and cluster articles in bignews by different media into groups, each consisting of pieces aligned on the same story. the resultant dataset, bignewsalign, contains 1,060,512 stories with aligned articles. next we train a new plm, politics, based on a pretraining objective leveraging inter-article triplet-loss using ideological content and story. concretely, we leverage continued pretraining @xcite , where we design an ideology objective operating over clusters of same-story articles to compact articles with similar ideology and contrast them with articles of different ideology. the learned representation can better discern the embedded ideological content. we further enhance it with a story objective that ensures the model to focus on meaningful content instead of overly relying on shortcuts, e.g., media boilerplate. both objectives are used together with our specialized masked language model objective that focuses on entities and sentiments to train politics. our main goal here is to create general-purpose tools for analyzing ideological content for researchers and practitioners in the broad community. furthermore, when experimenting on 11 ideology prediction and stance detection tasks using 8 datasets of different genres, including a newly collected dataset from allsides, politics outperforms both a strong svm baseline and previous plms on 8 tasks. notably, politics is particularly effective on long documents, e.g., achieving 10% improvements on both ideology prediction and stance detection tasks over roberta @xcite . we further show that our model is more robust in setups with smaller training sets. ## related work ideology prediction is a critical task for quantitative political science @xcite @xcite . both classical methods (e.g., naive bayes, svm; @xcite @xcite and deep learning models (e.g., rnn; @xcite have been used to predict ideology on a variety of datasets where ideology labels are available, such as legislative speeches @xcite and u.s. supreme court briefs @xcite . notably, @xcite pretrains a transformer-based language generator to minimize the ideological bias in generated text. as generative models are not as effective as masked language models (mlms) at text classification, our goal differs in that we train mlms to recognize ideological contents in various domains and tasks. stance detection is a useful task for ideology analysis because co-partisans are generally positive towards each other and negative towards counterpartisans @xcite . there has been a large body of work on identifying individuals' stances towards specific targets from the given text @xcite @xcite . on the methodology side, @xcite and @xcite apply statistical models, e.g., svm, with handcrafted text features. neural methods have also been widely investigated, including cnn @xcite , lstm @xcite , hierarchical networks @xcite , and representation learning @xcite . the washington post; nyt: the new york times; twt: the washington times. additional statistics of raw data size before downsampling and the corresponding publish dates can be found in table a4 . plms for predicting stances, e.g., incorporating extra features @xcite . kawintiranon and singh (2021) share a similar spirit with our work by upsampling tokens to mask. however, they pre-define a list of tokens customized for the given targets, which is hard to generalize to new targets. we aim to train plms relying on general-purpose sentiment lexicons and important entities, to foster model generalizability. ## politics via continued pretraining here we introduce our continued pretraining methods based on a newly proposed ideology objective that drives representation learning to better discern ideological content by comparing same-story articles ( §4.1), which is further augmented by a story objective to better focus on content. they are combined with the masked language model objective, which is tailored to focus on entities and sentiments ( §4.2), to produce politics ( §4.3). ## experiments given the importance of ideology prediction and stance detection tasks in political science @xcite @xcite , we conduct extensive experiments on a wide spectrum of datasets with 11 tasks ( §5.1). we then compare with both classical models and prior plms ( §5.2), and among our model variants ( §5.3). we present and discuss results in §5.5, where politics outperform all three baselines on 8 out of 11 tasks. for all models, mlm objectives are trained with bignewsbln, and ideology and story objectives are trained on bignewsalign. details are in appendix c.1. ## conclusion we study the problem of training general-purpose tools for ideology content understanding and prediction. we present politics, trained with novel ideology-driven pretraining objectives based on the comparisons of same-story articles written by media outlets of different ideologies. to facilitate model training, we also collect a large-scale dataset, bignews, consisting of news articles of different ideological leanings. experiments on diverse datasets for ideology prediction and stance detection tasks show that politics outperforms strong baselines, even with a limited amount of labeled samples for training, and state-of-the-art models. as well as the intellectual property and the privacy rights of the original authors of the texts, i.e., source owners. during data collection, the authors honored privacy rights of content creators, thus did not collect any sensitive information that can reveal their identities. all participants involved in the process have completed human subjects research training at their affiliated institutions. we also consulted section 107 foot_8 of the u.s. copyright act and ensured that our collection action fell under the fair use category.