Political Ideology and Polarization: A Multi-dimensional Approach (2022)

Abstract: Analyzing ideology and polarization is of critical importance in advancing our grasp of modern politics. Recent research has made great strides towards understanding the ideological bias (i.e., stance) of news media along the left-right spectrum. In this work, we instead take a novel and more nuanced approach for the study of ideology based on its left or right positions on the issue being discussed. Aligned with the theoretical accounts in political science, we treat ideology as a multi-dimensional construct, and introduce the first diachronic dataset of news articles whose ideological positions are annotated by trained political scientists and linguists at the paragraph level. We showcase that, by controlling for the author’s stance, our method allows for the quantitative and temporal measurement and analysis of polarization as a multidimensional ideological distance. We further present baseline models for ideology prediction, outlining a challenging task distinct from stance detection.

## introduction political ideology rests on a set of beliefs about the proper order of a society and ways to achieve this order @xcite @xcite . in western politics, these worldviews translate into a multi-dimensional construct that includes: equal opportunity as opposed to economic individualism; general respect for tradition, hierarchy and stability as opposed to advocating for social change; and a belief in the un/fairness and in/efficiency of markets @xcite . the divergence in ideology, i.e., polarization, is the undercurrent of propaganda and misinformation @xcite @xcite . it can congest essential democratic functions with an increase in the divergence of political ideologies. defined as a growing ideo-* equal contribution ordered by first name. ## setup many political scientists and political psychologists argue for the use of at least a bidimensional ideology for domestic politics that distinguishes between economic and social preferences @xcite @xcite . 3 we start with these two dimensions while adding a third dimension, "foreign", when the article tackles foreign issues. specifically, our annotation task entails examining a news article and annotating each dimension (detailed below) along three levels-liberal, conservative, neutral-for each paragraph. the neutral level for every dimension is reserved for paragraphs related to a specific dimension but either (a) contain both conservative and liberal elements that annotators were unable to ascertain an ideological dimension with confidence, or (b) do not portray any ideology. we additionally provide an irrelevant option if a dimension does not apply to the paragraph. the three dimensions are: social: while the (1) socially conservative aspect of this dimension is defined as respect for tradition, fear of threat and uncertainty, need for order and structure, concerns for personal and national security, and preference for conformity, its (2) socially liberal counterpart has been associated with a belief in the separation of church and state, tolerance for uncertainty and change @xcite . economic: similarly, while the (3) economically conservative aspect of this dimension refers to motivations to achieve social rewards, power, and prestige such as deregulation of the economy, lower taxes and privatization (i.e., being against deficit) spending and advocating for a balanced budget, its (4) economically liberal counterpart refers to motivation for social justice and equality such as issues related to higher taxes on rich individuals and businesses and more redistribution. foreign: after piloting the bidimensional approach on 300 articles, we find that using only 2 dimensions conflates two important aspects of ideology related to domestic economy and foreign trade. tariffs, import quotas, and other nontariff-based barriers to trade that are aimed at improving employment and the competitiveness of the us on the international market did not map well onto the bidimensional framework. after consulting several senior political scientists, we adopted a third dimension that dealt with the markets as well as the relations of the us with the rest of the world. while morals, values and traits such as freedom, safety, harm, care, reciprocity, in-group loyalty, authority, equality are formed. since, some scholars have used these traits to predict ideology whereas others have attempted to understand what traits unites people with the same ideology. (3) framing: frames are used in many ways in political science. they can refer to different ways scholars describe the same information or when scholars talk about different aspects of a single problem @xcite . the (5) globalist counterpart of this dimension accounts for free-trade, diplomacy, immigration and treaties such as the non-proliferation of arms, its (6) interventionist aspect is nationalist in its support for excise tax on imports to protect american jobs and economic subsidies and anti-immigration. with the annotated data, we demonstrate quantitative measures of polarization (section 5) and introduce the modeling task (section 6) of automatically identifying the ideology of the policy positions being discussed. ## data collection and annotation raw data since polarization is a process that needs to be analyzed over time @xcite , our annotated articles are sampled from a diachronic corpus of 1,749 news articles across nearly 3 decades @xcite . articles in this corpus are from political news articles of @xcite from the corpus of historical american english (coha, davies (2012)) covering years 1922-1986. these 1,749 articles are extracted such that: (1) they cover broad and politically relevant topics (ranging from education and health to economy) but still share discussions related to the federal budget to make our annotations tractable foot_3 ; (2) balanced in the number of articles across 5 news outlets with center-left, central, and center-right ideology (c.f. section 5): chicago tribune (ct), wall street journal (wsj), christian science monitor (csm), the new york times (nyt), and time magazine (tm). a detailed description of our curation process is in appendix a. the raw texts were not segmented into paragraphs, thus we used topic tiling @xcite for automatic segmentation. topic tiling finds segment boundaries again using lda and, thus, identifies major subtopic changes within the same article. the segmentation resulted in articles with 1 to 6 paragraphs. the average number of paragraphs per article was 4. annotation process our team (including a political science graduate student) developed an annotation protocol for expert annotators using definitions in section 2. the annotation process is independently reviewed by four political science professors from two universities in the us who are not authors two dimensions: socially and economically liberal ... secretary of defense robert s. [mc-namara] threw his full support today behind the administration's drive against poverty. ...mr. mc-namara said : "it is the youth that we can expect to be the most immediate beneficiaries of the war on poverty." he said he was endorsing the "entire program" both as a citizen and as a member of the cabinet. his endorsement came as his fellow republicans in congress continued to hammer away at parts of the administration's antipoverty program. . . . two dimensions: socially and economically conservative the antipoverty program, the republicans insisted, would undercut the authority of the cabinet members by making sargent shriver a "poverty czar." "i don't see how you can lie down and be a doormat for this kind of operation. "... we sampled on average 7.86 articles per year for annotation, for a total of 721 paragraphs across 175 articles. we divided the annotation task into two batches of 45 and 130 articles, the smaller batch was for training purposes. in addition to the political science graduate student, we recruited three annotators, all of whom are recent linguistics graduates in the us. the training sessions consisted of one general meeting (all annotators met) and six different one-on-one meetings (each annotator met with another annotator once). during initial training, the annotators were asked to highlight sentences based on which the annotation was performed. after the annotations of this batch were finalized, the annotators met with the political science student to create ground truth labels in cases of disagreement. then, the three annotators received the second batch and each article was annotated by 2 annotators. this annotation was composed of two stages to account for possible subjectivity. in stage 1, each annotator worked on a batch that zero dimension . . . "the committee is holding public hearings on president eisenhower's economic report, which he sent to congress last week. the secretary's [humphrey] appearance before the group provided an opportunity for political exchanges. one dimension: economically liberal senators paul h. douglas of illinois, j. w. fulbright of representative wright patman of texas, all democrats, were active in questioning mr. humphrey. the democrats asserted that the administration's tax reduction program was loaded in favor of business enterprises and shareholders in industry and against the taxpayer in the lowest income brackets. . . . one dimension: economically neutral senator fulbright .. declare[d] that the problem was to expand consumption rather than production. ... "production is the goose that lays the golden egg," mr. humphrey replied. "payrolls make consumers." agreement to assess the inter-annotator agreement of stage 1, we report krippendorf's α (hayes and krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39). the agreements among annotators for the economic and foreign dimensions are moderate & substantial @xcite , respectively; for social, the 'fair' agreement was noticed during annotation, and additional discussion for each paragraph was then held. afterwards, 25 more articles were independently annotated and assessed with an α of 0.53. although the agreements were not perfect and reflected a degree of subjectivity in this task, all dimensional labels were adjudicated after discussions between annotators. in total, creating this dataset cost ∼150 hours of manual multi-dimensional labeling. qualitative analysis of text highlights for the 25 articles used in training, all annotators highlighted the sentences that are relevant to each dimension they annotated. this helped annotators to focus on the sentences that drived their decision, and provided insights to the language of ideology, which we discuss here. on average, 21%-54% of the sentences in a paragraph were highlighted. we found entities such as "president" and "congress" were the most prevalent in the highlights, and they tackled social and economic issues combined. this is not surprising as it suggests that when the media quotes or discusses the "president" and "congress", they do so with reference to more complex policy issues. in contrast, individual congresspeople tackled mostly economic or social issues. this also is not surprising as it suggests that individual congresspeople are more concerned with specific issues. interestingly, "house" and "senate" almost always figured more in social issues. this suggest that when news media speaks about a specific chamber, they do so associating this chamber with social issues. finally, party affiliation was infrequent and was mostly associate with social issues. ## ideology analyses the number of paragraphs per dimension in total is: economics (558), social (291), forign (338), across the 175 articles. in we also analyze the percentage of articles that contain at least one pair of paragraph labels that lean in different directions; for instance, a paragraph with a label of globalist (i.e., liberal) in the foreign dimension and another paragraph with a label of conservative or neutral in the fiscal dimension. the percentage of such articles is 78.3%. out of these articles, we examine the average proportion of neutral, liberal, and conservative paragraph labels, and find neutral labels have the highest share (43.27%), followed by liberal (33.20%) and conservative (23.53%). in figure 2 (right), we visualize the percentage of articles where two dimensional labels co-occur within the same article. the figure indicates that ideology varies frequently within an article, showing that a single article-level label will not be fine-grained enough to capture variances within an article. in figure 2 (left) we also show paragraph-level label co-occurrence. unlike the article-level, the co-occurrences are less frequent and we are more likely to observe co-occurrences along the same side of ideology. still, we see interesting nuances; for example, on both the paragraph and the article level, the economic dimension is often neutral, and this tends to co-occur with both liberal and conservative positions in other dimensions. ## polarization in this section, we demonstrate how our framework can be used to analyze ideological polarization, quantitatively. to say that two groups are polarized is to say that they are moving to opposite ends of an issue on the multi-dimensional ideological spectrum while, at the same time, their respective political views on ideological issues converge within a group, i.e. socially liberals become also economically liberal @xcite . in political science when ideology is multidimensional, polarization is often quantified by considering three measures that capture complementary aspects @xcite : (1) sorting @xcite (the extent to which the annotated ideology deviates from an outlet's proclaimed ideological bias); (2) issue constraint @xcite (a correlational analysis between pairs of ideological dimensions); (3) ideological divergence (fiorina and abrams, 2008) (the magnitude of the distance between two groups along a single dimension). together these measures describe changes in the ideological environment over time: a concurrent increase in all three measures indicates polarization in media. limitations: we use only the fully adjudicated data and refrain from using model predictions, since our baseline experiments in section 6 show that predicting ideology is challenging. hence, the analysis are demonstrations of what our framework enables, which we discuss at the end of this section, and the conclusions are drawn for our annotated articles only. we group our data in four-year periods to reduce sparsity. measure 1: sorting we adapt the sorting principle of @xcite to our data and investigate the difference between the proclaimed ideological bias of a news outlet and the ideology of annotated articles from the outlet. to obtain the bias b j of a news outlet j, we average the ratings of each news outlet across common sites that rates media bias (adfontes, allsides, and mbfc), yielding: csm (-0.07), ct (0.15), nyt (-0.36), tm (-0.4), wsj (0.32) (c.f. to obtain the overall ideology i (j) i of article i from outlet j, we take the average of liberal (-1), neutral (0), and conservative (1) labels across its paragraphs in all three dimensions. thus, for each 4-year time period with m articles for outlet j, the sorting measure would be the absolute distance of article vs. outlet ideology |avg m @xmath0 in figure 3 , we plot the sorting measure, averaged across news outlets of the same proclaimed ideological bias. the figure shows that in our sample of articles, the left-leaning news outlets were closest to their proclaimed ideological bias measure over time, whereas the neutral outlets were more liberal before 1957 and after 1964. the rightleaning outlets were more conservative at that time than their proclaimed ideological bias. measure 2: issue constraint this measure refers to the tightness between ideological dimensions over time @xcite so as to assess, for example, if socially liberal dimensions are more and more associated with economic liberal dimensions for the news outlets. concretely, for each article we derive its ideology along a single dimension as the average of paragraph annotations along that dimension. we, then, calculate the pearson correlation between the article ideology of each pair of dimensions, over all articles from one outlet in the same period. results in figure 4 , again averaged across news outlets of the same ideological bias, show that for right-leaning media, the correlation between any two dimensions in the annotated data are largely positive (e.g., economically conservative were also socially conservative) until 1967 or 1970. however, for proclaimed left-leaning and neutral outlets, the correlations fluctuates especially when considering the foreign dimension. measure 3: ideological divergence this measures the distance between two ideological groups on a single dimension @xcite . we follow @xcite and calculate the bimodality coefficient @xcite per dimension over articles from all news outlets over the same time period. the bimodality coefficient ranges from 0 (unimodal, thus not at all polarized) to 1 (bimodal, thus completely polarized). figure 5 shows the evolution of the ideological divergence measure of every dimension. a bimodality measure assesses whether this divergence attained the threshold for the cumulative distribution to be considered bimodal. ideological distance, as a result, refers to the three bimodal coefficients. we note, for example, that the foreign dimension crossed this threshold between 1956 and 1968. this means that proclaimed left-leaning and right-leaning outlets grew further apart on foreign issues during this time period. discussion taken together, the graphs indicate that the years between 1957 and 1967 are the most noteworthy. during this period, from our sample of articles, we see that polarization was only present in conservative news media because it (1) sorted, as it was significantly more conservative than its composite bias measure, (2) constrained its issues, as evidence by high positive correlation values, and (3) became increasingly bimodal, as the ideological distance between their positions and those of their liberal counterpart on foreign issues increased over time. while this conclusion applies to only the set of articles in our dataset, the above analysis illustrates that our framework enables nuanced, quantitative analyses into polarization. we leave for future work, potentially equipped with strong models for ideology prediction, to analyze the data at scale. ## experiments we present political ideology detection experiments as classification tasks per-dimension on the paragraph level. we performed an 80/10/10 split to create the train, development, and test sets. the development and test sets contain articles uniformly distributed from our time period @xcite ) such that no particular decade is predominant. to ensure the integrity of the modeling task, all paragraphs belonging to the same article are present in a single split. the number of examples in the splits for each dimension for the adjudicated data are as follows: for the economic dimension, we had 450 training, 50 development, and 58 test examples. for the social dimension, we had 253 for training, 13 for development, and 25 for testing. for the foreign dimension, we had 266 for training, 33 for development, and 39 for testing. ## related work in contrast to our multi-dimensional approach that examines the ideology of the issue being discussed instead of the author stance, much of the recent work in computational linguistics has been dedicated to the latter (detection of ideological bias in news media) while collapsing ideology to one dimension @xcite @xcite @xcite @xcite . the proposed computational models classify the partiality of media sources without quantifying their ideology @xcite . other researchers interested in the computational analysis of the ideology have employed text data to analyze congressional text data at the legislative level @xcite and social media text at the electorate level @xcite . in political science, the relationship between (news) media and polarization is also an active area of research. prior work has studied media ideological bias in terms of coverage @xcite . @xcite argues there is no firm evidence of a direct causal relationship between media and polarization and that this relationship depends on preexisting attitudes and political sophistication. on the other hand, @xcite have established that polarization language snippets move from the legislature in the direction of the media whereas @xcite have shown that the media has an impact on agenda settings of legislatures. ## conclusion we take the first step in studying multi-dimensional ideology and polarization over time and in news articles relying on the major political science theories and tools of computational linguistics. our work opens up new opportunities and invites researchers to use this corpus to study the spread of propaganda and misinformation in tandem with ideological shifts and polarization. the presented corpus also provides the opportunity for studying ways that social context determines interpretations in text while distinguishing author stance from content. this work has several limitations. we only focus on news whereas these dynamics might be different in other forms of communication such as social media posts or online conversations, and the legislature. further, our corpus is relatively small although carefully annotated by experts. future work may explore semi-supervised models or active learning techniques for annotating and preparing a larger corpus that may be used in diverse applications.