Ideology Prediction from Scarce and Biased Supervision: Learn to Disregard the “What” and Focus on the “How”! (2023)

Abstract: We propose a novel supervised learning approach for political ideology prediction (PIP) that is capable of predicting out-of-distribution inputs. This problem is motivated by the fact that manual data-labeling is expensive, while self-reported labels are often scarce and exhibit significant selection bias. We propose a novel statistical model that decomposes the document embeddings into a linear superposition of two vectors; a latent neutral context vector independent of ideology, and a latent position vector aligned with ideology. We train an end-to-end model that has intermediate contextual and positional vectors as outputs. At deployment time, our model predicts labels for input documents by exclusively leveraging the predicted positional vectors. On two benchmark datasets we show that our model is capable of outputting predictions even when trained with as little as 5% biased data, and is significantly more accurate than the state-of-the-art. Through crowd-sourcing we validate the neutrality of contextual vectors, and show that context filtering results in ideological concentration, allowing for prediction on out-of-distribution examples.

## introduction political ideology prediction (pip) is motivated by a number of applications such as the need to understand government's policy making @xcite , a legislator's partisan/non-partisan actions @xcite @xcite , the general public's sentiment support for legislation @xcite @xcite etc. our overall goal is to infer ideology of legislators or the general public from written texts, media posts, and speeches. we can train a pip model with ground truth labels (e.g., liberal vs conservative) with a standard supervised learning approach. in some cases, such as for legislators, ground-truth labels can be inferred through party affiliation information, and these have served as good proxies for ideology @xcite . selection bias. in other situations, such as, for the domain of social media posts, ground truth is difficult to obtain, and is based on self-reported affiliation of the users @xcite . such selfreported affiliations are generally sparse, and when reported tend to be of extreme polarity @xcite . while manual labeling by annotators (e.g., amts) @xcite can be leveraged, selection bias is still an issue due to oversampling of posts from extreme vocal posters @xcite @xcite . justification. a black-box model trained exclusively on scarce and polarized group is likely to perform poorly on the under-observed moderates who are the majority and are reticent to disclose or discuss their politics online @xcite @xcite . inferring the majority's views is important to help us understand support for legislative/executive actions @xcite , or cultural group's real influence @xcite as in the recent kansas vote where policymakers over-estimated the public support for the abortion ban by overlooking the voice of silent/less vocal majority @xcite . proposed method. to account for scarcity and selection bias, we propose a novel method that, during training, enforces decomposition of text embedding into context and position, and trains a deep neural network model in a modified variational auto-encoder framework with bi-modal priors. contextual filtering. we propose to decompose document embeddings into a linear (orthogonal) superposition of two latent vectors consisting of a context vector and a position vector. we train a dnn model embedded with this decomposition to enforce the fact that context vectors be devoid of information that contains ideology, and the residual position vectors, obtained by filtering out the context, exclusively bears all ideological information. to ensure that trivial solutions are not found, we require that context vectors across all documents belong to a low-dimensional manifold. our perspective is that there are a sparse collection of common words (eg. guns, immigration, taxes, etc) whose neutral components serve to contextualize the document, and their corresponding embeddings constitute a low-dimensional space. de-noising. documents exhibit significant variance in scarce regimes, making it difficult to discern ideology. this is due to scarce document collections spread across diverse themes. position vectors, devoid of context, suppresses noise that accounts for large differences between document embeddings, but carries little relevance for pip, making learning representation of ideology easier. ## related work we describe prior works that focus on machine learning for predicting ideology from texts. supervised learning. @xcite propose a learning-based approach for ideology prediction on congressional report corpora. other studies on the same dataset apply modern learning methods under the same setting @xcite @xcite . research using social media data (tweets, forum posts) aims to map public users onto the liberal-conservative spectrum or simply predict their party labels @xcite . textual features. the majority of the aforementioned studies utilize off-the-shelf pretrained text representations such as bow, tf-idf, liwc, or word embeddings-glove, bert or roberta @xcite @xcite . scarce and biased data. the labels used as supervision are obtained from self-reported party affiliations (e.g., democrats or republicans) or manual labeling by annotators @xcite @xcite , and inherently suffer from label scarcity and selection bias. for example, the majority of the public are reticent to disclose their party affiliation or engage in political discourse online @xcite . furthermore, methods proposing to collect labeled texts to study opinions or ideology are prone to over-sampling the "vocal minority" while ignoring or down-sampling the "less vocal majority" @xcite @xcite . manual labeling is severely constrained by staffing and cognitive biases of the annotators themselves @xcite . finally, there are large domain differences between content generated by visibly opinionated and moderate users @xcite . for various reasons, models for learning ideology typically suffer from poor generalizablity @xcite @xcite . ## statistical model of text generation document decomposition. we first encode documents in a suitable embedding space (glove, roberta etc.), and let @xmath0, which serves as inputs to the learner. we decompose, x into two major components: low-dimensional and neutral encoding of context vectors. we proceed with the following intuition. for a certain choice of partisan theme θ i , say that there is a major polarization axis pa θ i depending on the theme, where people of different ideological groups (such as liberal or conservative, left or right) differentiate when giving speeches foot_0 . in principle we propose to seek orthogonality of context to the polarization axis pa. in practice, since pa is unobserved ex-ante, we enforce this constraint by careful initialization and empirically aligning position f with polarity during training, and we verify the orthogonality ex-post. initialization. to initialize context vectors, we adopt the following procedure. we first determine themes by generating a set of "neutral" seed words and phrases (hereafter, seeds), either through manual crafting or selecting frequent words with low χ foot_1 values (see appendix sec. a.4), and complement this set by expanding into the neighborhood of seeds, yielding a set of seeds for each theme i, a ij . themes are initialized as the pooled embedding of seed words t 0i = j a ij . context vectors are initialized as a mixture of themes, c 0 (θ 0 ) = t 0 θ 0 . ul-timately, both the theme choice, θ and the themes t are learned through training, starting with the initialization @xmath1.4 and a.5) and low-dimensionality results from the fact that the theme matrix is low-rank. this approach effectively removes variance that accounts for large differences between document embeddings, but carries little relevance for ideology learning and prediction, hence making learning representation of ideology easier. ## method mathematical background in addition to notation in sec. 3, we let @xmath2. let p s (x, y), p t (x, y), denote source and target joint distributions respectively (we drop these subscripts when clear from context). the learner gets access to a labeled set, d s of n s i.i.d data points (x i , y i ) ∼ p s and unlabeled set, d t , of n t i.i.d target input instances x i ∼ p t . inputs and outputs of model. the goal of the learner is to predict labels ŷ for the target given target inputs. additionally, our trained model also outputs for each input document, the context vector, c, the theme choices, θ, the position vector, f , the ideology vector, z. training loss. our loss function is a sum of reconstruction loss for inputs x @xmath3, and crossentropy loss, ce(y i , ŷ(z i ; γ)) on (x i , y i ) ∈ d s ; ŷ is the softmax output of a network governed by parameters γ, and taking the encoded latent representation z i (see below) as its input. reconstruction loss. our starting point is to optimize the marginal distribution, namely, e x∼q(x) log p(x), but as this is intractable, we fol-low by relaxing it to the elbo bound (kingma and welling, 2013): the first term in eq. 3 corresponds to the decoder that generates f ; the second term describes the generation of c from θ. the last term p λ (z) is the prior distribution of z. the first three terms minimize the reconstruction error (||x -(f + c)|| 2 ) via mean square loss. unlike traditional vae, we enforce multi-modal prior and model it as a k-modal gaussian following the approach in @xcite . we approximate our prior by sampling and feeding k pseudo-inputs into the same encoder, namely, p λ (z here k is a hyperparameter that specifies the modality of the prior. for the domain of the us politics we choose @xmath4, but our method can generalize to settings with different k. deep neural network (dnn) training. we train a dnn by optimizing the total loss, @xmath5, ϕ, λ) + e (x,y)∈d s ce(y, ŷ(z; γ)) end-toend by backpropagation over all the parameters (η, ϕ, λ, γ), while using a bi-modal prior on z, and call the resulting predictor bi-branch bi-modal gaussian variational auto-encoder (bbbg). our model differs from the traditional vae in that it uses a multi-modal prior and reconstructs the input using two separately learned components in the generative parts of the model (see appendix fig 5 , 6 ). the single branch ablation. as an ablation, and to understand the importance of eq. 1, we relax the neutrality constraint by deleting the neutral context learning branch of the model and we call this algorithm sbbg (single-branch variant). in the simplified version, the contexts are no longer estimated and we are reduced to a standard vae framework with a gaussian mixture prior, but with the same k-modal vamppriors . implementation and training details are described in sec. 5.5. justification for supervision. traditional vae models are unsupervised. but they tend to perform poorly due to over-regularization @xcite . in both of our experiments, the unsupervised vae variant appears to collapse into a prior of uni-modal gaussian. to solve this issue, we induced the model to converge to a bimodal prior by providing supervision with a few labeled examples. this means that, during training, we allow some ideology labels to be seen by the model, and the prediction loss of ideology are back-propagated to help tune the parameters in the encoders. the number of labels required to effectively train such a system turns out to be 5∼8% of the total samples. ## experiments we experiment with two benchmark datasets, congressional speeches and gun debate forum to baseline proposed bbbg, against well-known and prior ml methods. while we provide details of these datasets in the sec. a.1, we note that the reason for choosing these datasets is driven by the need to ascertain ground-truth extremity of texts and authors (ranging from extreme to neutral). other datasets are binary, and as such this information is not provided. the datasets we chose allow for sub-sampling of sub-populations with varying levels of ideological extremity. this allows us to validate proposed method under selection bias. ## conclusion we propose a novel deep-learning method to deal with label scarcity and selection bias that typically arise in political ideology prediction problems. our method learns to decompose text embeddings into neutral contexts and context filtered position vectors, which contain ideological information. in addition to demonstrating improved prediction accuracy on benchmark datasets, we also expose important aspects of our approach through ablative experiments. we validate context neutrality through crowd-sourcing, ideological concentration through context filtering, and knowledge-transfer to documents dealing with novel themes, and to people, whose ideology bears little similarity to training data. going forward, we can check if our model can extend to more general social platforms such as twitter, or learn and verify ideological representation on a continuum similar to dw-nominate.