UPPAM : A Unified Pre-training Architecture for Political Actor Modeling based on Language (2023)

Abstract: Modeling political actors is at the core of quantitative political science. Existing works have incorporated contextual information to better learn the representation of political actors for specific tasks through graph models. However, they are limited to the structure and objective of training settings and can not be generalized to all politicians and other tasks. In this paper, we propose a Unified Pre-training Architecture for Political Actor Modeling based on language (UPPAM). In UPPAM, we aggregate statements to represent political actors and learn the mapping from languages to representation, instead of learning the representation of particular persons. We further design structure-aware contrastive learning and behavior-driven contrastive learning tasks, to inject multidimensional information in the political context into the mapping. In this framework, we can profile political actors from different aspects and solve various downstream tasks. Experimental results demonstrate the effectiveness and capability of generalization of our method.

## introduction political actors are shaping our attitudes, opinions, and decisions toward public issues. for instance, on social platforms, politicians can select and emphasize certain aspects of content to bias the discussion, through which they can derive an opinion climate from user engagement and acquire direct feedback from potential voters and opinion leaders @xcite . political actor modeling is essential for quantitative political science and has applications in various downstream tasks such as roll call vote prediction @xcite , frame detection @xcite and bias detection @xcite . data-driven approaches utilize different kinds of information to profile political actors, including public statements, legislative behaviors and social figure 1 : an illustration of political actors. they not only participate in legislative activities, but also form relationships with others, and convey opinions through tweets, speeches and etc. we propose to represent political actors based on their statements and learn the mapping from language to their representations using social networks and behaviors as self-constructed supervision. generally speaking, previous research aims to learn representations for a certain group of political actors using supervision from specific downstream tasks as objectives. although they report positive results on target tasks, their models lack generalization ability in two aspects. (1) representations are learned on labeled data from specific tasks, e.g., state-level vote prediction, therefore they cannot be easily transferred to other tasks or scenarios. (2) the model is limited to the training setting and can not be adapted to dynamic social contexts. in other words, it's hard for the model to estimate new legislators, non-voting candidates and other political actors unseen. recently, large-scale pre-trained language models (plms) @xcite @xcite have demonstrated a strong generalization ability and achieved excellent performance in many language modeling tasks. motivated by plms, we explore representing political actors based on their statements and propose a unified pre-training architecture for political actor modeling based on language (uppam) 1 . we employ a two-stage training procedure following the fashion of plms. firstly, we pre-train our model to learn the mapping from statements to actor representation. we propose a multigranular method to represent political actors based on language, and information of political scenarios is further injected into our model via proposed structure-aware contrastive learning and behaviordriven contrastive learning tasks. secondly, we fine-tune the model for downstream tasks using the corresponding supervised objectives. uppam is novel in three points. (1) we learn the mapping from statements to the representation of political actors, instead of directly learning actor representations. by doing so, the mapping parameters can be transferred to any downstream tasks easily, learning representations for unseen political actors based on their statements. (2) we propose several self-training tasks to inject general knowledge in the political scenarios into mapping parameters in the pre-training stage. (3) we propose a multigranular actor representation model, that can capture nuances of both general ideology and specific preferences between different political actors. we evaluate our approach on three types of tasks in quantitative political science, i.e., profile of actors, prediction of behaviors and analysis of languages. uppam outperforms general plms and other political domain-specific plms on these tasks. our task-agnostic model also achieved competitive results compared to the task-specific models that employ architectures crafted for the vote prediction task. further analysis shows the effectiveness and robustness of uppam in few-shot settings and different aggregation settings. ## experiment setup we fine-tune our model on different kinds of downstream tasks in quantitative political science. we then compare it with prior general plms and political domain-specific plms. ## related work political actor modeling focuses on modeling attributes and behaviors of political actors, with special attention to estimating the ideology. because of the publicity and typicality, politicians like legislators have been the research subject for most work. the most widely used approach to estimate the ideology of legislators is ideal point model @xcite that represents legislators and bills as points in a one-dimension latent space from the rollcall data. after that, researchers further incorporate texts of bills @xcite to enhance the model, solving the problem of prediction on new bills. some embedding methods are also proposed to promote learning of legislators @xcite . more recently, external information including cosponsorship @xcite , donors @xcite , relevant stakeholders @xcite and expert knowledge @xcite is used to better learn legislator representation. they follow a mixed structure of textual encoder and graph encoder, to explicitly combine textual and structural information. despite outstanding performance on target tasks, these methods are limited to certain settings or data, behaving inefficient in dynamic political scenarios. thus they are hard to be transferred to all actors. by contrast, methods relying on texts @xcite provide more possibility for generalization. ## conclusion in this paper, we propose to learn political actors from languages and inject multidimensional domain knowledge into the plms through structureaware contrastive learning and behavior-driven con-trastive learning. experimental results validate the effectiveness and generalization capability of our approach.