An E mbedding M odel for E stimating L egislative P references from the F requency and S entiment of T weets (2020)

Abstract: Legislator preferences are typically represented as measures of general ideology estimated from roll call votes on legislation, potentially masking important nuances in legislators’ political attitudes. In this paper we introduce a method of measuring more specific legislator attitudes using an alternative expression of preferences: tweeting. Specifically, we present an embedding-based model for predicting the frequency and sentiment of legislator tweets. To illustrate our method, we model legislators’ attitudes towards President Donald Trump as vector embeddings that interact with embeddings for Trump himself constructed using a neural network from the text of his daily tweets. We demonstrate the predictive performance of our model on tweets authored by members of the U.S. House and Senate related to the president from November 2016 to February 2018. We further assess the quality of our learned representations for legislators by comparing to traditional measures of legislator preferences.

## introduction legislator preferences are typically estimated as general measures of ideology using roll-call votes on legislation. however, such measures fail to capture aspects of preferences not reflected in legislation, such as attitudes towards a sitting president. for instance, sen. bob corker (r-tn) famously referred to the trump white house as an "adult day-care center," john mccain (r-az) said trump "is often poorly informed,"and jeff flake (r-az) called him a "danger to a democracy," yet all of these republican senators cast more than 80% of their legislative votes in line with president @xcite . generally, the political science research recognizes that the public's views of the president have spillover effects on evaluations of legislators, which incentivizes strategic communication about the president. for example, senate majority leader mitch mcconnell recently encouraged republican senators in vulnerable re-election campaigns to distance themselves from trump. understanding legislators' attitudes toward the president enables greater understanding and measurement of such strategic communications. furthermore, these attitudes also matter for understanding the president's ability to pass his legislative agenda. in this paper, we propose a new method for estimating legislator preferences from the frequency and sentiment of their tweets using a novel combination of spatial models based on item response theory and the modeling of count data. we use this method to estimate legislator preferences about donald trump using tweets by members of congress and donald trump in the 15-month period following election day in november 2016. in our model, legislator embeddings interact with embedding representations of donald trump himself, constructed from a neural network using the text (and timing) of his tweets during the same time frame. thus, our model leverages the text feature extraction capabilities of neural networks and incorporates the legislator sentiment in tweets about trump as well as the strategic decision about whether and when to tweet about him. we quantitatively assess the quality of our learned legislator representations by demonstrating the model's predictive performance on a test set of tweets, and we also compare our model-obtained embeddings to dw-nominate scores, traditional measures of legislator ideology. 1 our analysis not only validates the modeling approach but also highlights that attitudes towards trump are not being entirely captured by legislative voting behavior. more broadly, a method for estimating domain-specific preferences, rather than general ideological ideal points, broadens the range of hypotheses than can be tested by political researchers. ## measuring legislator preferences the predominant method of measuring legislator preferences over the past half-century has been the modeling of the ideal point of a legislator from recorded votes on policy legislation. these ideal points constitute a spatial model for legislative behavior, as both legislators and policies are represented in a low-dimensional euclidean space @xcite . such ideal points are interpreted as measures of ideological preferences and have been used to test hypotheses on topics such as political polarization, political representation, and cross-institutional relationships @xcite . a key limitation of initial methods for estimating ideal points was the inability to perform out-ofsample predictions. thus, they could not be used to predict votes on new legislation. to address this shortcoming, @xcite extended the ideal point model by placing legislation into a "political space" based upon the latent topics of the legislation's text and perform prediction using these topics. @xcite use a nonparametric bayesian model to incorporate constituency data into a factor model for legislative roll calls and text, with the text again being analyzed using a topic model. further efforts to incorporate bill text using topic models come from @xcite , gerrish and blei ( 2012 efforts to incorporate text into vote prediction were improved by moving to an embedding paradigm rather than topic models. @xcite incorporate word embeddings into a model for vote prediction by representing a piece of legislation as the average of its word embeddings and further representing legislators using ideal vectors as a multi-dimensional extension to ideal points. @xcite augment bill text with bill metadata (i.e., bill sponsor information) to improve the predictive capabilities of legislator embeddings, and use a convolutional neural network (cnn, @xcite ), rather than the average over bill word em-beddings, to model bill text. while tweets have increasingly been used to measure political preferences of the mass public @xcite , little attention has been paid to the potential of using tweets to measure legislators' preferences. one notable exception, @xcite , uses the structure of social networks on twitter to learn ideological positions of both political elites and the general public, but does not incorporate information from the tweets themselves. as all legislators in the u.s. house and senate now use twitter to communicate with constituents on a wide variety of topics, we recognize an opportunity to observe nuances in legislator preferences not captured by broader ideological measures that rely on roll call votes. here we focus specifically on legislators' attitudes toward the sitting president. while attitudes toward the president are among the most frequently measured aspects of public opinion, there is currently no method for explicitly measuring these preferences among legislators. we develop an embedding model that jointly predicts the frequency and sentiment of legislator tweets about donald trump. similar to the @xcite modeling of legislator votes in response to the text of legislation, here legislator tweets are considered as a response to text features extracted from donald trump's tweets. whereas embedding models for vote prediction analyze only one outcome of legislator behavior (i.e., the vote itself), our embedding model is trained to predict multiple outcomes of legislator behavior in both tweet counts and content in the form of sentiment. moreover, because our model does not rely on votes casts by legislators, it could be used to estimate preferences among a wider range of political actors (e.g., candidates, cabinet members) on a variety of domains, and with texts other than tweets. ## tweet dataset we obtained all publicly-available tweets by members of congress from tweetcongress, a sunlight foundation initiative. we restricted the sample to only those tweets that contained any of a specific set of terms related to donald trump (in addition to his twitter handle): "donald trump," "trump," "realdonaldtrump," "maga," (an acronym for trump's campaign slogan "make america great again") "whitehouse," "whitehouse," "potus," (acronym for "president of the united states"), and "potus." of these, we further restricted the tweets to span in time from november 2016 to february 2018, when the data was collected. this culling process yielded 29,696 tweets from 451 legislators. the model also incorporates tweets from trump, which we obtained from the website www.trumptwitterarchive.com . for each day included in the dataset, the text of all tweets by donald trump was agglomerated and preprocessed by removing excess whitespace and lowercasing all letters. the text was tokenized and each word-token mapped to an integer identifier, with a vocabulary mapping of 2783 words. for each day, we obtain a sequence of integers representing the words composing the text of donald trump's tweets from that day, and these are the inputs to the model described in section 4.3. figure 1 plots the number of tweets by republicans, democrats, and trump over time for the period we examine. there were only 13 days for which trump did not tweet (2.79%), and the most tweets that he sent in a single day was 32. the most tweets by a democrat in a single day was 99, while the most tweets by a republican in a single day was 25. the variation in tweets across time highlights one of the key features of the model-the incorporation of not only the sentiment of tweets about trump by also the number of daily tweets. of the 29,696 trump-related legislator tweets, a subset of 4,661 tweets were randomly selected to be manually labeled with respect to their sentiment about trump, using a three-point "positive," "negative," "neutral" scale based on the text of the tweet foot_2 from november 2016 to february 2018. we divided the tweets temporally by day into disjoint training, validation, and test sets, such that all tweets from each day were randomly assigned to one of the three sets. the training, validation, and test sets contain 70%, 10%, and 20% of all days, respectively. ## legislator tweet model formulation our proposed model combines an embedding model for legislators with models for ordinal and count data, predicting both the number of daily tweets about donald trump sent by each legislator and the sentiment of labeled tweets. the joint nature of this model not only enables a more nuanced representation of legislators, but also accounts for the fact that a legislator who consistently tweets in favor (or against) the president is different from one who tweets occasionally, even if both express similar sentiment. the model also incorporates text features from trump's tweets to provide context for legislator tweets as reactions to trump. underpinning our model is the assumption of a latent political space of dimension k. in this space, we learn a set of "day embeddings" (or "trump embeddings") that interact with a set of legislator embeddings. for a particular day t, let τ @xmath0. indexing legislators by @xmath1, 2, . . . , n}, we endow a legislator i with a vector v @xmath2, which captures a legislator's propensity to react to trump regardless of how he presents himself via twitter. while the legislator embeddings are learned as free parameters of the model, the trump embeddings are constructed using the text of donald trump's tweets. we now describe how we use legislator and trump embeddings to predict tweet counts and sentiment. ## predictive results to demonstrate the efficacy of our model for legislator tweeting behavior with respect to president donald trump, we first show that the construction of trump embeddings from the language of his own tweets provides an informational signal for legislators to react to. we train our model using the days for the training set and present the predictive results for days in the test set. since the model seeks to capture two aspects of legislator tweeting behavior, we evaluate the model using two metrics: the negative-log likelihood of the count model and the mean-absolute-error (mae) of the sentiment model. overall model performance is also captured by the total loss of the model, which is the weighted negative-log likelihood of both the count and sentiment models, equation 8. mae is used rather than accuracy to account for the ordinal nature of the sentiment model. the hyperparameter γ controls the balance between the two components of our model, counts and sentiment. we present our results for three settings of γ, which allows us to analyze the two components of our model separately before analyzing the joint model. a full description of the process used to tune hyperparameters and a comparison of the model with linear and nonlinear text maps can be found in appendix b. for all results presented here, we set @xmath3, and use a linear text map. the number of epochs for which the model was trained varies depending on model setting, but in all cases each training batch comprises 128 tweets. the model was implemented in tensorflow @xcite and trained on a single nvidia titan x gpu. code can be found on the author's github at: github.com/gspell/congressionaltweets. when γ = 1, only the loss from the part of the model that handles tweet counts contributes to the total loss in equation 8. we present the final negative log-likelihood of the count model for both the poisson and negative binomial models described in section 4.1, and for both the case in which the text of donald trump's tweets is used to construct his daily embedding representation and the case in which the trump embeddings are free parameters of the model. for the negative binomial model, the model was trained for 75 epochs, which was the amount of training required to perform best on the validation (rather than test) set of tweets. the poisson model was trained for 100 epochs while using the text and 2000 epochs without text. the predictive results are shown in when γ = 0, only the loss from the part of the model that handles legislator tweet sentiment contributes to the total loss in equation 8. we present the final model loss -which is the negative loglikelihood of the sentiment model -as well as the model mae. again, we show results for the case in which trump's tweet text is used to construct embeddings and the case in which the text is not used. we also toggle an additional model setting for analysis: the inclusion of the legislator bias term, b i , from equation 4. we adjust the number of epochs to 150 for training with text. we train the model without text for 1000 and 3000 epochs, including and excluding the legislator bias term, respectively. the results are presented in ## legislator embeddings training our legislator tweeting model yields a key byproduct: the legislator embeddings. as with previous spatial representations of legislator preferences, our model enables the visualization of the positions of legislators in space. in figure 2 we plot the two dimensions of legislator embeddings from the model presented in in the figure we also see that embeddings are not simply an artifact of the number of tweets about trump authored by the legislator, nor whether the legislator is a member of the house or senate. legislators with more extreme values of twitter sentiment relative to other members of their party can be found in both chambers of congress and range from having authored fewer than 100 tweets about trump to over 500. 6 another initial validating characteristic of the embeddings is the clustering of prominent republican senators who have been publicly critical of trump. we examine the spatial positions of republican senators whom a 2017 washington post analysis identified as critical of the president based on their responses to controversial events in trump's presidency, such as trump's firing of fbi director james comey and response to the charlottesville protests, as well as overall rhetoric used when discussing trump @xcite . the positions of these senators are highlighted (dark black points) in figure 2. these senators are clustered together in the two-dimensional embedding space: john mc-cain, jeff flake, joe hoeven, bob corker, marco rubio, shelley capito, dean heller, dan sullivan, lamar alexander, and lisa murkowski. considering which democratic legislators are interspersed near the cluster of republicans in the two-dimensional embedding space is also informative. the two most extreme democratic outliers were angus king, an independent senator from maine who caucuses with the democratic party but has openly considered caucusing with the republican party and joe manchin, a notably conservative democratic senator in whose state trump won 68.5% of the vote. we observe fewer outliers among republicans. among the most extreme outliers, are ileana ros-lehtinen and carlos curbelo, whose districts hillary clinton won in 2016 by 19.6 and 16.3 percentage points, respectively. ros-lehtinen, in particular, tweeted many scathing responses to trump regarding his controversial stance on immigration. we next compare the embeddings to an existing measure of general legislator preferences. figure 3 illustrates the relationship between the first dimension of dw-nominate -a canonical measure of legislator ideology in political science -and the first dimension of our learned legislator embeddings. generally, legislators who are ideolog-ically conservative have lower embedding values, whereas liberals have higher values. at the same time, this comparison does identify legislators who are more or less critical of trump than might be expected based on ideology alone, thereby offering new empirical leverage to scholars examining the behavior and attitudes of legislators. figure 3 also compares our legislator embeddings to an approximation of how legislators might feel toward trump: the proportion of time that they vote in line with him during the period in which legislator tweets were collected. this metric was calculated using a dataset published by fivethirtyeight and includes only legislation on which the trump administration publicly expressed a clear position @xcite . while this measure is limited by many of the same constraints as other vote-based measures (e.g., dw-nominate), it is the closest existing measure of legislators' attitudes toward trump. in the right panel of figure 3 , we observe little variation in the extent to which legislators vote with trump, particularly for republicans. indeed, many of the president's most prominent critics frequently voted with the president during this time period. for instance, john mccain voted with trump 85% of the time, bob corker voted with trump 84% of the time, and jeff flake voted with trump 83% of the time. meanwhile, we observe far more variation in legislator embeddings among both republicans and democrats. in ap-pendix c, we further compare our embeddings to alternative measures of legislative preferences: campaign finance scores @xcite and trump vote-share in a legislator's constituency during the 2016 presidential election. ## conclusion in this paper, we modeled legislator tweeting behavior towards donald trump, predicting the frequency and sentiment of their tweets. the proposed model yields embedding representations for legislators that we interpret as measures of legislator attitudes towards trump. our application suggests that ideal points estimated from roll call votes can miss this critical aspect of political preferences for members of congress. whereas legislative voting might recover ideological similarities and differences with the president, it is not well suited to measure attitudes toward the president orthogonal to policy preferences, such as criticisms of his rhetoric and tone. to address this shortcoming and obtain representations of legislators' attitudes toward trump, we have proposed a model that assigns a vector to each legislator based on the content of their tweets about trump. we similarly represent donald trump with a vector for each day he tweets, constructed using the text of his daily tweets. legislator vectors and trump vectors interact to produce predictions of both the sentiment of legislator tweets about donald trump and the number of tweets produced each day. from this model we obtain representations of legislators that capture their attitudes toward the president. our model's predictive performance is robust to a variety of settings and achieves sentiment predictive performance of 0.127 mean-absolute-error and 89.3% accuracy, demonstrating its capability to predict legislator tweeting behavior. when visualizing the two dimensions of learned legislator embeddings we find that the model separates legislators across party lines (despite not being trained on the party of legislators) and groups together republican senators who are well-known critics of trump (despite overwhelmingly voting with him on legislation). though our model demonstrates the capability of representing legislators' attitudes toward trump and performs well with respect to predicting tweet counts and sentiment based upon donald trump's tweets, our method has some limitations. for one, as is the case for rheault and cochrane (2020), our model is not able to produce uncertainty bounds, as deriving uncertainty measures from neural networks remains an open area of research without a clear solution within the field of machine learning. 7 an avenue for improving the model is to allow it to capture legislators' dynamic attitudes toward trump over time. while legislator attitudes are currently modeled as static embeddings, allowing each legislator's embedding to change over time would enable the exploration of temporal dynamics and hypothesis testing about when legislators are more likely to tweet negatively about trump, what factors contribute to a legislator's decision to tweet about trump, and how the trump's tweets interact with legislator's tweets over time. while our aims in this paper were to develop a method of modeling attitudes toward trump beyond legislative policy preferences, this method can be used to test a wide range of hypotheses about modern u.s. politics. legislator embeddings can be used to explore how legislators appeal to different audiences, such as party leaders and constituents. the method presented here could similarly be used to evaluate how members of congress are punished and rewarded in elections for their criticism of praise of the president. moreover, because our model does not rely on roll call votes, it can also be used to model attitudes by any of the growing number of political elites using twitter, such as non-incumbent political candidates, state legislators, and pundits. possible extensions of this work could investigate enriching trump vectors by incorporating other sources of text, such as white house press releases and speeches. while we restrict ourselves to twitter data in this paper to maintain consistency across the sources of data for vectors representing trump and legislators, the incorporation of auxiliary text data could provide additional context.