Understanding Politics via Contextualized Discourse Processing (2021)

Abstract: Politicians often have underlying agendas when reacting to events. Arguments in contexts of various events reflect a fairly consistent set of agendas for a given entity. In spite of recent advances in Pretrained Language Models, those text representations are not designed to capture such nuanced patterns. In this paper, we propose a Compositional Reader model consisting of encoder and composer modules, that captures and leverages such information to generate more effective representations for entities, issues, and events. These representations are contextualized by tweets, press releases, issues, news articles, and participating entities. Our model processes several documents at once and generates composed representations for multiple entities over several issues or events. Via qualitative and quantitative empirical analysis, we show that these representations are meaningful and effective.

## introduction over the last decade political discourse has moved from traditional outlets to social media. this process, starting in the '08 u.s. presidential elections, has peaked in recent years, with former-president trump announcing the firing of top officials as well as policy decisions over twitter. this presents a new challenge to the nlp community, how can this massive amount of political content be used to create principled representations of politicians, their stances on issues and legislative preferences? this is not an easy challenge as in political texts perspective is often subtle rather than explicit @xcite . choices of mentioning or omitting certain entities or attributes can reveal the author's agenda. for example, tweeting "mass shootings are due to a huge mental health problem" in reaction to a mass shooting is likely to be indicative of opposing gun control measures, despite the lack of an explicit stance in the text. recent advances in pretrained language models (plms) in nlp @xcite @xcite have greatly improved word representations via contextualized embeddings and powerful transformer units, however such representations alone are not enough to capture nuanced biases in political discourse. two of the key reasons are: (i) they do not directly focus on entity/issuecentric data and (ii) they only represent linguistic context rather external political context. our main insight is that effectively detecting such bias from text requires modeling the broader political context of the document. this can include understanding relevant facts related to the event addressed in the text, the ideological leanings and perspectives expressed by the author in the past, and the sentiment/attitude of the author towards the entities referenced in the text. we suggest that this holistic view can be obtained by combining information from multiple sources, which can be of varying types, such as news articles, social media posts, quotes from press releases and historical beliefs expressed by politicians. for example, consider the following tweet in context of a school shooting: we need to treat our teachers better! we should keep them safe. if the author of the tweet is kamala harris (known to be pro-gun control), this tweet is likely to be understood as "ban guns to avoid mass shootings in schools". however, if the same tweet is from mike pence, whose stance on guns is: "firearms in the hands of law abiding citizens makes our communities safer", the tweet could mean "arming school teachers stops active shooters". this example demonstrates that depending on the context, the same text could signal completely different real-world actions. hence, we need to model the broader context of the text in order to understand its true meaning. visualization projecting the tweet representation into a 2d space is given in figure 1 , and shows how contextualization from our model helps disambiguate this example. first, we show the bert-base representation of the tweet (tweet-bert). we also show the bert-base representations of the known stances of pence and harris on gun control ({mike pence,kamala harris} stance-bert). finally, we apply our model, contextualizing the ambiguous tweet representation with speaker information ({mike pence,kamala harris} tweet-contextualized). the visualization captures how this representation can disambiguate the different interpretations of the same text, and capture their differences. a computational setting for this approach, combining text and context analysis, requires two necessary attributes: (i) an input representation that combines all the different types of information meaningfully and (ii) the ability to create a meaningful unified representation in one-shot, that captures the complementary strengths of the different inputs. we address the first challenge by introducing a graph structure that ties together first-person informal (tweets) and formal discourse (press releases and perspectives), third-person current (news) and consolidated (wikipedia) discourse. these documents are connected via their authors, the issues/events they discuss and the entities mentioned in them. as a clarifying example consider the tweet by former-president trump "the nra is under siege by cuomo". this tweet will be represented in our graph by connecting the text node to the author node (trump) and the referenced entity node (ny gov. cuomo). these settings are shown in fig. 2 . we propose a novel neural architecture that unifies all the information in the graph in one-shot. our architecture generates a distributed representation for each item in the graph that is contextualized by the representations of others. it can dynamically respond to queries, focusing the induced representation on a specific context. in our exam-ple, this results in a modified tweet representation helping us characterize trump's opinion of cuomo in the context of the guns issue. our architecture consists of an encoder combining all documents related to a given node to generate an initial node representation and a composer, a graph attention network (gat), composing the graph structure to generate contextualized node embeddings. we design two self-supervised learning tasks to train the model and capture structural dependencies over the rich discourse representation, predicting authorship and referenced entity links over the graph structure. intuitively, the model is required to understand subtle language usage; authorship prediction requires the model to differentiate between: (i) the language of one author from another and (ii) the language of the author in context of one issue vs another issue. referenced entity prediction requires understanding the language used by a specific author when discussing a particular entity, given the author's past discourse. we focus on a specific graph elementpoliticians, and evaluate their resulting discourse representation on several empirical tasks which capture their stances and preferences. our evaluation demonstrates the importance of each component of our model and usefulness of the learning tasks. to summarise, our research contributions include: 1. a novel graphical structure connecting various types of documents, entities, issues and events. 2. an effective neural architecture, compositional reader, processing all information in one-shot, and designing two effective tasks for training it. 2019) , we are now able to create rich textual representations, effective for many nlp tasks. although contextual information is captured by these models, they are not explicitly designed to capture entity/event-centric information. hence, tasks that require such information @xcite @xcite @xcite , would benefit from more focused representations. of late, several works attempted to solve such tasks, such as analyzing relationships and their evolution @xcite , analyzing political discourse on news and social media @xcite and political ideology @xcite @xcite . various political tasks such as roll call vote prediction @xcite @xcite @xcite , entity stance detection @xcite , hyper-partisan/fake news detection @xcite @xcite require a rich understanding of the context around the entities that are present in the text. but, the representations used are usually limited in scope to specific tasks and not rich enough to capture information that is useful across several tasks. the compositional reader model, that builds upon @xcite embeddings and consists of a transformer-based graph attention network inspired from @xcite and @xcite , aims to address those limitations via a generic entity-issue-event-document graph, which is used to learn highly effective representations. representing legislative preferences is typically done by modeling the ideal point of legislators represented in a euclidean space from roll-call records @xcite . recent approaches incorporate bill text information into this representation @xcite @xcite . most relevant to our work is @xcite ) which uses social media information. we significantly extend these approaches by contextualizing the social media content using a novel architecture. ## compositional reader in this section, we describe the architecture of the proposed 'compositional reader' model in detail. it contains 3 key components: graph generator, encoder and composer. given a query output of the query mechanism from sec. 3.3, graph generator creates a directed graph with entities, issues, events and documents as nodes. encoder is used to generate initial node embeddings for each of the nodes. composer is a transformer-based graph attention network (gat) followed by a pooling layer. it generates the final node embeddings and a single summary embedding for the query graph. each component is described below. ## learning tasks we design two learning tasks to train the compositional reader model: authorship prediction and referenced entity prediction. both the tasks are intuitively designed to train the model to learn the association between the author node representation and the language used by the particular author. these tasks are two variations of link prediction over the graph. the tasks are detailed below. ## evaluation we evaluate our model and pre-training tasks in a systematic manner using several quantitative tasks and qualitative analysis. quantitative evaluation includes grade paraphrase task, grade prediction on national rifle association (nra) and league of conservation voters (lcv) grades data followed by roll call vote prediction task. qualitative evaluation includes entity-stance visualization for issues and opinion descriptor generation. we compare our model's performance to bert representations, the bert adaptation baseline and representations from the encoder module. baselines and the evaluation tasks are detailed below. further evaluation tasks are in the appendix. ## conclusion we tackle the problem of understanding politics, i.e., creating unified representations of political figures capturing their views and legislative preferences, directly from raw political discourse data originating from multiple sources. we propose the compositional reader model that composes multiple documents in one shot to form a unified political entity representation, while capturing the real-world context needed for representing the interactions between these documents. we evaluate our model on several qualitative and quantitative tasks. we outperform bert-base model on both types of tasks. our qualitative evaluation demonstrate that our representations effectively capture nuanced political information. trump (@realdonaldtrump). harvard dataverse. felix biessmann. 2016. automating political bias prediction. arxiv preprint arxiv:1608.02195. janez brank, gregor leban, and marko grobelnik. 2017. annotating documents with relevant wikipedia concepts. in proceedings of slovenian kdd conference on data mining and data warehouses (sikdd). sihao chen, daniel khashabi, wenpeng yin, chris callison-burch, and dan roth. 2019. seeing things from a different angle: discovering diverse perspectives about claims. in proc. of the annual conference of the north american chapter of the association for computational linguistics (naacl). joshua clinton, simon jackman, and douglas rivers. 2003. the statistical analysis of roll call data. american political science review -amer polit sci rev, 98. maryam davoodi, eric waltenburg, and dan goldwasser. 2020. understanding the language of political agreement and disagreement in legislative texts. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 5358-5368, online. association for computational linguistics. dorottya demszky, nikhil garg, rob voigt, james zou, matthew gentzkow, jesse shapiro, and dan jurafsky. 2019. analyzing polarization in social media: method and application to tweets on 21 mass shootings. arxiv preprint arxiv:1904.01596. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. daniel diermeier, jean-françois godbout, bei yu, and stefan kaufmann. 2012. language and ideology in congress. british journal of political science, 42(1):31-55. lisa fan, marshall white, eva sharma, ruisi su, prafulla kumar choubey, ruihong huang, and lu wang. 2019. in plain sight: media bias through the lens of factual reporting. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6343-6349, hong kong, china. association for computational linguistics. wei fang, moin nadeem, mitra mohtarami, and james glass. 2019. neural multi-task learning for stance prediction. in proceedings of the second workshop on fact extraction and verification (fever), pages 13-19, hong kong, china. association for computational linguistics. sean m gerrish and david m blei. 2011. predicting legislative roll calls from text. in proceedings of the 28th international conference on machine learning, icml 2011. xavier glorot and yoshua bengio. 2010. understanding the difficulty of training deep feedforward neural networks. in proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. xiaochuang han, eunsol choi, and chenhao tan. 2019. no permanent friends or enemies: tracking relationships between nations from news. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1660-1676, minneapolis, minnesota. association for computational linguistics. mohit iyyer, anupam guha, snigdha chaturvedi, jordan boyd-graber, and hal daumé iii. 2016. feuding families and former friends: unsupervised learning for dynamic fictional relationships. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1534-1544, san diego, california. association for computational linguistics. kristen johnson and dan goldwasser. 2016. identifying stance by analyzing political discourse on twitter. in proceedings of the first workshop on nlp and computational social science, pages 66-75, austin, texas. association for computational linguistics. kristen johnson and dan goldwasser. 2018. classification of moral foundations in microblog political discourse. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 720-730, melbourne, australia. association for computational linguistics. anastassia kornilova, daniel argyle, and vlad eidelman. 2018a. party matters: enhancing legislative embeddings with author attributes for vote prediction. corr, abs/1805.08182. anastassia kornilova, daniel argyle, and vladimir eidelman. 2018b. party matters: enhancing legislative embeddings with author attributes for vote prediction. in proceedings of the 56th annual meet-thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, r'emi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. arxiv, abs/1910.03771. zhilin yang, zihang dai, yiming yang, jaime carbonell, russ r salakhutdinov, and quoc v le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in advances in neural information processing systems, pages 5754-5764.