C ommunity LM : Probing Partisan Worldviews from Language Models (2022)

Abstract: As political attitudes have diverged ideologically in the United States, political speech has diverged lingusitically. The ever-widening polarization between the US political parties is accelerated by an erosion of mutual understanding between them. We aim to make these communities more comprehensible to each other with a framework that probes community-specific responses to the same survey questions using community language models CommunityLM. In our framework we identify committed partisan members for each community on Twitter and fine-tune LMs on the tweets authored by them. We then assess the worldviews of the two groups using prompt-based probing of their corresponding LMs, with prompts that elicit opinions about public figures and groups surveyed by the American National Election Studies (ANES) 2020 Exploratory Testing Survey. We compare the responses generated by the LMs to the ANES survey results, and find a level of alignment that greatly exceeds several baseline methods. Our work aims to show that we can use community LMs to query the worldview of any group of people given a sufficiently large sample of their social media discussions or media diet.

## introduction political polarization is a prominent component of politics in the united states (poole and @xcite @xcite . previous studies have shown growing polarization in social media @xcite @xcite and substantial partisan and ideological differences in media diet @xcite @xcite . @xcite show that partisanship makes reliable predictions about an individual's word understanding. r. @xcite used modern machine-translation techniques to demonstrate that the left and right communities use english words differently. milbauer ## partisan twitter data we construct a twitter dataset containing 4.7m tweets (100m word tokens) by republican and democrat communities respectively. we first sampled 1m active u.s. twitter users before and after the 2020 presidential election. we adapt the standard method @xcite to estimate their political affiliation from the political accounts they follow and collect tweets of republican and democratic twitter users from 2019-01-01 to 2020-04-10. we pick this period because the anes 2020 survey was collected between 2020-04-10 to 2020-04-18 and we want to ensure the twitter training data does not leak information beyond 2020-04-10. we subsample 4.7m tweets from each side to achieve a balanced set and use nguyen et al. (2020)'s tweet tokenizer for data processing. details are described as follows. u.s. twitter user sampling. we first sample a subset of active twitter users from the "decahose", twitter's 10% sample of tweets. we define active u.s. users as those who posted at least 10 original tweets before and after the 2020 presidential election period (2020-07-01 to 2021-06-31). we then use litecoder foot_1 to extract user locations from their profile location strings and filter out users not based in the u.s. we construct the follow graph of the resulting set of 1,074,650 twitter users. partisan assignment. we follow previous studies @xcite to estimate the party affiliation of twitter users from the political accounts they follow. specifically, we update the list of twitter handles of us politicians from @xcite by adding current federal officeholders and governors from ballotpedia foot_2 . the final list has 457 republican and 473 democratic politician twitter handles. to identify committed partisan users, we adopt the following rules: a user is labeled as a democrat if they followed no fewer than 6 democratic politicians and no republican politician from the list in february 2022, whereas a person is labeled as a republican if they followed no fewer than 2 republican politicians and no democratic politicians. we choose these thresholds because there are 69% democratic users and 26% republican users on twitter foot_3 (2.65:1). this step predicts 182,788 democratic-leaning and 72,186 republican-leaning users (2.53:1). tweet pre-processing. we use the tweet tokenizer from @xcite to process all the data. this tokenizer converts user mentions and web/url links into special tokens @user and httpurl. we delete httpurl from the tweets because it does not contain useful community information. we do not lower the case but filter out tweets with less than 10 tokens, producing 7,554,409 democratic and 4,759,441 republican tweets. we randomly sample from democratic tweets to ensure both partisan communities have the same number of 4,759,441 tweets for training language models to ensure a fair community model comparison. ## framework we present a simple communitylm framework which adapts gpt-style language models to mine community insights. this framework consists of four steps: (1) fine-tune or train gpt language models on community data, (2) design prompts based on survey questions, (3) generate community responses with language models, (4) aggregate community stance based on responses. ## evaluation task formulation. the anes survey has selfreported party affiliation from participants. we use responses from republican and democratic participants and calculate their average ratings towards each of 30 items (persons and groups). these average ratings are provided in appendix b. if the average rating of republican participants is higher than that of democratic participants toward one item (e.g., joe biden), it is labeled as "r". otherwise, the item is labeled as "d". 70% items are labeled "d" and 30% "r". the 9 items with "r" label are donald trump, mike pence, marco rubio, nikki haley, clarence thomas, whites, capitalists, big business, and the republican party. the task asks a model to predict which community is more favorable towards an item. to address the data imbalance, we prefer weighted f1 to accuracy as a measure of model performance. baselines. we evaluate the performance of trained and fine-tuned communitylm (gpt-2) against 4 baselines. the first baseline is frequency model which counts the frequency of an item's name in each community's data and classifies the community with higher word frequency to be the label. the second baseline is keyword retrieval which uses keywords to retrieve tweets containing the keywords from each community's data, computes the average community stance, and selects the community with a higher stance score. keyword retrieval (full) means using the full names as keywords and keyword retrieval (surname) means using the surname of people. the third and fourth baselines use pre-trained gpt-2 and pre-trained gpt-3 curie respectively. "[context]" is a preceding context "as a democrat/republican, i think", which is concatenated with the prompts to generate partisan responses on each item. we compute the average community stance on 1000 synthetic responses and pick the community with a higher average stance score. it is noted that we also fine-tune or train gpt-2 on the aggregate partisan tweets and show their results in appendix d. overall performance. first, we observe that fine-tuned communitylm with "x is the" prompt achieves the best performance in both accuracy (97.33%) and weighted f1-score (97.29%) on the task. the same model's performance is sensitive to the prompt design and the longest prompt out of the four seems to work the best. "x" alone is bad, because it will result in many responses like "x @user", "x???", "x.", which are common twitter posts and are too short to interpret their attitudes. second, fine-tuned communitylm outperforms trained communitylm from scratch. it indicates that pre-training gpt-2 is helpful, probably because pre-training injects the general knowledge about the named entities into gpt-2. third, we find that keyword retrieval (surname) is a strong baseline in both accuracy (93.33%) and f1 (93.33%), but its performance is also sensitive to the selection of keywords. as we see, the weighted f1 performance of keyword retrieval (full), which uses a strict full name matching (e.g., "joe biden"), drops to 87.00%. in contrast, language models are able to learn the associations between different names for the same person and generalize without worrying about name forms. last, fine-tuned communitylm outperforms pre-trained gpt-2 and gpt-3 baselines. it is worth noting that the performance of pre-trained gpt-3 curie is consistently better than pre-trained gpt-2. gpt-3 with the "x is/are" prompt achieves the same score as the keyword retrieval (surname) baseline. ## conclusion in this paper, we present a simple commu-nitylm framework to evaluate the viability of fine-tuned gpt-2 community language models in mining community insights in the context of political polarization between republicans and democrats. we adopt anes survey questions and experiment with four types of prompts to generate community responses through gpt-2, showing that generated opinions are predictive about which community is more favorable towards selected public figures and groups. our results show that finetuned communitylm (gpt-2) outperforms the baseline methods. we analyze the model errors and run qualitative analyses to demonstrate that gpt-2 community language models can be used to rank public figures and probe word choices. there are a few limitations in the current approach. first, language models can synthesize unreliable responses. structured knowledge @xcite can be used to reduce nonsensical or unfaithful generation. therefore, it is important that we use statistical patterns rather than individual synthesized tweets to draw conclusions @xcite . second, language models are shown to be sensitive to prompt design in our experiments and are also vulnerable to negation and misprimed probes @xcite . in the future, we plan to develop a systematic approach to design effective prompts and evaluate the robustness of communitylm. third, we focus on the classic red and blue polarization and do not consider a more fine-grained segmentation of u.s. politics. we hope to extend this work to study multiple sociopolitical communities in america and surface their unheard voices.