Community-Cross-Instruct: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communities (2024)

Abstract: Social scientists use surveys to probe the opinions and beliefs of populations, but these methods are slow, costly, and prone to biases. Recent advances in large language models (LLMs) enable the creating of computational representations or “digital twins” of populations that generate human-like responses mimicking the population’s language, styles, and attitudes. We introduce Community-Cross-Instruct, an unsupervised framework for aligning LLMs to online communities to elicit their beliefs. Given a corpus of a community’s online discussions, Community-Cross-Instruct automatically generates instruction-output pairs by an advanced LLM to (1) finetune a foundational LLM to faithfully represent that community, and (2) evaluate the alignment of the finetuned model to the community. We demonstrate the method’s utility in accurately representing political and diet communities on Reddit. Unlike prior methods requiring human-authored instructions, Community-Cross-Instruct generates instructions in a fully unsupervised manner, enhancing scalability and generalization across domains. This work enables cost-effective and automated surveying of diverse online communities.

## introduction social scientists use surveys and focus groups to learn the opinions, needs, and concerns of diverse populations. however, designing surveys and recruiting participants is a slow and costly process, limiting the utility of these instruments for probing public opinion. surveys are prone to biases, such as the social desirability bias @xcite , where respondents may alter their responses to sensitive questions to appear more socially acceptable @xcite , non-response bias @xcite , where participants fail to answer questions, and self-selection bias due to the choices participants make to participate in the survey @xcite . in addition, social stigmas may taint responses @xcite , especially for hard-to-reach and marginalized groups. recent breakthroughs in generative ai and especially large language models (llms) enable new capabilities for creating computational representations of human populations -their digital twins @xcite -by ingesting vast textual data they create, for example, in online discussion forums. these llm-based models generate human-like responses that mimic the language, communication styles, and attitudes of populations they are aligned to, allowing us to probe their worldviews, biases, and sentiments in a cost-effective and automated manner. previous works have leveraged such llm-based representations to mine opinions and learn political attitudes of online communities @xcite . however, these studies typically finetune models like gpt-2 @xcite directly on raw textual data from the communities, resulting in models that can perform text continuation but are limited in their ability to answer questions in structured formats, such as multiple choice. moreover, raw community data often contains noise, irrelevant information, or off-topic discussions that can degrade model performance if not properly handled. to address these challenges, we aim to align llms to communities by finetuning the models with high-quality instructions where the community perspectives are embedded. however, curating high-quality instructional data is a non-trivial process. existing methods for self-supervised instruction generation rely on a seed set of instructions curated by domain experts @xcite , which is limited by the generalizability to a new domain. in this paper, we introduce community-cross-instruct: a fully unsupervised framework for aligning llms to online communities through instruction-tuning. it incorporates the readily available community data into the instruction generation pipeline, which no human supervision is needed. community-cross-instruct uses advanced llms to generate instruction-response pairs that better capture community perspectives in more structured and useful formats. these instructions-responses pairs are used to finetune foundational llms. the finetuned llms serve as digital twins of communities, which can be automatically surveyed to elicit their views. we show a high-level overview of our framework in figure 1 . specifically, given a corpus of comments and submissions in different forums on reddit, community-cross-instruct uses an advanced llm (gpt-4o) to automatically curate two instructional datasets: (1) comminst: a set of open-ended instructions for community-specific instruction tuning, and (2) commsurvey, a set of multi-choice survey questions for communityspecific survey completion. each instruction in comminst and each survey question in comm-survey are paired with responses from different communities (figure 2 ). we finetune foundational llms (gpt-3.5 or llama-3.1) on comminst, in order to align them to different communities and evaluate the finetuned llms on commsurvey to measure alignment. ## problem definition a topical domain (e.g., politics or diet) includes n communities {c 1 , c 2 , ...., c n } each with different views and beliefs. members of each community c i collectively author text corpus d i (e.g., discussions on reddit forums) expressing views and exhibiting behaviors. our goal is to align an llm f to each community c i using its texts d i , such that the aligned llm f ′ i learns the complex mindset of the community and responds to inputs in the community's voice. by administering surveys to the aligned llms {f ′ 1 , f ′ 2 , ..., f ′ n }, we obtain responses from different communities, thereby capturing their ideological differences. to align an llm f to a community c, we finetune it on a set of demonstrations (instructionresponse pairs) @xcite @xcite ) @xmath0, y j )}, where the instructions are open-ended questions probing the community's views on different topics, and the corresponding responses are aligned with each community's ideology. figure 2 (a) shows an example demonstration in the politics domain. we propose community-cross-instruct (figure 3 ), a framework to automatically generate communityspecific demonstrations i with an advanced llm f (gpt-4o) based on the community's text corpus d and use these demonstrations to instructiontune a foundational llm f (gpt-3.5 or llama-3.1) through a process we call "cross-instruct". ## online community corpora collection reddit is a vibrant social media platform hosting discussion forums (subreddits) on a wide range of topics @xcite . in this paper, we focus on two domains: politics and diet, which contain distinctive subreddits, or communities, with complex social dynamics. political subreddits are valuable expressions of diverse public opinions and viewpoints. in the age of rising polarization, llms can help researchers track the complex evolving ideological landscape and effectively elicit public opinions. meanwhile, the diet subreddits feature a wealth of sensitive, health-related conversations, and the latest diet and diet fads, often discussed using inscrutable insider jargon. we can track the direction of these discussions, identifying emerging trends, risks, and potential health misinformation. this insight allows public health officials to promptly address harmful health advice and intervene where necessary. additionally, it provides an early warning system to detect the spread of dangerous health practices, ensuring that corrective measures can be implemented swiftly to protect community health. we identify a set of representative subreddits for each domain based on personal knowledge and by querying chatgpt. we manually aggregate and filter results, obtaining the following five political online communities: r/liberal, r/neutralpolitics, r/anarcho_capitalism, r/conservative, and r/ask-the_donald; for diet, we investigate three communities: r/keto, r/weightlossadvice, and r/edanonymous. more details of these communities are presented in appendix a.1. we collect comments and submissions from january 2019 to december 2022 and remove those that were deleted by the author or the moderator through pushshift foot_0 . the statistics of the collected comments and submissions are shown in appendix a.1. we do not differentiate between comments and submissions, and treat each comment and submission as a separate document. in subsequent sections of this paper, we use "comment" to refer to both "comment" and "submission". ## instructional data generation for each domain, we curate a collection of (1) instructional datasets comminst for communityspecific instruction tuning, which is used to finetune llms to views of different communities, and (2) survey datasets commsurvey for evaluating the alignment of the finetuned llms' views to the communities using survey questions. the pipeline is depicted in figure 3 . ## experiments we align foundational llms to reddit communities by finetuning them on relevant demonstrations from comminst. after finetuning, we evaluate the models using surveys from commsurvey, comparing their responses to "semi-ground truths" to assess alignment. ## discussions we aim to align llms to online communities through instruction-tuning so that the aligned llms can be flexibly surveyed to probe the communities' views on different topics. to prepare data for instruction tuning, we propose community-cross-instruct, and the automatic creation of the open-ended instructionresponse pairs (comminst) is the core contribution. however, in order to demonstrate that the finetuned llms are indeed aligned to communities and can be treated as their digital twins, we propose to automatically generate survey questions and answers (commsurvey) to efficiently evaluate llm alignment. this is a secondary contribution of this paper, and we do not claim that commsurvey is the best way to measure alignment. therefore, our main goal is to propose a method to prepare llms as a proxy for surveying different online communities, rather than automatically creating such surveys. ## conclusion in this paper, we introduced and explored community-cross-instruct, a fully automated framework to represent online communities and elicit their views. by automating the process of surveying diverse online communities, community-cross-instruct offers a costeffective and efficient alternative to traditional survey methods used by social scientists. through finetuning llms based on community-specific instructions and answers, community-cross-instruct enables the generation of accurate responses that align with the beliefs and perspectives of different online communities. this innovative framework has demonstrated its effectiveness in representing various communities, including political and diet forums on reddit. community-cross-instruct opens up new possibilities for researchers to gain insights into online communities' diverse views and opinions, paving the way for deeper understanding and analysis of digital societies. moving forward, we will apply the framework to model organic online communities, such as those in the retweet network. in addition, we are interested in aligning llms to communities through reinforcement learning from human feedback (rlhf).