USDC : A Dataset of \underline{U} ser \underline{S} tance and \underline{D} ogmatism in Long \underline{C} onversations (2025)

Abstract: Analyzing user opinion changes in long conversation threads is extremely critical for applications like enhanced personalization, market research, political campaigns, customer service, targeted advertising, and content moderation. Unfortunately, previous studies on stance and dogmatism in user conversations have focused on training models using datasets annotated at the post level, treating each post as independent and randomly sampling posts from conversation threads. Hence, first, we build a dataset for studying user opinion fluctuations in 764 long multi-user Reddit conversation threads, called USDC. USDC contains annotations for 2 tasks: i) User Stance classification, which involves labeling a user’s stance in a post within a conversation on a five-point scale; ii) User Dogmatism classification, which involves labeling a user’s overall opinion in the conversation on a four-point scale. Besides being time-consuming and costly, manual annotations for USDC are challenging because: 1) Conversation threads could be very long, increasing the chances of noisy annotations; and 2) Interpreting instances where a user changes their opinion within a conversation is difficult because often such transitions are subtle and not expressed explicitly. Hence, we leverage majority voting on zero-shot, one-shot, and few-shot annotations from Mistral Large and GPT-4 to automate the annotation process. Human annotations on 200 test conversations achieved inter-annotator agreement scores of 0.49 for stance and 0.50 for dogmatism with these LLM annotations, indicating a reasonable level of consistency between human and LLM annotations. USDC is then used to finetune and instruction-tune multiple deployable small language models like LLaMA, Falcon and Vicuna for the stance and dogmatism classification tasks. We make the code and dataset publicly available [https://github.com/mounikamarreddy/USDC].

## introduction understanding fluctuations in a user's (or author's) opinions during a conversation is fundamental to successful interpersonal interactions. it is essential to develop better communication skills, fostering empathy, and making informed decisions. this understanding is particularly relevant in the context of dogmatism-a phenomenon observed in areas such as politics, religion, culture, intellect, and science-where rigid adherence to beliefs often hinders open-mindedness and empathy @xcite . by aligning with the opinions and stances of potential customers, advertisers can target their campaigns more effectively. companies can leverage this information for market research, tailoring products and services to meet consumer needs and preferences. similarly, political groups can gauge public reactions to policies and campaigns, adjusting their strategies accordingly. identifying differing opinions can facilitate conflict resolution by helping to understand the perspectives of all parties. by recognizing and respecting diverse opinions, society can promote tolerance and maintain social harmony. fig. 1 illustrates a sample reddit conversation on the topic of capitalism vs. socialism. in this context, an author's initial post-comprising the title and body-is referred to as a submission. multiple authors can then share their opinions as comments on this submission. specifically, this example contains 6 comments {c i } 6 @xmath0. we also display stance and dogmatism predictions from two llms: mistral large and gpt-4. some authors, like a 1 , change their views during the discussion based on the beliefs or opinions of others. at the beginning of the dialogue, author a 1 somewhat favors socialism (in submission and c 2 ). however, after considering the viewpoints of author a 2 in comments c 1 and c 3 , a 1 shifts their stance to somewhat favoring capitalism (in c 4 ), illustrating a firm yet openminded approach. on the other hand, author a 3 seems very flexible based on their comment c 5 . conversely, author a 3 appears very flexible based on their comment c 5 . understanding such conversations requires comprehending the fine-grained topics being discussed and the dynamic viewpoints of individual users. given the importance of understanding these user dynamics in conversations, training language models to perform this task automatically at scale is critical. while @xmath1. the submission content favors socialism and examines how the authors position their opinions regarding socialism vs. capitalism. crowdsourcing is one possible approach to address the need for a suitable dataset. however, manually annotating datasets for user opinions is time-consuming and costly, as annotators must read entire conversations to label each user's posts. additionally, manual annotation often faces challenges related to quality, as accurately labeling opinions requires understanding demographic details and domain-specific knowledge. given these limitations, achieving a comprehensive and accurate set of user opinions corresponding to posts about a topic often requires multiple annotators or iterative rounds of annotation. since users can change their opinion (often with subtle transitions and not with explicit statements) within a conversation, tracking such changes across multiple users manually becomes very cumbersome. recently, large language models (llms) @xcite @xcite , especially those built on transformer architectures @xcite and pretrained on large datasets, have resulted in state-of-the-art accuracies on several complex natural language processing (nlp) tasks @xcite . llms are also frequently used for synthetic dialog response generation @xcite @xcite . given the complex and cumbersome nature of conversation understanding, we hypothesize that llms can effectively capture the nuances involved in understanding user opinions and their shifts in multi-user conversational contexts. furthermore, since these models possess long-range memory capabilities, we believe they can reason over extended conversational threads involving numerous participants, as good as human annotators, if not better. we leverage llms like mistral large @xcite and @xcite to perform two tasks: i) user stance classification, which involves labeling a user's stance of a post in a conversation on a fivepoint scale; ii) user dogmatism classification, which deals with labeling a user's overall opinion in the conversation on a four-point scale. in addition to the class labels, we also obtain the reasoning behind the selection of each label. we experiment with these two models as human-like annotators, generating user opinions in full-length, multi-user reddit conversations in zeroshot, one-shot, and few-shot setups. thus, for every sample, we obtain annotations in six settings ({mistral large, gpt-4}×{zero-shot, one-shot, few-shot}). fig. 2 presents our llm-based annotation pipeline for user-level stance and dogmatism tasks. we consider majority voting over these six settings as our final annotations. this approach enables us to curate our usdc (user stance and dogmatism in conversations) dataset, which consists of 764 multi-user conversations from 22 subreddits, including 1,528 user-level dogmatism samples and 9,618 stance samples across all posts from selected users. annotations in usdc highlight specific user opinions in each post related to stance, track opinion fluctuations leading to dogmatism, and provide reasoning about why users hold specific opinions. we conduct a comprehensive evaluation of usdc by incorporating human annotations on 200 test conversations and measuring inter-annotator agreement between llm and human annotations. when we compare llm-generated annotations with human annotations, it becomes evident that the "lost in the middle" phenomenon @xcite is marginal in llms, whereas human annotators maintain a steady understanding and agreement throughout the conversation, regardless of its length or complexity. additionally, our "recency bias" @xcite analysis shows that llms rely heavily on full context to maintain better inter annotator agreement with human annotators. usdc addresses several weaknesses of existing post level stance and dogmatism datasets. first, the fulllength multi-user conversation aspect of usdc enables it to capture contextual and opinion shifts of multiple users. this feature allows it to serve as both an instruction-tuning user opinion dataset and an evaluation benchmark. we believe that the ability to perform instruction-tuning for user opinions at a large scale can bridge the gap between open-source and commercial user trait understanding models. additionally, the in-context learning annotations using state-of-the-art llms in usdc make it a more comprehensive measure of how current llms understand complex tasks like capturing opinions. further, the usdc dataset offers several use cases that extend its value in various domains, including, (i) improving moderation tools, (ii) analyzing public opinion dynamics, (iii) enhancing dialogue systems and (iv) creating dynamic contextual user representations. these aspects make it a valuable resource, especially for social media agents seeking deeper insights into user behavior. to demonstrate the utility of usdc, we utilize our dataset to fine-tune and instruction-tune open-source small language models (slms) for generating stance and dogmatism labels for users. we experiment with three pretrained slms like llama-2-7b, llama-3-8b @xcite , and falcon-7b @xcite . we also experiment with four instructiontuned slms like llama-2-chat-7b, llama-3-8binstruct, vicuna-7b-v.1.5, and falcon-7b-instruct. we make the following contributions: 1) we introduce usdc, a dataset of user stance and dogmatism in conversations. 2) we provide human annotations on 200 test conversations, achieving inter-annotator agreement scores of 0.49 for stance and 0.50 for dogmatism, indicating a reasonable level of consistency between human and llm annotations. 3) we benchmark initial results for the stance and dogmatism tasks using seven slms for udsc. we find that stance performance improves with instruction-tuning (f1-score of 56.2) compared to finetuning (f1-score of 54.9). however, dogmatism performs worse with instruction-tuning (f1-score of 49.2) compared to finetuning (f1-score of 51.4), highlighting the complexity of this task. 4) we apply transfer learning by fine-tuning slms on usdc and assessing their performance on existing post-level stance datasets, like spinos, mt-cds, and twitter-stance. we find that our transfer learning results are either comparable to or outperform prior studies. 5) we make the code, models and dataset publicly available 1 . similar to post level stance datasets, @xcite predicted user dogmatism on randomly sampled reddit posts from conversations, with each post limited to 200-300 characters. one major limitation of this work is the unavailability of a public dataset, and the treatment of each post as independent. overall, all these prior studies contrast with the usdc dataset, which focuses on tracking user-level opinions across long, multi-user conversations, capturing the evolution of stance and dogmatism over extended discussions rather than just on a specific target issues. ## training slms we train three pretrained slms (llama-2-7b, llama-3-8b, falcon-7b) and four instruction-tuned slms (llama-2-chat-7b, llama-3-8b-instruct, vicuna-7b-v.1.5, and falcon-7b-instruct). we finetune and instruction-tune these models using the proposed usdc dataset. we use pretrained model checkpoints from hugging face @xcite . all of these llms have a context length of 4096 tokens. model details and hyper-parameter settings are in appendix g. train-test setup. we conduct both finetuning and instruction-tuning of slms. we divide the dataset of 764 conversations into train (∼ 75%) and test splits (∼ 25%). the training dataset comprised 564 conversations, including 1128 samples of dogmatism labels and 7520 samples of stance labels. conversely, the testing dataset consisted of 200 conversations, with 400 sam-ples of dogmatism labels and 1831 samples of stance labels across two authors posts. finetuning of slms. for stance classification, we treat each user post as an independent sample. in contrast, for the dogmatism classification, we consider the entire user conversation as a single sample by concatenating all the threads from a user in that conversation. to load the pretrained slms, we perform 4-bit quantization, and we finetune the models by apply the lora technique @xcite , with sftt before saving the finetuned model. for finetuning, we used prompt for stance classification as shown in fig. 7 ## results baseline (un-fine-tuned) model performance and what constitutes a "reasonable" f1 score? to establish a reasonable f1-score benchmark for fine-tuning and instruction-tuning (discussed in the next subsections), we evaluated the un-fine-tuned slms, gpt-4 and mistral large, in few-shot settings. this evaluation includes both stance and dogmatism tasks, using majority voting to enhance reliability. the results are summarized in the tables 4 , 5 , 6 and 7 in appendix h. we make the following observations: (i) majority voting generally provides a slight improvement over individual few-shot configurations, which suggests the value of combining predictions from multiple models. (ii) the difference between gpt-4 and mistral large in un-finetuned few-shot settings is relatively small, indicating that both models are fairly comparable in performance on these tasks when using the llama-3-8b model. as shown by the un-fine-tuned model's performance for stance classification in ## discussion & conclusion we introduced usdc, a large-scale dataset of user stance and dogmatism in conversations, leveraging llms as human-like annotators. this dataset is used for various applications, including analyzing public opinions, enhancing dialogue systems, improving content moderation tools by identifying and flagging dogmatic or polarizing users in online discussions, and generating dynamic contextual user representations. the fulllength multi-user conversation aspect of usdc allows it to capture the contextual and opinion shifts of multiple users in a conversation. we believe that the ability to perform finetuning or instruction-tuning slms for user opinions at a large scale can bridge the gap between slms and commercial llms for understanding user traits. while finetuning slms shows good f1-score on both stance and dogmatism tasks, the f1-score remains below 60% (54.9% for stance and 51.4% for dogmatism). on the other hand, instruction-tuning of slms only improves f1-score performance on stance, not the dogmatism task. further, the performance still falls short of 60%, with weighted f1-scores of 56.2% for stance and 49.2% for dogmatism. these findings indicate that there is still significant room for improvement in understanding user opinions from a text segment. human evaluation showed an agreement of 0.57 for the stance and 0.52 for the dogmatism tasks between llm and human annotations. this indicates that llm-generated annotations in usdc are close to human labels. transfer-learning on 3 datasets also showed positive results.