“Tell me who you are and I tell you how you argue”: Predicting Stances and Arguments for Stakeholder Groups (2024)

Abstract: Argument mining has focused so far mainly on the identification, extraction, and formalization of arguments. An important yet unaddressedtask consists in the prediction of the argumentative behavior of stakeholders in a debate. Predicting the argumentative behavior in advance can support foreseeing issues in public policy making or help recognize potential disagreements early on and help to resolve them. In this paper, we consider the novel task of predicting the argumentative behavior of individual stakeholders. We present ARGENST, a framework that relies on a recommender-based architecture to predict the stance and the argumentative main point on a specific controversial topic for a given stakeholder, which is described in terms of a profile including properties related to demographic attributes, religious and political orientation, socio-economic background, etc. We evaluate our approach on the well-known debate.org dataset in terms of accuracy for predicting stance as well as in terms of similarity of the generated arguments to the ground truth arguments using BERTScore. As part of a case study, we show how juries of members representing different stakeholder groups and perspectives can be assembled to simulate the public opinion on a given topic.

## introduction debates on societally controversial topics typically involve different camps or stakeholder groups that have opposing views, interests, and goals. take the example of the debate on "abortion" @xcite , which is typically divided into a more "liberal" pro-choice camp and a more "conservative" pro-life camp. the so-called stance (pro/con) as well as the argumentative behaviour of stakeholders can often be predicted by knowing their political inclination (left/right), membership to a certain religious group, socio-economic backgrounds, etc. argumentative behavior is thus to some extent predictable given some knowledge about a group of stakeholders @xcite . this leads us to consider the new task of predicting stance and argumentative content given a certain controversial topic and a description of a particular stakeholder in terms of personal attributes. towards this goal, we present argenst, a framework for the prediction of stances and generation of arguments for stakeholder groups. our framework is inspired by the approach of @xcite , who focus on hate speech detection and model the opinion towards a controversial topic via a jury containing a number of ambassadors for each group. instead of predicting a general opinion for an entire group, we predict the stance and argumentative behavior of individual stakeholders, resulting in a distribution for groups assembled out of single stakeholders. understanding the key positions of stakeholders well in advance would allow us to recognize issues, conflicts, or even general opposition to public policies early on, thus helping to foresee, de-escalate, or prevent a conflict. more precisely, given a controversial topic t and a person p with a set of personal properties such as gender, income, or religion, we aim to predict p's stance on t and generate an argument (henceforth called: major claim) justifying the stance of p. we present in particular a fine-tuned architecture combining neural recommender systems with large language models (llms) based on @xcite as well as a prompt-based method utilizing pre-trained llms, gpt4 in particular. we evaluate our approach on data from the debate portal debate.org @xcite as described by @xcite . the dataset comprises of threaded discussions on controversial topics in addition to profiles of the different users of the portal including information about their political party, religion, education level, etc. we evaluate the predicted stance and argument against gold standard data extracted from debate. org in terms of accuracy and f 1 measure (stance) and bertscore (major claim). in this paper, we thus make the following contributions: ## related work summarization of debates and opinion analysis are prominent tasks in the field of argument mining @xcite . @xcite and @xcite introduced the field of key-point analysis as the task of identifying the most important aspects or arguments of a debate. while this task has received prominent attention in the form of shared tasks, e.g. by @xcite , these approaches are purely text-based and can only detect which core-points exist (including their frequency), but not who, that is, which stakeholder, stands in for which key-point. with the aim of establishing a relation between argumentative content and the personal standpoints/dispositions of individuals, recent research in the field of argumentation mining has focused on modeling the labeling behavior of single individuals, instead of aggregating them into a majority vote @xcite . indeed, current approaches attempt to predict the label dis-tribution @xcite or even labels for single individuals @xcite . the benefit of modeling the argumentative behavior of single individuals has recently been demonstrated by considering recommender-style architectures that embed single individuals on the basis of the arguments they share @xcite . however, @xcite did not consider any personal information related to political orientation, religious attitudes etc. in the computation of the embeddings. beyond the model proposed by @xcite al. (2023), gordon et al. (2022) also proposed a recommender-inspired model to develop a text classification system that relies on a component embedding the personal characteristics of annotators beyond considering their id and the text only. inspired by decision processes involving juries divided into different subgroups of individuals sharing a common characteristic (such as age, gender, ethnicity, political inclination, membership in a religious group, etc.), they determine the opinion of a whole group by taking the (predicted) decisions of their so-called ambassadors into account. as a first attempt to predict the stances of single individuals, @xcite have considered the task of predicting the stances of prominent persons on a given topic. for this purpose, they provide a large-scale resource, the expert stance graph from wikipedia, obtaining background information about the persons from articles that refer to the topics. as a drawback, the method is limited because it only applies to famous people with a wikipedia article. the approach can thus not predict stances for arbitrary persons based on information about socio-economic background. a step in this direction has been proposed by @xcite who have proposed the notion of a "digital representative" as a surrogate opinion generated by a fine-tuned llm. beyond this, @xcite have proposed to generate statements by fine-tuned llms as a way to foster consensus. concerning the prediction of stance and personal opinions in the narrower sense, @xcite have shown how stance can be predicted on the basis of a topic and mainly socio-economic factors. in contrast, @xcite have proposed a prompting-based approach to create "silicon samples" that have shown to successfully mirror human attitudes. in contrast to the abovementioned approaches, the method proposed in this paper can predict the stance of an arbitrary person that is represented via a set of personal and demographic attributes. for this, we propose two approaches: one fine-tuned recommender approach and one prompting approach that we both evaluate on the debate.org dataset, discussing their performance and limitations on a wide range of unseen topics and persons. ## dataset in this section, we describe the dataset used in our experiments that was provided by @xcite and originates from debate.org. in particular, we describe the personal and demographic attributes we use for our model. ## methods to predict the stance and to generate a major claim for a given individual described in terms of personal attributes, we propose two different approaches: i) a fine-tuned llm-based approach consisting of two models: one for predicting the stance, and another one for generating the textual major claim (section 4.1), as well as ii) promptingbased approaches relying on a pre-trained llm to predict both stance and major claim within one model (section 4.2) 4.1 fine-tuned llm following the idea of representing stakeholders as a group (jury) of individual persons having certain properties in common, we rely on a recommenderbased architecture following @xcite . since our goal is to predict the stance of a given person and to generate a major claim that can be regarded as an explanation for the stance, we introduce two major model components, the stance classifier and the major claim generator, trained in an end-to-end-fashion. the input for predicting the stance and major claim of a person foot_2 thus consists of three parts: (i) the topic, (ii) a person identifier that is embedded to capture similarities across persons based on their friendship network foot_3 , and (iii) all personal properties of that person (see section 3). this input is processed once by the stance classifier for predicting a binary label and by a generative llm (without the friendship connection) for generating the major claim. in order to integrate the stance prediction and friendship networks into the generation of the major claim, the hidden state of the stance prediction is fed into the llm generating the major claim. figure 1 provides an overview of the architecture. a detailed explanation of the major two components can be found in appendix b. ## experimental setup we automatically evaluate the stances and major claim produced as output of our models. we rely on a 70-10-20 split of the dataset described in section 3, corresponding to 5664, 847, and 1471 opinion arguments for the train, dev, and test set, respectively, without overlapping topics and persons. the dataset comprises of 1,253 unique persons. each sample consists of a topic title, a person with their attributes, stance, and major claim @xcite . ## conclusion and future work we have presented a framework and new task consisting of predicting the stance and major claims of individual stakeholders towards a given controversial topic. the proposed approach relies on a set of personal attributes provided as input to predict the stance and argumentative behaviour of a certain stakeholder (group) that is defined by demographic variables, political inclination, socioeconomic background, membership to a religious group, etc. we have presented and experimentally evaluated two specific approaches toward this end: a fine-tuning-based approach and a promptingbased approach. our experiments on the debate.org dataset using an automatic evaluation have shown that the prompting-based approach delivers better results than the fine-tuning approach on the prediction of stance (accuracy of .698 vs .521). in contrast, the fine-tuning approach seems to perform better than the prompting approach on the task of generating the actual arguments when compared to the ground truth arguments in terms of bertscore. the manual evaluation has relativized this, showing that the fine-tuning approach in many cases yields arguments that are not consistent with the given stance in 35% of cases, while the prompting-based approach has an error rate of only about 20%. we have nevertheless demonstrated that the approach is interesting from an application perspective as it enables to assemble juries of people that represent the perspectives of different stakeholders, thus allowing us to simulate the opinions and stances of a diverse set of people to understand their perspective on a given topic. this has the potential to support deliberation, allowing for the identification of potential issues, and to foster a better understanding of the rationale of different stakeholder groups. the approach has the potential to unveil the majority opinion of a certain stakeholder group as well as reveiling outliers. overall, our paper shows that the newly proposed task is indeed challenging. future work could consider a retrieval-augmented architecture that extracts additional background and commonsense knowledge and integrates them into the input of models to increase their ability to make more accurate predictions based on knowledge about the topic.